diff --git a/CHANGELOG.md b/CHANGELOG.md index cb52184cc..9fc6b0990 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -36,8 +36,12 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/). Pages still bound to the layout are **named in a warning and the drop proceeds**, matching every other DROP in mxcli (none refuse on dependents) and matching how a layout is normally corrected: re-create it under the same name and the pages rebind by qualified name — verified end to end, back to 0 errors. Left dropped, each page fails **CE1613** "The selected layout … no longer exists", which names the *page* and never the layout, so the warning names the pages and gives the `alter pages … where layout =` repoint. +- **`run --local --db-type hsqldb` — boot with no database server** — a local run can now use Mendix's built-in, file-based HSQLDB instead of PostgreSQL. The runtime already ships the driver and derives the file URL from `DatabaseType` and `DatabaseName`, so the data lands under `/deployment/data/database/hsqldb/`. `--db-host`, `--db-user` and `--db-password` are refused with HSQLDB (it is a file, not a connection) and `--ensure-db` is refused too (there is nothing to provision); PostgreSQL remains the default and is unchanged. Intended for local development and demos only — Mendix marks HSQLDB as local-testing-only. + ### Fixed +- **The runtime start cycle did not create a database that does not exist yet** — `RuntimeController.Start` only ran `execute_ddl_commands` when Mendix answered result 3 ("the database has to be updated"). A brand-new database answers result 2 ("the database to be used does not exist"), which was unhandled, so the first boot against the built-in HSQLDB database failed with `start failed: The database to be used does not exist.` Both results now trigger the schema step, and a start that still fails after it says so explicitly. + - **`mxcli run --local` hung forever on Windows at `Starting mxbuild --serve...`** — two POSIX assumptions in `cmd/mxcli/docker` combined. `ServeServer.alive()` (and `LocalRuntime.alive()`) asked `os.Process.Signal(syscall.Signal(0))`, which Go implements as `EWINDOWS` ("not supported by windows") for every signal but `Kill` — so a live mxbuild read as dead and `waitReady()` aborted the boot the instant it launched it. And `Stop()` waited on `cmd.Wait()` after a `killProcessGroup` that on Windows only called `p.Kill()`: mxbuild.exe is a wrapper that launches a Deno web-ext worker (`modeler/tools/deno/win-x64/deno.exe`) which inherits the stdout/stderr pipe `exec.Cmd` hands out, so the orphan kept the pipe open and `Wait()` never saw EOF — no error, no exit, just a hang. Windows now has a real liveness check (`OpenProcess(SYNCHRONIZE)` + `WaitForSingleObject`) and a tree kill (`taskkill /F /T`) behind `signalProcessGroup`/`killProcessGroup`; the POSIX implementations are unchanged. The Windows-only tests in `procgroup_windows_test.go` pin both halves — the tree-kill test fails on the pre-fix code, verified as a control — and a `windows-latest` CI job runs them (asserting they actually executed) so neither can come back. - **`raise error;` on a microflow's main flow passed check and exec, then failed the build** (mendixlabs/mxcli#1030) — with `[error] [CE0710] "The main flow cannot join an error flow or end in an error event."`, one per microflow. Mendix's error event *re-raises the error being handled*, so it is legal only where an error is in scope: inside an `on error { … }` handler. Studio Pro will not draw the connection from the normal flow to an error event; mxcli could, and did. It is now **MDL084**, at error severity, so `exec`'s pre-flight refuses the script with nothing written (`--no-check` still applies it, for reproducing the build failure). diff --git a/cmd/mxcli/cmd_run.go b/cmd/mxcli/cmd_run.go index a8c0f4bb9..02919d846 100644 --- a/cmd/mxcli/cmd_run.go +++ b/cmd/mxcli/cmd_run.go @@ -36,6 +36,9 @@ Requirements: - A reachable PostgreSQL (the devcontainer provides one); the database must already exist. Defaults: 127.0.0.1:5432, user 'mendix', db from the project name. Override with --db-host/--db-name/--db-user/--db-password. + - --db-type hsqldb runs against the built-in file database instead of + PostgreSQL, so no database server is needed. Its data lives under the + project's deployment directory and is meant for local development only. With --hub, the running app is exposed in a browser at a public URL through an mxcli tunnel-hub, without leaving this machine: a tunnel client reverse-tunnels @@ -147,6 +150,7 @@ Examples: adminPort, _ := cmd.Flags().GetInt("admin-port") servePort, _ := cmd.Flags().GetInt("serve-port") mxbuildPath, _ := cmd.Flags().GetString("mxbuild-path") + dbType, _ := cmd.Flags().GetString("db-type") dbHost, _ := cmd.Flags().GetString("db-host") dbName, _ := cmd.Flags().GetString("db-name") dbUser, _ := cmd.Flags().GetString("db-user") @@ -219,6 +223,7 @@ Examples: TraceService: traceService, TraceOTLP: traceOTLP, DB: docker.DBConfig{ + Type: dbType, Host: dbHost, Name: dbName, User: dbUser, @@ -313,6 +318,7 @@ func init() { runCmd.Flags().Int("admin-port", 0, "M2EE admin API port (default 8090)") runCmd.Flags().Int("serve-port", 0, "mxbuild --serve port (default 6543)") runCmd.Flags().String("mxbuild-path", "", "Path to the mxbuild to build with, overriding resolution (Studio Pro's bundled mxbuild on macOS/Windows, the cached CDN download on Linux)") + runCmd.Flags().String("db-type", "", "Database type for a local run: postgresql (default) or hsqldb (the runtime's built-in file database — no server, no --db-host/--db-user/--db-password, data under /deployment/data/database/hsqldb/)") runCmd.Flags().String("db-host", "", "Database host:port (IPv6: [::1]:5432; default 127.0.0.1:5432)") runCmd.Flags().String("db-name", "", "Database name (default derived from the project name)") runCmd.Flags().String("db-user", "", "Database user (default mendix)") diff --git a/cmd/mxcli/docker/dbtype.go b/cmd/mxcli/docker/dbtype.go new file mode 100644 index 000000000..331c679e5 --- /dev/null +++ b/cmd/mxcli/docker/dbtype.go @@ -0,0 +1,45 @@ +// SPDX-License-Identifier: Apache-2.0 + +package docker + +import ( + "fmt" + "strings" +) + +// Canonical database types a local run understands. The runtime's own enum also +// has Db2/MySql/Oracle/SapHana/SqlServer; those all need a server and are out of +// scope for `run --local`. +const ( + DBTypePostgreSQL = "postgresql" + DBTypeHSQLDB = "hsqldb" +) + +// NormalizeDBType maps a user-supplied --db-type to a canonical value. Empty +// means PostgreSQL, which is what a local run has always used, so the flag stays +// backwards compatible. +func NormalizeDBType(raw string) (string, error) { + switch strings.ToLower(strings.TrimSpace(raw)) { + case "", DBTypePostgreSQL, "postgres": + return DBTypePostgreSQL, nil + case DBTypeHSQLDB: + return DBTypeHSQLDB, nil + default: + return "", fmt.Errorf("unknown --db-type %q (want postgresql or hsqldb)", raw) + } +} + +// IsFileBasedDBType reports whether a canonical type is the runtime's built-in +// file database, which has no host to reach and no credentials. +func IsFileBasedDBType(canonical string) bool { + return canonical == DBTypeHSQLDB +} + +// RuntimeDatabaseType maps a canonical type to the spelling the runtime's +// DatabaseType parameter expects (the runtime enum is upper case). +func RuntimeDatabaseType(canonical string) string { + if canonical == DBTypeHSQLDB { + return "HSQLDB" + } + return "PostgreSQL" +} diff --git a/cmd/mxcli/docker/dbtype_test.go b/cmd/mxcli/docker/dbtype_test.go new file mode 100644 index 000000000..f3c024a0c --- /dev/null +++ b/cmd/mxcli/docker/dbtype_test.go @@ -0,0 +1,43 @@ +// SPDX-License-Identifier: Apache-2.0 + +package docker + +import "testing" + +func TestNormalizeDBType(t *testing.T) { + cases := []struct { + in string + want string + wantErr bool + }{ + {"", DBTypePostgreSQL, false}, + {"postgresql", DBTypePostgreSQL, false}, + {"PostgreSQL", DBTypePostgreSQL, false}, + {"postgres", DBTypePostgreSQL, false}, + {"hsqldb", DBTypeHSQLDB, false}, + {"HSQLDB", DBTypeHSQLDB, false}, + {" Hsqldb ", DBTypeHSQLDB, false}, + {"mysql", "", true}, + } + for _, c := range cases { + got, err := NormalizeDBType(c.in) + if c.wantErr { + if err == nil { + t.Errorf("NormalizeDBType(%q) = %q, want error", c.in, got) + } + continue + } + if err != nil || got != c.want { + t.Errorf("NormalizeDBType(%q) = %q, %v; want %q", c.in, got, err, c.want) + } + } +} + +func TestRuntimeDatabaseType(t *testing.T) { + if got := RuntimeDatabaseType(DBTypeHSQLDB); got != "HSQLDB" { + t.Errorf("RuntimeDatabaseType(hsqldb) = %q, want HSQLDB", got) + } + if got := RuntimeDatabaseType(DBTypePostgreSQL); got != "PostgreSQL" { + t.Errorf("RuntimeDatabaseType(postgresql) = %q, want PostgreSQL", got) + } +} diff --git a/cmd/mxcli/docker/localboot.go b/cmd/mxcli/docker/localboot.go index 06f10d482..673b2f1f4 100644 --- a/cmd/mxcli/docker/localboot.go +++ b/cmd/mxcli/docker/localboot.go @@ -33,7 +33,9 @@ import ( // already resolved, to /model/config.json — readDeploymentConstants // lifts them from there. -// DBConfig is the external Postgres the standalone runtime connects to. +// DBConfig is the database the standalone runtime connects to: either an external +// Postgres (Host/User/Password set) or the runtime's built-in file database +// (Type "HSQLDB", Host/User/Password empty — see applyDatabaseDefaults). type DBConfig struct { Type string // e.g. "PostgreSQL" Host string // "host:port", e.g. "127.0.0.1:5432" @@ -42,6 +44,13 @@ type DBConfig struct { Password string } +// IsFileBased reports whether this is the runtime's built-in file database, which +// has no host to reach. It keys on the runtime spelling that applyDatabaseDefaults +// sets (RuntimeDatabaseType), not the raw --db-type flag. +func (c DBConfig) IsFileBased() bool { + return c.Type == RuntimeDatabaseType(DBTypeHSQLDB) +} + // LocalRuntimeOptions configures StartLocalRuntime. type LocalRuntimeOptions struct { // DeployDir is the deployment directory (the runtime's BasePath). The mxbuild diff --git a/cmd/mxcli/docker/localboot_test.go b/cmd/mxcli/docker/localboot_test.go index 1fb48166e..bc40020e2 100644 --- a/cmd/mxcli/docker/localboot_test.go +++ b/cmd/mxcli/docker/localboot_test.go @@ -161,6 +161,24 @@ func TestRuntimeConfigParams_ApplicationRootUrl(t *testing.T) { } } +func TestRuntimeConfigParams_HSQLDB(t *testing.T) { + o := testLocalOpts() + o.DB = DBConfig{Type: "HSQLDB", Name: "appdb"} + p := runtimeConfigParams(o, nil) + checks := map[string]any{ + "DatabaseType": "HSQLDB", + "DatabaseHost": "", + "DatabaseName": "appdb", + "DatabaseUserName": "", + "DatabasePassword": "", + } + for k, want := range checks { + if p[k] != want { + t.Errorf("%s = %v, want %v", k, p[k], want) + } + } +} + func TestReadDeploymentConstants(t *testing.T) { dir := t.TempDir() modelDir := filepath.Join(dir, "model") diff --git a/cmd/mxcli/docker/runlocal.go b/cmd/mxcli/docker/runlocal.go index 199fdbd8b..1c5bc54d9 100644 --- a/cmd/mxcli/docker/runlocal.go +++ b/cmd/mxcli/docker/runlocal.go @@ -182,21 +182,6 @@ func (o *LocalRunOptions) applyDefaults() { if o.PollInterval == 0 { o.PollInterval = time.Second } - if o.DB.Type == "" { - o.DB.Type = "PostgreSQL" - } - if o.DB.Host == "" { - o.DB.Host = "127.0.0.1:5432" - } - if o.DB.User == "" { - o.DB.User = "mendix" - } - if o.DB.Password == "" { - o.DB.Password = "mendix" - } - if o.DB.Name == "" { - o.DB.Name = deriveDBName(o.ProjectPath) - } if o.ScreenshotPath == "" { o.ScreenshotPath = filepath.Join(filepath.Dir(o.ProjectPath), ".mxcli", "run-local.png") } @@ -214,6 +199,47 @@ func (o *LocalRunOptions) applyDefaults() { } } +// applyDatabaseDefaults validates --db-type and fills the connection settings the +// chosen database needs. It is separate from applyDefaults because it can fail: +// a flag combination that cannot work (--db-type hsqldb with --db-host) is a user +// error, not something to silently normalise away. +func (o *LocalRunOptions) applyDatabaseDefaults() error { + kind, err := NormalizeDBType(o.DB.Type) + if err != nil { + return err + } + if o.DB.Name == "" { + o.DB.Name = deriveDBName(o.ProjectPath) + } + if IsFileBasedDBType(kind) { + // The built-in database is a file: it has no host or credentials, and + // accepting them would imply a connection that never happens. + if o.DB.Host != "" || o.DB.User != "" || o.DB.Password != "" { + return fmt.Errorf("--db-type hsqldb uses the built-in file database and takes no " + + "--db-host, --db-user or --db-password") + } + if o.EnsureDB { + return fmt.Errorf("--ensure-db provisions PostgreSQL; the built-in HSQLDB database " + + "needs no provisioning — drop --ensure-db") + } + o.DB.Type = RuntimeDatabaseType(kind) + o.DB.Host, o.DB.User, o.DB.Password = "", "", "" + return nil + } + // PostgreSQL (unchanged behaviour). + if o.DB.Host == "" { + o.DB.Host = "127.0.0.1:5432" + } + if o.DB.User == "" { + o.DB.User = "mendix" + } + if o.DB.Password == "" { + o.DB.Password = "mendix" + } + o.DB.Type = RuntimeDatabaseType(kind) + return nil +} + // defaultOtelSpanFilters are the internal runtime spans suppressed under --trace. // Unfiltered per-activity tracing is ~10x slower; these bring it near baseline // while keeping the microflow-level spans (findings — OpenTelemetry). @@ -541,6 +567,9 @@ func sourceMTime(projectPath string) time.Time { // and hot-apply on every project change until interrupted. func RunLocal(opts LocalRunOptions) error { opts.applyDefaults() + if err := opts.applyDatabaseDefaults(); err != nil { + return err + } w, stderr := opts.Stdout, opts.Stderr // 0. Refuse fast if the loop's ports are already taken (a stale run/serve/ @@ -610,10 +639,12 @@ func RunLocal(opts LocalRunOptions) error { if err := EnsureDatabase(&opts.DB, w); err != nil { return fmt.Errorf("ensuring database: %w", err) } - } else if err := pingTCP(opts.DB.Host, 3*time.Second); err != nil { - return fmt.Errorf("database not reachable at %s: %w\n"+ - " Pass --ensure-db to provision it, or start Postgres and create the '%s' database (user %q).", - opts.DB.Host, err, opts.DB.Name, opts.DB.User) + } else if !opts.DB.IsFileBased() { + if err := pingTCP(opts.DB.Host, 3*time.Second); err != nil { + return fmt.Errorf("database not reachable at %s: %w\n"+ + " Pass --ensure-db to provision it, or start Postgres and create the '%s' database (user %q).", + opts.DB.Host, err, opts.DB.Name, opts.DB.User) + } } // Setup-only: prerequisites are ready (mxbuild+runtime cached, database up). diff --git a/cmd/mxcli/docker/runlocal_hsqldb_integration_test.go b/cmd/mxcli/docker/runlocal_hsqldb_integration_test.go new file mode 100644 index 000000000..df1536adb --- /dev/null +++ b/cmd/mxcli/docker/runlocal_hsqldb_integration_test.go @@ -0,0 +1,112 @@ +// SPDX-License-Identifier: Apache-2.0 + +//go:build integration + +package docker + +import ( + "bytes" + "io" + "net/http" + "os" + "os/exec" + "path/filepath" + "testing" + "time" +) + +// The point of the built-in database: an app boots with NO database server. +// Modeled on localapp_integration_test.go — scaffold a project when no fixture +// is given, build it with a real mxbuild serve, then boot the runtime with +// HSQLDB and require it to answer. Self-contained: HSQLDB is a file, so there is +// nothing else to stand up. +func TestRunLocal_HSQLDBNeedsNoDatabaseServer(t *testing.T) { + mprPath := os.Getenv("MXCLI_IT_PROJECT") + if mprPath == "" { + mxPath, err := ResolveMx("") + if err != nil { + t.Skipf("mx not resolvable and MXCLI_IT_PROJECT unset: %v", err) + } + dir, err := os.MkdirTemp("", "mxhsql") + if err != nil { + t.Fatal(err) + } + defer os.RemoveAll(dir) + scaffold := exec.Command(mxPath, "create-project") + scaffold.Dir = dir + if out, err := scaffold.CombinedOutput(); err != nil { + t.Skipf("mx create-project failed: %v\n%s", err, out) + } + mprPath = filepath.Join(dir, "App.mpr") + } + if _, err := os.Stat(mprPath); err != nil { + t.Skipf("no project fixture at %s: %v", mprPath, err) + } + + reader, err := openReadOnly(mprPath) + if err != nil { + t.Skipf("cannot open project: %v", err) + } + version := reader.ProjectVersion().ProductVersion + reader.Disconnect() + + installPath, err := resolveRuntimeInstall(version, io.Discard) + if err != nil { + t.Skipf("no runtime for %s: %v", version, err) + } + javaMajor, _ := ProjectJavaMajor(mprPath) + + serve, err := StartServe(ServeOptions{Version: version, JavaMajor: javaMajor, Host: "127.0.0.1", Port: 6549}) + if err != nil { + t.Skipf("cannot start mxbuild serve: %v", err) + } + defer serve.Stop() + + build, err := serve.Build(BuildRequest{Target: TargetDeploy, ProjectFilePath: mprPath}) + if err != nil { + t.Fatalf("build: %v", err) + } + if !build.OK() { + t.Fatalf("build failed: %s", build.Message) + } + + deployDir := filepath.Join(filepath.Dir(mprPath), "deployment") + var rtOut bytes.Buffer + rt, err := StartLocalRuntime(LocalRuntimeOptions{ + DeployDir: deployDir, + InstallPath: installPath, + JavaMajor: javaMajor, + AdminPass: defaultLocalAdminPass, + AppPort: 8087, + AdminPort: 8097, + DB: DBConfig{Type: "HSQLDB", Name: deriveDBName(mprPath)}, + Stdout: &rtOut, + Stderr: &rtOut, + }) + if err != nil { + t.Fatalf("boot with the built-in database: %v", err) + } + defer rt.Stop() + + client := &http.Client{Timeout: 5 * time.Second} + deadline := time.Now().Add(90 * time.Second) + for { + resp, err := client.Get(rt.AppURL()) + if err == nil { + resp.Body.Close() + if resp.StatusCode == 200 { + break + } + } + if time.Now().After(deadline) { + t.Fatalf("app did not answer 200 at %s\n--- runtime output ---\n%s", rt.AppURL(), rt.Log()) + } + time.Sleep(time.Second) + } + + hsqlDir := filepath.Join(deployDir, "data", "database", "hsqldb") + entries, err := os.ReadDir(hsqlDir) + if err != nil || len(entries) == 0 { + t.Fatalf("expected the built-in database files under %s (err=%v, n=%d)", hsqlDir, err, len(entries)) + } +} diff --git a/cmd/mxcli/docker/runlocal_test.go b/cmd/mxcli/docker/runlocal_test.go index 15c2e6d4f..e9b27699d 100644 --- a/cmd/mxcli/docker/runlocal_test.go +++ b/cmd/mxcli/docker/runlocal_test.go @@ -34,6 +34,9 @@ func TestDeriveDBName(t *testing.T) { func TestLocalRunOptions_Defaults(t *testing.T) { o := LocalRunOptions{ProjectPath: "/proj/App1112.mpr"} o.applyDefaults() + if err := o.applyDatabaseDefaults(); err != nil { + t.Fatalf("applyDatabaseDefaults: %v", err) + } if o.DeployDir != filepath.FromSlash("/proj/deployment") { t.Errorf("DeployDir = %q", o.DeployDir) } @@ -228,6 +231,9 @@ func TestLocalRunOptions_DefaultsRespectOverrides(t *testing.T) { DB: DBConfig{Host: "db:5432", Name: "custom", User: "u", Password: "p"}, } o.applyDefaults() + if err := o.applyDatabaseDefaults(); err != nil { + t.Fatalf("applyDatabaseDefaults: %v", err) + } if o.AppPort != 9000 { t.Errorf("AppPort override lost: %d", o.AppPort) } @@ -579,3 +585,62 @@ func TestURLPort(t *testing.T) { } } } + +func TestLocalRunOptions_DatabaseDefaults_HSQLDB(t *testing.T) { + o := LocalRunOptions{ProjectPath: "/proj/App1112.mpr", DB: DBConfig{Type: "hsqldb"}} + o.applyDefaults() + if err := o.applyDatabaseDefaults(); err != nil { + t.Fatalf("applyDatabaseDefaults: %v", err) + } + if o.DB.Type != "HSQLDB" { + t.Errorf("Type = %q, want HSQLDB", o.DB.Type) + } + if o.DB.Host != "" || o.DB.User != "" || o.DB.Password != "" { + t.Errorf("HSQLDB must carry no host/credentials, got %+v", o.DB) + } + if o.DB.Name != "app1112" { + t.Errorf("DB.Name = %q, want app1112", o.DB.Name) + } +} + +func TestLocalRunOptions_DatabaseDefaults_HSQLDBRejectsConnectionFlags(t *testing.T) { + for _, db := range []DBConfig{ + {Type: "hsqldb", Host: "127.0.0.1:5432"}, + {Type: "hsqldb", User: "u"}, + {Type: "hsqldb", Password: "p"}, + } { + o := LocalRunOptions{ProjectPath: "/proj/App.mpr", DB: db} + o.applyDefaults() + if err := o.applyDatabaseDefaults(); err == nil { + t.Errorf("hsqldb with %+v: want an error", db) + } + } +} + +func TestLocalRunOptions_DatabaseDefaults_HSQLDBRejectsEnsureDB(t *testing.T) { + o := LocalRunOptions{ProjectPath: "/proj/App.mpr", DB: DBConfig{Type: "hsqldb"}, EnsureDB: true} + o.applyDefaults() + if err := o.applyDatabaseDefaults(); err == nil { + t.Error("hsqldb + --ensure-db: want an error") + } +} + +func TestLocalRunOptions_DatabaseDefaults_UnknownType(t *testing.T) { + o := LocalRunOptions{ProjectPath: "/proj/App.mpr", DB: DBConfig{Type: "mysql"}} + o.applyDefaults() + if err := o.applyDatabaseDefaults(); err == nil { + t.Error("unknown db type: want an error") + } +} + +func TestDBConfig_IsFileBased(t *testing.T) { + if !(DBConfig{Type: "HSQLDB"}).IsFileBased() { + t.Error("HSQLDB should be file-based") + } + if (DBConfig{Type: "PostgreSQL"}).IsFileBased() { + t.Error("PostgreSQL is not file-based") + } + if (DBConfig{Type: "hsqldb"}).IsFileBased() { + t.Error("the check must use the runtime spelling set by applyDatabaseDefaults, not the raw flag") + } +} diff --git a/cmd/mxcli/docker/runtime_controller.go b/cmd/mxcli/docker/runtime_controller.go index cba0853d3..7fe2b8213 100644 --- a/cmd/mxcli/docker/runtime_controller.go +++ b/cmd/mxcli/docker/runtime_controller.go @@ -94,6 +94,7 @@ func (c *RuntimeController) Start() (*M2EEResponse, error) { if err != nil { return nil, err } + schemaAttempted := false if needsDBUpdate(resp) { ddl, err := CallM2EE(c.opts, "execute_ddl_commands", nil) if err != nil { @@ -102,12 +103,16 @@ func (c *RuntimeController) Start() (*M2EEResponse, error) { if msg := ddl.M2EEError(); msg != "" { return nil, fmt.Errorf("execute_ddl_commands failed: %s", msg) } + schemaAttempted = true resp, err = CallM2EE(c.opts, "start", nil) if err != nil { return nil, err } } if msg := resp.M2EEError(); msg != "" { + if schemaAttempted { + return resp, fmt.Errorf("start failed after creating or updating the database schema: %s", msg) + } return resp, fmt.Errorf("start failed: %s", msg) } // The runtime is up; wire the application log to a file (best-effort — a @@ -203,17 +208,32 @@ func (c *RuntimeController) ApplyBuild(build *BuildResult, restart func() error) return action, nil } -// needsDBUpdate reports whether a start response indicates the database schema -// must be updated before the runtime can serve (result 3 / "database has to be -// updated" / a synchronizationreason in the feedback). +// needsDBUpdate reports whether a start response indicates the database must be +// created or updated before the runtime can serve. +// +// Two results mean that, and both need the same response (execute_ddl_commands +// then start again): +// +// - result 3, "the database has to be updated" — the schema is out of date. +// - result 2, "the database to be used does not exist" — there is no schema at +// all. This is the normal first boot of the built-in HSQLDB database, whose +// JDBC URL carries ifexists=true and therefore never creates the file. +// +// Measured on Mendix 11.12.2: a fresh built-in HSQLDB project answers start +// with Result 2 and Message "The database to be used does not exist.". +// +// The synchronizationreason feedback is a third signal the runtime sometimes +// sends instead of the message. func needsDBUpdate(resp *M2EEResponse) bool { if resp == nil { return false } - if resp.Result == 3 { + if resp.Result == 2 || resp.Result == 3 { return true } - if strings.Contains(strings.ToLower(resp.Message), "database has to be updated") { + lower := strings.ToLower(resp.Message) + if strings.Contains(lower, "database has to be updated") || + strings.Contains(lower, "database to be used does not exist") { return true } if fb := resp.Feedback(); fb != nil { diff --git a/cmd/mxcli/docker/runtime_controller_test.go b/cmd/mxcli/docker/runtime_controller_test.go index 666c91f45..6c30ae954 100644 --- a/cmd/mxcli/docker/runtime_controller_test.go +++ b/cmd/mxcli/docker/runtime_controller_test.go @@ -344,6 +344,8 @@ func TestNeedsDBUpdate(t *testing.T) { {"clean", &M2EEResponse{}, false}, {"result3", &M2EEResponse{Result: 3}, true}, {"message", &M2EEResponse{Message: "The database has to be updated first"}, true}, + {"result2", &M2EEResponse{Result: 2}, true}, + {"no-existing-db-message", &M2EEResponse{Message: "The database to be used does not exist."}, true}, {"feedback", &M2EEResponse{RawFeedback: json.RawMessage(`{"synchronizationreason":"x"}`)}, true}, {"other-error", &M2EEResponse{Result: 1, Message: "unrelated"}, false}, } diff --git a/docs/11-proposals/PROPOSAL_run_local_builtin_database.md b/docs/11-proposals/PROPOSAL_run_local_builtin_database.md new file mode 100644 index 000000000..07e32a46d --- /dev/null +++ b/docs/11-proposals/PROPOSAL_run_local_builtin_database.md @@ -0,0 +1,121 @@ +--- +title: Built-in file database for run --local (HSQLDB) +status: proposed +date: 2026-09-21 +--- + +# Proposal: Built-in file database for `run --local` (HSQLDB) + +**Status:** Proposed +**Date:** 2026-09-21 +**Depends on:** the Windows process-helper fix (PR #1147) — this builds on that branch. + +`mxcli run --local` can only boot against PostgreSQL: `runlocal.go` hard-codes +`DatabaseType = "PostgreSQL"`, there is no `--db-type`, a TCP reachability check +runs before boot, and `--ensure-db` refuses anything else. That makes the tool +unusable without a database server, which is exactly the friction when handing a +build to a customer who only wants to click a `.exe` and see their app. + +Mendix ships an embedded, file-based database (HSQLDB) with the runtime, and +Studio Pro's own local run defaults to it. This proposal exposes it through +`mxcli run --local` so an app can boot with **no external database at all**, its +data in a local file. + +## Research findings + +Everything below was read out of the Mendix 11.12.2 runtime on this machine, not +assumed. + +1. **The runtime supports it.** `runtime/pad/etc/example.conf` lists `HSQLDB` as a + valid `DatabaseType` (`HSQLDB, MYSQL, ORACLE, POSTGRESQL, SAPHANA, SQLSERVER`), + and documents `DatabaseJdbcUrl` as a value that *"overrides the other database + connection settings"*. +2. **The driver is bundled.** `runtime/bundles/org.hsqldb.hsqldb.2.7.4.jar` ships + with the runtime; no extra download is needed. +3. **The runtime builds the file URL itself.** Decompiling + `com.mendix.datastorage-connectionbus.jar`: + - `HsqldbDataStoreConfigurator` contains `jdbc:hsqldb:file:`, + `builtInDatabasePath`, `hsqldb/`, `ifexists=true`, `databaseName`; + - `MainDatabaseConfiguration` contains `data/database`. + So with only `DatabaseType=HSQLDB` + `DatabaseName=` (and empty host / + user / password), the runtime derives + `jdbc:hsqldb:file:/data/database/hsqldb/;ifexists=true`. + `BasePath` is the deployment directory `run --local` already sets. +4. **Mendix marks it development-only.** `HSQLDB.yaml.hbs` states the setup is + *"intended for local testing only"*. This feature is for local dev / demo, not + production — and the docs must say so. +5. **mxcli's current behaviour.** `runlocal.go` defaults `o.DB.Type` to + `"PostgreSQL"`; `cmd_run.go` builds `DBConfig` from flags and never sets + `Type`; the pre-boot `pingTCP` and `EnsureDatabase` are PostgreSQL-only. + +## Design + +### CLI + +- New flag on `run`: `--db-type postgresql|hsqldb` (case-insensitive), default + `postgresql`. `postgresql` keeps today's behaviour exactly. +- With `--db-type hsqldb`: + - `--db-host`, `--db-user`, `--db-password` are refused (an explicit value is a + user error, not silently ignored) with a message naming the flag. + - `--db-name` is honoured (the HSQLDB file base name); default stays the + project-derived name. + - `--ensure-db` is refused: there is nothing to provision. + +### Boot config + +`DBConfig.Type` is taken from the flag instead of being forced to PostgreSQL. +For HSQLDB the runtime config carries `DatabaseType="HSQLDB"`, `DatabaseHost=""`, +`DatabaseUserName=""`, `DatabasePassword=""`, `DatabaseName=`. `mxcli` +does **not** set `DatabaseJdbcUrl`, so the runtime's own path rule applies. + +### Pre-boot reachability + +The `pingTCP` check is skipped for HSQLDB (there is no host:port). For PostgreSQL +it is unchanged. + +### Where the data lands + +By the runtime's own rule the files go to +`/deployment/data/database/hsqldb/.*`. mxcli keeps that default +(least surprise, matches Studio Pro) and documents it; no `DatabaseJdbcUrl` is +set and no `--db-path` flag is added. + +Accepted consequence: `deployment/` is a build output, so the database lives with +the rest of the local run artifacts. This is fine for the feature's purpose — a +local, disposable dev/demo database — and is stated in the docs. Anyone who needs +durable data uses a real database. + +## Testing + +Unit (any OS, no runtime needed): + +- `DBConfig` → runtime config mapping: HSQLDB yields empty host/user/password and + the right type; PostgreSQL is byte-for-byte unchanged. +- flag parsing: `--db-type hsqldb` (both cases), an invalid value is refused, + and `hsqldb` + `--db-host` / `--ensure-db` are refused. +- the reachability check is skipped for HSQLDB. + +Integration (Linux, reuses the existing `make test-integration` mxbuild + runtime +cache): + +- boot a blank project with `run --local --db-type hsqldb`; assert HTTP 200; +- assert the HSQLDB files exist under + `/deployment/data/database/hsqldb/`. + +Windows: the package must compile and the unit tests must pass (the existing +`windows-process-regression` job pattern); a full Windows boot integration is +optional and may be added as its own job. + +## Scope + +- In: `postgresql` and `hsqldb` only. Other Mendix database types are out of + scope. +- Out: production use, multi-node, `--ensure-db` for HSQLDB, `--db-jdbc-url` + passthrough. + +## Rollout + +- One PR on top of #1147: `feat: run --local with the built-in HSQLDB database`. +- CHANGELOG entry under `[Unreleased] / Added`. +- Docs: `--db-type` in the `run` help; a note that HSQLDB is for local testing + only and where its files live. diff --git a/docs/plans/2026-09-21-run-local-hsqldb-implementation-plan.md b/docs/plans/2026-09-21-run-local-hsqldb-implementation-plan.md new file mode 100644 index 000000000..37ce39708 --- /dev/null +++ b/docs/plans/2026-09-21-run-local-hsqldb-implementation-plan.md @@ -0,0 +1,794 @@ +# run --local with the built-in HSQLDB database — Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Let `mxcli run --local --db-type hsqldb` boot an app against Mendix's built-in, file-based HSQLDB database, so no external database server is needed. + +**Architecture:** A new `--db-type` flag feeds `DBConfig.Type`. A new `applyDatabaseDefaults()` validates the type and fills type-appropriate connection settings (PostgreSQL keeps today's defaults; HSQLDB clears host/user/password). The pre-boot reachability check is skipped for file-based databases. The runtime builds the HSQLDB file URL itself from `DatabaseType` + `DatabaseName`. + +**Tech Stack:** Go 1.26, cobra CLI, Mendix standalone runtime 11.12.2 (HSQLDB 2.7.4 bundled). + +**Spec:** `docs/11-proposals/PROPOSAL_run_local_builtin_database.md` + +**Branch:** `feat/run-local-hsqldb` (based on `fix/windows-local-run-process-helpers` / PR #1147). + +--- + +## File Structure + +| File | Responsibility | +|---|---| +| `cmd/mxcli/docker/dbtype.go` (new) | Canonical DB-type values + `NormalizeDBType` / `IsFileBasedDBType` / `RuntimeDatabaseType`. | +| `cmd/mxcli/docker/dbtype_test.go` (new) | Unit tests for the above. | +| `cmd/mxcli/docker/runlocal.go` | Move DB defaults out of `applyDefaults` into `applyDatabaseDefaults() error`; call it from `RunLocal`; skip TCP check for file-based DBs. | +| `cmd/mxcli/docker/runlocal_test.go` | Update two existing tests; add HSQLDB default/validation tests. | +| `cmd/mxcli/docker/localboot.go` | Add `DBConfig.IsFileBased()`. | +| `cmd/mxcli/docker/localboot_test.go` | Add `TestRuntimeConfigParams_HSQLDB`. | +| `cmd/mxcli/cmd_run.go` | Register `--db-type`, pass it into `DBConfig.Type`, document it. | +| `cmd/mxcli/docker/runlocal_hsqldb_integration_test.go` (new) | `//go:build integration`: boot with HSQLDB, assert HTTP 200 and the files exist. | +| `CHANGELOG.md` | `[Unreleased] / Added` entry. | + +--- + +### Task 1: DB-type helpers + +**Files:** +- Create: `cmd/mxcli/docker/dbtype.go` +- Test: `cmd/mxcli/docker/dbtype_test.go` + +- [ ] **Step 1: Write the failing test** + +Create `cmd/mxcli/docker/dbtype_test.go`: + +```go +// SPDX-License-Identifier: Apache-2.0 + +package docker + +import "testing" + +func TestNormalizeDBType(t *testing.T) { + cases := []struct { + in string + want string + wantErr bool + }{ + {"", DBTypePostgreSQL, false}, + {"postgresql", DBTypePostgreSQL, false}, + {"PostgreSQL", DBTypePostgreSQL, false}, + {"postgres", DBTypePostgreSQL, false}, + {"hsqldb", DBTypeHSQLDB, false}, + {"HSQLDB", DBTypeHSQLDB, false}, + {" Hsqldb ", DBTypeHSQLDB, false}, + {"mysql", "", true}, + } + for _, c := range cases { + got, err := NormalizeDBType(c.in) + if c.wantErr { + if err == nil { + t.Errorf("NormalizeDBType(%q) = %q, want error", c.in, got) + } + continue + } + if err != nil || got != c.want { + t.Errorf("NormalizeDBType(%q) = %q, %v; want %q", c.in, got, err, c.want) + } + } +} + +func TestRuntimeDatabaseType(t *testing.T) { + if got := RuntimeDatabaseType(DBTypeHSQLDB); got != "HSQLDB" { + t.Errorf("RuntimeDatabaseType(hsqldb) = %q, want HSQLDB", got) + } + if got := RuntimeDatabaseType(DBTypePostgreSQL); got != "PostgreSQL" { + t.Errorf("RuntimeDatabaseType(postgresql) = %q, want PostgreSQL", got) + } +} +``` + +- [ ] **Step 2: Run it and confirm it fails** + +Run: `go test -run 'TestNormalizeDBType|TestRuntimeDatabaseType' ./cmd/mxcli/docker/` +Expected: FAIL — `undefined: DBTypePostgreSQL` (and friends). + +- [ ] **Step 3: Write the implementation** + +Create `cmd/mxcli/docker/dbtype.go`: + +```go +// SPDX-License-Identifier: Apache-2.0 + +package docker + +import ( + "fmt" + "strings" +) + +// Canonical database types a local run understands. The runtime's own enum also +// has Db2/MySql/Oracle/SapHana/SqlServer; those all need a server and are out of +// scope for `run --local`. +const ( + DBTypePostgreSQL = "postgresql" + DBTypeHSQLDB = "hsqldb" +) + +// NormalizeDBType maps a user-supplied --db-type to a canonical value. Empty +// means PostgreSQL, which is what a local run has always used, so the flag stays +// backwards compatible. +func NormalizeDBType(raw string) (string, error) { + switch strings.ToLower(strings.TrimSpace(raw)) { + case "", DBTypePostgreSQL, "postgres": + return DBTypePostgreSQL, nil + case DBTypeHSQLDB: + return DBTypeHSQLDB, nil + default: + return "", fmt.Errorf("unknown --db-type %q (want postgresql or hsqldb)", raw) + } +} + +// IsFileBasedDBType reports whether a canonical type is the runtime's built-in +// file database, which has no host to reach and no credentials. +func IsFileBasedDBType(canonical string) bool { + return canonical == DBTypeHSQLDB +} + +// RuntimeDatabaseType maps a canonical type to the spelling the runtime's +// DatabaseType parameter expects (the runtime enum is upper case). +func RuntimeDatabaseType(canonical string) string { + if canonical == DBTypeHSQLDB { + return "HSQLDB" + } + return "PostgreSQL" +} +``` + +- [ ] **Step 4: Run the test and confirm it passes** + +Run: `go test -run 'TestNormalizeDBType|TestRuntimeDatabaseType' ./cmd/mxcli/docker/` +Expected: PASS. + +- [ ] **Step 5: Commit** + +```bash +git add cmd/mxcli/docker/dbtype.go cmd/mxcli/docker/dbtype_test.go +git commit -m "feat: add db-type normalization helpers for run --local" +``` + +--- + +### Task 2: Type-aware database defaults + +**Files:** +- Modify: `cmd/mxcli/docker/runlocal.go` (`applyDefaults` at ~166-200, `RunLocal` at ~542) +- Test: `cmd/mxcli/docker/runlocal_test.go` + +- [ ] **Step 1: Write the failing tests** + +Append to `cmd/mxcli/docker/runlocal_test.go`: + +```go +func TestLocalRunOptions_DatabaseDefaults_HSQLDB(t *testing.T) { + o := LocalRunOptions{ProjectPath: "/proj/App1112.mpr", DB: DBConfig{Type: "hsqldb"}} + o.applyDefaults() + if err := o.applyDatabaseDefaults(); err != nil { + t.Fatalf("applyDatabaseDefaults: %v", err) + } + if o.DB.Type != "HSQLDB" { + t.Errorf("Type = %q, want HSQLDB", o.DB.Type) + } + if o.DB.Host != "" || o.DB.User != "" || o.DB.Password != "" { + t.Errorf("HSQLDB must carry no host/credentials, got %+v", o.DB) + } + if o.DB.Name != "app1112" { + t.Errorf("DB.Name = %q, want app1112", o.DB.Name) + } +} + +func TestLocalRunOptions_DatabaseDefaults_HSQLDBRejectsConnectionFlags(t *testing.T) { + for _, db := range []DBConfig{ + {Type: "hsqldb", Host: "127.0.0.1:5432"}, + {Type: "hsqldb", User: "u"}, + {Type: "hsqldb", Password: "p"}, + } { + o := LocalRunOptions{ProjectPath: "/proj/App.mpr", DB: db} + o.applyDefaults() + if err := o.applyDatabaseDefaults(); err == nil { + t.Errorf("hsqldb with %+v: want an error", db) + } + } +} + +func TestLocalRunOptions_DatabaseDefaults_HSQLDBRejectsEnsureDB(t *testing.T) { + o := LocalRunOptions{ProjectPath: "/proj/App.mpr", DB: DBConfig{Type: "hsqldb"}, EnsureDB: true} + o.applyDefaults() + if err := o.applyDatabaseDefaults(); err == nil { + t.Error("hsqldb + --ensure-db: want an error") + } +} + +func TestLocalRunOptions_DatabaseDefaults_UnknownType(t *testing.T) { + o := LocalRunOptions{ProjectPath: "/proj/App.mpr", DB: DBConfig{Type: "mysql"}} + o.applyDefaults() + if err := o.applyDatabaseDefaults(); err == nil { + t.Error("unknown db type: want an error") + } +} +``` + +- [ ] **Step 2: Run them and confirm they fail** + +Run: `go test -run 'TestLocalRunOptions_DatabaseDefaults' ./cmd/mxcli/docker/` +Expected: FAIL — `o.applyDatabaseDefaults undefined`. + +- [ ] **Step 3: Remove the DB block from `applyDefaults` and add `applyDatabaseDefaults`** + +In `cmd/mxcli/docker/runlocal.go`, delete this block from `applyDefaults()`: + +```go + if o.DB.Type == "" { + o.DB.Type = "PostgreSQL" + } + if o.DB.Host == "" { + o.DB.Host = "127.0.0.1:5432" + } + if o.DB.User == "" { + o.DB.User = "mendix" + } + if o.DB.Password == "" { + o.DB.Password = "mendix" + } + if o.DB.Name == "" { + o.DB.Name = deriveDBName(o.ProjectPath) + } +``` + +Add this method immediately after `applyDefaults()`: + +```go +// applyDatabaseDefaults validates --db-type and fills the connection settings the +// chosen database needs. It is separate from applyDefaults because it can fail: +// a flag combination that cannot work (--db-type hsqldb with --db-host) is a user +// error, not something to silently normalise away. +func (o *LocalRunOptions) applyDatabaseDefaults() error { + kind, err := NormalizeDBType(o.DB.Type) + if err != nil { + return err + } + if o.DB.Name == "" { + o.DB.Name = deriveDBName(o.ProjectPath) + } + if IsFileBasedDBType(kind) { + // The built-in database is a file: it has no host or credentials, and + // accepting them would imply a connection that never happens. + if o.DB.Host != "" || o.DB.User != "" || o.DB.Password != "" { + return fmt.Errorf("--db-type hsqldb uses the built-in file database and takes no " + + "--db-host, --db-user or --db-password") + } + if o.EnsureDB { + return fmt.Errorf("--ensure-db provisions PostgreSQL; the built-in HSQLDB database " + + "needs no provisioning — drop --ensure-db") + } + o.DB.Type = RuntimeDatabaseType(kind) + o.DB.Host, o.DB.User, o.DB.Password = "", "", "" + return nil + } + // PostgreSQL (unchanged behaviour). + if o.DB.Host == "" { + o.DB.Host = "127.0.0.1:5432" + } + if o.DB.User == "" { + o.DB.User = "mendix" + } + if o.DB.Password == "" { + o.DB.Password = "mendix" + } + o.DB.Type = RuntimeDatabaseType(kind) + return nil +} +``` + +- [ ] **Step 4: Call it from `RunLocal`** + +In `RunLocal`, change the first two lines: + +```go + opts.applyDefaults() + w, stderr := opts.Stdout, opts.Stderr +``` + +to: + +```go + opts.applyDefaults() + if err := opts.applyDatabaseDefaults(); err != nil { + return err + } + w, stderr := opts.Stdout, opts.Stderr +``` + +- [ ] **Step 5: Fix the two existing tests that relied on DB defaults in `applyDefaults`** + +In `cmd/mxcli/docker/runlocal_test.go`, in `TestLocalRunOptions_Defaults`, after `o.applyDefaults()` add: + +```go + if err := o.applyDatabaseDefaults(); err != nil { + t.Fatalf("applyDatabaseDefaults: %v", err) + } +``` + +In `TestLocalRunOptions_DefaultsRespectOverrides`, after `o.applyDefaults()` add: + +```go + if err := o.applyDatabaseDefaults(); err != nil { + t.Fatalf("applyDatabaseDefaults: %v", err) + } +``` + +- [ ] **Step 6: Run the package tests** + +Run: `go test -run 'TestLocalRunOptions' ./cmd/mxcli/docker/` +Expected: PASS (all `TestLocalRunOptions_*`). + +- [ ] **Step 7: Commit** + +```bash +git add cmd/mxcli/docker/runlocal.go cmd/mxcli/docker/runlocal_test.go +git commit -m "feat: validate --db-type and fill type-aware database defaults" +``` + +--- + +### Task 3: Skip the TCP reachability check for a file database + +**Files:** +- Modify: `cmd/mxcli/docker/localboot.go` (add method near `DBConfig`) +- Modify: `cmd/mxcli/docker/runlocal.go` (the DB check at ~608) +- Test: `cmd/mxcli/docker/runlocal_test.go` + +- [ ] **Step 1: Write the failing test** + +Append to `cmd/mxcli/docker/runlocal_test.go`: + +```go +func TestDBConfig_IsFileBased(t *testing.T) { + if !(DBConfig{Type: "HSQLDB"}).IsFileBased() { + t.Error("HSQLDB should be file-based") + } + if (DBConfig{Type: "PostgreSQL"}).IsFileBased() { + t.Error("PostgreSQL is not file-based") + } + if (DBConfig{Type: "hsqldb"}).IsFileBased() { + t.Error("the check must use the runtime spelling set by applyDatabaseDefaults, not the raw flag") + } +} +``` + +- [ ] **Step 2: Run it and confirm it fails** + +Run: `go test -run TestDBConfig_IsFileBased ./cmd/mxcli/docker/` +Expected: FAIL — `c.IsFileBased undefined`. + +- [ ] **Step 3: Add the method** + +In `cmd/mxcli/docker/localboot.go`, immediately after the `DBConfig` struct, add: + +```go +// IsFileBased reports whether this is the runtime's built-in file database, which +// has no host to reach. It keys on the runtime spelling set by +// applyDatabaseDefaults. +func (c DBConfig) IsFileBased() bool { + return c.Type == RuntimeDatabaseType(DBTypeHSQLDB) +} +``` + +- [ ] **Step 4: Skip the check in `RunLocal`** + +In `cmd/mxcli/docker/runlocal.go`, change: + +```go + } else if err := pingTCP(opts.DB.Host, 3*time.Second); err != nil { +``` + +to: + +```go + } else if !opts.DB.IsFileBased() { + if err := pingTCP(opts.DB.Host, 3*time.Second); err != nil { + return fmt.Errorf("database not reachable at %s: %w\n"+ + " Pass --ensure-db to provision it, or start Postgres and create the '%s' database (user %q).", + opts.DB.Host, err, opts.DB.Name, opts.DB.User) + } + } +``` + +(Delete the original `return fmt.Errorf(...)` body that followed the old `else if`, so the block above replaces it exactly.) + +- [ ] **Step 5: Run the package tests** + +Run: `go test -run 'TestDBConfig_IsFileBased|TestLocalRunOptions' ./cmd/mxcli/docker/` +Expected: PASS. + +- [ ] **Step 6: Commit** + +```bash +git add cmd/mxcli/docker/localboot.go cmd/mxcli/docker/runlocal.go cmd/mxcli/docker/runlocal_test.go +git commit -m "feat: skip the TCP check for the built-in file database" +``` + +--- + +### Task 4: The runtime receives the HSQLDB configuration + +**Files:** +- Test: `cmd/mxcli/docker/localboot_test.go` + +`runtimeConfigParams` already forwards `o.DB.Type/Host/User/Password`; this task +pins the HSQLDB shape so a future edit cannot regress it. + +- [ ] **Step 1: Write the test** + +Append to `cmd/mxcli/docker/localboot_test.go`: + +```go +func TestRuntimeConfigParams_HSQLDB(t *testing.T) { + o := testLocalOpts() + o.DB = DBConfig{Type: "HSQLDB", Name: "appdb"} + p := runtimeConfigParams(o, nil) + checks := map[string]any{ + "DatabaseType": "HSQLDB", + "DatabaseHost": "", + "DatabaseName": "appdb", + "DatabaseUserName": "", + "DatabasePassword": "", + } + for k, want := range checks { + if p[k] != want { + t.Errorf("%s = %v, want %v", k, p[k], want) + } + } +} +``` + +- [ ] **Step 2: Run it** + +Run: `go test -run TestRuntimeConfigParams_HSQLDB ./cmd/mxcli/docker/` +Expected: PASS (the mapping already forwards these). + +- [ ] **Step 3: Commit** + +```bash +git add cmd/mxcli/docker/localboot_test.go +git commit -m "test: pin the HSQLDB boot configuration the runtime receives" +``` + +--- + +### Task 5: Wire up the `--db-type` flag + +**Files:** +- Modify: `cmd/mxcli/cmd_run.go` (flag registration ~316; flag read ~150; `DBConfig` ~221; long help ~38) + +- [ ] **Step 1: Read the flag** + +In `cmd/mxcli/cmd_run.go`, next to the other DB flags, add: + +```go + dbType, _ := cmd.Flags().GetString("db-type") +``` + +- [ ] **Step 2: Pass it into `DBConfig`** + +Change the `DB` literal: + +```go + DB: docker.DBConfig{ + Host: dbHost, + Name: dbName, + User: dbUser, + Password: dbPassword, + }, +``` + +to: + +```go + DB: docker.DBConfig{ + Type: dbType, + Host: dbHost, + Name: dbName, + User: dbUser, + Password: dbPassword, + }, +``` + +- [ ] **Step 3: Register the flag** + +Next to the other `db-*` registrations, add: + +```go + runCmd.Flags().String("db-type", "", "Database type for a local run: postgresql (default) or hsqldb (the runtime's built-in file database — no server, no --db-host/--db-user/--db-password, data under /deployment/data/database/hsqldb/)") +``` + +- [ ] **Step 4: Document it in the long help** + +In the `run` command's long help, after the line about `--db-host/--db-name/...`, add: + +``` + - --db-type hsqldb runs against the built-in file database instead of + PostgreSQL, so no database server is needed. Its data lives under the + project's deployment directory and is meant for local development only. +``` + +- [ ] **Step 5: Build and smoke-test the flag** + +Run: +```bash +go build -o bin/mxcli.exe ./cmd/mxcli +./bin/mxcli.exe run --help | grep -A2 -- --db-type +``` +Expected: the flag is listed with the HSQLDB description. + +- [ ] **Step 6: Verify the refusal is user-facing** + +Run: +```bash +./bin/mxcli.exe run --local -p EmptyApp/EmptyApp.mpr --db-type hsqldb --db-host 127.0.0.1:5432 +``` +Expected: exits non-zero with `--db-type hsqldb uses the built-in file database and takes no --db-host...`. + +- [ ] **Step 7: Commit** + +```bash +git add cmd/mxcli/cmd_run.go +git commit -m "feat: add the --db-type flag to run --local" +``` + +--- + +### Task 6: Integration test — HSQLDB needs no database server + +**Files:** +- Create: `cmd/mxcli/docker/runlocal_hsqldb_integration_test.go` + +- [ ] **Step 1: Write the test** + +Create `cmd/mxcli/docker/runlocal_hsqldb_integration_test.go`: + +```go +// SPDX-License-Identifier: Apache-2.0 + +//go:build integration + +package docker + +import ( + "io" + "net/http" + "os" + "os/exec" + "path/filepath" + "testing" + "time" +) + +// The point of the built-in database: an app boots with NO database server. +// Modeled on localapp_integration_test.go — scaffold a project when no fixture +// is given, build it with a real mxbuild serve, then boot the runtime with +// HSQLDB and require it to answer. Self-contained: HSQLDB is a file, so there is +// nothing else to stand up. +func TestRunLocal_HSQLDBNeedsNoDatabaseServer(t *testing.T) { + mprPath := os.Getenv("MXCLI_IT_PROJECT") + if mprPath == "" { + mxPath, err := ResolveMx("") + if err != nil { + t.Skipf("mx not resolvable and MXCLI_IT_PROJECT unset: %v", err) + } + dir, err := os.MkdirTemp("", "mxhsql") + if err != nil { + t.Fatal(err) + } + defer os.RemoveAll(dir) + scaffold := exec.Command(mxPath, "create-project") + scaffold.Dir = dir + if out, err := scaffold.CombinedOutput(); err != nil { + t.Skipf("mx create-project failed: %v\n%s", err, out) + } + mprPath = filepath.Join(dir, "App.mpr") + } + if _, err := os.Stat(mprPath); err != nil { + t.Skipf("no project fixture at %s: %v", mprPath, err) + } + + reader, err := openReadOnly(mprPath) + if err != nil { + t.Skipf("cannot open project: %v", err) + } + version := reader.ProjectVersion().ProductVersion + reader.Disconnect() + + installPath, err := resolveRuntimeInstall(version, io.Discard) + if err != nil { + t.Skipf("no runtime for %s: %v", version, err) + } + javaMajor, _ := ProjectJavaMajor(mprPath) + + serve, err := StartServe(ServeOptions{Version: version, JavaMajor: javaMajor, Host: "127.0.0.1", Port: 6549}) + if err != nil { + t.Skipf("cannot start mxbuild serve: %v", err) + } + defer serve.Stop() + + build, err := serve.Build(BuildRequest{Target: TargetDeploy, ProjectFilePath: mprPath}) + if err != nil { + t.Fatalf("build: %v", err) + } + if !build.OK() { + t.Fatalf("build failed: %s", build.Message) + } + + deployDir := filepath.Join(filepath.Dir(mprPath), "deployment") + rt, err := StartLocalRuntime(LocalRuntimeOptions{ + DeployDir: deployDir, + InstallPath: installPath, + JavaMajor: javaMajor, + AdminPass: defaultLocalAdminPass, + AppPort: 8087, + AdminPort: 8097, + DB: DBConfig{Type: "HSQLDB", Name: deriveDBName(mprPath)}, + Stdout: io.Discard, + Stderr: io.Discard, + }) + if err != nil { + t.Fatalf("boot with the built-in database: %v", err) + } + defer rt.Stop() + + deadline := time.Now().Add(90 * time.Second) + for { + resp, err := http.Get(rt.AppURL()) + if err == nil { + resp.Body.Close() + if resp.StatusCode == 200 { + break + } + } + if time.Now().After(deadline) { + t.Fatalf("app did not answer 200 at %s", rt.AppURL()) + } + time.Sleep(time.Second) + } + + hsqlDir := filepath.Join(deployDir, "data", "database", "hsqldb") + entries, err := os.ReadDir(hsqlDir) + if err != nil || len(entries) == 0 { + t.Fatalf("expected the built-in database files under %s (err=%v, n=%d)", hsqlDir, err, len(entries)) + } +} +``` + +- [ ] **Step 2: Compile the integration build** + +Run: `go test -tags integration -run TestRunLocal_HSQLDBNeedsNoDatabaseServer -count=1 ./cmd/mxcli/docker/ -c -o /dev/null` +Expected: compiles (no `undefined`). + +- [ ] **Step 3: Run it where a runtime is cached (Linux CI / this dev box)** + +Run: `go test -tags integration -run TestRunLocal_HSQLDBNeedsNoDatabaseServer -count=1 -v ./cmd/mxcli/docker/` +Expected: PASS (or SKIP with a clear reason when mxbuild/runtime are absent). If it fails because the HSQLDB files are not under `data/database/hsqldb/`, record the actual directory the runtime logged and update the assertion — that is the one empirical fact this test exists to pin. + +The directory assertion was empirically confirmed: the files land at +`deployment/data/database/hsqldb//app.properties` and `app.script`. The +Gradle bin directory must be on `PATH` (e.g. `/c/Program Files/Mendix/gradle-8.5/bin`) +with `MENDIX_GRADLE_HOME` at the Gradle root, or `mxbuild serve` cannot build. + +- [ ] **Step 4: Commit** + +```bash +git add cmd/mxcli/docker/runlocal_hsqldb_integration_test.go +git commit -m "test: prove run --local boots on the built-in database with no server" +``` + +--- + +### Task 6b: Create the schema when the database does not exist yet + +**Files:** +- Modify: `cmd/mxcli/docker/runtime_controller.go` (`needsDBUpdate`) +- Test: `cmd/mxcli/docker/runtime_controller_test.go` (`TestNeedsDBUpdate`) + +The integration test proved that a **fresh** HSQLDB database does not boot: Mendix's +HSQLDB URL carries `ifexists=true`, so it never creates the file, and `start` +answers M2EE result 2 ("The database to be used does not exist."). `needsDBUpdate` +only recognised result 3. + +- [ ] Add test cases `{"result2", &M2EEResponse{Result: 2}, true}` and + `{"no-existing-db-message", &M2EEResponse{Message: "The database to be used does not exist."}, true}`. +- [ ] In `needsDBUpdate`, return true for `resp.Result == 2 || resp.Result == 3`, and + match the lower-cased message `"database to be used does not exist"` as well as + `"database has to be updated"`. +- [ ] In `Start`, track that the schema step ran and report + `start failed after creating or updating the database schema: %s` if the second + start still fails. +- [ ] Commit: `fix: create the schema when the database does not exist yet`. + +--- + +### Task 7: CHANGELOG, full verification, PR + +**Files:** +- Modify: `CHANGELOG.md` + +- [ ] **Step 1: Add the changelog entry** + +Under `## [Unreleased]` → `### Added`, add: + +```markdown +- **`run --local --db-type hsqldb` — boot with no database server** — a local run + can now use Mendix's built-in, file-based HSQLDB instead of PostgreSQL. The + runtime already ships the driver and derives the file URL from `DatabaseType` + and `DatabaseName`, so the data lands under + `/deployment/data/database/hsqldb/`. `--db-host`, `--db-user` and + `--db-password` are refused with HSQLDB (it is a file, not a connection) and + `--ensure-db` is refused too (there is nothing to provision); PostgreSQL + remains the default and is unchanged. Intended for local development and + demos only — Mendix marks HSQLDB as local-testing-only. +``` + +- [ ] **Step 2: Run the full unit suite for the package** + +Run: `go test -count=1 ./cmd/mxcli/docker/` +Expected: PASS on Linux/CI. On Windows this package has pre-existing POSIX-only +failures (#897) — run the scoped set instead: +`go test -count=1 -run 'TestNormalizeDBType|TestRuntimeDatabaseType|TestLocalRunOptions|TestDBConfig_IsFileBased|TestRuntimeConfigParams' ./cmd/mxcli/docker/` + +- [ ] **Step 3: gofmt + vet** + +Run: +```bash +gofmt -l cmd/mxcli/docker/dbtype.go cmd/mxcli/docker/dbtype_test.go cmd/mxcli/docker/runlocal.go cmd/mxcli/docker/runlocal_test.go cmd/mxcli/docker/localboot.go cmd/mxcli/docker/localboot_test.go cmd/mxcli/cmd_run.go cmd/mxcli/docker/runlocal_hsqldb_integration_test.go +go vet ./cmd/mxcli/docker/ ./cmd/mxcli/ +``` +Expected: no output from `gofmt -l`; `go vet` clean. + +- [ ] **Step 4: Commit and push** + +```bash +git add CHANGELOG.md +git commit -m "docs: changelog for the built-in HSQLDB database" +git push -u origin feat/run-local-hsqldb +``` + +- [ ] **Step 5: Open the PR** + +Base it on `main` (or on `fix/windows-local-run-process-helpers` until #1147 merges): + +```bash +gh pr create --repo mendixlabs/mxcli --base main --head engalar:feat/run-local-hsqldb \ + --title "feat: run --local with the built-in HSQLDB database" \ + --body-file - <<'EOF' +Adds `--db-type postgresql|hsqldb` to `run --local`. With `hsqldb` the app boots +against Mendix's built-in file database, so no database server is needed; the +data lands under `/deployment/data/database/hsqldb/`. + +- `--db-type` defaults to `postgresql`; existing behaviour is unchanged. +- HSQLDB refuses `--db-host`/`--db-user`/`--db-password` and `--ensure-db`. +- The pre-boot TCP check is skipped for the file database. +- Unit tests pin the type mapping and validations; an integration test boots a + scaffolded app with HSQLDB and asserts HTTP 200 plus the data files. + +Spec: docs/11-proposals/PROPOSAL_run_local_builtin_database.md +Depends on #1147 (Windows process-helper fix). +EOF +``` + +Expected: a PR URL. + +--- + +## Self-Review + +- **Spec coverage:** CLI (`--db-type`) → Task 5; config mapping → Tasks 2-4; skip reachability → Task 3; `--ensure-db` refusal → Task 2; data location → Task 6; unit + integration tests → Tasks 1-6; CHANGELOG/PR → Task 7. No gaps. +- **Placeholders:** none — every code step carries the full code. +- **Type consistency:** `NormalizeDBType`, `IsFileBasedDBType`, `RuntimeDatabaseType`, `DBConfig.IsFileBased`, `applyDatabaseDefaults` are used with the same names and signatures throughout. +- **Known risk:** the exact HSQLDB directory (`data/database/hsqldb`) comes from strings read out of the runtime jars; Task 6 Step 3 is the checkpoint that confirms it and is the only place a correction would be needed.