Implement typed publish-only database environments

[cloutiertyler]

Description of Changes

Implements typed, publish-only environment variables: declared schemas, typed context accessors, CLI configuration resolution, and atomic replacement with the module on publish. Values are private and read-only to module code. Includes an ENV usage guide. Companion: SpacetimeDBPrivate#3940.

API and ABI breaking changes

Adds environment declarations to V10 metadata and new host imports. Modules using these additions require an updated host. No V11 ABI.

Rollback safety impact

n/a (no prerequisite PRs).

Hosts without ENV support cannot load modules using the new metadata. Rolling back those databases requires a migration.

Expected complexity level and risk

4/5. Changes span publication, durable storage, transaction consistency, secret access controls, and all four module libraries.

Testing

CLI smoke tests, schema validation, typed accessors, atomic publish/rollback, view refresh, submodule access restrictions, and restart/recovery tests.

[cloutiertyler]

Looks like the tags are being reordered here, was this a bug with the old code?

[cloutiertyler]

Yes. The old C++ reader had these reversed: canonical BSATN uses tag 0 for Some and tag 1 for None. This fixes the reader without changing the wire format. The regression covers missing, present-empty, and embedded-NUL values, and checks that the following field is still decoded correctly.

[cloutiertyler]

Is this an API breaking change? Should we consider just using HTTP headers instead of putting these in a map in the body?

[cloutiertyler]

Old clients remain supported by the new server: the JSON envelope is selected by application/vnd.spacetimedb.publish+json, while raw module bodies still work and supply an empty env map. There is a compatibility gap in the other direction: this CLI currently sends the envelope even for modules without env declarations. I'll retain raw-body publishing for those modules so they can still target older servers.

I would keep env values in the body. We allow up to 256 values of 8 KiB each, which is too large for typical HTTP header limits, and values can contain characters unsuitable for headers. A separate body format also keeps module bytes and the complete env map in one publish request.

[cloutiertyler]

Alright, I suppose. It just makes publishing more complicated for people who are currently publishing via HTTP directly (imagine publishing from a module in a procedure for example). We could also decrease the maximum allowed env value size, or otherwise decrease the number of env variables, or maybe just set a total size limit.

[cloutiertyler]

Agreed. A lower aggregate limit on the encoded environment would make headers feasible; the current limits are a design choice, not a reason headers are impossible. The body format adds base64/JSON work for direct HTTP publishers, including procedures, and I should have weighed that more explicitly. Raw-body publishing still works for requests without ENV values. I'll document the direct HTTP example and the header alternative, including the aggregate limit and encoding it would require, rather than treating the current envelope as inevitable.

[cloutiertyler]

Where will you document it?

[cloutiertyler]

In the existing HTTP database API reference, under a new "Publishing with environment values" subsection covering both POST and PUT. I've written the content type, JSON shape, complete-replacement rules, compatibility behavior, and a complete Python example. The ENV guide links to it. I also added the HTTP-header alternative and its encoding/aggregate-size tradeoffs to proposal #3942.

[cloutiertyler]

Why a Python example?

[cloutiertyler]

I chose Python because its standard library made the Base64 and JSON encoding easy to show in one script; it was an arbitrary example choice. I replaced it with a curl/jq example in the HTTP API reference and kept the exact wire format explicit for procedures and other HTTP clients. The example preserves quotes, newlines, and Unicode in values.

[cloutiertyler]

Note for other reviewers: are we cool with just not logging this?

[cloutiertyler]

The intent is to keep submitted SQL and parser diagnostics out of shared logs because either can contain secret values. We still log the SQL byte count and a generic rejection warning, and return the detailed error to the caller. This does reduce diagnostic detail; a structured error category would let us recover some of it without recording query text or values.

[cloutiertyler]

I don't think the log level should be warn here. I think it should be debug.

[cloutiertyler]

Changed the rejection message to debug. It still omits SQL text and parser diagnostics; the caller receives the detailed error.

[cloutiertyler]

Note for @coolreader18. The changes in this file seem a little complex to my eyes. I'm wondering if there's a better way to manage this.

[cloutiertyler]

There are a few separable changes here: passing the complete environment through publication, loading initial values only for a new database, and preserving the running host when publication fails. The cleanup additions came from actual rejected-publication and reset tests: an error could leave the host registry empty, and an unused candidate could wait on a scheduler that was never started. I agree the control flow could be clearer. A small candidate-cleanup helper would reduce duplication while retaining those guarantees.

[cloutiertyler]

My understanding is that the changes in this file make it so that a view that panics returns an error, rather than an empty view.

I'm not sure what the intended behavior @joshua-spacetime had in mind. I could see it going either way.

[cloutiertyler]

Yes. This changes ordinary SQL and subscription materialization to return an error when a view fails, instead of continuing with its backing table. That is a broader behavior change than environment support. I have not reproduced it on unmodified master or established that an empty result violates the intended contract, so I would separate this change and confirm the intended behavior with Joshua.

[cloutiertyler]

This is also related to that trap bug.

[cloutiertyler]

This part is separate from the empty-view/error change. A publish can change ENV and require client disconnection at the same time. The old branch skipped refreshing views in that case, including materializations created by ordinary SQL with no live subscriber to disconnect. This preserves client disconnection while refreshing surviving views against the newly published environment in the same transaction. I would keep that ENV consistency fix separate from the broader trap behavior changes.

[cloutiertyler]

Maybe we should suggest which of the various configuration files should contain secrets? I think basically any *.local.* file is not meant to be checked in, right?

[cloutiertyler]

Yes. I'll explicitly recommend spacetime.local.json or spacetime.{environment}.local.json for local secrets and show the corresponding .gitignore entries. The .local convention indicates personal configuration, but the documentation should still tell users to ensure those files are ignored. Checked-in spacetime.json and spacetime.{environment}.json should contain only non-secret defaults.

[cloutiertyler]

Added the specific recommendation to keep secrets in spacetime.local.json or spacetime.{environment}.local.json, together with the .gitignore patterns. Checked-in configuration is documented as containing non-secret defaults.

[lisandroct]

Extremely minor change to handle some collisions with other methods in C# (besides the one that are already being handled).

[JasonAtClockwork]

Ran through ~8 different test cases end to end with C++ and only HandlerContext was missing (which I've added), other than the Windows spacetime publish issue everything worked.

[JasonAtClockwork]

Suggested change

	let mut child = tokio::process::Command::new(extractor)
	.arg("extract-schema")
	.arg(&module)
	.arg("--host-type")
	.arg(host_type.to_ascii_lowercase())
	.env_clear()
	.stdin(Stdio::null())
	.stdout(Stdio::piped())
	.stderr(Stdio::null())
	.kill_on_drop(true)
	.spawn()
	.context("Cannot start local module schema inspection")?;
	let mut command = tokio::process::Command::new(extractor);
	command
	.arg("extract-schema")
	.arg(&module)
	.arg("--host-type")
	.arg(host_type.to_ascii_lowercase())
	.env_clear()
	.stdin(Stdio::null())
	.stdout(Stdio::piped())
	.stderr(Stdio::null())
	.kill_on_drop(true);
	// Windows uses PATH when resolving dependent DLLs for the child
	// process. Preserve only PATH while keeping the extractor's
	// environment otherwise isolated.
	#[cfg(windows)]
	if let Some(path) = std::env::var_os("PATH") {
	command.env("PATH", path);
	}
	let mut child = command
	.spawn()
	.context("Cannot start local module schema inspection")?;

During my testing spacetimedb-cli publish would fail because .env_clear() removed PATH.

[cloutiertyler]

Applied in e98ef53: schema inspection preserves PATH on Windows while clearing other inherited variables. Added a subprocess regression for environment isolation and the Windows PATH configuration. The five focused schema-extraction tests and strict CLI Clippy pass locally on macOS; Windows publishing verification is pending CI. Your HandlerContext change is included.