Database Packaging by isc-dchui · Pull Request #1121 · intersystems/ipm

isc-dchui · 2026-04-13T19:41:06Z

Description

Resolves #986

Overview

The feature adds a new lifecycle class, %IPM.Lifecycle.Database, that sits alongside the existing %IPM.Lifecycle.Module. It inherits from %IPM.Lifecycle.Base directly — not from Module — because Module has PACKAGING as a [Final] parameter and can't be subclassed to change it.
Two new commands are exposed in %IPM.Main:
- package-database → forces %IPM.Lifecycle.Database, routes to the Package phase
- publish-database → same, but routes to Publish (which calls Package first, then uploads to registry)
The -swap-db modifier is added to load/install/update to authorize the DB swap during install.

Key Details

Package = IRIS.DAT + metadata, not source. The .tgz contains the binary IRIS.DAT, an enriched module.xml (with database and a SHA-256 ), a deps/ directory of dependency manifests, and any non-compiled resource files (wheels, FileCopy, CPF). Nothing is recompiled at install time.
Remap-before-dismount during packaging. When building the package, the namespace is remapped to a fresh empty temp DB before the old DB is dismounted. This avoids any window where the namespace has no mounted routines DB.
Generated resources need special handling. Classes with Generated="true" are exported from the source DB to XML before the remap, then imported into the temp DB after. Once the old DB is dismounted, compiled generated classes are inaccessible.
Install skips Reload and Compile entirely. IsInstallContext() checks two conditions: Packaging=database AND IRIS.DAT is present in the module root. If both are true, %Reload sets SkipInvokes=1 and returns, %Compile is a no-op, and %Activate performs the swap. Pre-Activate elements are suppressed because the packaged classes don't exist until the DB is mounted.
Swap is a rename, not a copy. PerformDatabaseSwap dismounts the existing DB, renames the current IRIS.DAT to IRIS__.DAT as a backup, renames the packaged IRIS.DAT into place, and remounts. Renames are atomic and avoid copying multi-GB files.
Rollback is best-effort. On any failure after the swap starts, RollbackDatabaseSwap attempts dismount → delete new IRIS.DAT → rename backup back → remount. It never throws — it logs warnings and continues since it's called from a catch block.
Update step seeding on fresh install. %Activate calls HandleAllUpdateSteps(..., seedOnly=1) on a fresh install so a later update only runs steps introduced since that version. Skipped when params("Update")=1 (an actual upgrade).
One database package per namespace. ValidateBeforeSwap checks %IPM_Storage.ModuleItem for any other module with Packaging='database' and rejects the install if one exists.
Python deps are on by default for package-database. Unlike package (which defaults off), package-database defaults ExportPythonDependencies=1. Both requirements.txt-based deps (resolved to wheels via ExportPythonDependencies) and explicit resources are staged into the .tgz. PythonWheel install at runtime fires during Initialize phase, which has no Database.cls override.
Non-compiled resources are handled by resource processor hooks, not %Activate. FileCopy, WebApplication, and CPF processors run via OnBeforePhase/OnAfterPhase at the Module level before the lifecycle method is called. Overriding %Activate in Database.cls doesn't suppress them.
publish-database forces the lifecycle class. The handler sets tCommandInfo("data","Lifecycle") = "%IPM.Lifecycle.Database", then re-routes to the existing publish command handler. Base.%Publish calls ..Package(.pParams, 0) first (which populates ..Payload), then uploads to the registry. No Database.cls override of %Publish is needed.
dependencies.xml is actually a deps/ directory of per-dependency IRIS export XML files, each loadable with $system.OBJ.Load. RegisterDependencyMetadata loads them at install time to populate IPM storage — the code is already in the swapped-in DB, so only the IPM metadata records need registering. This is much easier to handle instead of using a fragile XSLT for a giant dependencies.xml
ValidateIPMNotInRoutinesDB is called at two points: immediately in ShellInternal (fail-fast before any work) and again inside %Package (defense-in-depth). It checks %Library.RoutineMgr.IsMapped("%IPM.Main.CLS").
The XSLT transform (InjectDatabasePackagingTransform XData) injects and into module.xml. The checksum placeholder REPLACECHECKSUM is substituted in ObjectScript before the stylesheet is compiled, since XSLT 1.0 has no parameterized element content.
When installing without swap-db, only installs source-packaged modules. If none exist, but database-packaged module(s) exist, informs user. When installing with swap-db, only installs database-packaged modules.
When publishing database-packaged modules to OCI registry, sets com.intersystems.ipm.packaging manifest annotation to "database" (for backwards compatibility, source packaged modules will not have this annotation) and use <module-version>_database__<IRIS-version> tag

Testing

All tests are in Test.PM.Integration.DatabasePackaging. The test infrastructure uses two shared lazy-initialized namespaces to avoid ~15 namespace create/teardown cycles per run:
- SharedNS1 (TESTDBPKGNS1): simple-db-module, module-with-invokes, module-with-dependencies
- SharedNS2 (TESTDBPKGNS2): all-resources-module, module-with-mixed-python, module-with-requirements; also has the zot ORAS registry configured
Per-test install namespaces are created fresh and torn down in OnAfterOneTest.

Test	What it covers
TestSimplePackageAndInstall	Package → verify IRIS.DAT + module.xml content (Packaging, checksum, SystemRequirements version) → install with `load -swap-db` → classes callable, module in list and history
TestPackageAndInstallWithDependencies	Package dep-module + main-with-deps; install main-with-deps → main class exists, dep appears in `list`
TestPackageAndInstallAllResourceTypes	Smoke test: module with class, include file, generated class, WebApp, FileCopy, PythonWheel packages and installs; include constant accessible, generated class callable, WebApp created
TestPackageAndInstallWithTestResources	`-include-test-resources` includes Scope=test (TestHelper) and UnitTest (Test) classes; default packaging excludes both
TestPackageWithUseCurrentDbFlag	`-use-current-db` completes without error and produces a package file
TestPackagingAndInstallValidation	Valid install succeeds; tampered IRIS.DAT (checksum mismatch), zero-sized IRIS.DAT, and missing `deps/` directory each fail validation
TestBackupAndRollback	Backup file created in routines DB directory after swap
TestUpgradeSourceToDatabase	Source-installed module upgraded via `update -path -swap-db`; Packaging changes to "database"; backup confirms DB swap occurred
TestUninstallDatabasePackage	Module removed from `list`; backup preserved; compiled classes gone from namespace
TestMixedPackaging	Database + source package coexist in same namespace; second database package in same namespace is rejected
TestPythonDependencyPackaging	module-with-mixed-python: explicit PythonWheel (lune) + requirements.txt wheel (pycparser) both included by default, excluded with `-no-export-python-deps`; lune importable after install. module-with-requirements: packaging wheel included by default, excluded with flag
TestInvokeBehaviorDuringDatabaseInstall	`<Invoke After="Compile">` skipped during database install; `<Invoke After="Activate">` runs; global markers confirm each
TestNonCompiledResourcesAppliedDuringInstall	CPF file staged in package; FileCopy target deleted before install, confirmed recreated after install — proves resource processor hooks fire independently of `%Activate` override
TestUpdateStepsWithDatabasePackaging	v1 fresh install seeds Step001 (not run); v2 upgrade runs Step002 once; Step001 still not run; backup count > 1 confirms second swap
TestPublishDatabaseWithORAS	publish-database + publish to zot; source install (no `-swap-db`) → Packaging=module; database install (`-swap-db`) → Packaging=database; main-with-deps install resolves dep as source; after unpublishing source tag, install without `-swap-db` fails with hint to use `-swap-db`
TestSystemRequirementsOverwrite	`package-database` overwrites a stale IRIS version in `<SystemRequirements>` with the current version

Not Yet Handled

Mirroring

Checklist

This branch has the latest changes from the main branch rebased or merged.
Changelog entry added.
Unit (zpm test -only) and integration tests (zpm verify -only) pass.
Style matches the style guide in the contributing guide.
Documentation has been/will be updated
- Source controlled docs, e.g. README.md, should be included in this PR and Wiki changes should be made after this PR is merged (add an extra issue for this if needed)
Pull request correctly renders in the "Preview" tab.

isc-jili

Looks great @isc-dchui ! I left some comments!

isc-jili · 2026-04-23T18:20:33Z

+}
+
+/// Remaps the namespace's routines to newDBName, then dismounts the old routines DB.
+/// Remap-before-dismount avoids a window where the namespace has no mounted routines DB.


Small nit: given that there is a nearly identical comment less than 10 lines down, suggest either removing that comment or removing this section from method description

isc-jili · 2026-04-23T20:21:01Z

+    // Resolve all module dirs before switching namespace (^UnitTestRoot only in original NS)
+    set depDir = ..GetModuleDir("db-packaging", "dependency-module")
+    set simpleDir = ..GetModuleDir("db-packaging", "simple-module")
+    set depsDir = ..GetModuleDir("db-packaging", "module-with-deps")


nit: suggest renaming depDir and depsDir to be more distinctive as currently their names are a bit confusing

isc-jili · 2026-04-23T20:28:18Z

+    // Extract and verify contents
+    set extractDir = ##class(%File).NormalizeDirectory(..PackageOutputDir _ "extract")
+    do ##class(%File).CreateDirectory(extractDir)
+    set sc = ##class(%IPM.General.Archive).Extract(packageFile, extractDir, .extractOutput)


Question: is there any info that needs to be checked in extractOutput? I don't see it being checked below. If it doesn't need to get checked, is there any value in storing it in a variable then?

isc-jili · 2026-04-27T19:59:21Z

+    set sc = ##class(%IPM.Main).Shell("package-database simple-db-module -path " _ ..PackageOutputDir)
+
+    // Restore original module.xml unconditionally before asserting, so a failure doesn't leave
+    // the shared NS in a broken state for subsequent tests


Question: if we want to restore module.xml unconditionally, would it be safer to have the following in a try/catch block in case the package-database fails?

Good call. Added a try/catch failsafe

isc-jili · 2026-04-27T20:05:47Z

+    set $namespace = installNS2
+
+    set sc = ##class(%IPM.Main).Shell("load " _ tamperedPackage _ " -swap-db")
+    do $$$AssertStatusNotOK(sc, "Tampered IRIS.DAT rejected by checksum validation")


Would it be more accurate to check for a substring of the expected error that is thrown as opposed to just the status not being OK? Since what if the status is not OK for a different reason than expected?

isc-jili · 2026-04-27T20:29:33Z

+        write:verbose !, "Computing SHA-256 checksum..."
+        set irisDAT = tempDBPath _ "IRIS.DAT"
+        set checksumHex = ..ComputeSHA256Hex(irisDAT)
+        write !, "  Checksum: ", checksumHex


Question: I'm noticing for print statements some are verbose and some are always printed. Just want to double check that this is the intent.
It seems that currently this is the state of print statements:

invoke warnings: always printed

checksum: always printed

package path: always printed

many progress steps: verbose-only

Right. I think the checksum and package path being always printed is helpful for the user in general

isc-jili · 2026-04-27T20:30:35Z

@@ -0,0 +1,1050 @@
+Include %syPrompt


Does anything in this class use %syPrompt?

The $$$SuccessResponse macro used to check the result of ##class(%Library.Prompt).GetYesNo() is defined in %syPrompt

isc-jili · 2026-04-27T20:31:02Z

+Method ExportGeneratedResources(
+	ns As %String,
+	includeTestResources As %Boolean,
+	ByRef params,


Nit: ExportGeneratedResources() takes ByRef params, but I don’t think the method actually uses it?

Good catch, removed it from the parameters

isc-jili · 2026-04-27T20:31:41Z

+/// Throws on any failure. Does not modify any state.
+Method ValidateBeforeSwap(
+	packageDir As %String,
+	ns As %String) [ Private ]


nit: param ns seems to be unused

Removed from parameters

isc-jili · 2026-04-27T20:36:34Z

+/// Dismounts the temp DB, remaps the namespace back to oldDBName, remounts it,
+/// and removes the temp database config entry.
+/// Logs warnings on partial failures but always attempts full restoration.
+ClassMethod RestoreRoutinesDB(


The new methods in this new class could all benefit from params documented in the method comments as well!

Added a bunch more documentation!

isc-eneil · 2026-05-06T20:36:11Z


  build:
-    timeout-minutes: 45
+    timeout-minutes: 60


Just curious-- why the timeout-minutes changes?

The new tests add enough time that we're running into the timeouts accidentally. There's an issue (#1195) to see if we can improve this overall

isc-eneil · 2026-05-06T20:38:31Z

+<parameter name="module" required="true" description="Name of module to package as database" />
+<modifier name="path" aliases="p" dataAlias="Path" value="true" description="Output directory for the .tgz file." />
+<modifier name="use-current-db" dataAlias="UseCurrentDB" dataValue="1" description="Package the namespace's current routines DB instead of creating a fresh one." />
+<modifier name="include-test-resources" dataAlias="IncludeTestResources" dataValue="1" description="Include test resources (Scope=test or Scope=verify) in the package. Excluded by default." />


I still vote calling this -dev instead of -include-test-resources to keep the modifier names consistent across commands and minimize confusion, but I won't die on that hill

isc-eneil · 2026-05-10T21:31:03Z

Nit: update the description to indicate that source packaging is the default for the "package" command

isc-eneil · 2026-05-10T21:40:44Z

+                // Identical to "package": route to the Package phase using current lifecycle
+                set tCommandInfo = "package"
+                do ..RunOnePhase(.tCommandInfo)
+            } elseif (tCommandInfo = "package-database") {


@isc-dchui I remember that adding package-** commands was part of your original design, but I can't remember why we went with that instead of flags on the existing package command so that it continues to be considered a module-action rather than separate commands. I'm not sure I have a strong preference either way, design-wise, but I'm noticing that the flow through the Shell is different.

It's because of the lifecycle phases. A separate command allows it to dispatch to the %IPM.Lifecycle.Database class directly

isc-eneil · 2026-07-20T20:52:21Z

        }
        set tModule.VersionString = ..Module.VersionString
-        if (..Module.Deployed) {
+        // Use the lifecycle's PACKAGING parameter to determine the publish type, not the module's


This is kind of confusing. Thoughts on changing the wording of this parameter to BuildPackaging or something like that?

…e in one call

… and packaging property

isc-kiyer

@isc-dchui few small notes. I love the description of the MR! Could you add some more information for each of the "Key Details"? In particular, for the following:

Can you explain why generated resources need special treatment? When packaging, wouldn't they be in the routine db already?
What is the reason for exporting the module manifests and reimporting them? Could that not be accomplished by setting up a global mapping/copying over the IPM globals from the globals db to the routine db during packaging? Then its one less step at deployment time.

isc-kiyer · 2026-04-14T03:41:10Z

                    // then it should have a package mapping of this specific resource to its namespace database
                    $$$ThrowOnError(..OnConfigureMappings(.pParams))
                }
-                if ..ResourceReference.Generated {


Why is this removed?

That's just dead code. If it's generated, it gets marked as handled at the bottom of the method anyway.

isc-kiyer · 2026-04-14T03:43:44Z

+<modifier name="path" aliases="p" dataAlias="Path" value="true" description="Output directory for the .tgz file." />
+<modifier name="use-current-db" dataAlias="UseCurrentDB" dataValue="1" description="Package the namespace's current routines DB instead of creating a fresh one." />
+<modifier name="include-test-resources" dataAlias="IncludeTestResources" dataValue="1" description="Include test resources (Scope=test or Scope=verify) in the package. Excluded by default." />
+<modifier name="export-python-deps" dataAlias="ExportPythonDependencies" dataValue="1" description="Include Python wheels (from &lt;PythonWheel&gt; resources) in the database package. On by default." />


Nit: could have this be value="true" instead? We need the concept of enums ideally to restrict values but that's out of scope for now so could simply set value true instead of 2 modifiers for it

isc-kiyer · 2026-04-14T03:44:59Z

+<description>
+Creates an IRIS.DAT database package bundled in a .tgz containing IRIS.DAT, module.xml (with SHA-256 checksum), and dependencies.xml.
+</description>
+<parameter name="module" required="true" description="Name of module to package as database" />


Should indicate that this module should not be a dependent of any other module (something we should check and throw an error for). We should also probably check for DB packaging that every module other than this one has at least 1 dependent because if not, you are packaging things that aren't intended.

isc-kiyer · 2026-04-14T03:49:37Z

 	pPhase As %String,
 	ByRef pParams) As %Status
 {
+    if $get(pParams("SkipInvokes")) { quit $$$OK }


What is this for? Is there a SkipInvokes option at the command line level?

isc-kiyer · 2026-04-14T03:53:52Z

+{
+    // Install context: IRIS.DAT already in place — no source to reload.
+    // Also skip <Invoke> elements in all subsequent phases: they ran during packaging.
+    // The new IRIS.DAT isn't mounted until %Activate, so any invoke referencing packaged


Why do we delay mounting of the IRIS.DAT? I feel like it should be mounted as early as possible (Reload is probably fine). Invokes should remain then as some may have to do with actions at deployment time. This is another important reason we need different install modes (dev, build and deploy)

isc-kiyer · 2026-05-10T23:55:40Z

+        write:verbose !, "Validating IPM mapping..."
+        do ..ValidateIPMNotInRoutinesDB()
+
+        // Warn about <Invoke> elements that won't run during database install.


Hmm so today this could result in some of our Payer Services solutions always issuing warnings because we don't yet have support for different build modes and during packaging time, there are certain reload invokes.
General question though as to why this needs to be done in activate and not in the Reload phase itself? What are the pros/cons of earlier vs later? I think the way we import deployed code is currently in reload.

isc-kiyer · 2026-05-10T23:57:18Z

+
+        if useCurrentDB {
+            // -use-current-db: use the existing routines DB directly, no remapping
+            // Note: globals mapping is unchanged throughout — IPM metadata stays in globals DB


Wouldn't it be better to have IPM metadata be put into routine db? Or I guess we achieve this by exporting the module manifests and loading them into the ns after mounting the IRIS.DAT on install from db packaged?

Right, the export/import is the mechanism and it avoids some trickiness around namespace/database mappings and possibly conflicting IPM data

isc-kiyer · 2026-05-11T00:04:09Z

+{
+    new $namespace
+    set $namespace = "%SYS"
+    $$$ThrowOnError(##class(Config.Namespaces).Get(ns, .nsProps))


Nit: instead of needing a ns switch, we can use ##class(%SYS.Namespace).GetAllNSInfo()

isc-jlechtne · 2026-08-05T20:04:59Z


        // get metadata from annotations
        set metadata = ..GetPackageMetadata(..Location, name, tag, "", client)
+        if (metadata = "") {


What is the metadata value if the tag doesn't exist in the tag list? So that it's not equal to ""

isc-jlechtne · 2026-08-05T20:59:40Z

 	ByRef pDependencyGraph,
-	pLog As %IPM.General.AbstractHistory = "") As %Status
+	pLog As %IPM.General.AbstractHistory = "",
+	pIPMPackaging As %String = "") As %Status


For consistency with other params the ordering would make more sense to have tReference.IPMPackaging following tReference.PlatformVersion and before params, dep graph, and log

isc-dchui requested review from isc-cborbonm, isc-egabhart, isc-eneil, isc-jili, isc-jlechtne, isc-kiyer, isc-pbarton and isc-tleavitt as code owners April 13, 2026 19:41

isc-dchui force-pushed the db-packaging branch from 6884e70 to ccc3b3e Compare April 13, 2026 19:42

isc-dchui added this to the v.0.11.0 milestone Apr 22, 2026

isc-jili requested changes Apr 27, 2026

View reviewed changes

isc-dchui force-pushed the db-packaging branch 2 times, most recently from bba26d8 to 216223e Compare June 1, 2026 19:58

isc-dchui force-pushed the db-packaging branch from 216223e to 079678d Compare July 6, 2026 19:28

isc-cborbonm reviewed Jul 13, 2026

View reviewed changes

Comment thread CHANGELOG.md Outdated

isc-eneil reviewed Jul 20, 2026

View reviewed changes

isc-dchui added 14 commits August 3, 2026 10:59

Add tests first

1ca7891

First pass implementation of package-database

987b618

First pass on installing database-packaged module

b178ee3

Bug fixing and test improvement

6f0fb95

Handle non-compiled files and test files with some bug fixes and tests

f468a9c

All tests pass now

9c12697

Refactor

04528b4

Fix python requirements.txt gap, add some tests, and minor refactoring

616d254

Cleanup after new tests

b81166d

Fix gap with invoke timing

3cf1c66

Refactor tests to carefully reuse namespace for performance gains

9a70a19

Minor refactor and remove test timing

78b0aaf

Properly handle update steps during database packaging/installing

b12f3f3

Add publish-database command for publishing a database-packaged modul…

428a637

…e in one call

isc-dchui added 10 commits August 3, 2026 10:59

Add changelog and merge two tests into one

dca9b2b

Readd comment

6334bbf

Adjust GitHub CI phase steps timing

69152be

Combine python tests into one

b8be80a

Add source vs database-packaged module filtering in ORAS with OCI tag…

9e94af7

… and packaging property

Address review comments

b2bfaf3

Reorganize methods

33ae2cf

Update to new version of IPM

bf5fdd4

Add check for mirrored databases

7c904bd

Address review comments

1193db1

isc-dchui force-pushed the db-packaging branch from 079678d to 1193db1 Compare August 3, 2026 17:16

isc-dchui requested review from isc-cborbonm, isc-eneil and isc-jili August 3, 2026 17:21

isc-kiyer requested changes Aug 3, 2026

View reviewed changes

isc-jlechtne reviewed Aug 5, 2026

View reviewed changes

Uh oh!

Conversation

isc-dchui commented Apr 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Overview

Key Details

Testing

Not Yet Handled

Checklist

Uh oh!

isc-jili left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

isc-kiyer left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

isc-dchui commented Apr 13, 2026 •

edited

Loading