Add "Sync" command

[isc-dchui]

Description

• Closes Sync command #1117

Motivation

The primary use case is LLM-assisted development: an LLM generates or edits .cls / .inc / .mac files on disk, then the developer runs sync to reload and recompile only what changed without a full reload that reinstalls the entire module. This makes the tight edit-compile-test loop fast enough to be practical inside an agentic coding session.

Secondary use case: any developer editing IPM-managed source who wants faster incremental feedback than reload provides.

Architecture

Entry point

%IPM.Main.Sync  →  ExecutePhases(name, $lb("Sync"), 1, .params)

Sync is a first-class lifecycle phase registered in %IPM.DataType.PhaseName. It fires the standard OnBeforePhase / OnAfterPhase / <Invoke> hooks for free. With no module name, Main.Sync loops over all DeveloperMode=1 modules and collects failures without throwing.

Flags: -delete (remove deleted files from server), -test/-t (run changed test cases), -verbose/-v.

10-step pipeline (%IPM.General.Sync.Pipeline.Run)

%IPM.Lifecycle.Base:%Sync is a thin instance-method delegator (must be an instance method so ExecutePhases' $method dispatch finds it); it just calls ##class(%IPM.General.Sync.Pipeline).Run(..Module, .params). The orchestration and its Sync* helpers live on %IPM.General.Sync.Pipeline.

Step	Method	What it does
1	SyncCheckModuleXml	Hash-compare module.xml; if changed, reload manifest immediately
2	FileHash.CollectScanDirs	GetSyncDirectory() on each SupportsSync()=1 processor; DeduplicateScanDirs; then WalkAndHashDirs targeted walk into allFiles/allHashes
3	SyncBuildReverseIndex	Map relPath → processor via (a) ResolveChildren+OnItemRelativePath for individually-declared files, (b) GetSyncDirectory() prefix scan for directory-owned resources
4	FileHash.ComputeChanges	3-pass change detection; exit early with "Nothing to sync" if clean
5	SyncRoutePathSet	Partition modified/deleted paths into syncByResource
6	SyncDispatchProcessors	Call OnSync on each routed processor
7	SyncCompile	Full compile of all in-scope compilable resources; IRIS automatically recompiles any dependents (subclasses, includers)
8	SyncApplyDeletes	If -delete: delete server-side docs, then recompile
9	FileHash.CommitChanges	Commit new baseline — skipped on exception so next sync retries
10	SyncRunTests	If -test: run changed test-phase cases via owning Test processor only

Targeted-walk architecture (GetSyncDirectory)

Each processor returns the single directory it owns as a normalized relative path (no leading slash; a trailing slash is fine). Sync walks only those directories, never the full module root. This avoids touching build/, data/, node_modules/, etc. entirely.

DeduplicateScanDirs removes any entry that is a subdirectory of an already-kept entry (alphabetical scan, prefix check) to prevent double-walking.

Abstract.GetSyncDirectory() returns "" (sentinel: no owned directory). All callers guard syncDir '= "".

Change detection (FileHash.ComputeChanges)

Signature: ComputeChanges(module, .modified, .deleted, .reverseIndex, .allFiles, .allHashes). Three-pass pipeline — SHA-1 hash only, no mtime or size:

• Pass 1 (walked compilable files, namespace-filtered): iterates allFiles directly; RelPathToDocName returns "" for extensions other than cls/inc/mac/int, so only those are considered here. Checks $$$comClassDefined / %RoutineMgr.Exists — only files compiled in this namespace count.
• Pass 2 (reverse-index relPaths): catches whatever Pass 1 skipped — uncompiled new files, non-compilable tracked files (e.g. test dirs), and tracked xml/rtn files. No I/O since paths already resolved.
• Pass 3 (stored rows → deletions): any FileHash row whose file is missing from disk → deleted. module.xml is skipped explicitly — Step 1 owns it, so it never routes through ComputeChanges.

No baseline row → new file → modified. Baseline exists → hash-only comparison.

Filesystem walking

WalkAndHashDirs dispatches to Python os.walk() (WalkAndHashFilesPython) with automatic SQL BFS fallback (WalkAndHashFilesSQL). Walk-once: step 2 walks all declared dirs into allFiles+allHashes, shared by both SyncBuildReverseIndex and ComputeChanges.

Stage rollout

Stage	Processors	Status
1	All AbstractCompilable derivatives, Test (test-phase only)	Implemented
2	FileCopy, WebApplication, PythonWheel	Planned
3	CPF, Copy, ArtifactoryTarball, LegacyLocalizedMessages, Default.Global	Planned
—	CSPApplication, SystemSetting, ModuleExport, LocalizationExport	Permanently out of scope

Peculiar implementation details

%-prefixed classes aren't caught by Pass 1. RelPathToDocName("src/cls/IPM/Main.cls") → "IPM.Main.CLS", but the real class is %IPM.Main; nothing recovers the %, so $$$comClassDefined("IPM.Main") is false and Pass 1 skips it. They're caught in Pass 2 instead, via their reverse-index entry (step 3a).

Test files are tracked by on-disk path, not by resolving to a server class. A test class at tests/unit/SyncTest/Tests/Trivial.cls maps to "tests.unit.SyncTest.Tests.Trivial.CLS" — a plausible name that isn't the real class, so $$$comClassDefined is false and Pass 1's namespace check would wrongly skip it. Two mechanisms compensate: test files reach the reverse index via the GetSyncDirectory() prefix scan (step 3b) and are detected in Pass 2 rather than Pass 1, and StampModule stamps every compilable file in a scan dir unconditionally, never consulting the namespace.

One helper owns scan-dir collection so stamping and syncing can't disagree. The "collect each processor's GetSyncDirectory() + DeduplicateScanDirs" loop is the single FileHash.CollectScanDirs, called by both StampModule (baseline stamping) and Pipeline.Run (step 2) — one source of truth for which dirs are in scope.

Deleting a test file needs Test.OnSync to delete the server doc itself. SyncApplyDeletes handles only AbstractCompilable resources, skipping Test, so OnSync must call $system.OBJ.Delete for removed test class files directly. Otherwise the server-side %UnitTest.TestCase subclass persists with no on-disk source and no hash record — permanently invisible to future syncs.

A failed compile leaves the baseline untouched so the next sync retries. If compile fails, CommitChanges is skipped; changed files keep their old hash rows (or none, if new), so the next sync re-detects and retries them once the error is fixed.

A pre-feature module self-heals its baseline on first sync. A module installed before this feature has zero FileHash rows. The first sync stamps a baseline from current disk state and exits; only the next sync after a real edit detects changes.

Performance: what was tried and why things changed

Approach	Outcome
mtime + size as fast path, hash on mismatch	Initial design. Dropped: mtime is unreliable on Docker bind mounts. Removed entirely. SHA-1 only.
Pure ObjectScript directory walk	Original implementation. Replaced by Python os.walk() because IRIS's %Library.File_FileSet SQL walk has severe per-row overhead on large directories. Python processes the entire tree in native C and returns results in bulk.
zsearch / $zsearch	Considered for file enumeration. Rejected: too slow on large trees.
Full-root BFS (walk everything, filter after)	Used before targeted-walk. Replaced by GetSyncDirectory() architecture: avoid entering build/, node_modules/, data/ entirely rather than filtering them after the I/O.
Separate walk + hash passes	Initial Python implementation walked directories first, then hashed separately. Merged into a single Python pass (c37ac2d) to avoid double-traversal overhead.
Flat/non-canonical resource paths	Early implementation attempted to support resources with files scattered across arbitrary directories. Dropped (080e6bed): too complex, too slow. Only canonical resource paths (declared Name/Directory attributes) are supported.

Testing

All integration tests in Test.PM.Integration.Sync. Fixture: tests/integration_tests/Test/PM/Integration/_data/sync-test/ is copied once to a native-filesystem "pristine" dir in OnBeforeAllTests (the bind-mounted _data/ is slow for the many small-file copies), then restored from there and loaded in dev mode before each test, uninstalled and deleted after. TestSyncAllProcessesDependenciesFirst uses a separate three-module fixture set (_data/sync-dep/{a,b,c}/).

Test	What it verifies
TestGetStoredPathsReturnsStampedPaths	StampModule records expected paths in FileHash after load
TestNoChangeIsNoOp	No-change sync exits with "Nothing to sync"
TestMigrationFromNoBaseline	Zero rows → self-heal baseline on first sync; edit before self-heal not retroactively detected; edit after baseline detected normally
TestModifiedClassRecompiles	Modified .cls file detected and recompiled
TestModifiedXmlClassRecompiles	Format="XML" class file syncs identically to UDL; reloaded class reflects edited value
TestSuperclassEditRecompilesSubclass	Editing a superclass causes IRIS to recompile its subclasses even though the subclass file is unchanged
TestIncludeEditRecompilesConsumer	Modified .inc causes consumer class to recompile
TestUntrackedFileIgnored	File outside declared resource dirs never seen, never stamped
TestDeleteSkippedByDefault	Deleted file left on server without -delete flag
TestDeleteTestClassRemovesFromServer	Deleted test class file removed from server with -delete
TestDeleteRecompilesDependents	Deleting a superclass causes dependents to fail recompile (surfaces error)
TestModuleXmlChangedWarning	Changed module.xml reloads manifest and emits warning
TestSyncTestFlag	-test runs changed test class; without flag, loads but doesn't run
TestSyncTestFlagBatchesMultipleChangedClasses	Multiple changed test classes in one resource → single batched RunTest invocation
TestSyncTestFlagOnlyRunsOwningResource	Changed test class dispatched only through its owning resource, not all UnitTest processors
TestSyncAllDevModeModules	No module name → syncs all dev-mode modules; [sync-test] specifically reports nothing to sync
TestSyncAllProcessesDependenciesFirst	No module name → modules synced dependency-first (least-dependent before most-dependent), even when dependency order diverges from install/row order
TestSyncModuleNotFound	Non-existent module name → error status
TestSyncNonDevModeModule	Non-dev-mode module → error status
TestModuleXmlAddsResourcePicksUpNewFile	module.xml change adding a new <Resource> causes that resource's files to be tracked in the same sync call
TestFailedCompileRetries	Syntax error → failed sync → baseline not committed → fixed file re-detected on retry
TestModuleXmlAndClassEditedTogether	Co-editing module.xml and a source class in one operation syncs both changes

Example

Changes made across four different modules, including modifying a test to make it fail and deleting a file. Note that the summary section appears when a namespace-sync (sync without a specified module) runs.

zpm:USER>sync -test -delete
[USER|sync-test]        Sync START
  Updated: tests/unit/SyncTest/Tests/Trivial.cls
  Deleted: src/cls/SyncTest/Deletable.cls
[sync-test] Sync complete: 1 file(s) updated, 1 deleted.
[sync-test] Sync done in 1.70s
Use the following URL to view the result:
http://172.19.0.4:52773/csp/sys/%25UnitTest.Portal.Indices.cls?Index=1&$NAMESPACE=USER
Some tests FAILED in suites:
  SyncTest/Tests

Test Results:

Test Run #1 (USER) .49786s 2026-08-03 18:29:15
Methods: 1 total, 0 passed, 1 failed
Assertions: 2 total, 1 passed, 1 failed

FAILED SyncTest\Tests:TestAlwaysPasses: AssertTrue - This test always passes.
[sync-test]     Sync FAILURE
ERROR #5001: 1 failure(s).
[USER|sync-dep-a]       Sync START
  Updated: src/cls/SyncDepA/Top.cls
[sync-dep-a] Sync complete: 1 file(s) updated.
[sync-dep-a] Sync done in 0.29s
[USER|sync-dep-a]       Sync SUCCESS
[USER|sync-dep-b]       Sync START
  Updated: src/cls/SyncDepB/Middle.cls
[sync-dep-b] Sync complete: 1 file(s) updated.
[sync-dep-b] Sync done in 0.26s
[USER|sync-dep-b]       Sync SUCCESS
[USER|sync-dep-c]       Sync START
  Updated: src/cls/SyncDepC/Base.cls
[sync-dep-c] Sync complete: 1 file(s) updated.
[sync-dep-c] Sync done in 0.26s
[USER|sync-dep-c]       Sync SUCCESS

================================================================
Sync Summary
================================================================
Modules checked:  4  (sync-test, sync-dep-a, sync-dep-b,
                      sync-dep-c)
Modules updated:  4
  sync-test   2 files   tests 0/1
  sync-dep-a   1 file
  sync-dep-b   1 file
  sync-dep-c   1 file
Warnings:  1
  [sync-test] 1 resource(s) skipped (no sync support): /static/.
================================================================

Checklist

• This branch has the latest changes from the main branch rebased or merged.
• Changelog entry added.
• Unit (zpm test -only) and integration tests (zpm verify -only) pass.
• Style matches the style guide in the contributing guide.
• Documentation has been/will be updated
  • Source controlled docs, e.g. README.md, should be included in this PR and Wiki changes should be made after this PR is merged (add an extra issue for this if needed)
• Pull request correctly renders in the "Preview" tab.