feat: Add a data importer for bringing cloud data into onPrem - #44
Open
twk3 wants to merge 2 commits into
Open
Conversation
Currents support is able to create an export of Org data and make it available for import to onPrem
📝 WalkthroughWalkthroughAdds an optional Helm-managed toolbox pod and retained PVC for Cloud-to-self-hosted organization imports. Introduces ClickHouse restore-mode configuration, propagates it to changestreams and toolbox environments, and documents import, incremental migration, recovery, configuration, and troubleshooting procedures. ChangesOrganization data import
Estimated code review effort: 3 (Moderate) | ~20 minutes Sequence Diagram(s)sequenceDiagram
participant Operator
participant Helm
participant ToolboxPod
participant MongoDB
participant ClickHouse
Operator->>Helm: Enable toolbox and ClickHouse restore mode
Helm->>ToolboxPod: Render pod with import configuration
Helm->>ToolboxPod: Mount persistent import state
Operator->>ToolboxPod: Copy artifact and run currents-import
ToolboxPod->>MongoDB: Restore and verify organization data
ToolboxPod->>ClickHouse: Restore and verify ClickHouse data
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 6
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@charts/currents/templates/changestreams/deployment.yaml`:
- Around line 50-53: Update the changestreams deployment env rendering around
clickhouseRestoreModeEnv so CURRENTS_CLICKHOUSE_RESTORE_MODE is emitted after
the concatenated global.env and changestreams.env entries, ensuring it remains
authoritative. Add or update the manifest test to verify the restore-mode
variable is last and overrides duplicate user-defined entries.
In `@charts/currents/templates/toolbox/pod.yaml`:
- Around line 29-34: Update the toolbox pod template near serviceAccountName to
set automountServiceAccountToken to false, preventing Kubernetes API credentials
from being mounted while retaining the existing chart service account.
In `@charts/currents/templates/toolbox/pvc.yaml`:
- Around line 20-23: The PVC retention policy in the toolbox chart must provide
a secure cleanup path for retained import data. Make helm.sh/resource-policy:
keep opt-in through an explicit configuration setting, or document and implement
deletion after successful import completion while preserving the PVC during
resumable or in-flight imports; ensure download.json, export artifacts, and
import state are not left indefinitely without an operator-controlled cleanup
path.
In `@docs/org-data-import.md`:
- Around line 172-181: Correct the fresh ClickHouse resume command in the
documentation by replacing the invalid kubectlexec invocation with kubectl -n
<ns> exec, preserving the existing POD reference and ch-import arguments.
- Around line 57-61: Update all commands in the org-data-import runbook to pass
the documented namespace parameter consistently: add --namespace <ns> to
both helm upgrade commands and -n <ns> to the toolbox get, cp, and exec
commands, including the referenced sections.
- Around line 63-67: Update the Helm command example in the artifact-import
documentation so the toolbox.persistence.size flag is included in the same
command before the final line continuation, or provide it as part of a separate
complete helm upgrade command; ensure no standalone shell token remains.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro
Run ID: 19ad337b-5c5e-428c-98ea-3c9963e40794
📒 Files selected for processing (8)
charts/currents/templates/_common.tplcharts/currents/templates/_helpers.tplcharts/currents/templates/changestreams/deployment.yamlcharts/currents/templates/toolbox/pod.yamlcharts/currents/templates/toolbox/pvc.yamlcharts/currents/values.yamldocs/README.mddocs/org-data-import.md
Comment on lines
+172
to
+181
| ```bash | ||
| # merge: | ||
| kubectl exec ${POD#pod/} -- \ | ||
| node /app/packages/scheduler/dist/orgImport/cli.js ch-import \ | ||
| --merge --targetOrgId=<24-hex-target-org> --dir=/data/export | ||
|
|
||
| # fresh (use the source org id, shown in the import logs / manifest): | ||
| kubectlexec ${POD#pod/} -- \ | ||
| node /app/packages/scheduler/dist/orgImport/cli.js ch-import \ | ||
| --orgId=<24-hex-source-org> --dir=/data/export |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win
Fix the fresh ClickHouse resume command.
kubectlexec is not a valid command, and the invocation also omits -n <ns>. Use kubectl -n <ns> exec ....
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@docs/org-data-import.md` around lines 172 - 181, Correct the fresh ClickHouse
resume command in the documentation by replacing the invalid kubectlexec
invocation with kubectl -n <ns> exec, preserving the existing POD reference and
ch-import arguments.
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@docs/configuration.md`:
- Line 219: Update the toolbox.env description in the configuration table to say
it applies to “the toolbox container” instead of “both toolbox containers,”
matching the single container rendered by the chart.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro
Run ID: 9345bdc8-0ef8-45ae-b8ce-7cc6eb661b70
📒 Files selected for processing (1)
docs/configuration.md
| | toolbox.enabled | bool | `false` | Create the toolbox pod and its scratch PVC. Leave disabled for normal installs — with this off, nothing in this section renders. | | ||
| | toolbox.name | string | `"toolbox"` | | | ||
| | toolbox.persistence | object | See [values.yaml] for default values | Scratch space for artifacts and the import state file. Size it at ~1.5x the export's total bytes (manifest totals.exportedBytes + clickhouseTotals.exportedBytes). | | ||
| | toolbox.env | list | `[]` | Additional environment variables for both toolbox containers. | |
There was a problem hiding this comment.
📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win
Document the actual toolbox container scope.
The chart renders a single toolbox container, so “both toolbox containers” is inaccurate and may confuse operators. Change this to “the toolbox container.”
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@docs/configuration.md` at line 219, Update the toolbox.env description in the
configuration table to say it applies to “the toolbox container” instead of
“both toolbox containers,” matching the single container rendered by the chart.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Currents support is able to create an export of Org data and make it available for import to onPrem
Summary by CodeRabbit
New Features
Documentation