feat(deployment)!: Decouple query_engine from deployment gating; Move to webui config and enable independent service control (resolves #2088).#2104
Draft
junhaoliao wants to merge 4 commits intoy-scope:mainfrom
Conversation
…ve to webui config and enable independent service control (resolves y-scope#2088).
Contributor
|
Important Review skippedDraft detected. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Organization UI Review profile: ASSERTIVE Plan: Pro Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
# Conflicts: # docs/src/user-docs/guides-k8s-deployment.md # tools/deployment/package-helm/templates/configmap.yaml
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Previously,
query_enginelived underpackage.query_engineand controlled two things at once:Setting
query_engine: "presto"would select a "base" compose file that excluded the entire Celeryquery pipeline (
query-scheduler,query-worker,reducer,mcp-server), making it impossible torun the API server (which depends on the query pipeline) alongside Presto. The two query engines were
mutually exclusive.
This PR decouples these concerns so that
query_engineonly controls the Web UI search interface,and individual services are enabled/disabled through their own config keys — aligning Docker Compose
with the Helm chart (#2004), where components are independently toggled.
Changes
Config schema (
clp_config.py):query_enginefromPackagetoWebUi(e.g.,webui.query_engineinstead ofpackage.query_engine).validate_query_engine_package_compatibilityfromPackage; add an equivalentcross-config validator at the
ClpConfiglevel.DeploymentTypefrom four variants (BASE,FULL,SPIDER_BASE,SPIDER_FULL) totwo (
CELERY,SPIDER) — since deployment type is now based solely oncompression_scheduler.type.query_scheduler,query_worker, andreducernullable (likeapi_serverandlog_ingestor). Setting them tonullinclp-config.yamldisables those services.Docker Compose (
controller.py, compose files):docker-compose-base.yamlanddocker-compose-spider-base.yaml(no longer needed).deploy.replicasenv var pattern toquery-scheduler,query-worker, andreducerindocker-compose-all.yamlso they can be toggled viaCLP_QUERY_SCHEDULER_ENABLED,CLP_QUERY_WORKER_ENABLED, andCLP_REDUCER_ENABLED.api-serverandmcp-servertodocker-compose-spider.yaml(previously missing).*_ENABLED=0when a service's config isnull.Helm chart:
query_enginefromclpConfig.packagetoclpConfig.webuiinvalues.yaml.configmap.yamlreferences accordingly.Documentation:
to reflect the new
webui.query_engineconfig location and simplified deployment types.Impact Assessment
package.query_engineno longer exists; users must move it towebui.query_engine. Existingclp-config.yamlfiles that setpackage.query_enginewill failvalidation.
selectively disable query pipeline services by setting
query_scheduler: null,query_worker: null, andreducer: null.clpConfig.package.query_engineis replaced byclpConfig.webui.query_engine.Checklist
breaking change.
Validation performed
Scenario 1: Default clp-json deployment (clp-s, Celery)
Task: Verify that the default deployment starts all services (including query pipeline) and
compression works.
Command:
Output:
Explanation: Deployment type is now
celery(wasfull). All services includingquery-scheduler,query-worker,reducer, andapi-serverstarted and became healthy.Task: Verify compression works.
Command:
Output:
Scenario 2: Presto query engine alongside Celery query pipeline
Task: Verify that setting
webui.query_engine: "presto"no longer excludes the Celery querypipeline services. Previously this would select
docker-compose-base.yamlwhich omittedquery-scheduler,query-worker, andreducer.Config (
etc/clp-config.yaml):Command:
Output:
Explanation: With
webui.query_engine: "presto", the deployment type is stillceleryand ALLservices start — including
query-scheduler,query-worker,reducer, andapi-server. Previouslythis config would have selected
docker-compose-base.yamlwhich excluded these services entirely.This validates the core goal of the issue: Presto and the Celery query pipeline can now run
simultaneously.
Task: Verify compression still works with Presto config.
Command:
Output:
TODO: test with Presto deployment