feat: logs search v2

[carderne]

Summary

Improves global Logs search with a smaller, bounded search representation that is indexed asynchronously. Source event inserts are no longer coupled to search indexing, and new logs can become searchable within seconds while a slower pass preserves complete coverage.

The Logs page reads the v2 search table directly.

Design

A best-effort preview pass processes complete five-second windows after a short safety delay. An authoritative pass processes one-minute windows after a longer delay and records each completed window as an append-only PostgreSQL checkpoint.

The initial projection boundary is also an append-only checkpoint, so PostgreSQL has no mutable projector state row. Redis coordinates projection work and stores the disposable preview watermark.

LOGS_SEARCH_PROJECTOR_ENABLED is the operational switch for projection. Both passes use LOGS_CLICKHOUSE_URL when configured and otherwise fall back to the primary CLICKHOUSE_URL. They share a strengthened, bounded projection fingerprint so retries and preview/finalized overlap do not produce duplicate results, while distinct logs that share a span and timestamp remain separate.

The destination is partitioned by insertion date to keep each projection insert localized, while retention remains based on the event timestamp.

[changeset-bot]

⚠️ No Changeset found

Latest commit: fdc20f4

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

[coderabbitai]

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

• @coderabbitai resume to resume automatic reviews.
• @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

• ▶️ Resume reviews
• 🔍 Trigger review

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Repository UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 010f945c-8ee2-41e7-a121-f32d69678646

📥 Commits

Reviewing files that changed from the base of the PR and between d789149 and cc3213c.

📒 Files selected for processing (1)

• internal-packages/clickhouse/schema/039_create_task_events_search_v2.sql

📜 Recent review details

⏰ Context from checks skipped due to timeout. (23)

• GitHub Check: report
• GitHub Check: webapp / 🧪 Unit Tests: Webapp (11, 12)
• GitHub Check: webapp / 🧪 Unit Tests: Webapp (12, 12)
• GitHub Check: webapp / 🧪 Unit Tests: Webapp (8, 12)
• GitHub Check: webapp / 🧪 Unit Tests: Webapp (7, 12)
• GitHub Check: webapp / 🧪 Unit Tests: Webapp (4, 12)
• GitHub Check: webapp / 🧪 Unit Tests: Webapp (6, 12)
• GitHub Check: webapp / 🧪 Unit Tests: Webapp (1, 12)
• GitHub Check: webapp / 🧪 Unit Tests: Webapp (5, 12)
• GitHub Check: webapp / 🧪 Unit Tests: Webapp (10, 12)
• GitHub Check: webapp / 🧪 Unit Tests: Webapp (9, 12)
• GitHub Check: webapp / 🧪 Unit Tests: Webapp (3, 12)
• GitHub Check: webapp / 🧪 Unit Tests: Webapp (2, 12)
• GitHub Check: obsmap / 🧪 Unit Tests: Observability Map
• GitHub Check: fk-cascade-guard / fk-cascade-guard
• GitHub Check: typecheck / typecheck
• GitHub Check: internal / 🧪 Unit Tests: Internal
• GitHub Check: e2e-webapp / 🧪 E2E Tests: Webapp
• GitHub Check: runops-guard / runops-guard
• GitHub Check: code-quality / code-quality
• GitHub Check: audit
• GitHub Check: audit
• GitHub Check: Analyze (javascript-typescript)

🧰 Additional context used

📓 Path-based instructions (1)

internal-packages/clickhouse/schema/[0-9][0-9][0-9]_*.sql

📄 CodeRabbit inference engine (internal-packages/clickhouse/CLAUDE.md)

internal-packages/clickhouse/schema/[0-9][0-9][0-9]_*.sql: Migration file numbering: name files as 0(N+1)_descriptive_name.sql where N is the largest existing migration number in schema/; rebase and renumber if main adds migrations before opening a PR
DDL in migrations must be idempotent: use ALTER TABLE ... ADD COLUMN IF NOT EXISTS, CREATE TABLE IF NOT EXISTS, DROP TABLE IF EXISTS, ADD INDEX IF NOT EXISTS, DROP INDEX IF EXISTS, and CREATE MATERIALIZED VIEW IF NOT EXISTS forms to allow out-of-order and retry-safe application

Files:

• internal-packages/clickhouse/schema/039_create_task_events_search_v2.sql

🔇 Additional comments (1)

internal-packages/clickhouse/schema/039_create_task_events_search_v2.sql (1)

3-3: LGTM!

---

Walkthrough

Added a ClickHouse V2 search table and projection pipeline with persistent state, leases, backfills, telemetry, scheduled processing, and admin controls. Updated log search normalization, minimum-length validation, bounded pagination, cursor handling, and period expansion. Centralized logs access checks across navigation and routes. Added ClickHouse command support, runtime configuration, database state storage, and integration tests.

🚥 Pre-merge checks | ✅ 3 | ❌ 2

❌ Failed checks (2 warnings)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 7.89% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.
Description check	⚠️ Warning	The description explains the design, but it omits the required issue reference, checklist, testing, changelog, and screenshots sections.	Add the required template sections, include the issue reference and checklist, document testing steps, provide a changelog entry, and add screenshots or state that they are not applicable.

✅ Passed checks (3 passed)

Check name	Status	Explanation
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Title check	✅ Passed	The title clearly identifies the main change: adding Logs search v2.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch feat/log-search-v2

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[github-actions]

Observability map

As of fdc20f4.

19/100 over 443 measured of 459 entry points (base 19, no change)

What this PR changed

route	base	head	now failing
/admin/api/v1/logs-search-projector	new	0	request-context
/resources/orgs/:organizationSlug/projects/:projectParam/env/:envParam/logs	0	50

FIX FIRST

• /api/v1/projects/:projectRef/envvars (sensitive) - auth-boundary, request-context
• /auth/sso (sensitive) - auth-boundary, request-context
• /_app/orgs/:organizationSlug/settings/team (sensitive) - error-classification, auth-scope, request-context

AUDIT 3 of 50 sensitive mutations record an actor. 47 without one.
CONTEXT 22 of 443 entry points name a tenant on a failure path. 342 appear only here, 39 of them sensitive, in the JSON rather than the fix list.

What the score is made of

CHECKS
  error-classification  179 applicable, 102 pass,   0 sole, global without it 12
  auth-boundary          62 applicable,  57 pass,   0 sole, global without it 16
  auth-scope             19 applicable,  17 pass,   0 sole, global without it 19
  request-context       443 applicable,  22 pass, 242 sole, global without it 64
  audit-trail            50 applicable,   3 pass,   0 sole, not in the score

The score and findings here are report-only and never gate the merge. Separately, a required test suite keeps this tool's symbol and route lists in sync with the code they name, and can fail a pull request that renames or removes a symbol they reference, or that adds the first route with a segment they anticipate. Each failure names the list to edit. The rules and their reasons: internal-packages/observability-map/README.md.

[pkg-pr-new]

Open in StackBlitz

@trigger.dev/build

npm i https://pkg.pr.new/@trigger.dev/build@7c8b416

trigger.dev

npm i https://pkg.pr.new/trigger.dev@7c8b416

@trigger.dev/core

npm i https://pkg.pr.new/@trigger.dev/core@7c8b416

@trigger.dev/python

npm i https://pkg.pr.new/@trigger.dev/python@7c8b416

@trigger.dev/react-hooks

npm i https://pkg.pr.new/@trigger.dev/react-hooks@7c8b416

@trigger.dev/redis-worker

npm i https://pkg.pr.new/@trigger.dev/redis-worker@7c8b416

@trigger.dev/rsc

npm i https://pkg.pr.new/@trigger.dev/rsc@7c8b416

@trigger.dev/schema-to-json

npm i https://pkg.pr.new/@trigger.dev/schema-to-json@7c8b416

@trigger.dev/sdk

npm i https://pkg.pr.new/@trigger.dev/sdk@7c8b416

commit: 7c8b416

[devin-ai-integration]

Devin Review found 1 new potential issue.