feat(connectors): add Bitbucket, Databricks, Google Chat, and Workday Help KB connectors - #6895
Conversation
|
The latest updates on your projects. Learn more about Vercel for GitHub. |
PR SummaryHigh Risk Overview Bitbucket reuses existing OAuth ( Databricks uses a PAT and allowlisted workspace hosts. Notebooks are a BFS walk of Google Chat is a new Workday Help is API-key style: tenant host + client ID, with Docs/icons/registries are updated (64 connectors). Google Chat must be listed in self-host OAuth docs and Reviewed by Cursor Bugbot for commit 327b8b7. Configure here. |
Greptile SummaryAdds Bitbucket, Databricks, Google Chat, and Workday Help knowledge-base connectors, including their authentication, registration, configuration, documentation, and test coverage.
Confidence Score: 5/5The pull request appears safe to merge. No blocking failure remains.
|
| Filename | Overview |
|---|---|
| apps/sim/connectors/google-chat/google-chat.ts | Implements Google Chat space discovery, transcript hydration, metadata mapping, and empty-window handling; the prior space-metadata issue is fixed. |
| apps/sim/connectors/bitbucket/bitbucket.ts | Implements Bitbucket repository-file and pull-request synchronization with pagination and deferred hydration. |
| apps/sim/connectors/databricks/databricks.ts | Implements Databricks workspace traversal and hydration for notebooks and saved SQL queries. |
| apps/sim/connectors/workday/workday.ts | Implements Workday Help authentication, article-version listing, validation, and document hydration. |
| apps/sim/lib/oauth/oauth.ts | Registers the Google Chat OAuth provider and its requested scopes. |
| apps/sim/connectors/registry.server.ts | Registers the four connector runtime implementations with the server-side sync registry. |
Sequence Diagram
sequenceDiagram
participant Sync as Knowledge Sync Engine
participant Registry as Connector Registry
participant Connector as Selected Connector
participant Service as External Service
participant KB as Knowledge Base
Sync->>Registry: Resolve connector configuration
Registry-->>Sync: Runtime connector
Sync->>Connector: listDocuments(config, cursor, context)
Connector->>Service: List source objects
Service-->>Connector: Paginated metadata
Connector-->>Sync: Deferred document stubs
Sync->>Connector: getDocument(externalId, context)
Connector->>Service: Fetch content
Service-->>Connector: Source content
Connector-->>Sync: Hydrated document
Sync->>KB: Reconcile and index document
Reviews (4): Last reviewed commit: "fix(connectors): keep the Bitbucket cap ..." | Re-trigger Greptile
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
… Help KB connectors Adds four knowledge base connectors, closing the gap where Sim shipped tool blocks for these services but could not index their content. - Bitbucket: repository source files and pull request descriptions over the existing Bitbucket OAuth credential - Databricks: notebooks (Workspace API) and saved SQL queries, PAT auth - Google Chat: spaces indexed as message transcripts, new google-chat OAuth service under the shared Google client - Workday Help: knowledge article versions via the public helpArticle/v1 API
Adversarial re-validation of all four connectors plus a combined-change regression audit. - bitbucket: stop declaring incremental sync (deletion reconciliation is disabled for incremental runs, so deleted files were never removed on the default code configuration); drop a wasted listing round-trip after the frontier drains; add 33 tests - databricks: reject an explicit maxDocuments of 0, which meant unlimited; add 34 tests - google-chat: correct the sender displayName documentation (user auth populates only name and type) and emit second-precision RFC-3339 in the message filter; 15 -> 24 tests - workday: fix a crash when maxVersions is persisted as a number; refuse a configuration whose status filter Workday did not honor; cap the unresolved-name error; 18 -> 27 tests - document the Google Chat service-account omission - docs: list all four connectors and correct the connector count
90ec33a to
d06ef0d
Compare
Review round 1. - orderBy takes a full ordering expression, not a bare direction. The reference documents the default as `createTime ASC`, so send `createTime DESC`; a bare `DESC` either 400s every hydration or is ignored, which would make the cap keep the oldest traffic and the later reverse render the transcript backwards. - getDocument no longer returns null when the message window is empty. A space with no messages is still a live space, and null is the "document is gone" signal the engine treats as last-known-good: returning it dropped spaces whose only prose is their description or guidelines, and left a stale transcript indexed after a space was cleared or lookbackDays was tightened past every message. - The transcript header is omitted when no message contributed text.
|
@cursor review |
…thheld something Review round 2. takeIndexableWithinCap reports capReached as soon as the running total equals maxItems, which is also true of a listing that ended at exactly that count. Setting listingCapped there suppressed deletion reconciliation for a complete listing, so upstream-deleted files and pull requests could stay in the knowledge base indefinitely. applyMaxItemsCap now takes whether Bitbucket had more content beyond the page -- a next link, or directories still queued on the frontier -- and flags the listing only when the cap actually withheld something, matching the Databricks, Google Chat, and Workday connectors.
|
@cursor review |
…ll request phase Review round 3. Fixes a regression from 622a21b. maxItems is shared across the code and pull request phases, so a code walk that ends with exactly maxItems documents and no next link or frontier stops pagination before the pull request phase runs. Scoping listingCapped to "this phase had more" left the flag unset in that case, and the engine then treated the run as a complete enumeration and could hard-delete previously indexed pr:* documents that were never listed. The cap flag now asks whether anything the connector was configured to list remains unlisted -- including a later phase the cap is about to stop us reaching.
|
@cursor review |
There was a problem hiding this comment.
✅ Bugbot reviewed your changes and found no new issues!
Comment @cursor review or bugbot run to trigger another review on this PR
Reviewed by Cursor Bugbot for commit 327b8b7. Configure here.
1 participant
Adds four knowledge base connectors. Each closes a gap where Sim already shipped tool blocks for the service but could not index its content — the same four sources Glean indexes natively that we did not.
What's here
bitbucketOAuth credentialmode: 'apiKey')google-chatOAuth serviceclientSecret:refreshTokenhelpArticle/v1)Every response shape is derived from an official spec — Atlassian's
swagger.v3.json, the generated Databricks Python SDK (same OpenAPI source as the rendered reference, which is an unfetchable SPA), Google's Chat REST reference, and Workday's publishedhelpArticle_v1Swagger. Nothing was inferred from partial docs; deliberate exclusions are documented in TSDoc at each site.Deliberate exclusions
wikiOAuth scope exists with no resource under it.issuescope guarding 18 paths, none of which this connector touches. Our Bitbucket OAuth service does not request that scope, and adding it would flag every existing Bitbucket credential as needing re-consent.FILEobjects — not prose, or not exportable.media.downloadplus Drive scopes and turns one document per space into unbounded per-message fan-out. Files shared in Chat belong to the Drive connector.Review history
Two validation passes ran over all four connectors, plus a combined-change regression audit. Highlights of what they caught:
Silent data loss (Bitbucket). The
555timeout fallback rewrote the opaquenextpagination URL, pairing a depth-5 cursor with a depth-1 result set — files drop out of the listing and, withlistingCappedunset, reconciliation then deletes them from the KB. Same class: PR paging usedsort=-updated_on, where a PR touched mid-pagination pushes another past the page boundary. Nowsort=-id.Silently missing content (Databricks). Notebooks inside Git folders were never indexed — the tree walk enqueued only
DIRECTORY, notREPO, so with the default/root everything under/Reposwas absent with no error.Crash on a valid config (Workday).
maxVersionswas read with(value as string)?.trim(), which does not guard a number — a config persisted withmaxVersions: 500threwvalue.trim is not a functionon every sync.Availability misreporting (Google Chat). The service was missing from
GOOGLE_OAUTH_SERVICES, soisOAuthServiceDeploymentAvailablefell through totrueand advertised the connector on self-hosts with no Google OAuth client configured.Tests that could not fail. Mutation testing on the Workday suite found six "verified" fixes whose tests passed regardless of the fix. Coverage was rewritten; every load-bearing test is now confirmed red-on-revert.
Test coverage across the four went from 33 tests to 118.
Sync fidelity
Three connectors have a change-detection limit rooted in a missing upstream revision field. None are papered over — each degrades loudly and self-heals on an explicit full resync (
rehydrateOnFullSync), and each logs a warning naming the consequence:lastActiveTimedoes not advance on edits or deletes.ObjectInfo.modified_atis documented as "only applicable to files", so notebooks may carry no timestamp.commitis explicitly not the commit that last modified the file. Files re-hydrate when the branch advances; path/extension filters are the intended bound.Bitbucket declares
supportsIncrementalSync: falsedeliberately: the sync engine disables deletion reconciliation for every incremental run, and the file walk is already full each run, so declaring support would leave upstream-deleted files indexed indefinitely on the defaultcodeconfiguration.Before merge
1. Google consent screen — operational, outside this repo. Google Chat requests
chat.messages.readonlyunder the sameGOOGLE_CLIENT_IDas our 13 existing Google services. In code this is safe: Better Auth sends each provider's own scopes at authorize time, and the computed scope-resolution table for all 58 pre-existing services is byte-identical before and after. Butchat.messages.readonlyis a Google restricted/sensitive scope, and adding it to a published consent screen forces the whole OAuth client back through verification — during which Google can throttle or warn on the client, affecting every existing Google connector and "Sign in with Google". Confirm the hosted project has the Chat API enabled and both scopes approved, or gate the service until it does.2. Workday
statusfilter — one live-tenant probe./articleVersionsreturns one row per revision with nolatest/currentfilter, so the required-status dropdown is the only guard against indexing every historical revision. Whetherstatusaccepts an/articleStatusesWID or a literal keyword cannot be settled from the spec:statushas nox-workday-populated-bywhileaudiencedoes, and sibling specs (learning_v1,recruiting_v4) document literal keywords for identically-shaped status params — which argues for the keyword form we did not implement. Mitigated so the wrong guess fails loudly:validateConfigreads the filtered response back and refuses a config whose returned status does not match the chosen one. The probe is one call —?limit=1&status=<WID>vs?limit=1&status=Published. If the keyword wins, delete the/articleStatuseslookup and pass the choice through.Known follow-ups (not blockers)
/articleStatuses[].descriptor, which is tenant-localized. A German or Japanese tenant cannot configure the connector, with no free-text escape hatch. Needs a UX decision.nameandtypeonUser, so transcripts carryusers/104512345678rather than display names. Resolving them needschat.memberships.readonly; adding it now would cost no re-consent (zero credentials issued) but compounds item 1 above.ConnectorMeta.auth.providerto the OAuth registry. Adding a Google service requires entries inoauth.ts,providers.ts, andGOOGLE_OAUTH_SERVICES; miss the third and the provider is silently filtered out of the Better Auth config, surfacing only as a 404 on the callback. The invariant holds across all 64 connectors today, but nothing enforces it.maxMessagesis unbounded on Google Chat, matching Slack;microsoft-teamsis the outlier that guards with a page cap. Worth a family-wide fix rather than a one-connector divergence.Verification
type-checkclean ·check:audits32/32 · 1190 tests acrossconnectors,lib/oauth, andlib/integrations. Rebased onto current staging (d9cfd7c68e) with all checks re-run after the rebase.