# Integration Builder — In-house Integration Framework **Authors:** Joana Maia **Date:** 2026-08-03 **Status:** Draft --- ## Problem statement CDP relies on Nango for OAuth, refresh-token lifecycle, and webhook routing across most integrations. Nango costs have grown to the point where the vendor dependency is no longer sustainable, and the existing pipeline — Kafka + `integration-run-worker` / `integration-stream-worker` / `data-sink-worker` + `integration.runs` / `.streams` / `.results` tables — has become complex enough that failures are hard to see end to end and hard to recover from. We need an in-house **Integration Builder** framework that owns OAuth, token rotation, webhook intake, and connector authoring, and that gives us full visibility and control over every step of ingestion. --- ## Goals Ordered by importance. Each goal is a framework requirement, testable per-integration once implemented. 1. **Full end-to-end visibility.** A developer or operator can inspect any ingestion run and see, per step, what happened: which API calls fired, which transformations ran, which activities were emitted, and — for every failure — whether the failure came from the third-party API, from transformation, from orchestration, or from the sink. Think of it as a factory floor: every station is inspectable. 2. **Data correctness / completeness validation.** The framework provides a mechanism — automatic on a schedule, or on demand — for a developer or operator to validate that the data CDP has collected matches the data on the remote (source) side. At CDP's scale, absence of this check makes silent data loss undetectable. The internal Loki initiative attempted this shape; the Integration Builder must deliver on it. A UI is not required in the initial phase — a scriptable, runnable check is enough to start. 3. **Strict, well-defined activity schemas.** Each integration declares the schema of every activity payload it emits. The framework enforces strict type checking at emit time; schema drift is caught before data reaches the sink. 4. **Retry on failure** at the right granularity — a single failing API call, mapper, or step retries without re-running the entire connection. 5. **Signal-not-noise error logging and alerting.** Errors are aggregated by root cause; recurring third-party failures do not page on every occurrence; alerts distinguish "the vendor is down" from "our code is broken". 6. **Daily aggregate stats** on ingestion volume (activities per integration per source) and on failures (per integration, per step, per error class). 7. **Agentic-development-ready.** Custom Claude skills and agents scaffold, author, and test new integrations. The developer flow is part of the framework, not an afterthought. 8. **100% test coverage** for every integration built in the new framework — unit tests for mappers, contract tests for schemas, fixture-based tests for full sync flows. 9. **GitHub scale.** The GitHub connector must scale to **~100,000 repositories using the same pool of installation tokens** we allocate today. Nango's current in-CDP token rotation queue (multi-connection pool, recently-used tracking, rate-limit-aware failover, pool cleanup on auth failure) is the reference for what "good" looks like. 10. **Spread API calls, avoid bursts.** Poll recently onboarded connections more frequently; poll older, quieter connections less frequently. No single-window stampedes. 11. **Re-onboarding.** Any connection can be re-onboarded on demand — full refresh or a specific time range — without a redeploy. 12. **Caching to reduce request volume**, particularly on re-onboarding, similar in spirit to Nango's cache but tuned for CDP scale. 13. **Third-party API version tracking.** The framework records the API version each connector targets, flags upcoming deprecations, and surfaces which connectors need updates. LinkedIn was the pain point that motivates this. 14. **Testable in dev/staging.** A developer can exercise a real (or recorded) integration against a dev/staging environment safely and cheaply. 15. **Maintenance as a first-class design criterion.** When choosing between designs — and between building a piece ourselves versus adopting an existing tool — long-term maintenance burden weighs heavily. Fewer moving parts, fewer bespoke pipelines, and reuse of well-supported tooling win over clever in-house builds when the observability and control requirements are met either way. 16. **24-hour ingestion freshness bar.** For the vast majority of integrations, data does not have to be real-time. Ingestion within **24 hours** is the minimum acceptable freshness. Webhook-fed events remain near-real-time where the provider supports them, but the framework is not optimized for streaming semantics — batching, backfill, and steady polling cadences are first-class. ### Non-goals - **Nango-style admin UI / user-facing connection dashboard — deferred, not rejected.** A visual surface is not a goal or constraint for the initial stages. If we later need a dashboard (in the spirit of the earlier internal Loki initiative), it lives inside CDP rather than in the framework itself. - **`git_integration` and `mailing_list_integration` — out of scope for this project.** They follow a substantially different pattern from Nango-backed integrations. Whether they should later migrate onto the Integration Builder framework is a separate evaluation, not a decision this project makes now. --- ## Background CDP integrations today run on Nango for OAuth, connection storage, refresh-token lifecycle, and webhook intake. On CDP's side, `integration-run-worker` orchestrates onboardings and incremental syncs, `integration-stream-worker` executes streams (the only component talking to third-party APIs), and `data-sink-worker` lands unified activities. The pipeline now uses **Kafka plus Sequin** to stream messages to all downstream consumers. Connector definitions live in `nango-integrations/*`. Outside Nango sit `git_integration` (Python), `mailing_list_integration`, and the legacy GitHub path in `backend/src/serverless/integrations/`. The Nango GitHub sync already runs an in-CDP token-rotation queue on top of Nango-owned tokens — the reference bar. See `inputs/confluence-integrations-framework.md` and `inputs/code-nango-integrations.md`. Failures in the current framework often surface only at the sink; root-causing means walking three worker logs and three tables. The rewrite is a chance to simplify. --- ## Proposed solution (context, not prescription) **This section frames the design space. Concrete topology is decided during Stage 1 POC and locked by an architecture document approved by the architecture team before Stage 2 development begins.** ### What is fixed - **Framework surface.** A new library (working name `services/libs/integration-builder`) owns three concerns end to end: - **Connector authoring** — a declarative model per connector (OAuth config, scheduled syncs, webhook handlers, activity schemas, mappers). No hand-rolled worker orchestration per connector. - **Auth & token pool** — OAuth grant flows, encrypted token storage, refresh scheduling, and a generalized token-pool primitive descended from today's Nango GitHub rotation queue. - **Webhook intake** — provider-agnostic route registry on top of the existing standalone webhook API. - **Observability contract.** Every step of every run — extract, transform, emit, sink — is inspectable in a common shape. Every emitted activity is validated against its declared schema before it leaves the connector. - **First-class developer tooling.** Repo-native Claude skills for scaffolding, running, and testing connectors ship with the framework. - **Freshness bar.** 24-hour minimum for polled integrations; webhook-fed events near-real-time where the provider supports them. ### What is deliberately open - **Language and runtime.** TypeScript / Node.js is the **recommended** starting point — it matches the current stack, and shared libs (`@crowd/common`, `@crowd/data-access-layer`, worker infrastructure) already live there. The final choice is Stage 1's call: if adopting an existing tool built in another runtime turns out to better satisfy the maintenance goal, that trade-off is on the table. - **Whether to reuse `integration-run-worker` / `integration-stream-worker` / `data-sink-worker` as-is, adapt them, or replace parts.** Same for the `integration.runs` / `.streams` / `.results` tables — see "Areas to investigate". - **How failed results are detected and how retries are structured.** The current path (rows in `integration.results` marked failed, replayed via Kafka) is complex. Alternatives — per-step trace records, load-package-style batches, an ops-only datastore — are on the table. - **Whether connector definitions follow Nango's folder layout.** Nango's shape (`syncs/`, `helpers/`, `mappers/`, `oauth.ts`) is a reasonable starting point since migration is easier if the shape is familiar. Not a requirement. - **Cache location and shape.** Redis is the incumbent; a per-connector on-disk cache or a shared object cache are on the table for re-onboarding workloads. ### Observability model — explore before building The observability + maintenance goals point strongly toward **evaluating existing tools before building an ingestion + observability spine from scratch**. Stage 1 should include an explicit exploration pass: what open-source or commercial tools can carry some or all of this shape, what their maintenance profile looks like, and how their cost model compares to running our own. **dlt** (https://dlthub.com) is one option worth considering — recommended by Joan as a starting point. Its `load_info` + `trace` model produces a structured per-run record (per-step status, per-job status, row counts, schema changes) that closely matches the "inspect any station on the factory floor" contract, and its schema-contract mode (`evolve` / `freeze` / `discard_row` / `discard_value`) directly answers the strict-schema goal. Caveats to weigh honestly: - Python runtime (our stack is TypeScript / Node — additional operational surface). - Destinations are warehouse-oriented; CDP's activities pipeline is not. - No OAuth / token-rotation surface — we would still own that. - **Cost.** dlt has a free plan, but running it against CDP volumes almost certainly means **self-hosting**, which has its own operational cost. dlt is one candidate; Stage 1 should survey the broader landscape (ingestion frameworks, workflow engines with connector ecosystems, observability tooling that plugs into a home-grown pipeline) and record the trade-offs in the architecture document. ### Token rotation and scale The GitHub connector must sustain the current token budget across ~100k repositories. The POC target (Stage 1) is 100 repositories — a validation slice, not the scale target. The path from 100 to 100k: - Multi-connection token pool per integration installation (already the shape today). - Recently-used tracking to keep the pool warm without starving cold tokens. - Rate-limit-aware failover: on 403 / secondary rate limit, park the token and rotate. - Pool cleanup when auth fails (invalid / revoked token removed, not recycled). - Poll cadence tiered by connection age: recently onboarded = frequent; older / quiet = infrequent. A global scheduler spreads calls across the day to avoid burst windows. Stage 1 proves these hold; the architecture doc records the locked design. --- ## Areas to investigate (Stage 1 exit criteria) Answered by the approved architecture doc before Stage 2 dev starts. 1. **Build vs. adopt.** Which parts of the ingestion + observability spine should be built in-house and which should be delegated to an existing tool (dlt or otherwise)? The answer is weighed against the maintenance goal and against a concrete cost/self-hosting comparison. 2. **Language and runtime.** TypeScript / Node stays the recommendation unless the adopt-vs-build decision points elsewhere. Lock the choice in the architecture doc. 3. **`data-sink-worker` future.** What, if anything, should change? Right single ingestion point for the new observability requirements, or doing too much? 4. **Operational storage for run/stream/result state.** Keep `integration.runs` / `.streams` / `.results` in Postgres, replace with something lighter (per-run trace record + small state table), or move operational data off the primary Postgres into a dedicated operational store. Current tables carry both semantic state and diagnostic churn; splitting is worth exploring. 5. **Failure-detection model.** Row-marking (today) vs. per-step trace with derived failure counters vs. event log with replay-from-offset. Pick one across integrations. 6. **Retry semantics.** Retry at the API call, the stream, or the run — different providers want different answers. 7. **Cache design.** What is cached (list responses, ETagged endpoints, mapper outputs?), where (Redis, disk, object store?), for how long — especially for re-onboarding. 8. **API-version tracking mechanism.** How connectors declare the vendor API version they target and how the framework surfaces deprecations. 9. **Dev/staging integration harness.** Recorded fixtures vs. sandbox tenants vs. shadow-connect against real accounts. --- ## Dependencies - **Nango** — must remain operational until Stage 3 completes (2026-12-31). - **Legacy GitHub integration** (`backend/src/serverless/integrations/`) — remains operational until Stage 4. - **Kafka + Sequin pipeline** — the downstream fan-out is shared with the rest of CDP; the framework produces into this pipeline, not around it. - **Provider APIs and OAuth apps** — GitHub, GitLab, Discourse, Groups.io, Jira Cloud/Server/DC, Confluence Cloud/Server/DC, Gerrit. App registration or existing-app reuse needed per provider before its stage. - **Architecture team review** — the Stage 1 architecture document requires their sign-off before Stage 2 development begins. --- ## Alternative solutions - **Stay on Nango.** Rejected — cost trajectory and inability to tune token pooling, streaming, and observability at CDP scale. - **Adopt dlt as the ingestion runtime.** Open evaluation, not rejected. Strong fit for the observability contract (`load_info` / `trace`) and schema enforcement. Honest caveats: Python runtime, warehouse-oriented destinations, no OAuth surface, and dlt's free tier likely means self-hosting at CDP volumes. Stage 1 weighs the maintenance win against the operational cost. - **Fork and self-host the Nango open-source core.** Open evaluation. Preserves the abstractions we already build around (connections, syncs, token rotation) and removes the vendor bill. The trade-off is maintenance burden of operating the fork — patching upstream, absorbing schema changes, running the Nango service ourselves — which must be measured directly against the maintenance goal. - **Per-integration bespoke services** (like `git_integration`). Rejected — reproduces the fragmentation this initiative closes. --- ## Risks | Risk | Likelihood | Impact | Mitigation | |---|---|---|---| | Token-pool regresses vs. Nango's GitHub rotation (rate-limit blowups at scale) | Medium | High | Stage 1 POC validates rotation against 100 repos; scale test a representative slice before Stage 2 cutover; keep Nango warm. | | OAuth flow bugs trigger user re-consent storms | Low | High | Structured refresh retries; alerting on refresh failure rates; parallel Nango path until Stage 3. | | Webhook double-delivery during cutover | Medium | Medium | Per-integration framework flag gates which path processes payloads; the other drops on read. | | Framework surface bloats to match Nango feature set | High | Medium | Non-goals enforced; each new capability needs an ADR. | | Legacy GitHub semantics differ from Nango GitHub | Medium | Medium | Stage 4 only after Stage 2 stable; treat legacy as a separate connector build. | | Custom skills/agents drift from framework | Low | Medium | Skills live in-repo, exercised on every Stage 1–5 build. | | Architecture doc slips, delaying Stage 2 | Medium | High | POC and architecture doc run concurrently; architecture team engaged from week 1. | | Build-vs-adopt decision locks us into a runtime we then have to maintain | Medium | High | Stage 1 records concrete cost + maintenance comparison in the architecture doc before commitment; decision is reversible until Stage 2 starts. | --- ## Implementation and rollout plan Migration required: **yes.** Integration Builder and the current framework run in parallel per integration. Cutover is per-connection and reversible via a per-integration framework flag. ### Stages - **Stage 1 — POC and architecture document.** - Exploratory build: Integration Builder scaffold (or adopted tool), OAuth grant + refresh, token-pool primitive, one connector (GitHub) as first user. - Explicit build-vs-adopt evaluation (dlt, self-hosted Nango fork, others) recorded in the architecture document. - POC validation target: **100 repositories** exercised end-to-end with token-rotation behavior on par with today's Nango GitHub sync. - **Deliverable before Stage 2 development begins: a "ready for development" architecture document approved by the architecture team.** - **Stage 2 — Nango GitHub fully migrated.** GitHub connector must scale to **~100k repositories** on the current token budget. **Deadline: 2026-09-30.** - **Stage 3 — Remaining Nango integrations migrated:** Jira (Cloud, Basic, DC API key, DC Basic), Confluence (Cloud, Basic, DC), Gerrit. **Deadline: 2026-12-31.** Nango decommissioned after this stage. - **Stage 4 — Legacy GitHub connections migrated** off `backend/src/serverless/integrations/` onto the new GitHub connector. No deadline stated. - **Stage 5 — All remaining integrations** (webhook-fed providers currently outside Nango: Discourse, GitLab, Groups.io; and any other in-house integrations except `git_integration` and `mailing_list_integration`, which stay out of scope for this project). No deadline stated. Experiment / feature flag: a per-integration framework flag drives rollout (single install → cohort → all), with an explicit rollback path. --- ### Acceptance Criteria Each stage is complete when every box below is checked. Criteria are per-stage and testable. #### Stage 1 — POC and architecture document - [ ] The POC collects data locally that matches the expected remote data for both a full historical sync and daily incremental runs on the target repositories, verified using the framework's correctness/completeness validation mechanism (Goal 2). - [ ] Zero unrecovered failures occur across the POC window. Failures are allowed; the framework retries and recovers every one without manual intervention. - [ ] The POC runs continuously for **1–2 weeks** covering incremental syncs and webhooks against **100 repositories**. - [ ] The token-rotation pool sustains rate-limit pressure with no manual intervention — tokens rotate, park, and clean up automatically under sustained load. - [ ] The architecture team approves the "ready for development" architecture document, which resolves every item in "Areas to investigate", records the build-vs-adopt decision, and locks the runtime, storage, retry, observability, and cache design. #### Stage 2 — Nango GitHub fully migrated (deadline 2026-09-30) - [ ] Every Nango GitHub connection runs on the new framework. - [ ] Nango receives **zero** GitHub API calls and **zero** GitHub webhooks. - [ ] Per-repository activity counts match pre- and post-migration — no data loss during cutover. - [ ] The team removes the GitHub integration from the Nango account and configuration. #### Stage 3 — Remaining Nango integrations migrated (deadline 2026-12-31) - [ ] Every remaining Nango integration — Jira (Cloud, Basic, DC API key, DC Basic), Confluence (Cloud, Basic, DC), Gerrit — runs on the new framework. - [ ] Nango serves **zero** live production traffic across all providers. - [ ] Per-connection activity counts match pre- and post-migration for each migrated integration. - [ ] The team decommissions the Nango account and removes the Nango dependency from CDP. #### Stage 4 — Legacy GitHub connections migrated - [ ] Every connection on `backend/src/serverless/integrations/` runs on the new GitHub connector. - [ ] Per-repository activity counts match pre- and post-migration. - [ ] The team removes the legacy serverless GitHub path from the codebase. #### Stage 5 — Remaining integrations migrated - [ ] Every remaining in-house integration — Discourse, GitLab, Groups.io, and any others outside `git_integration` and `mailing_list_integration` — runs on the new framework. - [ ] Per-connection activity counts match pre- and post-migration for each migrated integration. - [ ] No parallel legacy ingestion path remains for the migrated providers. --- ### Open Questions None blocking spec approval. See "Areas to investigate" for the questions Stage 1 must close before Stage 2 begins.