Link to this section#Integration Builder v2 (Nango deprecation)

Diagrams: view on Excalidraw

Link to this section#1. Where We Are Today

The situation: integrations run on three pipelines added at different points in the platform’s history, each solving the needs of its time. The practical consequence today: any capability we want across all integrations (visibility, retry, stats, correctness) has to be built three times, once per pipeline. So it usually gets built in one, or in none.

flowchart LR
    P["Provider APIs<br/>+ webhooks"]
    subgraph A["Path A: home-grown"]
        CRON["1-min<br/>cron ticks"] --> W["run + stream<br/>workers"]
        W --> Q[("3 queue tables<br/>on primary DB")]
    end
    subgraph B["Path B: Nango"]
        N["connector code<br/>in vendor cloud<br/>1 connection/repo"]
    end
    P --> CRON
    P --> N
    N -->|"cursor pull"| Q
    Q --> S["data-sink<br/>worker"] --> DB[("activity<br/>Relations")]

The limitations:

• Operational state lives in queue tables on the primary database: integration.runs, integration.streams, and integration.results hold per-record operational rows on the same instance as product data; processed rows are deleted after handling, run history has no retention policy, and incoming webhooks are stored in public.incomingWebhooks.
• Retries are unbounded: retry happens at stream level with a fixed backoff; there is no circuit breaker at the unit, connection, or provider level, and a rate limit on one stream delays its whole run.
• The sink has no feedback path: data_sink_worker processes results after a run’s state is already set; results published by the Nango path carry no run reference at all.
• Nothing verifies data against the provider: no mechanism compares what was stored with what the source platform holds.
• GitHub uses one vendor connection per repository: tracked in integration.nango_mapping, with connection creation rate-throttled by configuration.
• Credential storage and rotation are fragmented: credentials live in integrations columns and settings JSON, mostly plaintext, and are returned by list/find endpoints; token rotation exists as three separate implementations (the legacy GitHub connector, nango_worker, and the Nango-hosted sync code), none shared or persistent.
• High vendor cost, the main driver: Nango costs ~$200k/year today, with only part of the repositories migrated to it.
• No tests or metrics: neither connector codebase has automated tests; investigation relies on worker logs, the three state tables, and the vendor dashboard.

The business constraint driving the timeline: every Nango connection must be migrated, and the Nango account terminated, by 2026-12-31.

Link to this section#2. Architecture (validated by the POC)

Validated by the POC PR, and based on the requirements document. The POC focused specifically on the GitHub integration: it’s the majority of the data we currently ingest, and it was the main driver for adopting Nango in the first place.

Link to this section#The idea

One in-house TypeScript framework that owns connector authoring, credentials, scheduling, webhook intake, and observability. It runs entirely on infrastructure we already operate, and its only durable state is one new Postgres table. Every integration, GitHub at 100k repos or HackerNews alike, works the same way: a connector, an integration, sync units, runs, and activities.

flowchart LR
    P["Provider APIs<br/>GitHub · Jira · ..."]
    subgraph F["Integration framework (new)"]
        T[("sync_units<br/>(Postgres table)<br/>schedule · watermark")]
        D["Dispatcher<br/>(Temporal cron)<br/>rate budgets"]
        R["Sync run<br/>(Temporal workflow)<br/>fetch → validate → emit"]
        T --> D --> R
        R -->|"watermark"| T
    end
    R <-->|"fetch"| P
    R -->|"activities"| K["Kafka"]
    K --> S["data-sink<br/>worker"]
    S --> A[("activity<br/>Relations<br/>→ Tinybird")]
    R -.->|"logs"| DD["Datadog<br/>metrics + alerts"]
    S -.->|"logs"| DD
    V["Validation<br/>(Temporal, new)<br/>autonomous"] -.->|"count/IDs"| P
    V -.->|"compare"| A

(Step-by-step processing detail, including every failure scenario, lives in the sync-lifecycle diagram on Excalidraw.)

Link to this section#Four systems, each with one job

Link to this section#The sync_units table, the one new table

Column by column:

Unit identity is UNIQUE (integrationId, channelId, syncName); the dispatcher’s only query is served by one partial index (nextRunAt WHERE status = 'active').

Link to this section#What is a connector?

A connector is a declarative code package. It contains:

• a manifest: the syncs, their cadences, webhook handlers, API versions
• discover(): lists the integration’s channels (repos, projects, …)
• sync functions, Zod schemas, and mappers
• all built on one shared HTTP client that handles rate limits, retries, and tokens

One credential covers all of an integration’s channels. When an integration changes (connected, channels added or removed, deleted), its units are updated in the same flow; the database is the source of truth.

flowchart LR
    CON["Connector<br/>(code package, new)<br/>manifest · syncs<br/>· schemas"]
    INT["Integration<br/>connected account<br/>creds + settings"]
    U["sync_units<br/>(Postgres table, new)<br/>one row per<br/>channel × sync"]
    RUN["Runs<br/>(Temporal workflows)<br/>one each"]
    ACT["Activities<br/>identity-keyed"]
    CON --- U
    INT -->|"discover()"| U
    U --> RUN --> ACT

Link to this section#How does a sync run execute?

1. The dispatcher picks up due units when the rate budget allows.
2. One Temporal workflow starts per run.
3. The sync function loops through pages in-process: fetch → map → validate → emit.
4. The watermark moves forward only after Kafka confirms the batch.

Backfill and recovery fall out of the same mechanism:

• A backfill is just a watermark starting at NULL and sweeping oldest-to-newest; backfill vs incremental is the same code at different positions.
• A crash resumes from the watermark.
• The watermark also allows partial re-onboarding: set it to a specific date or event (whatever cursor the integration uses) and the unit re-syncs from that point only, instead of a full backfill.
• Correctness rests on one mechanism: every record carries an identity key and the sink upserts on it. Any failure just means re-fetching a bit of overlap, which the upsert absorbs.

The POC confirmed this works: deploys, restarts mid-backfill, parks and replays all resumed with no manual step and no duplicates.

Link to this section#How do we know the data is correct and complete?

Validation jobs run as ordinary sync units, asking the provider for the truth and comparing it with what CDP holds:

• Counts: a dedicated one-call lookup per connector (remoteCount), deliberately independent of the fetch code so it can catch our own bugs.
• Provider counts are compared against Tinybird data, the end of the pipeline, so a discrepancy surfaces problems on either side: integration fetching/mapping, or ingestion from Kafka to Tinybird.
• ID-level checks: reuse the sync itself in a no-emit mode, so there’s no second implementation to drift.
• Discrepancies alert; fixing a gap is a watermark rewind (one SQL update). A potential win down the line: letting validation trigger that rewind itself, making the pipeline self-healing.
• Migration cutover checks use the same mechanism.
• The provider is the referee; internal telemetry is never the proof.

The validator already ran in the POC, invoked by hand: it paged GitHub for per-activity-type counts on linuxfoundation/crowd.dev and compared them with an export of what CDP holds. It confirmed the POC data was complete against GitHub and found missing data in the Nango pipeline in the same run. The check that ran and the remoteCount unit that ships do the same thing: ask the provider for counts and compare. The final version only changes how it is triggered: scheduled as validation units, it runs on every unit and alerts instead of waiting to be invoked.

Rollout starts with count validation as a weekly scheduled unit: it runs during the weekend to validate the current week’s counts, if the rate budget allows, and is extended from there based on observed rate-limit usage.

Link to this section#How do webhooks fit in?

• A stateless endpoint: verify the signature, resolve the event to a unit, run the handler through the same emit path.
• Webhook-covered syncs drop to slow gap-fill polling.
• Webhooks are best-effort by design: polling remains the safety net, so intake needs no queue, no dedupe, and no retry machinery.

Link to this section#Observability

Operational state lives in three places, and nowhere else (no observability tables in Postgres):

1. Structured logs with bound ids → Datadog: every log line carries the integration, unit, and run ids; Datadog generates metrics at ingest, groups errors automatically, and drives monitors → Slack. Structured JSON logging follows ADR-0002; ADR-0003 also applies.
2. The sync_units row: an ordinary Postgres table, so plain SQL, Metabase dashboards, and Metaplane monitors work on it directly, with no instrumentation and no code change.
3. Temporal run history: workflow id = unit id; the per-run record.

Correctness is owned by the validation section above; observability answers the operational questions below. The split between the tools: Datadog on rates and absence, Metabase and Metaplane on which units and why (Datadog metrics stay tagged by platform and sync only, to keep cost flat, so they cannot answer per-unit questions).

Link to this section#A sync failed: where do we debug?

In order, from cheapest to deepest:

1. The unit row: lastErrorClass says what kind of failure, lastErrorMessage says why, consecutiveFailures says how long it has been happening.
2. Datadog logs filtered by the unit or run id: the bound ids give the full trail of one run; error grouping clusters the same failure across units.
3. The run’s Temporal history: the complete per-run record, found directly since the workflow id is the unit id.

In the POC, log events with bound ids plus the sync_units row were enough to diagnose every incident in the run (pool cold-start deadlock, silent headroom refusal, deep-path failures, label-nesting rejections) without reading code.

Link to this section#Is a unit alive and producing?

The unit row answers directly:

• lastRunAt / lastSuccessAt: last attempt and last success; stale lastSuccessAt against the unit’s cadence means it is falling behind
• emittedCount: activities from the last successful run (not cumulative; totals are counted at the sink)
• lastRunComplete: did the latest run finish its pages
• status: units stuck at dead_letter need a human

Link to this section#How do we keep track of daily runs?

• Every run logs a run summary, and every dispatcher cycle logs a dispatch summary (claimed, admitted, deferred, started, failed, duration).
• In Datadog those become ingest-time metrics and a per-platform dashboard, built from the existing log fields with no code change.
• Fleet-level counts are plain SQL on sync_units for Metabase: counts grouped by lastErrorClass, consecutiveFailures above a threshold, units whose lastSuccessAt is older than their cadence.

Link to this section#How far along is a backfill?

watermark.phase on the unit row (backfill → incremental) is the progress signal for the whole fleet: every sync starts at backfill and flips when it completes. Fleet-wide backfill progress is one SQL query.

Link to this section#How do we get alerted?

All of these fire alerts to Slack:

• Datadog monitors on rates and absence.
• Metaplane monitors on sync_units for per-unit conditions: stale lastSuccessAt, failure streaks, dead_letter units.
• Validation discrepancies (see the correctness section above): the provider-vs-CDP count and ID checks alert when data is missing, not just when runs fail.

Link to this section#Infrastructure

v2 introduces no new infrastructure. It runs on the Oracle (OKE) setup CDP already operates (the same Kubernetes cluster, Postgres, Redis, and Kafka), deployed and operated like any existing worker. The Oracle arrangement runs until the end of 2027.

Risk flag: the infrastructure may need to migrate to another provider after 2027. That decision affects the entire CDP system, not this project specifically, but it is worth flagging here: v2 self-hosts the sync pipeline on that infrastructure, whereas Nango runs vendor-hosted and depends on it far less. The framework itself carries no provider coupling (standard Kubernetes workloads on Postgres, Redis, and Kafka), so it moves with the rest of the platform.

Link to this section#GitHub token rotation: the shared installation pool

This mechanism is GitHub-specific for now — it’s built around GitHub App installations and their rate budget. The pattern is designed to generalize: the same pooling and rotation logic could cover any other token-based integration (GitLab, for example) once one needs it. This is also the part the POC tested hardest.

Link to this section#The model: a pool entry is an installation, not a token

• GitHub’s rate budget is per installation and survives token re-mints, so a new token value is not a new budget.
• Using tokens as entries was an early bug: it let one run revive an installation another had just parked.
• Installation tokens are minted on demand from the App credential, cached, and re-minted when within 2 minutes of expiry.

State lives in Redis, per platform:

• an entry hash keyed by installation id: {token, tokenExpiresAtMs, parkedUntil, quarantined}
• an LRU sorted set over the entries
• one budget hash per entry: limit, remaining, resetAt, probedAt
• a Lua script does read-merge-write in one step, so two runs cannot overwrite each other’s parks and quarantines

Link to this section#How a request gets a token, and what happens when calls fail

flowchart TD
    REQ["Provider request<br/>needs a token"] --> WALK["Walk pool in LRU order<br/>(channel's own installation first)"]
    WALK --> CHK{"Entry usable?<br/>not quarantined · not parked<br/>token mintable · budget left"}
    CHK -->|"no"| MORE{"More entries?"}
    MORE -->|"yes"| CHK
    MORE -->|"no: pool out"| OUT["One rate-limit error with the<br/>earliest park/budget reset;<br/>run parks, dispatcher defers the unit"]
    CHK -->|"yes"| USE["Take entry,<br/>update LRU score"]
    USE --> CALL["Provider call"]
    CALL -->|"success"| DONE["Response"]
    CALL -->|"rate limit<br/>(status or GraphQL body)"| PARK["Park entry until reset,<br/>retry same request<br/>on next entry"] --> MORE
    CALL -->|"auth failure"| REMINT["Drop cached token<br/>(not the entry),<br/>retry once with fresh mint"]
    MINT["Mint failure<br/>(provider.auth /<br/>provider.contract)"] -.-> Q["Quarantine entry"]

Acquisition:

• walks the pool in LRU order, the channel’s own installation first if it has one
• takes the first entry that is not quarantined, not parked, can mint a token, and has budget left
• budget comes from GitHub’s rate-limit endpoint, re-probed when the cached value is older than 90 seconds or past its reset
• a successful acquire updates the entry’s LRU score, which spreads load across the pool instead of draining one installation

On failure the client rotates instead of failing; the run only reports an error when the whole pool is out:

Link to this section#The dispatcher checks budget before starting work

• It sums remaining budget across healthy entries against a per-run estimate and defers units it cannot cover, so a starved pool produces deferrals instead of runs that fail on their first call.
• Entries are re-seeded hourly from the App’s installation list, filtered by CROWD_GITHUB_POOL_INSTALLATION_IDS, using set-if-absent so a reseed does not clear live park or quarantine state.

Link to this section#What the POC proved, and the main argument for v2 over Nango

Nango can reach the same installations, but:

Link to this section#Why not an existing connector ecosystem (Singer, dlt)

Both were tested hands-on (standalone micro-POCs, re-runnable scripts) against the three mechanisms v2 rests on. Both are Python-only, so either would run as a Python sidecar inside the TS Temporal workers.

Custom GraphQL, where the GitHub budget economics live, is hand-written code in both. Testing also hit real silent-data bugs in one of the best-maintained community taps (issues crash; a stream that silently re-pulls everything every run). Net: they supply the cheap layer (pagination, config), we’d still build the hard layer (pool, rotation, watermark) against their internals, and the POC’s pool evidence would have to be re-proven on a fork.

It seems even that both of these gives us what Nango gave us which is: for every single connector that Nango had to build for us, they couldn’t use their default platform connector. They always had to customize it, either because we want/need specific data, or our authentication mechanism is also not standard. Being github the main driver.

Link to this section#Authentication beyond GitHub

GitHub is the hardest case and was built first. Every other platform’s auth is a simpler instance of the same machinery: a pool whose “mint” step is the scheme-specific part. The connector manifest declares the scheme; the framework owns storage, refresh, and supply. Sync functions never see a credential: the shared HTTP client acquires one per request, reports the outcome back, and parks or quarantines by the same rules as the GitHub pool.

Pool scope follows the credential. GitHub’s pool is platform-scoped because installations are interchangeable budget. Every other scheme is integration-scoped: a single-entry pool per integrations row, holding that instance’s own credential, captured where it is captured today (user-entered credentials in the CDP connect flow for Jira/Confluence/Gerrit; the existing backend OAuth endpoints, which store the instance’s refresh token) and stored on the integrations row. Same storage, park, and quarantine semantics. No rotation, because none of these platforms has needed it (their rate ceilings are comfortable relative to our volume), with rotation available if one ever does.

Derived tokens are produced inside the acquisition walk itself. When an entry’s cached derived token is missing or near expiry, acquire runs the scheme’s mint synchronously, caches the result with its expiry, and hands it out; single-flight behind the pool’s Lua lock, so concurrent runs cannot race a one-time-use refresh token. There is no background token-maintainer process. Only the GitHub mint path (App JWT → installation token) is POC-proven; the others reuse the same slot as design.

None of the schemes is new. Each reuses the exact credential the platform’s connections use today; v2 changes who attaches it to the request, not what the user provides:

Link to this section#AI: skills and autonomous agents

AI is a deliberate part of the architecture, in two forms: skills (defined procedures that carry a workflow end to end, verified by the framework’s gates) and autonomous agents (continuously running monitors). The POC connector was already written mostly by AI against staging.

The point is maintainability. The long-term cost of an in-house framework is not building it, it is maintaining a growing set of connectors: keeping up with provider API changes, watching daily runs, porting the remaining platforms, adding new ones. The table assigns each of those to a skill or an agent, so the recurring human work per integration shrinks to reviewing data mappings and handling escalations, and does not grow with the number of connectors.

This is a requirement on the framework, not an add-on: the manifest, schemas, mappers and CI gates must be standard enough that a skill can generate and verify a connector without improvising, and agents work through the same interfaces humans use (structured logs with bound ids, sync_units, one-line SQL operations).

Link to this section#What no longer exists at the end state

• Path A’s three workers, integration.runs, integration.streams (and, post-migration, potentially integration.results, pending the sink rework review)
• the 1-minute cron ticks and the bin scripts
• all of Nango: cloud, webhook API, worker, nango_mapping, nango_cursors
• incomingWebhooks
• per-record operational rows of any kind
• the three separate token-rotation implementations

Link to this section#3. What the POC validated

The full GitHub connector (9 syncs) ran in staging against ~100 public repos on real APIs, from onboarding to activityRelations, polling only:

Link to this section#4. Migration

The one fixed point: every Nango connection migrated and the Nango account terminated by 2026-12-31.

How one connection moves: freeze the legacy connector for that platform → set a migration flag on its integrations row (legacy → shadow → migrated, read by both pipelines) → shadow-run the new pipeline, recording counts and IDs instead of emitting, while legacy keeps feeding production → flip only when the validator is clean against both the provider and what legacy ingested → watch ~7 days, flipping back on any surprise. Nango keeps fetching through the watch window, which is what makes rollback instant; once the connection is confirmed it is paused or deleted on the Nango side, and its rollback expires there.

The migrations are AI-driven. The GitHub port was already done that way. Once the core pipeline is stable, each remaining platform migrates through the migration skill described in §2 under “AI: skills and autonomous agents”, which carries the steps above end to end.

Link to this section#5. Cost at scale

Link to this section#Datadog

The product requirement is 24h data latency; syncs run twice daily. All figures below are a deliberate ceiling, every assumption priced at its harsh end, so the real bill lands under the estimate: 50% retry allowance (the scale test showed failure streaks averaging 10 runs during incidents), 12 lines per run (2.5x the measured 4.8, covering data-heavy windows and contention skip lines), and on-demand indexing rates with no annual commit assumed. Measured basis: a staging window on one connectors-worker pod (14,752 log lines) plus production fleet counts. 30-day retention.

If Datadog gets too expensive, the fallback is Grafana, Loki and S3; the framework only emits structured JSON logs, so switching backend needs no code change.

Link to this section#Logs: linear in units × cadence

The acquire line is 58% of volume; an index exclusion filter on it (ingest-only) roughly halves the bill. The 43,577 non-GitHub units add ~$0.2k/month at the same cadence. Fleet-wide load at 100k repos is ~85 req/s, 2 to 3 pods.

Link to this section#Metrics: flat in fleet size if the tag set holds

$0.05 per distinct timeseries (metric name × tag values). Tag only bounded categories, never identifiers, and the bill does not move as the fleet grows.

The host tag is billed by default (@crowd/telemetry submits via per-node DogStatsD), so the ceiling multiplies the fleet by 10 nodes: ~13,800 timeseries → $690/month. Measured usage sits within the pooled per-host allotment (avg 3,284 of a ≥5,530 pool at 55 hosts); the ceiling does not bank on it.

Link to this section#Total (ceiling)

Logs include the flat ~$0.2k non-GitHub share. Initial backfill of the full 100k fleet adds a one-time ~$5k in logs. Not included: infra hosts ($15-23/host/month).

Link to this section#Temporal

Temporal Cloud bills per action (workflow start, activity start, each retry attempt) plus history storage. A sync run costs 3 actions as coded (workflow start, one executeSync activity, the dispatcher’s startRun activity); the ceiling prices 4 to absorb dispatcher overhead: its 5 fixed activities per cycle (~24k cycles/day at target), deferUnit loops when the pool is parked, and failed-start retries. Same run basis as Datadog: twice daily +50% retries, plus the ~4M monthly non-GitHub unit runs. Contract rates (order form 2025-12): $23.28/M actions, $0.00105/GBh retained storage, plus a Business plan fee of 10% of usage. History per run at 15KB (2x the assumed steady-state size), 7-day retention.

Baseline from the August 2026 invoice: the entire platform today consumes 134M actions/month for $3.6k; the v2 ceiling at the 100k long-term goal adds ~2.3x that volume, against a $53k/yr credit commitment. Initial backfill of the full 100k fleet adds a one-time ~$5k in actions at 500 runs/repo. Cost scales linearly with cadence: hourly on the 5 hot syncs would be ~$35k/month Temporal alone.

Link to this section#Bottom line

Min is the optimized floor: measured 4.8 lines/run, acquire-line exclusion filter on, committed Datadog rates, metrics inside the allotment, 3 actions/run, minimal retries. Max is the ceiling above.

Initial backfill is a separate one-time cost, paid once per repo when it is onboarded, never again after: ~$10k total to backfill the full 100k fleet at ceiling assumptions (~$5k Temporal actions + ~$5k Datadog logs), scaling proportionally with the repos onboarded.