# Integration Builder v2 (Nango deprecation)


**Diagrams:** [view on Excalidraw](https://excalidraw.com/#json=wDNGxu5dmarqxxpY7oq6k,NpdzD_1nW0WZVpYJmdafNg)

---

## 1. Where We Are Today

**The situation:** integrations run on **three pipelines added at different points in the platform's history**, each solving the needs of its time. The practical consequence today: any capability we want across all integrations (visibility, retry, stats, correctness) has to be built three times, once per pipeline. So it usually gets built in one, or in none.

```mermaid
flowchart LR
    P["Provider APIs<br/>+ webhooks"]
    subgraph A["Path A: home-grown"]
        CRON["1-min<br/>cron ticks"] --> W["run + stream<br/>workers"]
        W --> Q[("3 queue tables<br/>on primary DB")]
    end
    subgraph B["Path B: Nango"]
        N["connector code<br/>in vendor cloud<br/>1 connection/repo"]
    end
    P --> CRON
    P --> N
    N -->|"cursor pull"| Q
    Q --> S["data-sink<br/>worker"] --> DB[("activity<br/>Relations")]
```

**The limitations:**

- **Operational state lives in queue tables on the primary database:** `integration.runs`, `integration.streams`, and `integration.results` hold per-record operational rows on the same instance as product data; processed rows are deleted after handling, run history has no retention policy, and incoming webhooks are stored in `public.incomingWebhooks`.
- **Retries are unbounded:** retry happens at stream level with a fixed backoff; there is no circuit breaker at the unit, connection, or provider level, and a rate limit on one stream delays its whole run.
- **The sink has no feedback path:** `data_sink_worker` processes results after a run's state is already set; results published by the Nango path carry no run reference at all.
- **Nothing verifies data against the provider:** no mechanism compares what was stored with what the source platform holds.
- **GitHub uses one vendor connection per repository:** tracked in `integration.nango_mapping`, with connection creation rate-throttled by configuration.
- **Credential storage and rotation are fragmented:** credentials live in `integrations` columns and settings JSON, mostly plaintext, and are returned by list/find endpoints; token rotation exists as three separate implementations (the legacy GitHub connector, `nango_worker`, and the Nango-hosted sync code), none shared or persistent.
- **High vendor cost, the main driver:** Nango costs ~$200k/year today, with only part of the repositories migrated to it.
- **No tests or metrics:** neither connector codebase has automated tests; investigation relies on worker logs, the three state tables, and the vendor dashboard.

**The business constraint driving the timeline:** every Nango connection must be migrated, and the Nango account terminated, by **2026-12-31**.

---

## 2. Architecture (validated by the POC)

> Validated by the [POC PR](https://github.com/linuxfoundation/crowd.dev/pull/4463), and based on the [requirements document](https://linuxfoundation-dxwx.dsp.so/R446nfTq-integration-builder-tech-spec#integration-builder--in-house-integration-framework). The POC focused specifically on the GitHub integration: it's the majority of the data we currently ingest, and it was the main driver for adopting Nango in the first place.

### The idea

One in-house TypeScript framework that owns connector authoring, credentials, scheduling, webhook intake, and observability. It runs entirely on infrastructure we already operate, and its only durable state is **one new Postgres table**. Every integration, GitHub at 100k repos or HackerNews alike, works the same way: a connector, an integration, sync units, runs, and activities.

```mermaid
flowchart LR
    P["Provider APIs<br/>GitHub · Jira · ..."]
    subgraph F["Integration framework (new)"]
        T[("sync_units<br/>(Postgres table)<br/>schedule · watermark")]
        D["Dispatcher<br/>(Temporal cron)<br/>rate budgets"]
        R["Sync run<br/>(Temporal workflow)<br/>fetch → validate → emit"]
        T --> D --> R
        R -->|"watermark"| T
    end
    R <-->|"fetch"| P
    R -->|"activities"| K["Kafka"]
    K --> S["data-sink<br/>worker"]
    S --> A[("activity<br/>Relations<br/>→ Tinybird")]
    R -.->|"logs"| DD["Datadog<br/>metrics + alerts"]
    S -.->|"logs"| DD
    V["Validation<br/>(Temporal, new)<br/>autonomous"] -.->|"count/IDs"| P
    V -.->|"compare"| A
```

*(Step-by-step processing detail, including every failure scenario, lives in the [sync-lifecycle diagram on Excalidraw](https://excalidraw.com/#json=wDNGxu5dmarqxxpY7oq6k,NpdzD_1nW0WZVpYJmdafNg).)*

### Four systems, each with one job

| System | Job |
|---|---|
| **Postgres** (`integration.sync_units`) | The control table: schedule, status, watermark, last emitted count; one row per (channel, sync). Every operational action is a SQL UPDATE. |
| **Temporal** | Runs the work: one dispatcher cron + one sync-run workflow type. Retries, heartbeats, and timeouts come built-in. Workflow id = unit id, so a unit can never run twice at once. Run history doubles as the debugging record. |
| **Kafka** | One job: carry validated activities to the data-sink, through the existing publishing path. Changes to the sink and to `integration.results` are out of scope: reworking `data_sink_worker` is a post-migration proposal that needs its own architecture review |
| **Redis** | Short-lived coordination: token pool, rate budgets, OAuth nonces, derived tokens. |

### The `sync_units` table, the one new table

Column by column:

| Column | Represents |
|---|---|
| `id` | Unit identifier (uuid) |
| `integrationId` | The owning integration (FK to `public.integrations`) |
| `platform` | Platform name, denormalized so ops actions can filter by it (`'github'`, `'jira'`, …) |
| `channelId` | The provider's **stable** identifier for the channel (GitHub repo node ID, Slack channel ID, …); unit identity, survives renames |
| `channelName` | The human-readable channel name (repo URL, `#channel`); used for display, ops queries, and stamped as `activity.channel` on emit |
| `syncName` | Which of the connector's syncs this unit runs (`issues`, `pullRequests`, …) |
| `status` | `active` \| `paused` \| `dead_letter` \| `decommissioned` |
| `nextRunAt` | When the unit runs next; the scheduling truth the dispatcher reads. Exactly one writer: the run's own terminal outcome |
| `lockedAt` | Claim marker, set at claim and cleared by the run's outcome write. "Is this unit running" is answered by Temporal, not by this column |
| `lastRunAt` / `lastSuccessAt` | Last attempt / last success; freshness at a glance |
| `consecutiveFailures` | Current failure streak; drives the circuit breaker |
| `lastErrorClass` / `lastErrorMessage` | What kind of failure, and why. The message column was added during the POC; the class alone sent every diagnosis into the logs |
| `watermark` | The only cursor (JSONB, connector-owned; `NULL` = never synced → full backfill). `watermark.phase` (`backfill` → `incremental`) is the **progress signal for the whole fleet**: every sync starts at `backfill` and flips when it completes |
| `lastRunComplete` | Did the latest run finish its pages. Added during the POC. **Latest run only**: if a backfilled unit's next incremental run parks, it flips back to false, so it does not mean "backfill done" |
| `emittedCount` | Activities from the **last successful run** ("is this unit producing?"). Not cumulative; totals must be counted at the sink |
| `createdAt` / `updatedAt` | Bookkeeping |

Unit identity is `UNIQUE (integrationId, channelId, syncName)`; the dispatcher's only query is served by one partial index (`nextRunAt WHERE status = 'active'`).

### What is a connector?

A connector is a declarative code package. It contains:

- a **manifest**: the syncs, their cadences, webhook handlers, API versions
- **`discover()`**: lists the integration's channels (repos, projects, ...)
- **sync functions**, **Zod schemas**, and **mappers**
- all built on one **shared HTTP client** that handles rate limits, retries, and tokens

One credential covers all of an integration's channels. When an integration changes (connected, channels added or removed, deleted), its units are updated in the same flow; the database is the source of truth.

```mermaid
flowchart LR
    CON["Connector<br/>(code package, new)<br/>manifest · syncs<br/>· schemas"]
    INT["Integration<br/>connected account<br/>creds + settings"]
    U["sync_units<br/>(Postgres table, new)<br/>one row per<br/>channel × sync"]
    RUN["Runs<br/>(Temporal workflows)<br/>one each"]
    ACT["Activities<br/>identity-keyed"]
    CON --- U
    INT -->|"discover()"| U
    U --> RUN --> ACT
```

### How does a sync run execute?

1. The dispatcher picks up due units when the rate budget allows.
2. One Temporal workflow starts per run.
3. The sync function loops through pages in-process: fetch → map → validate → emit.
4. The watermark moves forward only after Kafka confirms the batch.

Backfill and recovery fall out of the same mechanism:

- A backfill is just a watermark starting at NULL and sweeping oldest-to-newest; backfill vs incremental is the same code at different positions.
- A crash resumes from the watermark.
- The watermark also allows partial re-onboarding: set it to a specific date or event (whatever cursor the integration uses) and the unit re-syncs from that point only, instead of a full backfill.
- Correctness rests on one mechanism: every record carries an identity key and the sink upserts on it. Any failure just means re-fetching a bit of overlap, which the upsert absorbs.

**The POC confirmed this works**: deploys, restarts mid-backfill, parks and replays all resumed with no manual step and no duplicates.

### How do we know the data is correct and complete?

Validation jobs run as ordinary sync units, asking the provider for the truth and comparing it with what CDP holds:

- **Counts**: a dedicated one-call lookup per connector (`remoteCount`), deliberately independent of the fetch code so it can catch our own bugs.
- **Provider counts are compared against Tinybird data**, the end of the pipeline, so a discrepancy surfaces problems on either side: integration fetching/mapping, or ingestion from Kafka to Tinybird.
- **ID-level checks**: reuse the sync itself in a no-emit mode, so there's no second implementation to drift.
- **Discrepancies alert**; fixing a gap is a watermark rewind (one SQL update). A potential win down the line: letting validation trigger that rewind itself, making the pipeline self-healing.
- **Migration cutover checks** use the same mechanism.
- The provider is the referee; internal telemetry is never the proof.

**The validator already ran in the POC**, invoked by hand: it paged GitHub for per-activity-type counts on `linuxfoundation/crowd.dev` and compared them with an export of what CDP holds. It **confirmed the POC data was complete against GitHub** and **found missing data in the Nango pipeline** in the same run. The check that ran and the `remoteCount` unit that ships do the same thing: ask the provider for counts and compare. The final version only changes how it is triggered: scheduled as validation units, it runs on every unit and alerts instead of waiting to be invoked.

**Rollout starts with count validation as a weekly scheduled unit**: it runs during the weekend to validate the current week's counts, if the rate budget allows, and is extended from there based on observed rate-limit usage.

### How do webhooks fit in?

- A stateless endpoint: verify the signature, resolve the event to a unit, run the handler through the same emit path.
- Webhook-covered syncs drop to slow gap-fill polling.
- Webhooks are best-effort by design: polling remains the safety net, so intake needs no queue, no dedupe, and no retry machinery.

### Observability

Operational state lives in three places, and nowhere else (no observability tables in Postgres):

1. **Structured logs with bound ids → Datadog**: every log line carries the integration, unit, and run ids; Datadog generates metrics at ingest, groups errors automatically, and drives monitors → Slack. Structured JSON logging follows [ADR-0002](https://github.com/linuxfoundation/lfx-architecture-decisions/blob/main/decisions/0002-structured-json-logging.md); [ADR-0003](https://github.com/linuxfoundation/lfx-architecture-decisions/blob/main/decisions/0003-opentelemetry-instrumentation.md) also applies.
2. **The `sync_units` row**: an ordinary Postgres table, so plain SQL, Metabase dashboards, and Metaplane monitors work on it directly, with no instrumentation and no code change.
3. **Temporal run history**: workflow id = unit id; the per-run record.

Correctness is owned by the validation section above; observability answers the operational questions below. The split between the tools: Datadog on rates and absence, Metabase and Metaplane on which units and why (Datadog metrics stay tagged by platform and sync only, to keep cost flat, so they cannot answer per-unit questions).

#### A sync failed: where do we debug?

In order, from cheapest to deepest:

1. **The unit row**: `lastErrorClass` says what kind of failure, `lastErrorMessage` says why, `consecutiveFailures` says how long it has been happening.
2. **Datadog logs filtered by the unit or run id**: the bound ids give the full trail of one run; error grouping clusters the same failure across units.
3. **The run's Temporal history**: the complete per-run record, found directly since the workflow id is the unit id.

In the POC, log events with bound ids plus the `sync_units` row were enough to diagnose every incident in the run (pool cold-start deadlock, silent headroom refusal, deep-path failures, label-nesting rejections) without reading code.

#### Is a unit alive and producing?

The unit row answers directly:

- `lastRunAt` / `lastSuccessAt`: last attempt and last success; stale `lastSuccessAt` against the unit's cadence means it is falling behind
- `emittedCount`: activities from the last successful run (not cumulative; totals are counted at the sink)
- `lastRunComplete`: did the latest run finish its pages
- `status`: units stuck at `dead_letter` need a human

#### How do we keep track of daily runs?

- Every run logs a **run summary**, and every dispatcher cycle logs a **dispatch summary** (claimed, admitted, deferred, started, failed, duration).
- In Datadog those become ingest-time metrics and a per-platform dashboard, built from the existing log fields with no code change.
- Fleet-level counts are plain SQL on `sync_units` for Metabase: counts grouped by `lastErrorClass`, `consecutiveFailures` above a threshold, units whose `lastSuccessAt` is older than their cadence.

#### How far along is a backfill?

`watermark.phase` on the unit row (`backfill` → `incremental`) is the progress signal for the whole fleet: every sync starts at `backfill` and flips when it completes. Fleet-wide backfill progress is one SQL query.

#### How do we get alerted?

All of these fire alerts to Slack:

- **Datadog monitors** on rates and absence.
- **Metaplane monitors on `sync_units`** for per-unit conditions: stale `lastSuccessAt`, failure streaks, `dead_letter` units.
- **Validation discrepancies** (see the correctness section above): the provider-vs-CDP count and ID checks alert when data is missing, not just when runs fail.

### Infrastructure

**v2 introduces no new infrastructure.** It runs on the Oracle (OKE) setup CDP already operates (the same Kubernetes cluster, Postgres, Redis, and Kafka), deployed and operated like any existing worker. The Oracle arrangement runs until the end of 2027.

**Risk flag: the infrastructure may need to migrate to another provider after 2027.** That decision affects the entire CDP system, not this project specifically, but it is worth flagging here: v2 self-hosts the sync pipeline on that infrastructure, whereas Nango runs vendor-hosted and depends on it far less. The framework itself carries no provider coupling (standard Kubernetes workloads on Postgres, Redis, and Kafka), so it moves with the rest of the platform.

### GitHub token rotation: the shared installation pool

This mechanism is GitHub-specific for now — it's built around GitHub App installations and their rate budget. The pattern is designed to generalize: the same pooling and rotation logic could cover any other token-based integration (GitLab, for example) once one needs it. This is also the part the POC tested hardest.

#### The model: a pool entry is an installation, not a token

- GitHub's rate budget is per installation and survives token re-mints, so a new token value is not a new budget.
- Using tokens as entries was an early bug: it let one run revive an installation another had just parked.
- Installation tokens are minted on demand from the App credential, cached, and re-minted when within 2 minutes of expiry.

State lives in Redis, per platform:

- an entry hash keyed by installation id: `{token, tokenExpiresAtMs, parkedUntil, quarantined}`
- an LRU sorted set over the entries
- one budget hash per entry: `limit`, `remaining`, `resetAt`, `probedAt`
- a Lua script does read-merge-write in one step, so two runs cannot overwrite each other's parks and quarantines

#### How a request gets a token, and what happens when calls fail

```mermaid
flowchart TD
    REQ["Provider request<br/>needs a token"] --> WALK["Walk pool in LRU order<br/>(channel's own installation first)"]
    WALK --> CHK{"Entry usable?<br/>not quarantined · not parked<br/>token mintable · budget left"}
    CHK -->|"no"| MORE{"More entries?"}
    MORE -->|"yes"| CHK
    MORE -->|"no: pool out"| OUT["One rate-limit error with the<br/>earliest park/budget reset;<br/>run parks, dispatcher defers the unit"]
    CHK -->|"yes"| USE["Take entry,<br/>update LRU score"]
    USE --> CALL["Provider call"]
    CALL -->|"success"| DONE["Response"]
    CALL -->|"rate limit<br/>(status or GraphQL body)"| PARK["Park entry until reset,<br/>retry same request<br/>on next entry"] --> MORE
    CALL -->|"auth failure"| REMINT["Drop cached token<br/>(not the entry),<br/>retry once with fresh mint"]
    MINT["Mint failure<br/>(provider.auth /<br/>provider.contract)"] -.-> Q["Quarantine entry"]
```

Acquisition:

- walks the pool in LRU order, the channel's own installation first if it has one
- takes the first entry that is not quarantined, not parked, can mint a token, and has budget left
- budget comes from GitHub's rate-limit endpoint, re-probed when the cached value is older than 90 seconds or past its reset
- a successful acquire updates the entry's LRU score, which spreads load across the pool instead of draining one installation

On failure the client rotates instead of failing; the run only reports an error when the whole pool is out:

| Condition | What happens |
|---|---|
| Rate limit: HTTP status **or** GraphQL body (GitHub returns 200 with `errors[].type: 'RATE_LIMIT'`) | Park the entry until the reset time from the error or the headers, then retry the same request on the next entry. Rotation stops when the pool reports itself out, not after a fixed number of tries |
| Pool out of entries | One rate-limit error carrying the earliest park or budget reset. The run parks and the dispatcher defers the unit until then |
| Auth failure on a request | Drop that entry's cached token, not the entry, and retry once with a fresh one |
| Mint failure classed `provider.auth` / `provider.contract` | Quarantine the entry. Kept for inspection, never revived automatically in the POC |
| Provider unavailable | 3 attempts with exponential backoff, then fail the run |

#### The dispatcher checks budget before starting work

- It sums remaining budget across healthy entries against a per-run estimate and defers units it cannot cover, so a starved pool produces deferrals instead of runs that fail on their first call.
- Entries are re-seeded hourly from the App's installation list, filtered by `CROWD_GITHUB_POOL_INSTALLATION_IDS`, using set-if-absent so a reseed does not clear live park or quarantine state.

#### What the POC proved, and the main argument for v2 over Nango

| Finding | Evidence |
|---|---|
| Installation tokens are interchangeable at scale | Pool at 104 installations: 1,206 mints served; 42 different installations seen serving arbitrary repos in one 3-minute window; zero `provider.auth` failures |
| Because they are interchangeable, the budgets add up | The production App's 502 installations across 16,586 repos give about **4.67M GraphQL points/hour** (5,000/hr per installation, +50 per repo or user, capped at 12,500, floor of 10,000 on Enterprise Cloud), usable against any public repo |
| What that means at 100k repos | GitHub is one shared budget to schedule against, not a token to provision per connection |

Nango can reach the same installations, but:

| | Nango | v2 pool |
|---|---|---|
| Adding an installation | Two manual scripts | Reseeded hourly from the App's installation list, no scripts |
| New capacity usable | Up to a month later: the list is cached for 30 days with no invalidation but the TTL | Within the hour |
| Rotation state | Vendor-hosted code with no shared state; a drained installation cannot be parked fleet-wide | Parks and quarantines shared in Redis, fleet-wide |

### Why not an existing connector ecosystem (Singer, dlt)

Both were tested hands-on (standalone micro-POCs, re-runnable scripts) against the three mechanisms v2 rests on. Both are Python-only, so either would run as a Python sidecar inside the TS Temporal workers.

| v2 needs | Singer / tap-github (v1.29.2) | dlt (1.30.0) |
|---|---|---|
| Bounded, resumable runs | ❌ State every 10k records; a killed run loses all progress; forced mid-run checkpoints are marked not resumable | ✅ only at the low-level client (page-by-page, caller keeps cursor); ❌ the pipeline layer runs the whole sync itself |
| Watermark saved only after Kafka ack | ✅ State is stdout lines; the runner saves when it wants (tested) | ❌ Cursor is saved with the data automatically, no hook to gate it |
| Redis-shared token pool, rotate on 429 | ❌ In-memory list per process, crash when empty, no injection point; Redis-backing means forking | ❌ Proven: a retried request reuses the same token even after the pool switches; rotation requires owning the request loop |

Custom GraphQL, where the GitHub budget economics live, is hand-written code in both. Testing also hit real silent-data bugs in one of the best-maintained community taps (`issues` crash; a stream that silently re-pulls everything every run). Net: they supply the cheap layer (pagination, config), we'd still build the hard layer (pool, rotation, watermark) against their internals, and the POC's pool evidence would have to be re-proven on a fork.

It seems even that both of these gives us what Nango gave us which is: for every single connector that Nango had to build for us, they couldn't use their default platform connector. They always had to customize it, either because we want/need specific data, or our authentication mechanism is also not standard. Being github the main driver.

### Authentication beyond GitHub

GitHub is the hardest case and was built first. Every other platform's auth is a simpler instance of the same machinery: **a pool whose "mint" step is the scheme-specific part.** The connector manifest declares the scheme; the framework owns storage, refresh, and supply. Sync functions never see a credential: the shared HTTP client acquires one per request, reports the outcome back, and parks or quarantines by the same rules as the GitHub pool.

**Pool scope follows the credential.** GitHub's pool is platform-scoped because installations are interchangeable budget. Every other scheme is integration-scoped: a single-entry pool per `integrations` row, holding that instance's own credential, captured where it is captured today (user-entered credentials in the CDP connect flow for Jira/Confluence/Gerrit; the existing backend OAuth endpoints, which store the instance's refresh token) and stored on the `integrations` row. Same storage, park, and quarantine semantics. No rotation, because none of these platforms has needed it (their rate ceilings are comfortable relative to our volume), with rotation available if one ever does.

**Derived tokens are produced inside the acquisition walk itself.** When an entry's cached derived token is missing or near expiry, acquire runs the scheme's mint synchronously, caches the result with its expiry, and hands it out; single-flight behind the pool's Lua lock, so concurrent runs cannot race a one-time-use refresh token. There is no background token-maintainer process. Only the GitHub mint path (App JWT → installation token) is POC-proven; the others reuse the same slot as design.

None of the schemes is new. Each reuses the exact credential the platform's connections use today; v2 changes who attaches it to the request, not what the user provides:

| Scheme | Mint step (in-acquire) | Pool scope | Platforms | Reused from |
|---|---|---|---|---|
| Static secret (basic auth, API key, PAT) | none, used as-is; auth failure quarantines and alerts, a human rotates | integration | Jira Cloud/DC, Confluence DC, Gerrit | Nango's per-auth-method configs (`jira-basic`, `jira-data-center-api-key`, `confluence-basic`, …) map 1:1; migrated connections keep their existing credentials |
| Static bot token | none | integration | Slack, Discord | legacy connectors, unchanged |
| OAuth2 + refresh token | refresh-token exchange → short-lived access token, cached in Redis with its expiry | integration | GitLab, Confluence Cloud (OAuth) | the legacy GitLab connector's refresh flow, centralized; existing backend OAuth connect endpoints keep issuing the initial grant |
| Session login | login call → session token, re-minted on expiry | integration | groups.io | legacy groups.io connector's login flow |
| App installation pool | JWT → installation token | platform | GitHub | the POC pool (previous section) |

### AI: skills and autonomous agents

AI is a deliberate part of the architecture, in two forms: **skills** (defined procedures that carry a workflow end to end, verified by the framework's gates) and **autonomous agents** (continuously running monitors). The POC connector was already written mostly by AI against staging.

| Where | Form | What it does | Human involvement |
|---|---|---|---|
| Nango → v2 migration | Skill | Carries the §4 flow end to end: freeze, shadow-run, validate against provider and legacy, flip, watch | Reviews validation results before the flip |
| New integrations | Skill | Generates the connector (manifest, syncs, schemas, mappers) and verifies it through the CI gates | Minimal review, data mapping only |
| Provider changes | Autonomous agent | Monitors provider docs and API version deprecations; flags what affects our connectors | Acts on the flags |
| Daily operations | Autonomous agent | Monitors the logs and alerts from daily runs | Handles escalations |

**The point is maintainability.** The long-term cost of an in-house framework is not building it, it is maintaining a growing set of connectors: keeping up with provider API changes, watching daily runs, porting the remaining platforms, adding new ones. The table assigns each of those to a skill or an agent, so the recurring human work per integration shrinks to reviewing data mappings and handling escalations, and does not grow with the number of connectors.

This is a requirement on the framework, not an add-on: the manifest, schemas, mappers and CI gates must be standard enough that a skill can generate and verify a connector without improvising, and agents work through the same interfaces humans use (structured logs with bound ids, `sync_units`, one-line SQL operations).

### What no longer exists at the end state

- Path A's three workers, `integration.runs`, `integration.streams` (and, post-migration, potentially `integration.results`, pending the sink rework review)
- the 1-minute cron ticks and the bin scripts
- all of Nango: cloud, webhook API, worker, `nango_mapping`, `nango_cursors`
- `incomingWebhooks`
- per-record operational rows of any kind
- the three separate token-rotation implementations

---

## 3. What the POC validated

The full GitHub connector (9 syncs) ran in staging against ~100 public repos on real APIs, from onboarding to `activityRelations`, polling only:

| Criterion | Validated in the POC | Out of POC scope, in the final version |
|---|---|---|
| Data is correct and complete | Complete against GitHub on the validated repo; zero field drift against Nango across the fleet | The validator (built and tested) plugged into the scheduler, running on every unit |
| Failures recover on their own | Deploys, restarts, parks and replays all resumed unattended | Control-plane seeding so the pool reseeds even when fully out; `provider.contract` failures dead-letter instead of retrying |
| Token handling matches Nango | Installations proved interchangeable at scale; rotations, parks and quarantines visible in logs | A per-repo eligibility rule for private repos and `repository_selection: selected` installations |
| Observability works under fire | Logs plus `sync_units` diagnosed every real incident without reading code | A log event for headroom refusals; per-request cost read from each response |
| Operations are one-liners | Re-onboard, watermark rewind, pause and status change: each a single SQL `UPDATE` | Nothing to add |
| The dev loop holds | The POC connector was written mostly by AI against staging | The harness, CI gates and skills that make AI-written connectors repeatable and verified |


---

## 4. Migration

**The one fixed point: every Nango connection migrated and the Nango account terminated by 2026-12-31.**

| Milestone | What | Target |
|---|---|---|
| **GitHub** | Finalize the core pipeline while migrating github-nango connections in parallel, ramped one org → 10% → 50% → 100% | **2026-09-30 → 2026-10-16** |
| **Jira, Confluence, Gerrit** | 32 connections, direct-flip behind the validation gate | **2026-12-31** |
| **Nango off** | Connections and config deleted, account terminated | **2026-12-31** |
| **Post-migration** | Webhook intake, secret manager, sink rework, legacy-github cleanup | to be scheduled and prioritized in 2027 |
| **Remaining platforms** | Path A platforms migrated, dormant ones decommissioned, queue tables dropped | to be scheduled and prioritized in 2027 |

**How one connection moves:** freeze the legacy connector for that platform → set a migration flag on its `integrations` row (`legacy` → `shadow` → `migrated`, read by both pipelines) → shadow-run the new pipeline, recording counts and IDs instead of emitting, while legacy keeps feeding production → flip only when the validator is clean against both the provider and what legacy ingested → watch ~7 days, flipping back on any surprise. Nango keeps fetching through the watch window, which is what makes rollback instant; once the connection is confirmed it is paused or deleted on the Nango side, and its rollback expires there.

**The migrations are AI-driven.** The GitHub port was already done that way. Once the core pipeline is stable, each remaining platform migrates through the migration skill described in §2 under "AI: skills and autonomous agents", which carries the steps above end to end.

---

## 5. Cost at scale

### Datadog

The product requirement is 24h data latency; syncs run twice daily. All figures below are a deliberate ceiling, every assumption priced at its harsh end, so the real bill lands under the estimate: 50% retry allowance (the scale test showed failure streaks averaging 10 runs during incidents), 12 lines per run (2.5x the measured 4.8, covering data-heavy windows and contention skip lines), and on-demand indexing rates with no annual commit assumed. Measured basis: a staging window on one `connectors-worker` pod (14,752 log lines) plus production fleet counts. 30-day retention.

If Datadog gets too expensive, the fallback is Grafana, Loki and S3; the framework only emits structured JSON logs, so switching backend needs no code change.

#### Logs: linear in units × cadence

| Step | Ceiling value | Where it comes from |
|---|---|---|
| Acquire lines per API request | 0.94 | measured: 14,092 `pool_token_acquired` lines against 14,999 requests |
| Lines per run | 12 | 4.8 measured on caught-up hourly polls; a 12h window pages more data, contended pods log skip-and-retry lines, incidents log error trails |
| Bytes per line | 657 | measured: 9.7 MB over 14,752 lines |
| Runs per repo per day | 24 | 8 syncs twice daily, +50% retries |
| Events per repo per month | ~8,800 | |
| Indexing / ingest | $3.75 per M (on-demand, 30-day retention) / $0.10 per GB | Datadog list, no commit discount assumed |
| **Per repo per month** | **$0.033** | indexing outweighs ingest ~55:1 at this line size |

The acquire line is 58% of volume; an index exclusion filter on it (ingest-only) roughly halves the bill. The 43,577 non-GitHub units add ~$0.2k/month at the same cadence. Fleet-wide load at 100k repos is ~85 req/s, 2 to 3 pods.

#### Metrics: flat in fleet size if the tag set holds

$0.05 per distinct timeseries (metric name × tag values). Tag only bounded categories, never identifiers, and the bill does not move as the fleet grows.

| Metric group | Tags | Combinations | Timeseries |
|---|---|---|---|
| `runs`, `run_failures` | sync × outcome, sync × error class | 24, 32 | 56 |
| `run_duration`, `run_requests`, `run_emitted` | sync | 8 each | 120 |
| pool acquires, skips, mints | borrowed, reason | 4 | 4 |
| 6 dispatcher gauges and timers | none | 6 | 10 |
| **GitHub subtotal** | | | **~190** |
| 17 further platforms at 3 syncs each | | | ~1,190 |
| **Fleet** | | | **~1,380 → $69/month** |

The host tag is billed by default (`@crowd/telemetry` submits via per-node DogStatsD), so the ceiling multiplies the fleet by 10 nodes: **~13,800 timeseries → $690/month**. Measured usage sits within the pooled per-host allotment (avg 3,284 of a ≥5,530 pool at 55 hosts); the ceiling does not bank on it.

#### Total (ceiling)

| Scale | Logs | Metrics | Monthly | Yearly |
|---|---|---|---|---|
| 15,434 repos (today) | ~$0.7k | ~$0.7k | ~$1.4k | ~$17k |
| 50,000 repos | ~$1.9k | ~$0.7k | ~$2.6k | ~$31k |
| **100,000 repos** (long-term goal) | **~$3.5k** | **~$0.7k** | **~$4.2k** | **~$50k** |

Logs include the flat ~$0.2k non-GitHub share. Initial backfill of the full 100k fleet adds a one-time ~$5k in logs. Not included: infra hosts ($15-23/host/month).

### Temporal

Temporal Cloud bills per action (workflow start, activity start, each retry attempt) plus history storage. A sync run costs 3 actions as coded (workflow start, one `executeSync` activity, the dispatcher's `startRun` activity); the ceiling prices 4 to absorb dispatcher overhead: its 5 fixed activities per cycle (~24k cycles/day at target), `deferUnit` loops when the pool is parked, and failed-start retries. Same run basis as Datadog: twice daily +50% retries, plus the ~4M monthly non-GitHub unit runs. Contract rates (order form 2025-12): $23.28/M actions, $0.00105/GBh retained storage, plus a Business plan fee of 10% of usage. History per run at 15KB (2x the assumed steady-state size), 7-day retention.

| Scale | Runs/month | Actions/month | Actions | Retained history | Plan fee | Monthly | Yearly |
|---|---|---|---|---|---|---|---|
| 15,434 repos (today) | 15.3M | ~61M | $1.4k | ~$0.04k | ~$0.15k | ~$1.6k | ~$19k |
| 50,000 repos | 40.5M | ~162M | $3.8k | ~$0.1k | ~$0.39k | ~$4.3k | ~$51k |
| **100,000 repos** (long-term goal) | **77M** | **~308M** | **$7.2k** | **~$0.2k** | **~$0.74k** | **~$8.1k** | **~$97k** |

Baseline from the August 2026 invoice: the entire platform today consumes 134M actions/month for $3.6k; the v2 ceiling at the 100k long-term goal adds ~2.3x that volume, against a $53k/yr credit commitment. Initial backfill of the full 100k fleet adds a one-time ~$5k in actions at 500 runs/repo. Cost scales linearly with cadence: hourly on the 5 hot syncs would be ~$35k/month Temporal alone.

### Bottom line

Min is the optimized floor: measured 4.8 lines/run, acquire-line exclusion filter on, committed Datadog rates, metrics inside the allotment, 3 actions/run, minimal retries. Max is the ceiling above.

| Scale | Datadog | Temporal | Monthly | Yearly |
|---|---|---|---|---|
| 15,434 repos (today) | $0.06-1.4k | $0.8-1.6k | **$0.9-3.0k** | **$10-36k** |
| **100,000 repos** (long-term goal) | $0.3-4.2k | $4.0-8.1k | **$4.3-12.5k** | **$52-150k** |

Initial backfill is a separate one-time cost, paid once per repo when it is onboarded, never again after: ~$10k total to backfill the full 100k fleet at ceiling assumptions (~$5k Temporal actions + ~$5k Datadog logs), scaling proportionally with the repos onboarded.