# Sched Historical Import – Specification
| | |
| ----------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Status** | **Draft.** Development blocked on questions 1 and 2; running the import also needs question 2d |
| **Requester** | Nirav (original request: registrations); David (attendance tracking, for other stakeholders) |
| **Product owner** | Jonathan |
| **Prepared by** | Joana Maia, 2026-10-16 (rescoped to a one-time import on 2026-10-22) |
| **POC** | [github.com/joanagmaia/connectors-poc](https://github.com/joanagmaia/connectors-poc), folder `sched/`. Private repository; ask Joana Maia for access |
**Status labels:** ✅ **Validated in POC** (tested in the proof of concept) · 📄 **Provider docs only** (in Sched's public docs, not tested) · ⚠️ **Not proven** (neither docs nor POC confirm it) · ❌ **Not available** (docs or POC show it can't be obtained).
**Why an import and not a connector.** LF events stop using Sched in 2027, so CDP won't build an ongoing Sched connector. Past Sched data is still worth having: it tells CDP who registered for and spoke at which sessions, with their email and company. That helps **identity resolution** (recognising that a Sched person is a person CDP already knows from another source) and adds company information. So CDP imports past Sched events once, with code kept in crowd.dev so the import can be run again later, for chosen events, dates or data types.
## Open questions and blockers
Every open decision and question lives here, once, most important first. **Blocks** says what can't happen until it's answered. The rest of the document refers to these by number.
| # | Question | Context | Blocks | Who decides |
| --- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------- | -------------------------------- |
| 1 | **Organizations: how far do we go?** | Sched has only a free-text company name, and CDP creates no organization without a domain. Three options in §5 "Organizations" | Organization mapping | Joana, Jonathan |
| 2 | **Product sign-off**
(a) Read-Only or Read/Write Sched keys?
(b) Activity types and scores?
(c) Which time dates a registration?
(d) Which CDP project gets each event? | (a) Only Read/Write returns emails, and emails are what link Sched people to people CDP already knows (§7).
(b) Proposed: `registered-session`, score 3 (as Cvent's `registered-event`); `spoke-at-session`, score 8 (as Cvent's `attended-event`).
(c) Options and trade-offs in §5 "Registration time".
(d) Proposed: the project that owns the same event in LF's Cvent registration data, for example CNCF for KubeCon. How to match each Sched event to its Cvent event is not yet checked | Development (a–c); running the import (d) | Jonathan, Nirav |
**Left to the development team** (not blockers): Oracle Vault setup with CloudOps (whether CDP pods can already read Vault secrets with their own identity, who creates the vault and its access policy, and whether keys can sync from 1Password automatically or are loaded by hand; §7); pacing calls and streaming the large registration call (§8); tracking progress so a stopped run can resume; the order and batching of events; whether usernames count as verified; where in crowd.dev the script lives; checking field coverage on each new event during its dry run.
**Already decided:** no ongoing Sched connector. LF has 112 events on Sched; Joana Maia collects all their keys. The data in scope is the same as before the rescope: `registered-session` and `spoke-at-session`, the same identities and attributes. Check-in attendance (`attended-session`) is left out. The import is a crowd.dev script, well documented (§6). No raw copy of the downloaded data is kept. Sessionize is scoped separately. Sched data doesn't reach Snowflake today. CDP does nothing about removals: someone who un-registered after an earlier run keeps their activity. The first run covers one event (proposed: `kccnceu2021`, the POC event), to prove the flow end to end.
## 1. Overview
Sched is a conference schedule platform. Every event has its own Sched site (for example `kccnceu2021.sched.com`) and its own REST API. Attendees add sessions to their personal schedule and speakers are linked to sessions.
Objectives:
- Import past LF Sched events once, as `registered-session` and `spoke-at-session` activities.
- Link Sched people to people CDP already knows, and to their organizations where CDP can tell which one (question 1).
- Keep the import code in crowd.dev, so it can be run again with filters (proposed: events, dates, data types; §6).
- Import all 112 LF events on Sched, mapped to their CDP projects in one file (§6).
## 2. Summary
- **Can we do it?** Yes. The registrations, speakers and profiles in scope were read live from a May 2021 event in October 2026 (§3).
- **Flow (decided):** a well-documented crowd.dev script reads each event's API and sends the rows through CDP's normal processing, with filters by event, date and data type, and a _dry run_ that only counts (§6). No ongoing sync, and no keys stored in CDP.
- **Re-runs don't duplicate** as long as every activity keeps the same id, time, event and project between runs (§5).
- **Scale:** 112 events on Sched today, each with its own key. One mapping file lists every event with its CDP project (§6). Joana Maia collects the 112 keys into Oracle Vault, where the import reads them (§7).
- **The key type decides most of the value:** only a Read/Write key returns emails (question 2a).
- **Organizations:** Sched has no company domain; options in question 1.
## 3. Data we import
Two details matter for the mapping:
- **Two ids per session.** Sched identifies a session by a short _session key_ (for example `372`) and by a 32-character _session id_. Registrations use the key, speakers the id; the session list carries both.
- **Ids are per event.** Session keys are short numbers within one event's site, so they may repeat across events (⚠️ Not proven; one event tested). Every id in CDP therefore includes the event (§5).
### Activities
| Proposed CDP activity type | Meaning in CDP | Source in Sched | Status | Notes |
| ----------------------------------------------------- | --------------------------------------------------------------------- | ----------------------------------- | ------------------- | ------------------------------------------------------------------------------------------------------------------- |
| `registered-session` (proposed – Jonathan to confirm) | A person added a session to their schedule; one per (person, session) | Registration list (`user/sessions`) | ✅ Validated in POC | 116,553 registrations, 4,686 people, 351 sessions, no duplicates. Same with both key types |
| `spoke-at-session` (proposed – Jonathan to confirm) | A person speaks at a session; one per (speaker, session) | Speaker export (`role/export`) | ✅ Validated in POC | 351 speakers, 344 with a session list. Assumes the listed speaker actually spoke: Sched records no speaker no-shows |
### Where the data comes from
| Endpoint | What it gives | Size on the test event | Status |
| --------------- | ---------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------- | ------------------- |
| `session/list` | Sessions: name, times, type, venue | 351 sessions, 0.5 MB | ✅ Validated in POC |
| `role/export` | Speakers. One call returns usernames and session lists, another returns emails | 351 speakers | ✅ Validated in POC |
| `user/list` | Profiles: username, name, company, position, location, avatar; email with Read/Write only | 6,647 people | ✅ Validated in POC |
| `user/sessions` | Registrations | 116,553 rows, 48–56 MB, one call, no paging | ✅ Validated in POC |
| `site/sync` | Event settings, including the time zone (`Europe/Brussels`) | — | ✅ Validated in POC |
| `session/seats` | Per-session sign-up list with a sign-up time; Read/Write adds email. Only needed if question 2c picks the sign-up time | One call per session; 1,627 rows on the largest | ✅ Validated in POC |
### Historical data
| Scope | What Sched allows | Status |
| ------------------ | ------------------------------------------------------------------------------------------------------------- | ----------------------------------------- |
| Which events | Only events whose keys we get; each past event (for example each KubeCon) is a separate site with its own key | 📄 Provider docs only (one key per event) |
| Old events | A May 2021 event still returned its full data in October 2026 | ✅ Validated in POC |
| How much per event | Registration, profile and speaker calls return the event's full current lists, with no date filter (the session list's documented `since` filter was unreliable in the POC) | ✅ Validated in POC |
| After 2027 | Whether past sites and keys keep working once LF leaves Sched. No raw copy is kept, so a re-run needs them | ⚠️ Not proven |
### Identities and attributes
| Data | Coverage on the test event | Depends on | Status |
| -------------------------- | ------------------------------------------------------------------------------------------------- | ------------------------------------------ | ------------------- |
| Sched username | 100% of profiles, speakers and registrations. 321 registration rows (9 usernames) have no profile | Nothing | ✅ Validated in POC |
| Email | 99.5% of profiles, 90.3% of speakers, 100% of registration rows | Read/Write key; Read-Only returns no email | ✅ Validated in POC |
| Company | Free text, no domain or identifier. 49,327 of 116,553 registration rows (42%) | Nothing | ✅ Validated in POC |
| Position, location, avatar | Sparse; for example position on 41,535 of 116,553 registration rows | Nothing | ✅ Validated in POC |
## 4. Scale
**Per event**, measured on `kccnceu2021` (a large virtual KubeCon): 116,553 registrations, 4,686 registrants, 6,647 profiles, 351 sessions, 351 speakers. The registration call is 48–56 MB and takes 21–25 seconds. ✅ Validated in POC.
**Calls per event:** about 5–6, whatever the event's size, so about half a minute at the POC pace of 12 calls a minute. Choosing the sign-up time (question 2c) adds one call per session: 351 more on the test event, about 30 minutes. Whether Sched counts its rate limit per key, per event or per IP is ⚠️ Not proven, so the import paces every key as if they share one limit until that is known.
**How many events:** 112 on Sched today, about 650 read calls in all, roughly an hour at the POC pace. How much data depends on which events they are. LF's own registration data gives a sense of it (⚠️ Not proven as a Sched figure):
| Measure | Value |
| --------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------- |
| Past LF events registered through Cvent, 2019–2026 | 633 (54 to 100 a year) |
| Accepted Cvent registrations on them | about 470,000 |
| Test event: Cvent vs Sched | 26,713 Cvent registrations; 6,647 Sched profiles (25%); 116,553 Sched registrations (4.4 per Cvent registration) |
| If every Cvent event had used Sched at those ratios | about 2.1 million `registered-session` activities and 120,000 profiles |
That last figure covers all past Cvent events, not the 112 Sched ones, and comes from one virtual event. Once the list is known, the same ratio applied to those events' Cvent registrations gives a better estimate. Value is concentrated: the 50 largest Cvent events hold 68% of registrations, and KubeCon and Open Source Summit events alone hold 63%.
Cvent registrations are already in CDP, so many Sched people may already be known by the same email. That overlap is where the identity-resolution value comes from.
## 5. Mapping to CDP
Platform `sched`, new to CDP. Each event's subdomain, display name and project come from the event mapping file (§6).
| CDP field | `registered-session` | `spoke-at-session` |
| ------------------- | --------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------- |
| Timestamp | Question 2c (below) | Session start; Sched has no speaker-assignment date |
| Source id | Proposed `{subdomain}:{username}:{session key}`; the (username, session key) pair is unique on all rows ✅ Validated in POC | Proposed `{subdomain}:speaker:{username}:{session id}` |
| Parent id | `{subdomain}:{session key}` | `{subdomain}:{session id}` |
| Channel | Proposed: the event (subdomain) | Same |
| URL | `https://{subdomain}.sched.com/event/{session id}` | Same; returned HTTP 200 in the May 2026 POC run (not re-checked) ⚠️ Not proven |
| Member identities | Sched username; email with a Read/Write key. Proposed: mark emails _verified_ (trusted enough to merge people and link organizations), as CDP does for Cvent | Same |
| Member attributes | Display name, company, position, location, avatar | Same |
| Activity attributes | Session name, type, start and venue; event name | Same |
**Times.** No Sched timestamp carries a zone; each event's zone is in its settings ✅ Validated in POC. That the times are stored in that zone is ⚠️ Not proven, so the import reads them in the event's zone until shown otherwise.
### Registration time (question 2c)
| Option | What it is | Trade-off | Status |
| ---------------- | --------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ------------------- |
| Last-update time | `updated` on the registration row | Free. For this May 2021 event it runs to November 2023, so activity can land years after the event. What changes it is not known; if it changes between runs, the re-run duplicates the activity | ✅ Validated in POC |
| Sign-up time | `signup_date` on the per-session sign-up list | One call per session; Read/Write needed to match rows to people. 346 of 4,630 rows checked fall after their session started | ✅ Validated in POC |
| Session start | The session's start time | Free, and always on the event's dates. Not the moment the person registered | ✅ Validated in POC |
### Organizations (question 1)
What CDP does today, for every source:
- For each **verified email**, CDP links the person to the organization of the email's domain, creating it if it doesn't exist. Free-mail domains (Gmail and similar) are skipped.
- For a **company name with no domain**, CDP links to an existing organization with exactly that name, and creates nothing.
Options:
1. Company name stays a member attribute. Organizations come only from work emails, so only with a Read/Write key.
2. As option 1, plus send the company name, so CDP links people to existing organizations with exactly that name. No new organizations, but an exact-name match can be the wrong company. The Cvent import doesn't do this.
3. Create organizations from the company name. This changes CDP's rule for every source.
### Running it more than once
- **Same values, no duplicates.** CDP treats an activity as the same one when its time, type, source id, event and project all match, and updates it. A re-run duplicates an activity if any of those changed: a different registration time (question 2c), a session moved to another time, or an event mapped to another project. Whether past sessions ever change time is ⚠️ Not proven.
- **Matching people.** CDP first matches by Sched username, which nobody has on the first run, then by verified email from any source. With a Read/Write key, a Sched person whose email CDP already has joins that person. Without emails, every Sched person becomes a new, unlinked member.
- **Removals.** A re-run only adds or updates.
- **Cancelled and deleted sessions.** Skip sessions marked `[CANCELLED]` and speakers on sessions that no longer exist. Seen in the May 2026 POC run, not re-checked ⚠️ Not proven.
## 6. Import flow
**Decided: a crowd.dev import script, well documented.** It reads each event from the Sched API, turns the rows into activities and members, and sends them into CDP's normal processing queue. That is the same path the _Snowflake connectors_ use: the crowd.dev code that loads LF data such as Cvent registrations and meetings from LF's data warehouse. From there, deduplication, people matching and organization linking work as for any source.
Two alternatives were set aside: loading Sched into the data warehouse first (two systems to build, no filters), and running CDP's new connector framework once (no per-event key storage, no run-once mode).
**A run, as proposed:**
- **Event mapping file:** every event with its project (below). Keys come from Oracle Vault at run time, never from CDP or the repository (§7).
- **Filters:** events (one, a list, or all in the file); a date range on the event's dates; data type (registrations, speakers or both); dry run.
- **Safety:** read calls only; under 30 calls a minute; stop on the first rate-limit response; check the response body for a bad key, which returns HTTP 200 with an error message.
- **Order:** the first event end to end, then the rest of the mapping file, largest events first (below).
- **Documentation:** a usage guide next to the script: how to add events to the configuration file, store their keys, check them, dry-run, run, and re-run.
### Event mapping file (proposed)
With 112 events, one file in crowd.dev maps every event to its project; its key is found in Oracle Vault by the event's subdomain (§7). It holds no secrets and is reviewed like code, so the event-to-project mapping is visible and agreed before anything is imported.
| Field | Example | Why |
| ------------- | ---------------------------------------------------------- | ----------------------------------------------------------------------------------------------------- |
| Subdomain | `kccnceu2021` | Builds the API calls, the ids and the session links |
| Display name | KubeCon + CloudNativeCon Europe 2021 Virtual | Shown on activities |
| CDP project | CNCF | Where the event's activities go (question 2d). Proposed starting point: the project of the same event in LF's Cvent data |
Steps:
1. **Fill the file** with the 112 events and their projects; review it as a pull request.
2. **Collect the keys** in one pass: Joana Maia copies each event's key from its Control Panel into Oracle Vault, one secret per event, named `sched-`.
3. **Check keys:** one read call per event says whether the key works and whether it returns emails (Read/Write). About 112 calls, around ten minutes.
4. **Dry run** every event: counts only, nothing sent.
5. **Import** `kccnceu2021` first, then the rest, largest first, one event at a time. Each event is marked done when finished, so a stopped run resumes and a repeated run skips or updates.
6. **Report:** events imported, events skipped (no key, bad key, Read-Only, no API access) and counts.
**What CDP needs added:** the `sched` platform, its two activity types and their scores, and the script. The script creates one _integration_ (CDP's record that a project receives data from a source) per project on first use, as the Snowflake connectors do.
Where to start in crowd.dev (public repository):
- [How the Snowflake connectors send results for processing](https://github.com/linuxfoundation/crowd.dev/blob/ae691f916c7db33b791874346129c75f2ffd9bec/services/apps/snowflake_connectors/src/consumer/transformerConsumer.ts#L41-L125)
- [How they find a project and create its integration](https://github.com/linuxfoundation/crowd.dev/blob/ae691f916c7db33b791874346129c75f2ffd9bec/services/apps/snowflake_connectors/src/core/integrationResolver.ts#L67-L158)
- [A backfill script with filters and a dry run (Octolens)](https://github.com/linuxfoundation/crowd.dev/blob/ae691f916c7db33b791874346129c75f2ffd9bec/services/apps/connectors_worker/src/bin/backfill-octolens-mentions.ts#L65-L120)
## 7. Access and keys
- 📄 "All paid plans include API access"; whether the cheapest plan has it is not documented, which may rule out some past events.
- 📄 One API key per event, in the event's Sched Control Panel under Settings > API. Sched's docs don't say which admin role can see it. The key is sent with every call, and Sched filters out requests without a User-Agent header.
- Sched's docs don't mention key types, but on the test event a Read-Only and a Read/Write key returned different data:
| What | Read-Only key | Read/Write key | Status |
| --------------------------------- | ------------------------- | ---------------------------------------------------------------------------- | ------------------- |
| Email | None | 99.5% of profiles, 90.3% of speakers, 100% of registration rows | ✅ Validated in POC |
| Registrations via `user/sessions` | Complete; 48.5 MB, 21.7 s | Complete; 56.4 MB, 24.5 s | ✅ Validated in POC |
| Speakers, usernames, company | Same | Same | ✅ Validated in POC |
| Risk | — | The name suggests it can also change event data. The POC made no write calls | ⚠️ Not proven |
The import only calls read endpoints. Which key type to ask for is question 2a; Joana Maia collects the keys.
### Key handling (proposed)
The import runs in a Kubernetes pod on Oracle Cloud, where CDP runs. Keys live in **Oracle Vault**:
- **One secret per event**, named `sched-` (for example `sched-kccnceu2021`), all in one vault. Keys can be added one at a time as they are collected, and the mapping file needs no key column. Only the vault's id is passed to the script, as a setting.
- **Read at run time by the pod's own identity.** An access policy lets only the import job read secrets in that vault. The script fetches each key when it reaches that event, so keys never sit in a file, an environment variable or a log.
- **Never log Sched responses as they are.** The settings call (`site/sync`) echoes the API key in its response ✅ Validated in POC. Log counts and statuses only.
- **Deleted after the import.** These keys may be able to change event data, so they stay only as long as the import needs them. A later re-run first puts the keys back in the vault.
- **Checked before importing.** The key check reports any event in the mapping file with no secret in the vault.
Running it looks like:
```
SCHED_OCI_VAULT_ID= pnpm run script:import-sched -- --all
```
Nothing in crowd.dev uses Oracle Vault yet, so this is the first use; the open setup items are under "Left to the development team".
## 8. Limits and edge cases
| Limit or case | Value | Impact on the import | Status |
| ------------------------------ | ------------------------------------------------------------------------------------------------ | --------------------------------------------- | -------------------------------------------------------- |
| Rate limit | Under 30 calls a minute, "cumulative and sustained"; going over gets the key revoked for a while | Pace well under it (the POC used 12 a minute) | 📄 Provider docs only (limit header ✅ Validated in POC) |
| Large call | `user/sessions` is 48–56 MB in one call | Stream it | ✅ Validated in POC |
| Bad keys return HTTP 200 | The body reads "ERR: Authorization failed." | Check the body, not only the status code | ✅ Validated in POC |
| Registration without a profile | 321 rows, 9 usernames | Import with empty profile attributes | ✅ Validated in POC |
| Sparse profile fields | Many are empty | Treat all as optional | ✅ Validated in POC |
| Other events | Only one event was tested | Check counts in each event's dry run | ⚠️ Not proven |
## 9. Acceptance criteria
Draft; not yet agreed with Jonathan.
- [ ] A dry run on `kccnceu2021` reports 116,553 registrations and one activity per (speaker, session), and sends nothing to CDP.
- [ ] A real run on `kccnceu2021` creates one `registered-session` per (person, session) and one `spoke-at-session` per (speaker, session), in the agreed project.
- [ ] Running it again creates no new activities (holds unless question 2c picks the last-update time, §5).
- [ ] With a Read/Write key, a Sched person whose email CDP already has joins that person instead of becoming a new member.
- [ ] Organizations follow the option chosen in question 1.
- [ ] A run can be limited to chosen events, a date range and a data type.
- [ ] The mapping file lists every event's subdomain and project, with no keys.
- [ ] A key check reports, per event, whether the key works and whether it returns emails, without importing anything.
- [ ] A stopped run resumes where it left off; events without a key are skipped and reported.
- [ ] Keys are read from Oracle Vault at run time and never stored in CDP, the repository or logs; only read endpoints are called, under 30 calls a minute.
- [ ] The import code is merged in crowd.dev with a usage guide: adding events, storing keys, checking keys, dry runs, runs and re-runs.
## 10. Proof of concept
**Where:** [github.com/joanagmaia/connectors-poc](https://github.com/joanagmaia/connectors-poc), folder `sched/`. You can test it there:
1. Copy `sched/.env.example` to `sched/.env` and fill in 1Password references for `API_KEY` (Read-Only) and `API_KEY_RW` (Read/Write). Ask Joana for the keys.
2. From the repo root, run a probe, for example:
```
op run --env-file=sched/.env -- npx tsx sched/probes/11-attendee-registrations-live.ts --event=kccnceu2021
```
Probes call read endpoints only, pace themselves at 12 calls a minute, stop on the first 429, and write redacted results to `sched/evidence/`.
**Based on:** event `kccnceu2021` ([kccnceu2021.sched.com](https://kccnceu2021.sched.com/)); live runs on 2026-10-16 with a Read-Only and a Read/Write key; an earlier POC run from May 2026 on the same event; Sched's public docs, read on 2026-10-16. Event counts in §4 come from LF's event registration data, read on 2026-10-22.
**Tested:** registration sources with both key types; profile and speaker fields per key type; both registration-time candidates; the bad-key response and rate-limit headers. A probe uses the Read/Write key when `API_KEY_RW` is set, otherwise Read-Only.
**Not tested:** a second event (needs its key); a real rate-limit response (would need stress calls, not approved); write behaviour (never allowed); an import into CDP.
## 11. References
- Sched: [API](https://sched.com/api) · [pricing](https://sched.com/pricing)
- [crowd.dev](https://github.com/linuxfoundation/crowd.dev) · [POC repository](https://github.com/joanagmaia/connectors-poc) (folder `sched/`, private)