simocracy-indexer · dual-tap-epds-firehose

Simocracy Indexer — dual-tap fan-in from certified.one ePDS firehose

@daviddao

Routed the indexer's firehose ingest around bsky.network's 100-account-per-PDS cap by adding a second tap pointed directly at the ePDS, then proved the path end-to-end with a fresh ePDS account.

Date: 2026-05-06 · Target: simocracy-indexer-production.up.railway.app · pass

Headline

Why this matters

Bluesky's reference relay (bsky.network) applies a default 100-account-maximum quota to new PDS instances it crawls. The simocracy-indexer's tap consumer was previously connected only to that relay. So as soon as certified.one (the ePDS used by simocracy's email-login flow) crossed account #100, accounts #101+ would be invisible to our indexer — silently breaking the GraphQL/lookup paths the simocracy-v2 webapp relies on.

The com.atproto.sync.subscribeRepos lexicon is identical on every PDS and every relay — a PDS just emits its own accounts' commits instead of the whole network. So we can subscribe directly to wss://certified.one/xrpc/com.atproto.sync.subscribeRepos for a complete view of certified.one accounts, no matter how many of them exist, with zero coordination required (the firehose is public — see the lexicon: "Public and does not require auth; implemented by PDS and Relay.").

Topology change

What changed

Code (simocracy-indexer @ 3074513)

• index.js: TAP_WS_URL now accepts a comma-separated list. Each upstream gets its own WebSocket with independent reconnect state. The single ws module-level was replaced with a tapConnections[] array of slots.
• /stats: reports tapUpstreams[] and tapConnected[] alongside the legacy wsConnected (preserved as any-upstream-connected so existing uptime checks keep working).
• docker-compose.yml: added a tap-epds service for local-dev parity, mounted on its own ./data-epds volume so the two taps don't fight over the same SQLite cursor file.
• README.md: documented the dual-upstream production topology.

Railway (project simocracy-indexer, env production)

Smoke test

The smoke test exercises the new path end-to-end: a real new account on certified.one, created via simocracy.org's email-login flow, all the way through to a successful GraphQL lookup against the indexer.

1 — Disposable inbox

Acquire a temporary email from mail.tm (temp-mail.org refused with a Cloudflare challenge). The inbox came pre-allocated as 2920iseabal@deltajohnsons.com.

01mail.tm provisions a disposable address with no account creation.

2 — Email login on simocracy.org

Open simocracy.org, click Sign In, switch to the EMAIL tab, paste the disposable address. The webapp calls GET /api/oauth/login?email=…, which performs the ATProto OAuth PAR dance and 302-redirects to auth.certified.one with login_hint=<email>.

03Login dialog with the email tab selected.

04Email pasted in; clicking Continue 302s to certified.one.

3 — certified.one account creation

certified.one emails a 6-digit code (102231 in this run) to the inbox, which we read back from mail.tm.

05Verification challenge on auth.certified.one.

06The "Welcome to Certified" email with the code.

After verification, certified.one prompts for a handle (epds-smoke-test) and renders the OAuth consent screen for the https://www.simocracy.org/client-metadata.json client.

08Handle picker — entered epds-smoke-test.

10OAuth consent: "Grant access to your epds-smoke-test.certified.one account."

Resolving the new handle confirms the PDS placement is exactly what we need:

$ curl -s "https://public.api.bsky.app/xrpc/com.atproto.identity.resolveHandle?handle=epds-smoke-test.certified.one"
{"did":"did:plc:njw357tdisvgovexfks6xem2"}

$ curl -s https://plc.directory/did:plc:njw357tdisvgovexfks6xem2 | jq '.service[0].serviceEndpoint'
"https://certified.one"

4 — Mint a sim

Authorising returns to simocracy.org/my-sims as the new account. Clicking MINT YOUR FIRST SIM opens the avatar editor; we name the sim ePDS Smoke 045633, randomise the appearance, and hit SAVE TO ATPROTO to write a org.simocracy.sim commit to certified.one.

14Sim configured, ready to save.

15Toast: "Sim 'ePDS Smoke 045633' created! URL: simocracy.org/sims/epds-smoke-045633"

5 — Verify on the PDS

The record lands on certified.one. Listing the repo confirms the AT-URI:

$ curl -s "https://certified.one/xrpc/com.atproto.repo.listRecords?repo=did:plc:njw357tdisvgovexfks6xem2&collection=org.simocracy.sim" | jq '.records[0] | {uri,cid,name:.value.name}'
{
  "uri": "at://did:plc:njw357tdisvgovexfks6xem2/org.simocracy.sim/3ml7oyn62wk2x",
  "cid": "bafyreidxg7l63xhiodrabidfdrf6qu373dez3nntpywyxu7x66i6kkodwu",
  "name": "ePDS Smoke 045633"
}

6 — Verify via the indexer

The indexer's /stats endpoint reports both upstreams as live, and the GraphQL compat layer returns the new record. Both fields are new in this run — tapUpstreams[] and tapConnected[] are the proof that the multi-upstream code path is the one running in production.

$ curl -s https://simocracy-indexer-production.up.railway.app/stats
{
  "recordCount": 957,
  "knownDids": 51,
  "tapUpstreams": [
    "ws://tap.railway.internal:2480/channel",
    "ws://tap-epds.railway.internal:2480/channel"
  ],
  "tapConnected": [true, true],
  "wsConnected": true,
  "inFlightBackfills": 0,
  ...
}

$ curl -s https://simocracy-indexer-production.up.railway.app/graphql \
  -H 'Content-Type: application/json' \
  -d '{"query":"query Q($collection:String!,$first:Int){records(collection:$collection,first:$first){edges{node{uri did rkey value}}}}",
       "variables":{"collection":"org.simocracy.sim","first":1000}}' \
  | jq '.data.records.edges | map(select(.node.did == "did:plc:njw357tdisvgovexfks6xem2"))'
[
  {
    "node": {
      "uri":  "at://did:plc:njw357tdisvgovexfks6xem2/org.simocracy.sim/3ml7oyn62wk2x",
      "did":  "did:plc:njw357tdisvgovexfks6xem2",
      "rkey": "3ml7oyn62wk2x",
      "value": { "name": "ePDS Smoke 045633", "$type": "org.simocracy.sim", ... }
    }
  }
]

The CID and AT-URI match the PDS-side listRecords exactly.

7 — Profile page renders

The simocracy webapp resolves the slug epds-smoke-045633 via its alias map and the indexer feeds the profile route the sim record. The rendered page is end-to-end proof that the user-facing path is healthy:

16simocracy.org/sims/epds-smoke-045633 — profile served from indexer data sourced via tap-epds.

Which upstream actually delivered the record?

Two paths could have produced the record-in-DuckDB outcome: (a) tap-epds's firehose subscription, or (b) the simocracy webapp's post-write ping to GET /backfill-did?did=…. They log differently:

The production logs from this run show only the firehose log line, which is conclusive:

simocracy-indexer:
  Connected to tap[1] (ws://tap-epds.railway.internal:2480/channel)
  Connected to tap[0] (ws://tap.railway.internal:2480/channel)
  ...
  First record from new DID did:plc:njw357tdisvgovexfks6xem2 — auto-backfilling
  Auto-backfill did:plc:njw357tdisvgovexfks6xem2: inserted=1

tap-epds:
  [INFO] connecting to firehose ... url="wss://certified.one/xrpc/com.atproto.sync.subscribeRepos" cursor=0
  [INFO] connected to firehose
  ...
  [INFO] processing resync did="did:plc:njw357tdisvgovexfks6xem2"
  [INFO] fetching repo from PDS pds="https://certified.one"
  [INFO] resync repo complete

Findings

Reproducing