simocracy-indexer · tap-epds-volume-fix

Simocracy Indexer — tap-epds volume fix + cleanup discipline

@daviddao

Closed the volume gap from the previous run by attaching a persistent volume to tap-epds, proved cursor persistence across restarts, re-ran the end-to-end smoke test (this time deleting the test sim afterwards), and along the way fixed a real backfillDid bug that left deleted sims visible in the indexer.

Date: 2026-05-06 · Target: simocracy-indexer-production.up.railway.app · pass

---

Headline

What this run picked up

The previous run, dual-tap-epds-firehose, validated the dual-tap topology end-to-end but left two loose ends:

• Volume gap (Info finding): tap-epds was running without a Railway volume mounted at /data, so its SQLite cursor was on the ephemeral container filesystem. A redeploy reset it to zero and forced a full backfill window re-pull every time.
• Stranded sim: the test sim at://did:plc:njw357tdisvgovexfks6xem2/org.simocracy.sim/3ml7oyn62wk2x was created with an anonymous mail.tm inbox that vanished when the agent session ended. Without inbox access, that account can never be re-authenticated, and the sim record can't be cleaned up via the normal user flow.

Both got addressed in this run.

1 — Persistent volume for tap-epds

$ cd ~/Projects/simocracy-indexer
$ railway service tap-epds
$ railway volume add --mount-path /data
Volume "tap-epds-volume" created for service tap-epds in production at "/data"

$ railway volume list
Volume: tap-epds-volume         Attached to: tap-epds            Mount path: /data
Volume: tap-volume              Attached to: tap                 Mount path: /data
Volume: simocracy-indexer-volume Attached to: simocracy-indexer  Mount path: /app/data

Railway hot-attaches volumes — they're visible to the running container only after a restart, so an explicit railway redeploy --service tap-epds is required to actually mount the new persistent /data.

Proving cursor persistence (3 successive restarts)

This is the core of why the volume matters — the SQLite firehose_cursors table in /data/tap.db needs to outlive the container.

Restart	When	Pre-existing cursor in DB?	Connected with cursor
#1	~22:14, after first redeploy	no — fresh volume	cursor=0
#2	~22:16, ~90s after #1	no — too fast for RunCursorSaver to advance lastSeq ≥ 1	cursor=0
#3	~22:20, ~4m after #2	yes — cursor row present	cursor=7122

# Restart #3 startup log — proof of resume
[INFO] connecting to firehose
       url="wss://certified.one/xrpc/com.atproto.sync.subscribeRepos?cursor=7122"
       cursor=7122
[INFO] connected to firehose
Starting Container

This is exactly what we wanted: with TAP_CURSOR_SAVE_INTERVAL=1s (default) and a persistent /data volume, tap survives restarts without losing its place. Restart #2 came too quickly to be a fair test — saveCursor only writes when seq ≥ 1, and the bsky/atproto firehose's lowest sequence on the cursor save path needs a few moments to advance — so it produced a false-looking "no cursor" but doesn't reflect normal operation.

2 — End-to-end smoke test, this time with cleanup

Same path as the previous run — disposable email → certified.one OAuth → mint a sim → verify it reaches the indexer — but this time:

• Used a registered mail.tm account (with a saved password) instead of anonymous, so the inbox is recoverable across agent-browser sessions.
• Deleted the sim via the simocracy webapp UI at the end of the run.
• Verified the deletion propagated all the way through to the indexer's GraphQL.

Walkthrough

Indexer verification

$ curl -s 'https://simocracy-indexer-production.up.railway.app/graphql' \
  -H 'Content-Type: application/json' \
  -d '{"query":"query Q($collection:String!,$first:Int){records(collection:$collection,first:$first){edges{node{uri did rkey value}}}}",
       "variables":{"collection":"org.simocracy.sim","first":1000}}' \
  | jq --arg did 'did:plc:hgs7ptdnin6u6l3hzw4kh47y' \
      '.data.records.edges | map(select(.node.did == $did))'

[
  {
    "node": {
      "uri":  "at://did:plc:hgs7ptdnin6u6l3hzw4kh47y/org.simocracy.sim/3ml7pv2x44k2x",
      "did":  "did:plc:hgs7ptdnin6u6l3hzw4kh47y",
      "rkey": "3ml7pv2x44k2x",
      "value": { "name": "VolFix Smoke 051226", "$type": "org.simocracy.sim", ... }
    }
  }
]

And the same delivery proof as last run — the Auto-backfill log line (firehose-path trigger) rather than Backfill-did (webapp-ping path):

simocracy-indexer logs:
  Connected to tap[1] (ws://tap-epds.railway.internal:2480/channel)
  Connected to tap[0] (ws://tap.railway.internal:2480/channel)
  …
  First record from new DID did:plc:hgs7ptdnin6u6l3hzw4kh47y — auto-backfilling
  Auto-backfill did:plc:hgs7ptdnin6u6l3hzw4kh47y: inserted=1 deleted=0

tap-epds logs:
  [INFO] processing resync did="did:plc:hgs7ptdnin6u6l3hzw4kh47y"
  [INFO] fetching repo from PDS pds="https://certified.one"

3 — Cleanup, and the bug it surfaced

This is where things got interesting. Deleting the first test sim ("VolFix Smoke 051226") via simocracy.org's /my-sims page did delete it from the PDS, but it stayed in the indexer:

Root cause

Two things lined up to swallow the delete:

1. tap triggered a resync, not a delete commit. Looking at cmd/tap/firehose.go in indigo — when a commit's prevData doesn't match tap's tracked MST root for that repo (which can happen after multi-op commits, like simocracy's "delete sim then write a delete-history record" sequence), tap marks the repo Desynchronized and triggers a resync worker, but doesn't process the commit itself. So the action="delete" never reaches the downstream WebSocket. The indexer sees the resync logs but no delete event:

   tap-epds:
     [INFO] processing resync did="did:plc:hgs7ptdnin6u6l3hzw4kh47y"
     [INFO] fetching repo from PDS  size=837
     [INFO] iterating repo records  rev="3ml7qxcykpc2x"
     [INFO] pre-loaded existing records count=0   ← repo is now empty
     [INFO] resync repo complete                   ← but no delete forwarded

2. backfillDid was a pure upsert. The simocracy webapp pings GET /backfill-did?did=… after every record write, including deletes. backfillDid would refetch the user's repo, see 0 sim records, and… do nothing. There was no diff-against-DuckDB step. So neither the firehose path nor the backfill path cleaned up.

Fix — make backfillDid a real reconciliation

Shipped two commits to simocracy-indexer:

• 619cc95 — backfillDid is now a true reconciliation (upsert + delete-missing). After enumerating each (did, collection) from the PDS, diff against DuckDB and writer.deleteRecord any URIs that aren't in the freshly-fetched set. Guarded on a per-collection collectionFetchOk flag so a transient PDS error never produces false deletes. Also adds deleted=N to the Auto-backfill / Backfill-did log lines so the operational signal is visible.
• 3b01369 — fix duck.all binding. The first commit passed [did, collection] as a single argument to duck.all — which uses positional spread args, not a binding array — and silently failed in the try/catch. Verified locally with a minimal repro:

  positional args: [{"a":"foo","b":"bar"}]      ✓
  array arg:       Invalid Input Error: Values were not provided   ✗

  The fix: spread did, collection, plus log if the diff itself ever errors so a future regression is observable.

Verification — the second sim, deleted cleanly

Created a second sim ("Cleanup Verify Test"), confirmed it landed in the indexer, then deleted it. This time the firehose path delivered the delete cleanly (no resync edge case), so the indexer caught it via handleTapMessage:

simocracy-indexer logs:
  Connected to tap[0] (ws://tap.railway.internal:2480/channel)
  Connected to tap[1] (ws://tap-epds.railway.internal:2480/channel)
  Deleted: at://did:plc:hgs7ptdnin6u6l3hzw4kh47y/org.simocracy.sim/3ml7rwip35c2x (via tap[1])

PDS:    {"records": []}                                    ✓
Indexer: sim records for that DID: 0 (expected 0)          ✓

The first sim's stale row is also gone now — the new backfillDid reconciliation tidied it up on a subsequent webapp-triggered backfill ping.

4 — Recorded the discipline as a skill

To prevent the "stranded test account" problem from recurring, codified the smoke-test workflow + cleanup discipline as a Pi agent skill at ~/.pi/agent/skills/simocracy-smoke-test/SKILL.md. Key rules baked in:

• Use a mail.tm account registered with a password, never anonymous — anonymous boxes are unrecoverable across browser/agent sessions and re-registering the same address fails with 422 already used.
• Delete the sim at the end of the run, in the same agent session, while still authenticated.
• Verify the deletion propagated through both the PDS-side listRecords and the indexer-side GraphQL.
• Capture the firehose-vs-webapp-ping log distinction: only the Auto-backfill/Deleted: … via tap[N] log lines are proof that the firehose path actually delivered the event — Backfill-did alone could be the webapp's post-write ping doing the work.
• If a sim ends up stranded anyway (lost email access), document it as a cleanup gap and coordinate with the ePDS operator instead of leaving it silent.

Findings

Reproducing