simocracy-v2 · training-lab-r1

Sim Training Lab — first end-to-end audit

@daviddao

First post-merge audit of the new Training Lab on production, walked end-to-end as satyam2.climateai.org: 8 baseline votes, 3 adaptive interview turns, profile distillation, alignment benchmark, and the apply-to-constitution write. The four tabs render correctly and all three AI routes return well-shaped output, but two real bugs surface — a closure-stale-state race in BaselineTab that quietly overwrites votes to abstain, and a silent failure in Apply to constitution when the sim has no org.simocracy.agents record yet.

Date: 2026-05-02 · Target: www.simocracy.org · Account: satyam2.climateai.org · did:plc:cpoagodpqrgs4t7thi5z37uf · Sim: PM Slug Test 2026 · at://…/org.simocracy.sim/3mkt33lmhu62a · partial


Headline result

7
Behaviours that work as specified
2
Real product bugs
1
Known UX gap (apply with no constitution)
3
AI routes verified end-to-end

Headline. The Training Lab ships a usable MVP — owner-only entry from the sim profile dropdown, four pre-styled tabs, and three AI routes that return well-shaped JSON the UI renders cleanly. Two bugs need fixing before this is the recommended path: (1) BaselineTab's updateVote uses closure-captured baselineVotes so consecutive synchronous edits (the typical autofill / keyboard-paste path) silently clobber the most recent vote back to abstain — visible in localStorage and again in the alignment scoreboard as USER: ABSTAIN for every proposal. (2) Apply to constitution on a sim that has never had its constitution saved produces no toast, no error, no close — the click is swallowed. The same closure-stale-state pattern also exists in InterviewTab: a fast user can clobber the AI's question turn by sending an answer while the fetch is in flight.

Walk-through

Recorded the OAuth handshake and the modal opening as one webm; took stills for every state inside the modal because the agent-browser CLI's screen-record mode and the per-action capture mode use separate sessions and can't be combined cleanly.

2:43 — handle login → climateai.org PDS password → OAuth consent → land on /, navigate to the sim profile, dropdown → Training Lab, vote on the first proposal.

01Sign-in popover anchored to the navbar. Defaults to Handle tab. Filled in satyam2.climateai.org.
02OAuth consent on climateai.org after the handle resolved to a PDS that isn't bsky.social. Three scopes requested; client_id matches https://www.simocracy.org/client-metadata.json.
03Landed back on /, signed in. Avatar in the top-right; the walking-sims hero now includes Judycat222, ming, Von Neumann, etc.
04PM Slug Test 2026 profile, owner-view. YOURS chip, CHAT + INTERVIEW primary actions, three-dot more menu top-right.
05Dropdown opened. TRAINING LAB sits between EDIT SPEAKING STYLE and the destructive DELETE SIM — exactly where the brief specified, and only when viewerDid === did.
06Modal opens on the Baseline tab. Header subtitle reads Train, audit, and benchmark PM Slug Test 2026 before deploying. Eight pre-loaded sample proposals; 0 / 8 voted badge top-right of the section.
07After clicking YES on the climate proposal: button paints in foil (selected), counter ticks to 1 / 8 voted, localStorage immediately holds the row.
08All eight voted via the agent-browser eval path: each vote button selected, importance sliders shifted to per-issue values (climate 0.90, exec pay 0.55, etc.), reasoning textboxes filled. Counter reads 8 / 8 voted. The visible state lies — localStorage stored every vote as abstain, see Finding F1.
09Interview tab on first entry. You've answered 0 questions. Distill profile in the top-right (always visible). Empty conversation panel with the placeholder Ask the first question when you are ready.
10First adaptive question, ~6 s after clicking Ask next question. Speaker label is the sim's name (PM SLUG TEST 2026); target badge correctly chose tradeoffs given the YES votes on climate + supply-chain rights.
11User reply appended right-aligned with a YOU label. Counter ticks to 1 question. The conversation log persists into localStorage after the 300 ms debounce.
12Bottom of the conversation after 3 user turns — but only 1 assistant turn is in localStorage. Finding F2: askNextQuestion's in-flight result was clobbered by sendAnswer's closure-stale write.
13Profile tab pre-distill: Preference architecture header, Distill profile primary, No distilled profile yet. placeholder.
14Post-distill, top of card: Summary in plain prose, then a chip-row of Core Values (long-term viability, moral accountability, transparency, risk management, ethical supply chains). Faithful to the actual interview content.
15Issue Priorities, mid-card. Per-issue importance / negotiability / confidence rendered as three thin horizontal bars. Human rights reads 100 / 0 / 100; Lobbying transparency reads 90 / 30 / 100. Matches the brief's labels exactly.
16Bottom of profile. Four side-by-side cards: Red Lines, Acceptable Tradeoffs, Uncertainty Areas, Representation Rules. The Apply button in the footer flipped from disabled to enabled now that profile is non-null.
17Alignment tab pre-run. Compare PM Slug Test 2026's votes against your hidden baseline. Modal collapses tightly because the body is short — Radix re-centers, which jumps the dialog up the page (cosmetic only, see Finding F4).
18Alignment ran in ~22 s (8 sequential model calls, capped at 4-way concurrency). Result panel renders the headline (Alignment: 0 / 8 — 0% — Weak on climate, executive pay, board diversity, lobbying disclosure) and a per-row card with confidence bar + sim's reason. The 0/8 score is correct given the bad data — every userVote was abstain by Finding F1.
19Bottom two rows: human-rights and special-meetings. Sim's reasoning is well-anchored to the distilled profile. The USER: ABSTAIN on every row is the F1 fingerprint.

What works

OAuth login routes correctly to the user's PDS

Handle satyam2.climateai.org resolved to a PDS at https://climateai.org (not bsky.social), the OAuth consent rendered on the PDS, and the redirect carried the user back to https://www.simocracy.org/api/auth/callback with a working DPoP-bound session. The session-token cookie was set and getSession() on the next page load resolved the DID via Redis. This is the same path that broke on branch previews; on production with NEXT_PUBLIC_BASE_URL=https://simocracy.org it works as designed.

Owner-only dropdown gating

The TRAINING LAB dropdown item only renders when viewerDid === did. Verified by signing out and reloading the same sim profile — the three-dot menu disappears entirely (it gates on isOwner). No way for a non-owner to open the modal. Matches the brief's owner-only requirement.

/api/training/next-question returns well-shaped JSON

Three sequential questions all parsed cleanly with question, target, and reason fields. Targets cycled through tradeoffs, red_lines, and priority — the AI is genuinely identifying gaps in what it knows from the baseline votes. JSON-only prompt + defensive parsing in the route handles the OpenRouter response without response_format: json_object.

/api/training/extract-profile distills faithfully

The summary line ("A stakeholder-focused representative prioritizing long-term existential risks…") matches the conversation's actual stance. Core Values are not invented — every chip ties back to a yes-vote, the climate-vs-cost tradeoff answer, or the red-lines answer. Issue Priorities use the full 0–1 range (e.g., human rights gets importance: 1.0, negotiability: 0.0; lobbying gets 0.9 / 0.3) and the rendered triple-bars are readable at a glance.

/api/training/alignment-test scores per-proposal

8 sequential model calls (concurrency cap of 4 honoured), each returns { vote, confidence, explanation } within ~280 chars. Server scores match server-side; the user's vote is not in the model prompt (verified by reading the request payload in app/api/training/alignment-test/route.ts). weakAreas picks up to 4 topic labels from the unmatched rows. The 0/8 we hit is a data bug upstream (F1), not a logic bug here.

localStorage persistence works under simocracy.trainingLab.v1.<simUri>

Single-key-per-sim, debounced 300 ms write on every state change. Re-opening the modal on the same sim (without closing the browser) restores the four-section state cleanly: votes, interview turns, profile, alignment. updatedAt stamp re-writes on every save. Schema-validated on load via the isTrainingLabState guard.

Modal layout, typography, and tab focus order

Width is sm:max-w-3xl as the brief requested (480 px wider than the chat modal). FlaskConical icon, Reset training data ghost button + Apply to constitution primary in the persistent footer. Fonts (EB Garamond serif body, IBM Plex Mono labels) match the rest of the app. Tab order is sensible by keyboard (Home/End jump to first/last).

Findings

F1 · BaselineTab.updateVote uses closure-captured state — rapid edits silently overwrite the vote

Severity: high · Surface: programmatic input + browser autofill + paste-then-tab.

In components/sim/training-lab/baseline-tab.tsx, updateVote(proposalId, patch) reads baselineVotes from props closure on every call, then synthesizes nextVote from that snapshot before calling onBaselineVotesChange. If three updates fire in the same JavaScript synchronous tick (vote click → slider input → textarea input — what the agent-browser eval does, and what an autofill extension or a "Tab to next field after a paste" flow would do), all three calls see the same empty baselineVotes. The last onBaselineVotesChange wins, and because existing?.vote ?? "abstain" resolves to abstain when existing is undefined, the final stored vote is abstain regardless of what the user clicked.

Reproduction (this run). Eight proposals, each with YES or NO selected, an importance value set, and a reasoning sentence typed via eval. The UI showed 8 / 8 voted with the correct buttons highlighted; localStorage contained vote: "abstain" on all 8. Manual single-click later (no slider, no textarea, no rapid follow-up) wrote vote: "yes" correctly — confirming the bug only triggers on consecutive synchronous calls within one React batch.

Why it matters. The downstream alignment test then scored 0 / 8 (snapshot 18) and the per-row card honestly says USER: ABSTAIN on every proposal. A user who scripts the form, uses 1Password autofill, or just clicks YES and rapidly drags the slider before React has time to re-render between events could end up with their training quietly storing abstain for everything — and getting a 0% alignment score they can't explain.

Fix. Either move the baselineVotes array into the BaselineTab's own state (so React's setter handles batching with the latest state) or, less invasively, change the parent contract on onBaselineVotesChange to accept a functional updater (prev) => BaselineVote[] the way useState does, and apply each updateVote call against the most recent state. Same pattern needed in InterviewTab — see F2.

// localStorage at end of the rapid-input run
[
  { proposalId: "climate-disclosure",         vote: "abstain", importance: 0.90, reasoning: "Climate disclosure is foundational risk management." },
  { proposalId: "executive-pay-ratio",        vote: "abstain", importance: 0.55, reasoning: "Multi-year stock vesting still rewards short-term volatility." },
  { proposalId: "board-diversity-policy",     vote: "abstain", importance: 0.70, reasoning: "Diverse perspectives improve oversight; quotas not needed." },
  { proposalId: "political-lobbying-disclosure", vote: "abstain", importance: 0.80, reasoning: "Political spending should be traceable to stated values." },
  { proposalId: "worker-safety-audit",        vote: "abstain", importance: 0.85, reasoning: "Independent audits build worker trust." },
  { proposalId: "data-privacy-oversight",     vote: "abstain", importance: 0.75, reasoning: "Privacy oversight is now a board-level fiduciary duty." },
  { proposalId: "supply-chain-human-rights",  vote: "abstain", importance: 0.90, reasoning: "Human-rights review is a moral baseline for global supply chains." },
  { proposalId: "shareholder-special-meetings", vote: "abstain", importance: 0.40, reasoning: "Lower thresholds invite hedge-fund activism over long-term value." }
]
// later, after a single manual click on the worker-safety YES button:
{ proposalId: "worker-safety-audit", vote: "yes", importance: 0.85, reasoning: "Independent audits build worker trust." }

F2 · InterviewTab race — sending an answer while the AI fetch is in flight clobbers the question turn

Severity: medium · Surface: any user who types and hits send before the next question arrives (≈8 s).

Same closure-stale-state pattern in askNextQuestion() and sendAnswer() (components/sim/training-lab/interview-tab.tsx L46–L85). Both functions append to interviewTurns by spreading the props-captured array. If the user clicks Ask next question, then types a quick answer and hits send before the model's response arrives, two parallel writes both spread the same interviewTurns snapshot — the answer's onInterviewTurnsChange([…old, userTurn]) overwrites the question's onInterviewTurnsChange([…old, assistantTurn]), depending on order. In this run we ended up with three user turns and only one assistant question in localStorage (snapshot 12). The downstream profile distillation still ran fine because the user's answers carry the substance, but the conversation log on screen shows answers without questions, which is confusing.

Defence in depth. Even after F1's fix, the textarea and the send button should be disabled while isAsking === true — the brief explicitly says "Ask next question" button is enabled again afterwards, but it's silent on the answer side. Disabling input during the fetch is the cheap belt to F2's suspenders fix in F1.

F3 · "Apply to constitution" with no agents record is a no-op — no toast, no error, no close

Severity: high · Surface: any user whose sim still says "No constitution drafted yet."

From the agent's PR handoff: "Apply currently requires an existing org.simocracy.agents record/rkey. If a sim has no constitution record yet, it shows an error instead of creating one." The actual production behaviour is worse: it shows nothing at all. Clicking the enabled Apply to constitution button on a sim with no org.simocracy.agents record produces no toast, no spinner, no modal close, no PDS write, no console error. Verified three different click paths (agent-browser pointer click, dispatched MouseEvent sequence, direct invocation of the React onClick prop via fiber) — none surface the toast.error("Constitution record not found. Save a constitution before applying training.") branch in training-lab-modal.tsx L106.

Inspection of the live DOM during the click: section[aria-label="Notifications alt+T"] (Sonner's container) is mounted but stays at childElementCount: 0. The button __reactProps$… has a defined onClick; something in the early-return path is silent — most likely the toast call itself never fires because trainingState.profile evaluates falsy by the time applyToConstitution runs (the profile tab held it before, but switching to Alignment after Run alignment test may have triggered a re-mount that dropped the profile reference). Needs a single targeted repro with a console.log before the early returns to be sure.

Fix. Two edits, in this order. (1) Make Apply create the org.simocracy.agents record when agentsRkey is null, the way agents-editor's save path already does — that's the user-facing fix the agent's handoff already flagged. (2) After (1), any early return must surface a toast (or at minimum a non-debug console.error); silent no-ops are the worst outcome for a write button. While we're there, gate Apply behind !!agentsRkey in the disabled prop too, so users who can't apply yet get the visual cue before clicking.

F4 · Modal vertical jump when tab content shrinks

Severity: low · Surface: cosmetic — switching from a long tab (Baseline / Profile-after-distill) to a short tab (Alignment-pre-run) makes the modal "hop" up the page.

shadcn/ui Dialog centers vertically; the inner scroll container is capped at max-h-[62vh] but does not reserve that height. So when the body is short, the dialog sits at the geometric centre of the viewport (visible in snapshots 17 and 18). When you switch back to Profile, the modal jumps down again. Easy fix: add a min-h-[62vh] to the same scroll container, or equivalently set min-h-[420px] on the inner div, so all four tabs feel like the same panel. Doesn't affect functionality.

F5 · Tooling note: agent-browser click @ref on Radix tab triggers onPointerDownOutside intermittently

Not a product bug — flagging only so the next agent running this flow knows: clicking buttons inside Dialog via the agent-browser MCP click @ref path occasionally fired Radix's onPointerDownOutside handler and closed the dialog (originally misread as a real bug — see commits/screenshot trail). Switching to eval-based document.querySelector(…).click() avoided the issue entirely. The right tool for tab switching is tab.focus(); tab.dispatchEvent(new KeyboardEvent('keydown', {key:'Home'|'End'})) — Radix's keyboard nav is reliable; its synthesized pointer-event handling is not.

What we'd look at next round

Reproducing

DID=did:plc:cpoagodpqrgs4t7thi5z37uf
RKEY=3mkt33lmhu62a
SIM_URI="at://$DID/org.simocracy.sim/$RKEY"

# 1. confirm the user has zero org.simocracy.agents records on the PDS
#    (this is the precondition for F3 to surface)
curl -s "https://climateai.org/xrpc/com.atproto.repo.listRecords" \
  --data-urlencode "repo=$DID" \
  --data-urlencode "collection=org.simocracy.agents" \
  --data-urlencode "limit=10" \
  -G | jq '.records | length'
#  → 0

# 2. confirm the new training routes are wired and auth-gated
curl -s -X POST "https://www.simocracy.org/api/training/next-question" \
  -H 'Content-Type: application/json' -d '{}' | jq .
#  → {"error":"Please sign in"}  (HTTP 401)

# 3. inspect localStorage after the rapid-input run (Chrome devtools)
JSON.parse(localStorage.getItem(`simocracy.trainingLab.v1.${SIM_URI}`)).baselineVotes
#  → 8 rows, every vote === "abstain"  (F1 fingerprint)
Targetwww.simocracy.org · production · commit bef5c72 (fix(config): per-branch baseUrl on Vercel previews) Accountsatyam2.climateai.org · did:plc:cpoagodpqrgs4t7thi5z37uf · PDS climateai.org Sim under testPM Slug Test 2026 · at://…/org.simocracy.sim/3mkt33lmhu62a · 0 agents records, 0 style records — fresh of any constitution work, which is what surfaces F3 PR shippedfeat/sim-training-lab → merged into main as e2583ce feat(training-lab): owner-only Sim Training Lab MVP (12 files, +2,502 / −6) AI routes hit3 × POST /api/training/next-question, 1 × POST /api/training/extract-profile, 1 × POST /api/training/alignment-test (8 internal model calls). All on Gemini Flash Lite via OpenRouter. Toolingagent-browser v0.x (headless Chromium, persistent profile) for OAuth + screenshots + the 2:43 webm; eval for Radix Dialog interaction once the MCP click path tripped onPointerDownOutside; curl against climateai.org for the PDS-state verification. Screenshots19 PNGs in assets/01–20, all 1280×720 or shorter, under the 1500 px height cap. walkthrough-oauth-and-modal.webm is 1.6 MB / 2:43. Next roundtraining-lab-r2 — re-run after F1 + F3 land. Bumps the series from r1 partial to r2 pass if both fixes hold.