simocracy-v2 · training-lab-r1
First post-merge audit of the new Training Lab on production, walked end-to-end as satyam2.climateai.org: 8 baseline votes, 3 adaptive interview turns, profile distillation, alignment benchmark, and the apply-to-constitution write. The four tabs render correctly and all three AI routes return well-shaped output, but two real bugs surface — a closure-stale-state race in BaselineTab that quietly overwrites votes to abstain, and a silent failure in Apply to constitution when the sim has no org.simocracy.agents record yet.
Headline. The Training Lab ships a usable MVP — owner-only entry from the sim profile dropdown, four pre-styled tabs, and three AI routes that return well-shaped JSON the UI renders cleanly. Two bugs need fixing before this is the recommended path: (1) BaselineTab's updateVote uses closure-captured baselineVotes so consecutive synchronous edits (the typical autofill / keyboard-paste path) silently clobber the most recent vote back to abstain — visible in localStorage and again in the alignment scoreboard as USER: ABSTAIN for every proposal. (2) Apply to constitution on a sim that has never had its constitution saved produces no toast, no error, no close — the click is swallowed. The same closure-stale-state pattern also exists in InterviewTab: a fast user can clobber the AI's question turn by sending an answer while the fetch is in flight.
Recorded the OAuth handshake and the modal opening as one webm; took stills for every state inside the modal because the agent-browser CLI's screen-record mode and the per-action capture mode use separate sessions and can't be combined cleanly.
2:43 — handle login → climateai.org PDS password → OAuth consent → land on /, navigate to the sim profile, dropdown → Training Lab, vote on the first proposal.
satyam2.climateai.org.
climateai.org after the handle resolved to a PDS that isn't bsky.social. Three scopes requested; client_id matches https://www.simocracy.org/client-metadata.json.
viewerDid === did.
eval path: each vote button selected, importance sliders shifted to per-issue values (climate 0.90, exec pay 0.55, etc.), reasoning textboxes filled. Counter reads 8 / 8 voted. The visible state lies — localStorage stored every vote as abstain, see Finding F1.
localStorage after the 300 ms debounce.
localStorage. Finding F2: askNextQuestion's in-flight result was clobbered by sendAnswer's closure-stale write.
profile is non-null.
userVote was abstain by Finding F1.
Handle satyam2.climateai.org resolved to a PDS at https://climateai.org (not bsky.social), the OAuth consent rendered on the PDS, and the redirect carried the user back to https://www.simocracy.org/api/auth/callback with a working DPoP-bound session. The session-token cookie was set and getSession() on the next page load resolved the DID via Redis. This is the same path that broke on branch previews; on production with NEXT_PUBLIC_BASE_URL=https://simocracy.org it works as designed.
The TRAINING LAB dropdown item only renders when viewerDid === did. Verified by signing out and reloading the same sim profile — the three-dot menu disappears entirely (it gates on isOwner). No way for a non-owner to open the modal. Matches the brief's owner-only requirement.
/api/training/next-question returns well-shaped JSONThree sequential questions all parsed cleanly with question, target, and reason fields. Targets cycled through tradeoffs, red_lines, and priority — the AI is genuinely identifying gaps in what it knows from the baseline votes. JSON-only prompt + defensive parsing in the route handles the OpenRouter response without response_format: json_object.
/api/training/extract-profile distills faithfullyThe summary line ("A stakeholder-focused representative prioritizing long-term existential risks…") matches the conversation's actual stance. Core Values are not invented — every chip ties back to a yes-vote, the climate-vs-cost tradeoff answer, or the red-lines answer. Issue Priorities use the full 0–1 range (e.g., human rights gets importance: 1.0, negotiability: 0.0; lobbying gets 0.9 / 0.3) and the rendered triple-bars are readable at a glance.
/api/training/alignment-test scores per-proposal8 sequential model calls (concurrency cap of 4 honoured), each returns { vote, confidence, explanation } within ~280 chars. Server scores match server-side; the user's vote is not in the model prompt (verified by reading the request payload in app/api/training/alignment-test/route.ts). weakAreas picks up to 4 topic labels from the unmatched rows. The 0/8 we hit is a data bug upstream (F1), not a logic bug here.
localStorage persistence works under simocracy.trainingLab.v1.<simUri>Single-key-per-sim, debounced 300 ms write on every state change. Re-opening the modal on the same sim (without closing the browser) restores the four-section state cleanly: votes, interview turns, profile, alignment. updatedAt stamp re-writes on every save. Schema-validated on load via the isTrainingLabState guard.
Width is sm:max-w-3xl as the brief requested (480 px wider than the chat modal). FlaskConical icon, Reset training data ghost button + Apply to constitution primary in the persistent footer. Fonts (EB Garamond serif body, IBM Plex Mono labels) match the rest of the app. Tab order is sensible by keyboard (Home/End jump to first/last).
Severity: high · Surface: programmatic input + browser autofill + paste-then-tab.
In components/sim/training-lab/baseline-tab.tsx, updateVote(proposalId, patch) reads baselineVotes from props closure on every call, then synthesizes nextVote from that snapshot before calling onBaselineVotesChange. If three updates fire in the same JavaScript synchronous tick (vote click → slider input → textarea input — what the agent-browser eval does, and what an autofill extension or a "Tab to next field after a paste" flow would do), all three calls see the same empty baselineVotes. The last onBaselineVotesChange wins, and because existing?.vote ?? "abstain" resolves to abstain when existing is undefined, the final stored vote is abstain regardless of what the user clicked.
Reproduction (this run). Eight proposals, each with YES or NO selected, an importance value set, and a reasoning sentence typed via eval. The UI showed 8 / 8 voted with the correct buttons highlighted; localStorage contained vote: "abstain" on all 8. Manual single-click later (no slider, no textarea, no rapid follow-up) wrote vote: "yes" correctly — confirming the bug only triggers on consecutive synchronous calls within one React batch.
Why it matters. The downstream alignment test then scored 0 / 8 (snapshot 18) and the per-row card honestly says USER: ABSTAIN on every proposal. A user who scripts the form, uses 1Password autofill, or just clicks YES and rapidly drags the slider before React has time to re-render between events could end up with their training quietly storing abstain for everything — and getting a 0% alignment score they can't explain.
Fix. Either move the baselineVotes array into the BaselineTab's own state (so React's setter handles batching with the latest state) or, less invasively, change the parent contract on onBaselineVotesChange to accept a functional updater (prev) => BaselineVote[] the way useState does, and apply each updateVote call against the most recent state. Same pattern needed in InterviewTab — see F2.
// localStorage at end of the rapid-input run
[
{ proposalId: "climate-disclosure", vote: "abstain", importance: 0.90, reasoning: "Climate disclosure is foundational risk management." },
{ proposalId: "executive-pay-ratio", vote: "abstain", importance: 0.55, reasoning: "Multi-year stock vesting still rewards short-term volatility." },
{ proposalId: "board-diversity-policy", vote: "abstain", importance: 0.70, reasoning: "Diverse perspectives improve oversight; quotas not needed." },
{ proposalId: "political-lobbying-disclosure", vote: "abstain", importance: 0.80, reasoning: "Political spending should be traceable to stated values." },
{ proposalId: "worker-safety-audit", vote: "abstain", importance: 0.85, reasoning: "Independent audits build worker trust." },
{ proposalId: "data-privacy-oversight", vote: "abstain", importance: 0.75, reasoning: "Privacy oversight is now a board-level fiduciary duty." },
{ proposalId: "supply-chain-human-rights", vote: "abstain", importance: 0.90, reasoning: "Human-rights review is a moral baseline for global supply chains." },
{ proposalId: "shareholder-special-meetings", vote: "abstain", importance: 0.40, reasoning: "Lower thresholds invite hedge-fund activism over long-term value." }
]
// later, after a single manual click on the worker-safety YES button:
{ proposalId: "worker-safety-audit", vote: "yes", importance: 0.85, reasoning: "Independent audits build worker trust." }
Severity: medium · Surface: any user who types and hits send before the next question arrives (≈8 s).
Same closure-stale-state pattern in askNextQuestion() and sendAnswer() (components/sim/training-lab/interview-tab.tsx L46–L85). Both functions append to interviewTurns by spreading the props-captured array. If the user clicks Ask next question, then types a quick answer and hits send before the model's response arrives, two parallel writes both spread the same interviewTurns snapshot — the answer's onInterviewTurnsChange([…old, userTurn]) overwrites the question's onInterviewTurnsChange([…old, assistantTurn]), depending on order. In this run we ended up with three user turns and only one assistant question in localStorage (snapshot 12). The downstream profile distillation still ran fine because the user's answers carry the substance, but the conversation log on screen shows answers without questions, which is confusing.
Defence in depth. Even after F1's fix, the textarea and the send button should be disabled while isAsking === true — the brief explicitly says "Ask next question" button is enabled again afterwards, but it's silent on the answer side. Disabling input during the fetch is the cheap belt to F2's suspenders fix in F1.
Severity: high · Surface: any user whose sim still says "No constitution drafted yet."
From the agent's PR handoff: "Apply currently requires an existing org.simocracy.agents record/rkey. If a sim has no constitution record yet, it shows an error instead of creating one." The actual production behaviour is worse: it shows nothing at all. Clicking the enabled Apply to constitution button on a sim with no org.simocracy.agents record produces no toast, no spinner, no modal close, no PDS write, no console error. Verified three different click paths (agent-browser pointer click, dispatched MouseEvent sequence, direct invocation of the React onClick prop via fiber) — none surface the toast.error("Constitution record not found. Save a constitution before applying training.") branch in training-lab-modal.tsx L106.
Inspection of the live DOM during the click: section[aria-label="Notifications alt+T"] (Sonner's container) is mounted but stays at childElementCount: 0. The button __reactProps$… has a defined onClick; something in the early-return path is silent — most likely the toast call itself never fires because trainingState.profile evaluates falsy by the time applyToConstitution runs (the profile tab held it before, but switching to Alignment after Run alignment test may have triggered a re-mount that dropped the profile reference). Needs a single targeted repro with a console.log before the early returns to be sure.
Fix. Two edits, in this order. (1) Make Apply create the org.simocracy.agents record when agentsRkey is null, the way agents-editor's save path already does — that's the user-facing fix the agent's handoff already flagged. (2) After (1), any early return must surface a toast (or at minimum a non-debug console.error); silent no-ops are the worst outcome for a write button. While we're there, gate Apply behind !!agentsRkey in the disabled prop too, so users who can't apply yet get the visual cue before clicking.
Severity: low · Surface: cosmetic — switching from a long tab (Baseline / Profile-after-distill) to a short tab (Alignment-pre-run) makes the modal "hop" up the page.
shadcn/ui Dialog centers vertically; the inner scroll container is capped at max-h-[62vh] but does not reserve that height. So when the body is short, the dialog sits at the geometric centre of the viewport (visible in snapshots 17 and 18). When you switch back to Profile, the modal jumps down again. Easy fix: add a min-h-[62vh] to the same scroll container, or equivalently set min-h-[420px] on the inner div, so all four tabs feel like the same panel. Doesn't affect functionality.
agent-browser click @ref on Radix tab triggers onPointerDownOutside intermittentlyNot a product bug — flagging only so the next agent running this flow knows: clicking buttons inside Dialog via the agent-browser MCP click @ref path occasionally fired Radix's onPointerDownOutside handler and closed the dialog (originally misread as a real bug — see commits/screenshot trail). Switching to eval-based document.querySelector(…).click() avoided the issue entirely. The right tool for tab switching is tab.focus(); tab.dispatchEvent(new KeyboardEvent('keydown', {key:'Home'|'End'})) — Radix's keyboard nav is reliable; its synthesized pointer-event handling is not.
EDIT CONSTITUTION save first, then re-open the lab, distill again, and check the merged markdown lands at org.simocracy.agents.description with a clean ## Training Lab Profile section. Re-applying should replace, not append.window.confirm path — the headless agent-browser session hung on window.confirm(), which we should swap for a Radix AlertDialog anyway. UX win and unblocks automated testing.DID=did:plc:cpoagodpqrgs4t7thi5z37uf
RKEY=3mkt33lmhu62a
SIM_URI="at://$DID/org.simocracy.sim/$RKEY"
# 1. confirm the user has zero org.simocracy.agents records on the PDS
# (this is the precondition for F3 to surface)
curl -s "https://climateai.org/xrpc/com.atproto.repo.listRecords" \
--data-urlencode "repo=$DID" \
--data-urlencode "collection=org.simocracy.agents" \
--data-urlencode "limit=10" \
-G | jq '.records | length'
# → 0
# 2. confirm the new training routes are wired and auth-gated
curl -s -X POST "https://www.simocracy.org/api/training/next-question" \
-H 'Content-Type: application/json' -d '{}' | jq .
# → {"error":"Please sign in"} (HTTP 401)
# 3. inspect localStorage after the rapid-input run (Chrome devtools)
JSON.parse(localStorage.getItem(`simocracy.trainingLab.v1.${SIM_URI}`)).baselineVotes
# → 8 rows, every vote === "abstain" (F1 fingerprint)
bef5c72 (fix(config): per-branch baseUrl on Vercel previews)
Accountsatyam2.climateai.org · did:plc:cpoagodpqrgs4t7thi5z37uf · PDS climateai.org
Sim under testPM Slug Test 2026 · at://…/org.simocracy.sim/3mkt33lmhu62a · 0 agents records, 0 style records — fresh of any constitution work, which is what surfaces F3
PR shippedfeat/sim-training-lab → merged into main as e2583ce feat(training-lab): owner-only Sim Training Lab MVP (12 files, +2,502 / −6)
AI routes hit3 × POST /api/training/next-question, 1 × POST /api/training/extract-profile, 1 × POST /api/training/alignment-test (8 internal model calls). All on Gemini Flash Lite via OpenRouter.
Toolingagent-browser v0.x (headless Chromium, persistent profile) for OAuth + screenshots + the 2:43 webm; eval for Radix Dialog interaction once the MCP click path tripped onPointerDownOutside; curl against climateai.org for the PDS-state verification.
Screenshots19 PNGs in assets/01–20, all 1280×720 or shorter, under the 1500 px height cap. walkthrough-oauth-and-modal.webm is 1.6 MB / 2:43.
Next roundtraining-lab-r2 — re-run after F1 + F3 land. Bumps the series from r1 partial to r2 pass if both fixes hold.