Eval Wiki · educator audit · round 3
Test Report v3
Post-fix audit after (a) renaming Plurality Wiki → Eval Wiki, (b) adding a one-sentence summary beneath every page title, and (c) populating a structured Eval Card on every one of the 71 synthesis pages. Format inspired by evalcards.evalevalai.com.
What changed in round 3
- Rename — every Plurality Wiki string replaced with Eval Wiki across masthead, title,
<title>tag, sources page, and the homepage wordmark. - One-sentence summaries — every metric page now opens with a plain-English definition right under the title (before tags or any other layout). 71 / 71 pages covered.
- Eval Card block — between the hero and the prose body, every page shows a structured glanceable card with up to 11 fields: kind, unit of evaluation, inputs, output, range, direction, formula, timing, introduced by (year), license/access, main caveat. Only populated fields render.
- Schema — content collection now accepts
summary: stringandcard: { … }frontmatter (optional, backward-compatible). - Three title polishes —
incidence-rate-from-reports,relative-risk-from-reports,reporting-overrepresentation-testhad their formulas baked into the title (# IR_G ≥ (β − γ^FR)/(γ^TR − γ^FR)). Formulas now live in the card'sformulafield; titles read clean.
Educator-perspective evaluation
A student new to the field should be able to answer — for any metric, within ten seconds of opening its page — these four questions:
- What is this metric, in one sentence?
- What do I need to compute it?
- What does it produce, and in what units/range/direction?
- Who introduced it, and what's the main pitfall?
Before round 3, answers required reading paragraph-style prose. Now every page surfaces them in the card grid directly below the title.
Card field semantics
| Field | Purpose | Example value |
|---|---|---|
summary | One-sentence plain-English definition shown in the hero | "The classical inequality index: half the mean absolute difference between every pair of agents' outcomes, normalised by the mean — ranges from 0 to 1." |
kind | Rough type (coder-based index, axiom, framework, impossibility theorem, algorithm family…) | "coder-based index" |
unit_of_evaluation | What the metric scores | "speech act (sentence / short utterance)" |
inputs | Data needed to compute it | "verbatim transcript of deliberation" |
output | Shape + type of the result | "five-dimensional ordinal score per speech act" |
range | Numeric range if scalar | "[0, 1]" |
direction | Higher-is-better / lower-is-better / depends | "higher is better" |
formula | Operational definition | "G = (1 / (2 n² μ)) Σᵢ Σⱼ |y_i − y_j|" |
timing | Pre- / during / post-deployment | "post-deployment" |
introduced_by (+ year) | Canonical attribution | "Corrado Gini, 1912" |
license | Access / licensing model of the metric definition | "public" |
caveat | One-line main failure mode or pitfall | "Insensitive to tail shape — two very different distributions can share a Gini" |
All fields are optional. A tension page, for instance, only fills kind, caveat, and introduced_by — rendering a short card with just those rows.
Coverage
- 71 / 71 synthesis pages have a
summary. - 71 / 71 have a
cardblock (with at minimumkind; see field semantics above for the full set). - 0 broken internal links across 4,100+ links (postbuild link checker).
- 0 unprocessed wikilinks.
- 0 raw
$$pages (all math via KaTeX). - 88 pages built.
Screen recording
32-step tour of the post-round-3 site — focuses on card pages across all 11 families + the two hubs + a tension page + search.
Does the structure make sense from an educator's perspective?
Ten-second rule: can a student name this metric after a glance?
Across 32 sampled pages, the answer is yes. The layout now reads top-to-bottom as a definitional pyramid: (1) which family? (eyebrow), (2) what is it called? (H1), (3) what is it in one sentence? (summary), (4) tags, (5) card with 9–11 structured fields, (6) prose with history and context, (7) backlinks in the sidebar. Each layer adds context the previous one leaves out.
Every card names the originator
Like Eval Cards itself, provenance is a first-class field. A student can see in one line that quadratic funding is Buterin, Hitzig, Weyl 2019; that DRO is Hashimoto, Srivastava, Namkoong, Liang 2018; that Ostrom's design principles are 1990 with the 2021 blockchain extension by Rozas et al. Attribution never requires reading the prose.
Main failure mode in every card
The biggest pedagogical win: almost every card has a caveat row that names the single most common pitfall. Examples from the corpus:
- Gini: "Insensitive to the shape of the tails — two very different distributions can share a Gini."
- BERTScore: "Measures surface similarity, not factual correctness — a BERTScore-high summary can still hallucinate."
- Algorithmic fidelity (twin): "Aligning to a majority view is easy; the minority-preservation adaptation is what matters in practice."
- Quadratic funding: "Requires strong sybil resistance; collusion-resistance is the central open problem."
- AIR framework: "Aggregation requires a faithful ex-ante-policy model; mis-specifying it leaks into every downstream number."
This is the research-educator move Eval Cards gets right — reporting conditions travel with the number, so readers don't over-interpret.
Structural disagreements between families are their own pages, with cards
Four pages (tension-bridging-vs-jr, tension-endorsement-vs-dqi, tension-fidelity-vs-strategyproof, tension-fast-vs-severe-harm-detection) explicitly name when not to trust any single metric. Their cards use kind: structural tension and a caveat field with the recommended resolution. This is the most educator-friendly contribution: it teaches the limits of the ontology, not just the ontology.
Post-deployment family fits the card format without special-casing
The §11 cluster (AIR framework, IR/RR/over-representation/stopping-time) was added after round 1; every §11 page now has a card in the same format as §1–§10 pages. Fields like timing: post-deployment and formula: IR_G ≥ (β − γ^FR) / (γ^TR − γ^FR) make clear how the family differs from pre-deployment metrics — without visually breaking the ontology.
48 dead-link spans in §11 pages remain
Still unchanged from round 2. The §11 family's prose references concepts/* and sources/* pages that don't exist yet — the renderer correctly marks them as dead-link spans. Fixing them is a content task (create 14 missing concept/source pages), not a renderer bug. The cards themselves use the existing citations.
A few card fields could use a tighter word budget
Two rows occasionally wrap awkwardly on narrow viewports (e.g. "five-dimensional ordinal score per speech act; aggregate to group mean"). They remain readable, but a future pass could trim to ≤ 70 characters per cell. Not a correctness issue.
Add a "download card" action
Eval Cards allows exporting a card as a standalone artefact. We could emit a /{slug}/card.json endpoint per page and a small "Export card" link in the sidebar — so researchers can cite or embed the structured metric definition without scraping HTML. Easy next step; not required for educator use.
All ontology families at a glance (post-round-3)
| Family | Metrics | Card coverage |
|---|---|---|
| §1 Deliberation quality | 7 | 7 / 7 with summary + card |
| §2 Bridging, representation, consensus | 12 | 12 / 12 |
| §3 Summarization faithfulness | 6 | 6 / 6 |
| §4 Digital twin fidelity | 7 | 7 / 7 |
| §5 Mechanism robustness | 12 | 12 / 12 |
| §6 Distributive justice | 8 | 8 / 8 |
| §7 Meta-evaluation | 5 | 5 / 5 |
| §8 Institutional design | 1 | 1 / 1 |
| §9 Policy uptake | 1 | 1 / 1 |
| §10 Tensions | 4 | 4 / 4 |
| §11 Post-deployment evaluation | 5 + 1 hub | 6 / 6 |
| Hub pages (_ontology, _mve-stack) | 2 | 2 / 2 |
| Total | 71 | 71 / 71 (100%) |
Reproducing
cd wiki/frontend rm -rf .astro dist # cache-bust npm run build python3 -m http.server 8000 -d dist & # Re-run the educator audit: bash tests/scripts/tour.sh # existing tour # Or just populate the cards end-to-end: python3 tests/scripts/add-cards.py # idempotent; skips pages that already have a card