Gold Annotation

1 · What you are doing here

We are auditing how benchmark papers describe what their benchmark measures. For each paper you answer a fixed set of questions (“fields”) — e.g. does the paper claim the benchmark measures a construct? quote the sentence that says so. Your answers come only from the paper itself (left pane), never from prior knowledge about the benchmark, and never from guessing the “desired” answer.

You always code your own copy, blind: nobody else can see your answers, and you cannot see theirs. There is no “correct rate” shown anywhere — honest disagreement is expected and useful.

2 · Anatomy of one card

Field name + type (top): the question ID from the codebook and whether it wants a categorical pick or free text.
Yellow codebook box: the authoritative definition — what it is, the coder question to ask yourself, and how to answer. Read it every time until it is second nature.
Find in paper (above the PDF): type any term, press Enter, click a hit to jump the PDF to that page. The grey search ideas chips under the codebook box are generic starting points for the current field — they are the same for every paper and are not hints about this paper's answer.
Your annotation: pill buttons (pick one; click again to clear), checkboxes (multi-select), or a free-text box. Free-text quote fields have a “no quote in paper / N/A” checkbox — use it when the paper genuinely contains no such statement; it is different from leaving the field unfilled.
Notes: optional; write down anything you were unsure about — notes are gold when we adjudicate disagreements.

Everything autosaves ~0.4 s after you stop editing (watch the saved / unsaved indicator).

3 · Gate rows (auto-filled fields)

Some fields only make sense conditional on another: if you answer construct_claim_present = no, the two quote fields under it are automatically filled with [none]. You will see an orange “auto-gated row” banner on such rows — don't fight it; if you think the value is wrong, go back to the gate field it names and change that.

4 · Navigation & keyboard

j / → next row · k / ← previous
u jump to your next unfilled row (also the next unfilled button)
1–9 pick the Nth pill (auto-advances) or toggle the Nth checkbox
0 toggle “no quote in paper / N/A” on free-text fields
ctrl/cmd+S force save

The unfilled filter shows only rows you haven't answered; the benchmark dropdown shows your per-paper progress. Your position is remembered, so closing the tab is safe.

5 · Your practice task

Code the practice paper below end-to-end (all fields, in order). Work exactly as you would for real: read the codebook box, search the PDF, quote precisely (copy the paper's own words), use notes when torn between two answers. Expect 45–90 minutes. We will go over everyone's practice answers together before the real assignment starts.

No valid access key

Gold annotation

source PDF: —

Annotator answers (read-only)

Your annotation

Notes (optional)