Gold Annotation

1 · What you are doing here

You will read research papers, one at a time. Each paper introduces a test for AI systems. Your job is to answer a fixed list of questions about what the paper itself says — for example: does the paper say what ability its test is supposed to measure? If yes, copy the exact sentence where it says so.

One note on wording. The menus and buttons on this page say paper, because you work through one paper at a time. Some of the questions say benchmark instead. That is the usual research word for the test a paper introduces. So when a question asks about “the benchmark”, it is asking about the test described in the paper you are reading right now.

Two ground rules:

Only the paper counts. Answer from what is written in the PDF on the left — not from anything you may already know or assume. If the paper doesn't say it, the answer is “no” or “not stated”, even if it seems obvious.
There is no right answer to match. Nobody can see your answers and you can't see anyone else's. Later, answers are compared and differences discussed — honest disagreement is a normal and useful part of the process. Never guess what “they” want; say what you see in the paper.

2 · The screen, piece by piece

Left half: the paper (PDF). You can scroll, zoom, and search inside it.
Right half: one question at a time (a “card”).
Yellow box: the official instructions for the current question — what it means, what to ask yourself, and how to answer. When in doubt, the yellow box wins. Read it every time at first; it becomes second nature quickly.
“Find in paper” box (above the PDF): type a word, press Enter, and you get a list of every place it appears. Click one and the PDF jumps to that page with the word highlighted; the list tucks itself away — click the blue “N matches” label to bring it back and try another. The small grey buttons under “search the paper for” are words taken from the wording of the current question. They are the same for every paper and every coder, so they never hint at this paper's answer — they are only a shortcut for the search box. A button shows a whole word (say capability) but searches for its opening letters, so every form of the word is found.
Answer area: either round buttons (click one to choose; click it again to un-choose), a list of checkboxes (pick all that apply), or a text box. The heading above it tells you which kind of answer is wanted. When it reads Exact quote, the box wants the paper's own words: select the sentence in the PDF, copy it, and paste it in — do not retype it, shorten it, or put it in your own words. Every text box also has an N/A checkbox under it — “no quote in paper” on a quote question, “the paper doesn't say” on the short-phrase one. Tick it when you are confident the paper simply doesn't contain such a statement. That is a real answer; an empty box just means “not done yet”.
Notes box: optional, and free-form — a comment on why you answered as you did, or a note that you were unsure between two answers. Nobody is marking these; they are very helpful later when we compare answers.

Saving is automatic — about half a second after you stop typing or clicking (watch the little saved / unsaved word at the bottom of the card). You never need to press a save button.

3 · Questions that fill themselves in

Some questions only apply if you answered “yes” to an earlier one. Example: if you answer no to “does the paper claim to measure an ability?”, then “copy the sentence with that claim” obviously has no answer — so the system fills it with [none] for you. Those rows show an orange “auto-filled” banner. Don't edit them: if you think the auto-filled value is wrong, go back to the earlier question the banner names and change your answer there.

4 · Getting around quickly

→ next question · ← previous one
1–9 choose the 1st–9th round button without the mouse (moves on automatically)
0 tick/untick the N/A checkbox under a text box

The unfilled button at the top shows only what you haven't answered yet, and the drop-down menu shows how far along you are in each paper. Your place is remembered — closing the tab or shutting the laptop loses nothing.

5 · Your practice task

Answer all the questions for the practice paper shown below, in order, exactly as you would for real: read the yellow box, search the paper, and when a question asks for a quote, copy the paper's own words exactly. Use the notes box whenever you're torn. Plan for roughly 45–90 minutes.

This is a sandbox — nothing you do here touches the real study. We will go over your practice answers together on a short follow-up call before the real work starts.

No valid access key

Gold annotation

source PDF: —

Annotator answers (read-only)

Your annotation

Notes (optional) — e.g. a comment, or that you were unsure