IR 2 — Grading sheets

Three sheets, matching the 40 / 40 / 20 split in program. Each criterion is scored 0–4:

Score Meaning
0 absent
1 attempted, but wrong or superficial
2 adequate, no more
3 good: clear, correct, thoughtful
4 excellent: a staff engineer or a reviewer would be impressed

Sheet totals are rescaled to the component weight. Grade the pair on sheets 1 and 2; adjust individuals by up to ±10 % of the pair mark when the two contributions were visibly unequal (say so in the comment).

1. Presentation (40 %) — pair

Paper: ______________________ Pair: ______________________ Session: ____ Date: ________

# Criterion What a 4 looks like 0–4
P1 Problem and positioning The audience knows in two minutes what problem the paper solves, why the spring pipeline does not solve it, and what the one-sentence claim is
P2 Method explained correctly The mechanism is explained, not just named; one worked example or diagram; a non-expert could re-derive the key step
P3 Evidence read critically Baselines and datasets are named; the key table is shown and interpreted; the presenters say which numbers they believe and which they do not (saturated benchmark, missing baseline, unfair comparison)
P4 Own results integrated At least one number the pair produced themselves appears in the talk and is compared with the paper's; the testbed row is shown
P5 Live demo Three queries side by side against the spring baseline on precomputed indexes: a hero query with the why, a failure query explained rather than excused, a wildcard taken from the room; runs inside 10 minutes with no notebook scrolling
P6 Connections Explicit links to the spring course and to at least one other seminar paper; the framing lecture's questions are picked up
P7 Delivery Time kept (25 + 10); slides readable; both presenters speak; no reading from notes
P8 Dialogue The prepared answers to the discussants' questions are short and precise; "we don't know" is used when true; the pair defends or concedes points about its own results and lets the room in

Total /32 → ×1.25 = /40. Comment: ____________________________________________

2. Implementation and report (40 %) — pair

# Criterion What a 4 looks like 0–4
I1 Reproduction target chosen well The reproduced result is a central claim of the paper, scaled down honestly (smaller model / corpus), with the scaling decision justified in the reading plan and the report
I2 Reproduction executed The experiment runs end to end from the README; numbers are reported against the paper's; discrepancies are diagnosed, not hidden. A failed reproduction with a convincing diagnosis scores as high as a successful one
I3 Testbed result Track A: nDCG@10 and Recall@100 on SciFact, NFCorpus and ArguAna against the closest baseline, with the shared script's p-value and CI. Track B (efficiency papers): QPS, build time, index size, peak RAM and Recall@10 vs exact on the 92k MS MARCO subset, plus Track A to show ranking quality survived. results.json validates
I4 Experimental hygiene Fixed seeds; no tuning on the test queries; the same evaluation code as everyone else; the difference is read with its p-value and CI, and a +0.01 with p = 0.4 is called noise; multiple comparisons acknowledged when several variants were tried
I5 Analysis Error analysis with concrete queries: where the method helps, where it hurts, and a hypothesis why. Not just a table
I6 Code quality Readable, runnable in ≤ 10 lines from the README, dependencies pinned, no dead code, no 3 GB checkpoints in git
I7 Report 4–6 pages: problem, method in the pair's own words, reproduction, testbed, analysis, limitations, "what we would do with 100 GPU-hours". Precise, no padding
I8 Honesty and scope Everything claimed was done; what was skipped is stated and why; use of AI assistants is declared

Total /32 → ×1.25 = /40. Comment: ____________________________________________

Report deadline: seven days after the presentation; Dec 18 at the latest. Late: −2 points per day on the /40. Materials for the discussants (slides, code, results, demo queries) are due on the Wednesday before the session, 18:00: −2 points per day on sheet 1.

3. Discussant role and participation (20 %) — individual

Each student is discussant once (two discussants per paper). 15 points for the discussant role, 5 for participation across the seminar.

3a. Discussant note (written, due Thursday 18:00 before the session) — two templates

Each paper has two discussants with different briefs (see program §8). One page maximum. The instructor forwards both notes to the presenting pair the same Thursday evening so they can prepare short written answers — this is what happens with real reviews, and it is what makes the session a dialogue.

Discussant A — the auditor (reads the code and results as closely as the paper)

Paper: ...                           Discussant A: ...            Session: ...

1. The claim the pair chose to reproduce, in one sentence — and is it central to the paper?
2. Baselines: were they tuned? Is the comparison fair at the scale used? (cite the README/results.json)
3. The testbed row: does the difference survive its p-value / CI? Track B numbers plausible?
4. One anomaly or gap in the implementation (a hyperparameter, a query, a missing failure case):
5. What was skipped, and was it said?
6. Your two questions for the pair (about the numbers or the code); they will answer them in the session:

Discussant B — the skeptic (reads the paper against the field)

Paper: ...                           Discussant B: ...            Session: ...

1. The paper in two sentences (your words, not the abstract):
2. Real problem or benchmark artefact? Who would use this, at what cost?
3. Main limitation or unconvincing point (a specific table, assumption, or dataset):
4. One connection: to the spring pipeline, to another seminar paper, or to a real system:
5. What would change your mind (an experiment the paper did not run)?
6. Your two questions for the pair (about validity, cost or connections); they will answer them in the session:

3b. Discussant sheet

Paper: ______________________ Discussant: ______________________ Session: ____

# Criterion What a 4 looks like 0–4
D1 Preparation The note shows the paper and the pair's code/results were read; specifics are cited (a table, a hyperparameter, a query); the note follows the role's template
D2 Critical depth The limitation is substantive (a confound, a missing baseline, a claim the data does not support), not "the dataset could be bigger"
D3 Question quality Questions are specific, answerable, and open something up; at least one targets the pair's own results
D4 Question round Puts the two prepared questions clearly, follows up once when an answer dodges, does not lecture or re-summarise the talk, no overlap with the other discussant; the exchange would help the pair improve the report
D5 Moderation A moderates the open discussion: brings in the pre-posted class questions, gets at least three other people to speak, keeps time, does not let the instructor take over; B reopens with an uncovered question if the room is quiet

Total /20 → ×0.75 = /15.

3c. Participation (5 points)

Points Evidence
0–3 Pre-session questions posted (one per session, Thursday 18:00): 3 = all five sessions, 2 = four, 1 = two or three, 0 = fewer
0–2 In-session contributions: 2 = asks or answers substantively in most sessions, 1 = occasionally, 0 = silent

Total sheet 3: /20.

4. Instructor checklist per session