Information Retrieval 2 — Recent Advances in Information Retrieval

Seminar, M2 (AI track), Université Paris-Saclay — Fall 2026
Instructor: Kim Gerdes · Language: English · Format: reading seminar with implementations
When: six Friday mornings, 9:00–12:30 · Nov 6, 13, 20, 27 · Dec 4, 11 (2026)
Who: 28 students in 14 pairs · who presents what, and who discusses it

Companion documents: the annotated literature review (13 themes, 161 papers), the paper menu and the bidding page, the grading sheets.

1. Positioning

The spring course Information Retrieval (March–April 2026; slides and notebooks) built a complete modern retrieval pipeline: IR metrics → BM25 and variants → bi-encoders and FAISS → cross-encoder reranking, RRF, doc2query, ColBERT → BEIR/MTEB → a mini-RAG, with a scientific-article Codabench challenge.

IR 2 takes the pipeline as given and asks what has changed since 2023 and what the open problems are: LLMs as rerankers, query rewriters and search agents; how embedding models are actually trained and where they break; learned sparse and multi-vector retrieval in production; RAG beyond "retrieve then prompt"; scholarly and patent retrieval; evaluation with LLM judges; poisoning and bias; multimodal documents.

Each pair adopts one paper (a modern classic or a 2024–2026 paper), presents it, reproduces a key result at small scale, and applies the method on a shared testbed so the whole seminar produces one comparable table.

For newcomers, meaning students who took neither the spring course nor a comparable IR course elsewhere (BM25, dense retrieval, reranking, RAG): the spring course page is your self-study package — the slides of the five classes and the notebooks without solutions (the slides contain everything needed to fill them in) — and you are paired with a returning student. Session 1 includes a 45-minute recap. Expect roughly 15 hours of self-study before Nov 6. If you followed a similar IR course elsewhere, say so when you bid: the notes are still worth a skim, and you count as a returning student when pairs are formed.

Inside a mixed pair, a sensible split: the newcomer owns the testbed run, the results table and the live demo (the most scaffolded parts, with starter code in the repo); the returning student owns the reproduction. Both present, both answer questions, and the report says who did what.

Each student presents once (one talk, with their partner) and is discussant once (for a paper in another session), and posts one question per session. That is the whole workload; nothing else is graded.

2. Calendar and deadlines

Pairs presenting on Nov 13 have one week from the assignment (Nov 4); pairs presenting on Dec 11 have five. In exchange, late presenters are expected to relate their paper to what was presented earlier in the seminar.

3. A session (3h30)

The 15 minutes after the demo are a dialogue, not a review. The two discussants open with the two questions each sent on Thursday; the presenters answer them, prepared, in two minutes at most per question, and a discussant follows up once when an answer dodges. Then the moderator opens the floor with the questions the class posted, and anyone may intervene at any point, including during the question round. The instructor stays silent for the first five minutes.

Rules of thumb for presenters: assume the audience has read the abstract and the spring notes, nothing else. One slide of "what the spring pipeline does" versus "what this paper changes". Show at least one number you produced yourself. End with what did not work.

Live demo protocol (10 minutes, three queries). The demo runs on the presenters' laptop against indexes computed beforehand: nothing is trained or indexed live, Colab is used only if it was tested that morning, and a two-minute screen recording is kept as a fallback. Show side-by-side rankings, spring baseline versus your method, for exactly three queries: a hero query where the method clearly wins and you can say why; a failure query where it collapses; and a wildcard query proposed on the spot by the discussants or the audience. Three queries fill ten minutes and leave no room for scrolling through a notebook.

Before every session, every student posts one question about one of the session's papers on the question page (deadline Thursday 18:00). Each question stays hidden until the session starts at 9:00, so nobody can copy. The session's presenters and discussants can read them from Thursday 18:00, once posting has closed; the instructor uses them to grade participation.

4. Session 1 — kickoff (Nov 6)

Slides (PDF): kickoff · model presentation

5. Sessions 2–6 — themes (provisional)

The themes below are the default grouping; the final mapping follows the papers the pairs choose. About three presentations per session, two in the last one, which fits 27 students in 13–14 groups with one spare slot; the exact count follows the final number of pairs.

Session 6 ends with a 45-minute synthesis: the assembled results table, what generalised across papers, what did not, and course feedback.

6. The shared testbed

Everything a pair implements is evaluated twice: once as a reproduction (the paper's own setting, scaled down) and once on the shared testbed (the seminar's setting, identical for everyone). The testbed has two tracks, because a 5,000-document corpus can show whether a method ranks better but not whether it is faster.

Repo: github.com/kimgerdes/ir2-testbed · shared results table

Track A, effectiveness: the spring baseline on three BEIR datasets already on every laptop.

The scientific-article set from the spring challenge (graded relevance) can be added for methods aimed at scholarly search; it is not in the repo. Baselines in the repo: BM25, all-MiniLM-L6-v2, BAAI/bge-small-en-v1.5, RRF(BM25 + dense), cross-encoder ms-marco-MiniLM-L-6-v2 rerank of the BM25 top-100. Metrics: nDCG@10 and Recall@100, and, computed by the shared script, a paired t-test and a bootstrap 95 % confidence interval of the per-query difference against the closest baseline (the paired t-test is the most reliable choice for IR, Urbano et al. 2019). Every pair adds one row per dataset. A +0.01 with p = 0.4 is reported as noise, not as a win.

Track B, efficiency: the MS MARCO subset (92,565 passages, 1,000 queries) that every returning student has from the spring. For papers about indexing, quantisation, learned-sparse or multi-vector engines (Seismic, RaBitQ, HNSW hubs, MUVERA, WARP, ConstBERT, ACORN), 5,000 documents show nothing: a flat NumPy dot product takes milliseconds and any index is slower. On 92k passages the speed-ups become visible. Metrics: queries per second on the laptop CPU, index build time, index size on disk, peak RAM, and Recall@10 against exact search, so that the effectiveness cost of the speed-up sits on the same row. Track B pairs also report Track A nDCG@10 to show that ranking quality survived. No paper assigned this year needs Track B, so the repo does not include it.

Papers whose object is neither ranking nor speed (evaluation, security, RAG generation, agents) still use the Track A corpora: LLM relevance judgments on SciFact versus the human qrels, poisoning of NFCorpus, a RAG loop over ArguAna. The report must state explicitly which testbed number the method changes and by how much.

Starter kit in the repo. Nobody in the spring course trained an encoder, and about a third of the menu asks for it. Besides the baseline scripts, the repo ships starter-contrastive-training.ipynb: a sentence-transformers v3 training loop with MultipleNegativesRankingLoss and in-batch negatives, hard-negative mining with the spring cross-encoder, an optional LoRA adapter via peft on BGE-small, and ten lines of NumPy for the alignment and uniformity metrics of Wang & Isola (2020). If your paper involves training, run it once first.

Compute guide. Laptops (CPU, 16 GB) handle everything at BEIR scale with models ≤ 150M parameters. Free Colab (T4, 16 GB VRAM) handles fine-tuning MiniLM / BGE-small / ModernBERT-base with small batches, and inference with 1–8B open LLMs (Qwen 2.5/3, Llama 3.x, Mistral 7B in 4-bit). Not feasible: training LLMs, indexing millions of documents, commercial API spend beyond a few euros. Choose the small-scale version of the experiment and say so in the report.

Repository conventions. One folder per pair in the shared GitHub repo, added by pull request: README.md (how to run, in ≤ 10 lines), reproduce.ipynb, method.py (writes your runs; python testbed.py score turns them into results.json), demo.md (the three demo queries and the link to the fallback recording), slides.pdf, report.pdf. results.json follows the schema in the repo: per dataset, the method's nDCG@10 and Recall@100, the closest baseline's, the delta, the paired-t p-value and the bootstrap CI. The results table builds itself.

7. Grading

Each student presents once (with their partner) and is discussant exactly once, for a paper in a different session and theme than their own. Two discussants per paper; 27 students cover 13–14 papers. Discussant assignments are published with the paper assignments on Nov 4 so everyone knows their week early.

8. The discussant role

Discussants are the seminar's peer reviewers. Two per paper, with distinct briefs so they do not repeat each other:

• Discussant A, the auditor, reads the pair's code, README and results.json as closely as the paper. Were the baselines tuned? Is the scaled-down reproduction faithful to the claim it tests? Does the testbed row hold up (p-value, both tracks where relevant)? Were failures reported? Their question targets an anomaly in the implementation or the numbers.
• Discussant B, the skeptic, reads the paper against the field. Does it solve a real problem or a benchmark artefact? What does it cost to deploy? How does it relate to the spring pipeline and to another seminar paper? Their question is about validity and use.

Both discussants:

1. Read the paper in full, plus the pair's README, results table and demo queries, posted on the Wednesday before the session (18:00). Presenters who post late lose 2 points per day on the presentation sheet: without the material the discussants cannot do their job.
2. Submit a one-page discussant note by Thursday 18:00, one day after the material, using the role-specific template in grading sheets §3. The note ends with two questions for the presenters. The instructor forwards both notes to the pair the same evening; the pair prepares short written answers.
3. After the demo, the question round (8 minutes): A then B put their two questions; the presenters answer each in two minutes at most; a discussant follows up once when an answer dodges. Anyone in the room may cut in. No slides, no summary of the talk.
4. Then the open discussion (7 minutes): A moderates: brings in the questions the class posted, keeps time, makes sure at least three other people speak; B reopens with a question that was not covered if the room is quiet.

The instructor stays out of the discussion for the first five minutes. This is deliberate.

9. What the instructor prepares

• Sept: the menu (from full review); the testbed repo with baseline scripts, testbed.py (including the paired t-test and bootstrap CI) and the results.json schema; the contrastive-training starter notebook; the self-study package for newcomers.
• Nov 4–10: assignments, the discussant schedule with A/B roles, the reading-plan feedback (one paragraph per pair).
• Per session: 30-minute framing lecture; the framing pointers are listed per theme in full review.
• Session 1: the 15-minute primer on articles and patents and the model presentation of Beyond Abstracts in seminar format, with the three-query demo done properly.
• Session 6: the synthesis of the shared results table.

10. Risks and mitigations