IR 2 — Annotated literature review (candidate papers for the seminar)

Purpose. The pool from which the student menu is drawn and from which the instructor's 30-minute framing lectures are built. Thirteen themes plus an addendum, 161 papers, each with a verified link, a summary, its connection to the spring course or to the instructor's research, a concrete pair implementation sized for a laptop or free Colab, and discussion hooks. Themes are pre-assigned to a default session (see program §5); the final session mapping follows the papers the pairs choose.

Reading the entries. Tag classic = pre-2024 canonical reference; recent = 2024–2026. Difficulty 1–3 is for a pair of M2 students with the spring course behind them. Compute laptop / Colab / Colab + small API budget. Testbed = SciFact, NFCorpus, ArguAna (BEIR), the MS MARCO subset, and the scientific-article set from the spring challenge; see program §6.

Instructor's own papers, used as the model presentation in session 1 and as connection points throughout: Djemmal, Olutola & Gerdes (2026), Beyond Abstracts: Learning Scientific Paper Embeddings from Full-Text Windows; Zuo, Gerdes et al. (2025/26), Patent Representation Learning via Self-supervision (arXiv:2511.10657).

Contents

• Session 2 (Nov 13) — Representations · Text embedding models: timeline, training recipes, theory (12 papers)
• Session 2 (Nov 13) — Representations · Learned sparse retrieval, multi-vector retrieval, efficiency and compression (12 papers)
• Session 3 (Nov 20) — Domains · Scholarly document retrieval and representation (12 papers)
• Session 3 (Nov 20) — Domains · Patent retrieval, patent representation learning, prior-art search (12 papers)
• Session 3 (Nov 20) — Domains · Domain adaptation and synthetic training data (12 papers)
• Session 3 (Nov 20) — Domains · Multilingual and cross-lingual retrieval; temporal aspects and diachronic embeddings (12 papers)
• Session 4 (Nov 27) — LLMs in the loop · LLM rerankers, LLM query rewriting, reasoning-intensive retrieval (12 papers)
• Session 4 (Nov 27) — LLMs in the loop · Generative retrieval, learning to rank, click models, distillation (the classics the spring course skipped) (12 papers)
• Session 5 (Dec 4) — RAG and agents · RAG architectures and the generation side (12 papers)
• Session 5 (Dec 4) — RAG and agents · Agentic search, deep-research agents, RL-trained search agents (12 papers)
• Session 6 (Dec 11) — Trust · Evaluation of retrieval and RAG in the LLM era (12 papers)
• Session 6 (Dec 11) — Trust · Security, robustness and bias of neural retrieval and RAG (12 papers)
• Session 6 (Dec 11) — Trust · Multimodal and visually rich document retrieval, document parsing, code and tables (12 papers)
• Addendum · Papers added after the Gemini / Google Scholar pass (5 papers)

Theme 1: Text embedding models: timeline, training recipes, theory

Default slot: Session 2 (Nov 13) — Representations.

1. SimCSE: Simple Contrastive Learning of Sentence Embeddings (Gao et al., EMNLP 2021)

• Link: https://arxiv.org/abs/2104.08821 (arXiv:2104.08821) · Code: https://github.com/princeton-nlp/SimCSE
• Tag: classic · Difficulty: 1 · Compute: laptop / Colab
• Summary: Unsupervised SimCSE feeds each sentence twice through the encoder and uses the two dropout-perturbed outputs as a positive pair, with in-batch negatives; supervised SimCSE uses NLI entailment pairs as positives and contradictions as hard negatives. BERT-base reaches 76.3 (unsup.) / 81.6 (sup.) Spearman on STS, +4.2/+2.2 over prior best. The analysis section explains the gains with Wang & Isola's alignment/uniformity metrics: the contrastive objective flattens the singular-value spectrum and fixes anisotropy.
• Why it matters: The canonical self-supervised embedding recipe, and the exact baseline whose "over-dispersion" on patents the instructor's own work diagnoses and repairs with section-based views.
• Pair implementation: Reproduce unsup-SimCSE on all-MiniLM-L6-v2/BGE-small using SciFact+NFCorpus abstracts as unlabeled text (minutes on Colab), evaluating nDCG@10 zero-shot vs the untrained model and BM25. Then swap dropout positives for "title vs abstract" views of the same paper and report alignment/uniformity and retrieval, testing whether the instructor's patent finding transfers to scientific text.
• Discussion hooks: Why does pure dropout noise work at all, and when does it over-disperse? Is dropout augmentation a sentence-level trick that fails for documents?

2. Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hypersphere (Wang & Isola, ICML 2020)

• Link: https://arxiv.org/abs/2005.10242 (arXiv:2005.10242); https://proceedings.mlr.press/v119/wang20k.html · Code: https://github.com/SsnL/align_uniform
• Tag: classic · Difficulty: 3 (proofs) · Compute: laptop
• Summary: Decomposes what InfoNCE optimises into two measurable properties of normalised features: alignment (positives close) and uniformity (features spread over the hypersphere, measured via a Gaussian-potential kernel). Proves the contrastive loss asymptotically optimises both and shows that directly optimising the two metrics matches or beats InfoNCE downstream.
• Why it matters: The geometric lens students need for the InfoNCE material they saw in spring; the metrics the instructor uses to analyse S2Full and patent embeddings.
• Pair implementation: Compute L_align/L_uniform for 8–10 off-the-shelf models (MiniLM, BGE, E5, Contriever, Qwen3-Embedding-0.6B, ...) on SciFact/NFCorpus query–positive pairs and plot them against nDCG@10, reproducing the paper's alignment–uniformity scatter. Then fine-tune BGE-small on the MS MARCO subset with the direct L_align + λ·L_uniform loss instead of MNRL and compare.
• Discussion hooks: Does uniformity matter for retrieval (only the top-k ordering counts) or only for STS? Companion reading: Godey et al., "Anisotropy Is Inherent to Self-Attention in Transformers" (EACL 2024, https://arxiv.org/abs/2401.12143) — if anisotropy is inherent, is contrastive learning fighting the architecture?

3. Unsupervised Dense Information Retrieval with Contrastive Learning — Contriever (Izacard et al., TMLR 2022)

• Link: https://arxiv.org/abs/2112.09118 (arXiv:2112.09118) · Code: https://github.com/facebookresearch/contriever
• Tag: classic · Difficulty: 2 · Compute: Colab
• Summary: Trains a dense retriever with no labels: positives are two independent random crops of the same document, negatives come from a MoCo momentum queue so that large negative sets do not require large batches. Zero-shot on BEIR it beats BM25 on 11 of 15 datasets at Recall@100, and the multilingual variant retrieves across scripts (Arabic queries to English docs). E5 (Wang et al. 2022, https://arxiv.org/abs/2212.03533) is the weak-supervision sibling: CCPairs + "query:/passage:" prefixes, the first model to beat BM25 on BEIR without labels.
• Why it matters: The bridge from DPR (supervised) to the whole E5/BGE/GTE generation; "two windows of the same document are positives" is precisely the instructor's S2Full training signal.
• Pair implementation: Implement independent cropping on the 92k MS MARCO passages plus the three BEIR corpora and train MiniLM with sentence-transformers' CachedMultipleNegativesRankingLoss as a stand-in for the MoCo queue; evaluate zero-shot on SciFact/NFCorpus/ArguAna vs BM25 and vs the released facebook/contriever. Ablate crop length and queue/batch size.
• Discussion hooks: When does "same document" mean "relevant"? Why did unsupervised retrievers lose to BM25 for years before this?

4. Improving Text Embeddings with Large Language Models — E5-Mistral (Wang et al., ACL 2024)

• Link: https://aclanthology.org/2024.acl-long.642/ ; https://arxiv.org/abs/2401.00368 (arXiv:2401.00368) · Code: https://github.com/microsoft/unilm/blob/master/e5/README.md
• Tag: recent · Difficulty: 2 · Compute: Colab + small API budget
• Summary: Uses GPT-4 in two steps (brainstorm task types, then generate query/positive/hard-negative triples with instructions) to produce synthetic data for hundreds of thousands of tasks in ~100 languages, then LoRA-fine-tunes Mistral-7B with a plain contrastive loss and last-token pooling for under 1k steps. Strong MTEB results with synthetic data alone; adding labeled data gave the then-SOTA. The template for NV-Embed, Gecko, Qwen3-Embedding and most 2024–2026 leaderboard toppers.
• Why it matters: Extends the spring InfoNCE/hard-negative material to decoder LLMs and introduces "data synthesis as the main lever"; Qwen3-Embedding-0.6B (Zhang et al. 2025, https://arxiv.org/abs/2506.05176; https://github.com/QwenLM/Qwen3-Embedding) is the open, laptop-sized descendant.
• Pair implementation: Generate ~2k domain-specific triples for SciFact/NFCorpus with a local Qwen2.5-7B-Instruct (4-bit) or a few euros of API; fine-tune BGE-small and, more ambitiously, Qwen2.5-0.5B with last-token pooling + LoRA using instructions. Compare with zero-shot BGE-small and Qwen3-Embedding-0.6B on the testbed.
• Discussion hooks: Is this distillation of the teacher LLM's notion of relevance rather than learning from data? How much of the MTEB gain is benchmark contamination via the synthetic prompts?

5. LLM2Vec: Large Language Models Are Secretly Powerful Text Encoders (BehnamGhader et al., COLM 2024)

• Link: https://arxiv.org/abs/2404.05961 (arXiv:2404.05961) · Code: https://github.com/McGill-NLP/llm2vec (pip package; supports Llama, Mistral, Gemma, Qwen-2)
• Tag: recent · Difficulty: 2 · Compute: Colab
• Summary: Three cheap steps turn any decoder into an encoder: enable bidirectional attention, adapt it with masked-next-token prediction, then unsupervised SimCSE. No synthetic GPT-4 data needed; unsupervised SOTA on MTEB at the time and large gains on word-level tasks. The counterpoint is Weller et al., "Seq vs Seq: An Open Suite of Paired Encoders and Decoders" (Ettin, ICLR 2026, https://arxiv.org/abs/2507.11412; https://github.com/jhu-clsp/ettin-encoder-vs-decoder): with identical data and 17M–1B paired models, purpose-built encoders still beat converted decoders, and a 400M encoder beats a 1B decoder on classification.
• Why it matters: The cleanest explanation of why decoder LLMs embed at all, and a direct test of "are encoders obsolete?"
• Pair implementation: Apply LLM2Vec to Qwen2.5-0.5B/1.5B on Colab (MNTP on the testbed corpora, then SimCSE), ablating each of the three steps on SciFact/NFCorpus. Compare with the same-budget fine-tune of an equally sized encoder (ettin-encoder-150m or ModernBERT-base) to test the Ettin claim.
• Discussion hooks: Is causal attention the real bottleneck, or is it pooling? If a decoder trained on 2T tokens loses to a 150M encoder, why does the leaderboard say the opposite?

6. Smarter, Better, Faster, Longer: A Modern Bidirectional Encoder — ModernBERT (Warner et al., ACL 2025)

• Link: https://arxiv.org/abs/2412.13663 (arXiv:2412.13663) · Code: https://github.com/AnswerDotAI/ModernBERT ; models answerdotai/ModernBERT-base/-large
• Tag: recent · Difficulty: 2 · Compute: Colab
• Summary: Retrains the BERT recipe with 2024 tooling: 2T tokens including code, 8,192-token context, RoPE, alternating local/global attention, GeGLU, unpadding and FlashAttention. Reported as a Pareto improvement over older encoders on GLUE, single-vector and ColBERT-style retrieval (BEIR) and code, at base (149M) and large (395M) sizes. EuroBERT (Boizard et al. 2025, https://arxiv.org/abs/2503.05500) is the multilingual European analogue (210M–2.1B, Llama-3-style, 8k context).
• Why it matters: The long-context encoder the instructor trains on full-text windows; the "encoders are back" moment in the timeline.
• Pair implementation: Fine-tune ModernBERT-base, BERT-base and MiniLM as bi-encoders with the same MNRL budget on the MS MARCO subset and evaluate on SciFact/NFCorpus/ArguAna. Then exploit the 8k window: embed full-text papers (a handful from arXiv or S2Full) as one vector vs chunked, and measure throughput/memory on CPU vs BERT.
• Discussion hooks: What does a single vector lose when the input grows from 512 to 8k tokens? Is ModernBERT's gain architecture, data, or just the code in the pretraining mix?

7. Promptriever: Instruction-Trained Retrievers Can Be Prompted Like Language Models (Weller et al., arXiv 2024)

• Link: https://arxiv.org/abs/2409.11136 (arXiv:2409.11136) · Code: https://github.com/orionw/promptriever
• Tag: recent · Difficulty: 2 · Compute: Colab
• Summary: Builds a bi-encoder (RepLLaMA-style, LLaMA-2-7B) trained on MS MARCO augmented with ~500k LLM-written instance-level instructions plus "instruction negatives" (passages relevant to the query but violating the instruction). Result: +14.3 p-MRR / +3.1 nDCG on FollowIR, +12.9 Robustness@10 on InstructIR, and prompting alone yields +1.4 average nDCG on BEIR.
• Why it matters: Turns "query" into "query + relevance definition", extending the spring dense-retrieval material toward LLM-style prompting; complements the name-dropped BRIGHT/instruction retrieval.
• Pair implementation: Run the released Promptriever (4-bit on Colab) on SciFact with and without a relevance instruction ("a document is relevant only if it supports or refutes the claim") and on ArguAna ("retrieve the counter-argument"), against BGE/Qwen3-Embedding with the same instructions. Then fine-tune BGE-small on a 20k slice of their instruction-augmented MS MARCO and measure instruction sensitivity.
• Discussion hooks: Can one vector honour both the query and an arbitrary constraint (see paper 9)? Are prompt-tuned retrievers just rerankers in disguise?

8. Contextual Document Embeddings (Morris & Rush, ICLR 2025)

• Link: https://arxiv.org/abs/2410.02525 (arXiv:2410.02525) · Code: https://github.com/jxmorris12/cde ; model jxm/cde-small-v1
• Tag: recent · Difficulty: 2–3 · Compute: Colab
• Summary: Argues that a document embedding should depend on its corpus, as BM25's IDF does. Two methods: (1) contextual batching, clustering the training corpus so in-batch negatives are hard and domain-consistent without mining; (2) a two-stage encoder that first embeds a sample of the corpus into context tokens that condition the second-stage encoder for both queries and documents. cde-small-v1 (<400M) reached 65.0 MTEB average, best among small models, with no hard-negative mining or huge batches. Late Chunking (Günther et al., Jina 2024, https://arxiv.org/abs/2409.04701; https://github.com/jina-ai/late-chunking) is the within-document cousin: embed the whole document, then pool per chunk.
• Why it matters: Reconnects dense retrieval to BM25's corpus statistics and to the instructor's interest in document context (full-text windows).
• Pair implementation: Evaluate cde-small-v1 on SciFact/NFCorpus/ArguAna with context sampled from the target corpus vs from MS MARCO (wrong corpus) vs none; reproduce the clustered-batching gain when fine-tuning BGE-small on the MS MARCO subset. Optional: late chunking with jina-embeddings-v2-small-en on long documents as a second contextualisation baseline.
• Discussion hooks: Is CDE "learned IDF"? Contextual embeddings break the "embed once, compare anywhere" contract of vector databases — is that acceptable?

9. On the Theoretical Limitations of Embedding-Based Retrieval — LIMIT (Weller et al., ICLR 2026)

• Link: https://arxiv.org/abs/2508.21038 (arXiv:2508.21038) · Code: https://github.com/google-deepmind/limit ; dataset https://huggingface.co/datasets/orionweller/LIMIT
• Tag: recent · Difficulty: 2 (theory 3) · Compute: laptop
• Summary: Uses sign-rank results to show that the number of distinct top-k document subsets a single-vector embedding of dimension d can return is bounded by d; "free embedding" experiments (optimising vectors directly against qrels) locate the critical corpus size for each d. The LIMIT dataset (50k docs, trivially simple queries like "who likes apples?") makes state-of-the-art embedders fail while sparse and multi-vector methods do much better. A 2026 rebuttal, "R^2k is Theoretically Large Enough for Embedding-based Top-k Retrieval" (Wang et al., ICML 2026, https://arxiv.org/abs/2601.20844), argues the minimal dimension is Θ(k), independent of corpus size.
• Why it matters: The theoretical framing for Matryoshka/dimensionality and for the ColBERT/SPLADE material; directly relevant to patent search with combinatorial relevance definitions.
• Pair implementation: Reproduce the free-embedding experiment on SciFact/NFCorpus qrels (smallest d that fits the qrels exactly) and on the small LIMIT split; evaluate BGE-small, Qwen3-Embedding-0.6B, BM25 and a MaxSim late-interaction scorer on LIMIT-small on a laptop.
• Discussion hooks: LIMIT vs R^2k: existence of an embedding vs learnability by a fixed encoder — who is right? Is this the strongest argument yet for hybrid retrieval?

10. Text Embeddings Reveal (Almost) As Much As Text — Vec2Text (Morris et al., EMNLP 2023)

• Link: https://aclanthology.org/2023.emnlp-main.765/ ; https://arxiv.org/abs/2310.06816 (arXiv:2310.06816) · Code: https://github.com/vec2text/vec2text
• Tag: classic · Difficulty: 2 · Compute: Colab
• Summary: Frames embedding inversion as controlled generation: a zero-step hypothesis model plus iterative correction that re-embeds and refines. Recovers 92% of 32-token inputs exactly from gtr-base, and inverts OpenAI ada-002 well enough to leak names from clinical notes; Gaussian noise on embeddings is studied as a defence.
• Why it matters: The geometric complement to LIMIT (a single vector cannot express all relevance patterns, yet carries nearly the whole text) and a concrete privacy issue for vector databases of patents or manuscripts (Questel).
• Pair implementation: Run the released gtr-base inverters on NFCorpus/SciFact abstracts and plot BLEU/exact-match vs length and vs correction steps (students know BLEU); test the noise defence against retrieval nDCG@10 on the same corpus. Train a small zero-step inverter (T5-small) for BGE-small on the 92k MS MARCO passages.
• Discussion hooks: If embeddings are invertible, are they compressed text or semantic abstractions? Does Matryoshka truncation protect privacy?

11. MMTEB: Massive Multilingual Text Embedding Benchmark (Enevoldsen et al., ICLR 2025)

• Link: https://arxiv.org/abs/2502.13595 (arXiv:2502.13595) · Code: https://github.com/embeddings-benchmark/mteb ; leaderboard https://huggingface.co/spaces/mteb/leaderboard
• Tag: recent · Difficulty: 1–2 · Compute: laptop
• Summary: Community expansion of MTEB (Muennighoff et al. 2022, https://arxiv.org/abs/2210.07316) to 500+ tasks and 250+ languages, adding instruction-following, long-document and code retrieval; introduces an inter-task-correlation downsampling that keeps model rankings while cutting compute, and the MTEB(eng, v2)/Multilingual leaderboards with a zero-shot flag. HUME (El Assadi et al. 2025, https://arxiv.org/abs/2510.10062) adds human baselines on 16 MTEB datasets: humans ~77.6% vs 80.1% for the best model, exposing dataset issues.
• Why it matters: Students used MTEB hands-on in spring; this is how the leaderboard they trust was rebuilt, and how it can be gamed.
• Pair implementation: Wrap the shared testbed as custom mteb tasks and evaluate 12–15 models spanning 2019–2026; test whether the paper's correlation-based subset selection preserves rankings (Spearman between testbed and public MTEB(eng, v2) rankings). Annotate 50 SciFact queries themselves à la HUME.
• Discussion hooks: Goodhart's law: has MTEB become training data (zero-shot flag vs "trained on everything")? What should an IR-specific leaderboard (BEIR vs BRIGHT) look like in 2026?

12. History slot: "Static embeddings, 2013 to 2025" — Train 400x Faster Static Embedding Models (Aarsen, Hugging Face blog, Jan 2025) with Model2Vec

• Link: https://huggingface.co/blog/static-embeddings ; https://github.com/MinishLab/model2vec · Code: same (sentence-transformers StaticEmbedding; models static-retrieval-mrl-en-v1, minishlab/potion-*)
• Tag: recent (non-peer-reviewed) · Difficulty: 1 · Compute: laptop
• Summary: Two 2025 results bring word2vec-style static vectors back: Aarsen trains bag-of-token-embeddings retrievers with MNRL + Matryoshka on 30 datasets, reaching 87.4% of all-mpnet-base-v2 at 397x CPU speed (multilingual: 92.3% of multilingual-e5-small at 125x); Model2Vec distils any sentence transformer into static token vectors by forwarding the vocabulary, PCA and Zipf weighting, giving models up to 50x smaller and 500x faster. Background reading for the timeline: Zhang et al. 2025 survey (https://arxiv.org/abs/2507.20783) and Kashyap et al., EACL 2024 (https://arxiv.org/abs/2305.12641).
• Why it matters: Explicitly suited to a "history of embeddings 2013–2026" talk: word2vec → SBERT → contrastive pretraining → LLM embedders → static again, answering "what did the transformer add?"
• Pair implementation: "One model per era on one testbed": GloVe/word2vec averaging, all-MiniLM (2019), Contriever (2021), E5/BGE (2022–23), LLM2Vec-Qwen or Qwen3-Embedding-0.6B (2024–25), static-retrieval-mrl (2025), plotting nDCG@10 vs release year vs CPU latency on SciFact/NFCorpus/ArguAna. Then distil BGE-small with Model2Vec and train their own static retriever on the MS MARCO subset.
• Discussion hooks: If static vectors get ~85–90% of a transformer, is contextualisation worth 400x? Is progress since 2022 mostly data rather than architecture?

Framing resources for the instructor's 30-min intro

• Zhang et al. 2025, "On The Role of Pretrained Language Models in General-Purpose Text Embeddings: A Survey" (45 pp., LLM era) — https://arxiv.org/abs/2507.20783
• Kashyap et al., "A Comprehensive Survey of Sentence Representations: From the BERT Epoch to the ChatGPT Era and Beyond", EACL 2024 — https://arxiv.org/abs/2305.12641 ; Cao 2024, review of MTEB top methods — https://arxiv.org/abs/2406.01607
• MTEB leaderboard — https://huggingface.co/spaces/mteb/leaderboard ; Aarsen, "Training and Finetuning Embedding Models with Sentence Transformers v3" (May 2024, the recipe pairs will use) — https://huggingface.co/blog/train-sentence-transformers
• Vicki Boykis, "What are embeddings?" (free PDF, historical narrative for a general audience) — https://vickiboykis.com/what_are_embeddings/

Notes

• The session's web-search budget ran out mid-task; remaining checks were done by fetching arXiv/GitHub/HF pages directly, so every URL above was fetched or seen in a search result.
• Venues not confirmed on the arXiv page: ModernBERT (listed as arXiv 2024) and Promptriever (arXiv 2024). MTEB original is cited as arXiv 2022 (published at EACL 2023).
• Static embeddings/Model2Vec have no arXiv paper (arXiv search returns nothing); item 12 is a blog + repo plus two surveys, and could be split into a standalone "static retrieval" project and a pure history talk.
• Near-misses (all verified): NV-Embed (Lee et al., ICLR 2025, https://arxiv.org/abs/2405.17428; 7B, CC-BY-NC, latent-attention pooling — good ablations but not fine-tunable by students); GritLM (https://arxiv.org/abs/2402.09906); Gecko (https://arxiv.org/abs/2403.20327, no code); EmbeddingGemma 300M (https://arxiv.org/abs/2509.20354); Nomic Embed (TMLR, https://arxiv.org/abs/2402.01613, fully open data — an alternative to Contriever if "reproducible recipe" is the goal); jina-embeddings-v3 task-LoRA (https://arxiv.org/abs/2409.10173); "Patent Language Model Pretraining with ModernBERT" (Yousefiramandi & Cooney 2025, https://arxiv.org/abs/2509.14926, classification only but relevant to the instructor's patent work); Zhou et al. 2026 chunking taxonomy (https://arxiv.org/abs/2602.16974); Bernas et al. 2026 anisotropy geometry (https://arxiv.org/abs/2604.08764).
• Overlaps with other themes: Promptriever (instruction/LLM retrieval), LIMIT (multi-vector/ColBERT, SPLADE), CDE/late chunking (RAG chunking), Vec2Text (privacy), MMTEB (evaluation).

Theme 2: Learned sparse retrieval, multi-vector retrieval, efficiency and compression

Default slot: Session 2 (Nov 13) — Representations.

Every link below was seen in a search result or fetched directly during this session.

1. SPLADE: Sparse Lexical and Expansion Model for First Stage Ranking (Formal et al., SIGIR 2021)

• Link: https://arxiv.org/abs/2107.05720 (arXiv:2107.05720) · Code: https://github.com/naver/splade ; models: https://huggingface.co/naver/splade-v3
• Tag: classic · Difficulty: 1 · Compute: laptop (inference) / Colab (fine-tuning)
• Summary: Uses BERT's MLM head to project each token onto the 30k vocabulary, applies log-saturation and a FLOPS sparsity regulariser, and max-pools to get a sparse bag-of-words with expansion terms that lives in a classical inverted index. Follow-ups: SPLADE++ "From Distillation to Hard Negative Sampling" (Formal et al., SIGIR 2022, https://arxiv.org/abs/2205.04733) adds MarginMSE distillation and hard-negative mining; SPLADE-v3 (Lassance et al., 2024, https://arxiv.org/abs/2403.06789) reaches 40.2 MRR@10 on MS MARCO dev and 51.7 nDCG@10 on BEIR-13 with 100 negatives per batch and KL+MarginMSE.
• Why it matters: The direct neural successor to BM25 and Robertson & Zaragoza; the "InfoNCE + hard negatives" recipe students saw only on slides is here applied to sparse vectors. Instructor link: swap the backbone to SciBERT/ModernBERT and study which expansion terms appear on scientific/patent text.
• Pair implementation: Load naver/splade-v3 with SparseEncoder (sentence-transformers) and reproduce BEIR SciFact/NFCorpus/ArguAna numbers; then fine-tune a small SPLADE (MiniLM or BGE-small backbone) on the MS MARCO subset on Colab, sweeping the FLOPS regulariser to plot the effectiveness-vs-index-size curve and inspecting expansion terms on the scientific-article set.
• Discussion hooks: If SPLADE "expands" documents with vocabulary terms, is it still lexical search, or dense retrieval in a 30k-dim disguise? Why does query-side expansion help so little (Nguyen et al. 2023) while document-side weighting matters most?

2. PromptReps: Prompting LLMs to Generate Dense and Sparse Representations for Zero-Shot Document Retrieval (Zhuang et al., EMNLP 2024)

• Link: https://arxiv.org/abs/2404.18424 (arXiv:2404.18424); https://aclanthology.org/2024.emnlp-main.250/ · Code: https://github.com/ielab/PromptReps
• Tag: recent · Difficulty: 2 · Compute: Colab
• Summary: No training at all: the LLM is prompted to "represent this text with one word"; the last-token hidden state gives a dense vector and the next-token logits over the vocabulary give a SPLADE-like sparse vector, fused into a hybrid retriever. On BEIR it matches or beats trained LLM embedders, with the gap growing with LLM size.
• Why it matters: A bridge between the spring course's SBERT/DPR embeddings and LLM-era retrieval; it makes learned sparse representations available with zero fine-tuning, which is what the compute budget allows.
• Pair implementation: Reproduce the dense/sparse/hybrid ablation with Qwen2.5-1.5B and 3B (4-bit) on SciFact and NFCorpus; then compare PromptReps sparse output with SPLADE-v3 on the same testbed, measuring index size (non-zeros per doc) and nDCG@10, and test prompt variations on the scientific-article set.
• Discussion hooks: Is a logit distribution over the vocabulary a principled sparse representation or a lucky artefact of the MLM/CLM head? Does scaling the LLM substitute for contrastive training?

3. CSPLADE: Learned Sparse Retrieval with Causal Language Models (Xu et al., IJCNLP-AACL 2025)

• Link: https://arxiv.org/abs/2504.10816 (arXiv:2504.10816); https://aclanthology.org/2025.ijcnlp-long.7/ · Code: none found
• Tag: recent · Difficulty: 3 · Compute: Colab
• Summary: Extends SPLADE to decoder-only LLMs and identifies two obstacles: instability early in contrastive training and the unidirectional-attention bottleneck. A lightweight adaptation phase and two bidirectional variants fix both, scaling LSR to 8B while shrinking the index, plus an analysis of model quantization trade-offs. Context: Mistral-SPLADE (Doshi et al., 2024, https://arxiv.org/abs/2408.11119, no code) and "Scaling Sparse and Dense Retrieval in Decoder-Only LLMs" (Zeng et al., 2025, https://arxiv.org/abs/2502.15526) show sparse beating dense at every scale 1B-8B.
• Why it matters: Directly extends the course's SPLADE name-drop into the LLM era and touches the instructor's interest in how the backbone shapes embedding geometry.
• Pair implementation: Reproduce the causal-vs-bidirectional ablation at small scale: LoRA-adapt Qwen2.5-0.5B/1.5B with a SPLADE head on the MS MARCO subset (Colab, small batches), compare with and without the adaptation phase, and evaluate on SciFact/NFCorpus against SPLADE-v3.
• Discussion hooks: Why would sparse retrieval scale better than dense with LLM backbones? Is an 8B sparse encoder ever deployable in an inverted index at query time?

4. Efficient Inverted Indexes for Approximate Retrieval over Learned Sparse Representations — "Seismic" (Bruch et al., SIGIR 2024, best paper runner-up)

• Link: https://arxiv.org/abs/2404.18812 (arXiv:2404.18812) · Code: https://github.com/TusKANNy/seismic (Rust, pip install pyseismic-lsr)
• Tag: recent · Difficulty: 2 · Compute: laptop
• Summary: SPLADE vectors break classical dynamic-pruning assumptions ("wacky weights"), so Seismic partitions each inverted list into geometrically cohesive blocks with a summary vector, skips blocks by summary score, and uses a forward index for exact scoring. On MS MARCO it reaches sub-millisecond query latency, one to two orders of magnitude faster than inverted-index baselines and beating the graph-based winners of the BigANN 2023 sparse track.
• Why it matters: Answers the question left open in the spring course: does SPLADE actually run fast in an inverted index? Also ties the FAISS IVF idea (clusters + summaries) to sparse vectors.
• Pair implementation: Encode the 92k MS MARCO passages with SPLADE-v3, build Seismic vs. an exact sparse dot product (scipy) vs. a plain rank_bm25-style inverted index; reproduce the recall@10-vs-latency curve, then run the same on SciFact/NFCorpus and report nDCG@10 loss at each speed setting.
• Discussion hooks: If approximate sparse retrieval is now needed, has learned sparse lost its "exactness" advantage over dense ANN? Where does the sparse/dense distinction still matter?

5. SPLATE: Sparse Late Interaction Retrieval (Formal et al., SIGIR 2024)

• Link: https://arxiv.org/abs/2404.13950 (arXiv:2404.13950); https://doi.org/10.1145/3626772.3657968 · Code: none found
• Tag: recent · Difficulty: 2 · Compute: Colab (train adapter), laptop (retrieval)
• Summary: Freezes ColBERTv2 and learns a small "MLM adapter" mapping its token embeddings to a SPLADE-style sparse vocabulary vector, so candidate generation runs in an inverted index and only the final MaxSim scoring needs the token vectors. Re-ranking just 50 candidates retrieved in under 10 ms on CPU matches PLAID's effectiveness.
• Why it matters: Joins the course's two conceptual threads (SPLADE and ColBERT/PLAID) into one CPU-friendly pipeline; an alternative to the BM25 -> dense -> RRF -> cross-encoder baseline.
• Pair implementation: Reproduce the "sparse candidates + MaxSim rerank" idea using answerai-colbert-small-v1 (or ColBERTv2) token embeddings and a learned adapter trained on the MS MARCO subset; compare against BM25-candidates + MaxSim and against PLAID-style retrieval on SciFact/ArguAna, reporting nDCG@10 vs. candidate depth.
• Discussion hooks: If BM25 candidates + MaxSim rerank is nearly as good, what exactly does the learned sparse adapter buy? Is late interaction becoming a reranker rather than a retriever?

6. WARP: An Efficient Engine for Multi-Vector Retrieval (Scheerer et al., SIGIR 2025) with XTR as prerequisite

• Link: https://arxiv.org/abs/2501.17788 (arXiv:2501.17788) · Code: https://github.com/jlscheerer/xtr-warp (CPU retrieval supported)
• Prerequisite: XTR, "Rethinking the Role of Token Retrieval in Multi-Vector Retrieval" (Lee et al., NeurIPS 2023, https://arxiv.org/abs/2304.01982, code https://github.com/google-deepmind/xtr)
• Tag: recent (XTR: classic) · Difficulty: 3 · Compute: Colab (index) + laptop (query)
• Summary: XTR trains ColBERT so that token retrieval alone is sufficient and imputes missing similarities, making scoring 2-3 orders of magnitude cheaper (+2.8 nDCG@10 on BEIR without distillation). WARP builds an engine for XTR/ColBERT with dynamic similarity imputation (WARP-SELECT), implicit decompression, and two-stage reduction: 41x lower latency than XTR's reference implementation, 3x faster than PLAID. A 2026 replicability study (Jha et al., https://arxiv.org/abs/2605.00646) confirms XTR's token-matching claims but not its effectiveness advantage over ColBERT under controlled training.
• Why it matters: Continues ColBERTv2/PLAID from the spring course; WARP is the current state of the art engine, and the replicability study is a ready-made discussant angle.
• Pair implementation: Index SciFact and NFCorpus with WARP (Colab for index build), then on a laptop measure latency vs. nDCG@10 for varying k' (tokens retrieved) and compare against exact MaxSim over all tokens; replicate the XTR "score imputation" ablation on the MS MARCO subset.
• Discussion hooks: Is the multi-vector speed problem solved by better engines or by better training objectives? What does the failed replication tell us about how we evaluate retrieval papers?

7. MUVERA: Multi-Vector Retrieval via Fixed Dimensional Encodings (Dhulipala et al., NeurIPS 2024)

• Link: https://arxiv.org/abs/2405.19504 (arXiv:2405.19504); https://papers.nips.cc/paper_files/paper/2024/hash/b71cfefae46909178603b5bc6c11d3ae-Abstract-Conference.html · Code: C++ https://github.com/google/graph-mining/tree/main/sketching/point_cloud ; Python port https://github.com/sionic-ai/muvera-py
• Tag: recent · Difficulty: 2 · Compute: laptop
• Summary: Randomly partitions embedding space (SimHash-style) and sums/averages token vectors per bucket, asymmetrically for queries and documents, producing a single fixed-dimensional vector whose inner product approximates the Chamfer/MaxSim score with epsilon guarantees. It retrieves 2-5x fewer candidates than PLAID heuristics for the same recall and on BEIR gives ~10% higher recall at 90% lower latency. Adopted in Weaviate 1.31 (blog: ~70% memory reduction, 3-6x faster import, some recall loss).
• Why it matters: Reduces multi-vector search to the FAISS single-vector search the students already master; the geometric argument (space partitioning) links to the instructor's embedding-geometry interest.
• Pair implementation: Implement FDEs in numpy on answerai-colbert-small-v1 token embeddings for SciFact/NFCorpus/ArguAna; search FDEs with FAISS, re-rank top-k with exact MaxSim, and reproduce the recall-vs-candidates curve against brute-force MaxSim; vary the number of partitions/repetitions to show the dimension-vs-recall trade-off.
• Discussion hooks: If a single vector can approximate MaxSim, why train multi-vector models at all? Where does the approximation break (long documents, many query tokens)?

8. Efficient Constant-Space Multi-Vector Retrieval — "ConstBERT" (MacAvaney et al., ECIR 2025, best short paper honourable mention)

• Link: https://arxiv.org/abs/2504.01818 (arXiv:2504.01818); https://link.springer.com/chapter/10.1007/978-3-031-88714-7_22 · Code: https://github.com/pisa-engine/ConstBERT ; model https://huggingface.co/pinecone/ConstBERT
• Tag: recent · Difficulty: 2 · Compute: laptop (inference) / Colab (fine-tune)
• Summary: Instead of one vector per token, learns a projection to a fixed number (32) of vectors per document, giving predictable storage and about 2x compression vs. ColBERTv2 with near-equal effectiveness. Companion: "Towards Lossless Token Pruning in Late-Interaction Retrieval Models" (Zong & Piwowarski, SIGIR 2025, https://arxiv.org/abs/2504.12778) keeps only 30% of tokens with regularisation losses. A SIGIR 2026 reproduction (Ghosh et al., https://arxiv.org/abs/2604.09982) reproduces MS MARCO within 0.05% MRR@10 but finds 86-97% degradation on long TREC ToT queries for both ConstBERT and ColBERTv2.
• Why it matters: The Matryoshka idea ("fixed representation budget") applied to multi-vector models; relevant to the instructor's full-text scientific and patent documents, where per-token storage explodes.
• Pair implementation: Reproduce ConstBERT vs. ColBERTv2 (or answerai-colbert-small) on SciFact/NFCorpus with index size and nDCG@10; then fine-tune ConstBERT-style pooling with k in {8, 16, 32} on the MS MARCO subset (Colab) and test on the scientific-article set, adding the long-query stress test from the reproduction paper.
• Discussion hooks: Do 32 learned vectors capture "aspects" of a document, and can we interpret them? Why do both models collapse on queries longer than about 20 words?

9. Fantastic (small) Retrievers and How to Train Them: mxbai-edge-colbert-v0 Tech Report (Takehi, Clavié et al., arXiv 2025) with answerai-colbert-small

• Link: https://arxiv.org/abs/2510.14880 (arXiv:2510.14880); blog "Small but Mighty" https://www.answer.ai/posts/2024-08-13-small-but-mighty-colbert.html · Code: https://huggingface.co/answerdotai/answerai-colbert-small-v1 ; https://github.com/answerdotai/ragatouille
• Tag: recent · Difficulty: 2 · Compute: Colab
• Summary: answerai-colbert-small-v1 (33M params) distils from BGE-M3 and Gemma2-lightweight rerankers on ~3.8M triplets and scores 53.79 BEIR average vs. 50.02 for ColBERTv2 (110M) and 53.25 for bge-base. The mxbai-edge-colbert-v0 report (17M and 32M variants) gives extensive ablations of this recipe and surpasses ColBERTv2 on BEIR with strong long-context efficiency.
• Why it matters: The only theme entry where students train a late-interaction model themselves; it reuses the course's cross-encoder (ms-marco-MiniLM) as teacher and MiniLM/BGE-small as student, exactly the models in the baseline pipeline.
• Pair implementation: Distil a MiniLM-sized ColBERT on the MS MARCO subset using teacher scores from the course cross-encoder (Colab, small batches); reproduce one ablation from the report (e.g. teacher choice or projection dimension) and compare with answerai-colbert-small and the dense baseline on SciFact/NFCorpus/ArguAna.
• Discussion hooks: Why does a 33M multi-vector model beat 110M single-vector models: architecture or distillation data? Is "BEIR average" still a meaningful target when models are trained on BEIR-adjacent data?

10. RaBitQ: Quantizing High-Dimensional Vectors with a Theoretical Error Bound for ANN Search (Gao & Long, SIGMOD 2024)

• Link: https://arxiv.org/abs/2405.12497 (arXiv:2405.12497); https://dl.acm.org/doi/abs/10.1145/3654970 · Code: https://github.com/VectorDB-NTU/RaBitQ-Library (pip install rabitqlib, IVF and HNSW backends)
• Tag: recent (the PQ-successor classic) · Difficulty: 2 · Compute: laptop
• Summary: Randomly rotates normalised vectors and keeps one bit per dimension, but derives an unbiased distance estimator with a provable, asymptotically optimal error bound, unlike naive sign binarisation. It underlies Elastic's "Better Binary Quantization" in Lucene/Elasticsearch 8.16 (blog: ~95% memory reduction, 90% recall with 3x oversampling on 1M Cohere v3 vectors; https://www.elastic.co/search-labs/blog/better-binary-quantization-lucene-elasticsearch). The practitioner baseline is the Hugging Face binary/int8 quantization post (Shakir, Aarsen, Lee 2024, https://huggingface.co/blog/embedding-quantization): 32x memory, ~25x speed, ~96% performance retained with float rescoring.
• Why it matters: Extends FAISS PQ from the spring course with a theory-first alternative that industry actually adopted within months.
• Pair implementation: Implement 1-bit RaBitQ in numpy (rotation, sign codes, norm, estimator) over BGE-small embeddings of the 92k MS MARCO passages; reproduce the estimator's error-vs-dimension plot and compare recall@100 and nDCG@10 against naive binary, int8 and FAISS PQ, with and without float rescoring, on SciFact/NFCorpus.
• Discussion hooks: Why does a random rotation turn a crude 1-bit code into a principled estimator? When does the rescoring step hide most of the quantisation loss, and what does that imply for cost-per-query?

11. Down with the Hierarchy: The 'H' in HNSW Stands for "Hubs" (Munyampirwa et al., ECIR 2026)

• Link: https://arxiv.org/abs/2412.01940 (arXiv:2412.01940); https://link.springer.com/chapter/10.1007/978-3-032-21324-2_3 · Code: https://github.com/BlaiseMuhirwa/flatnav (pip install flatnav)
• Tag: recent (HNSW analysis) · Difficulty: 2 · Compute: laptop
• Summary: Benchmarks show that a flat navigable-small-world graph matches HNSW's latency and recall on high-dimensional embeddings with ~38% less peak memory. The "Hub Highway Hypothesis": in high dimensions a few hub nodes form a well-travelled highway that plays the role of the hierarchy. Companion: "Breaking the Curse of Dimensionality: On the Stability of Modern Vector Retrieval" (Lakshman et al., 2025/26, https://arxiv.org/abs/2512.12458) explains via stability theory why ANN works for multi-vector, filtered and sparse search.
• Why it matters: Revisits the FAISS HNSW index students used hands-on, and hubness is an embedding-geometry phenomenon (related to the alignment/uniformity analysis in the instructor's S2Full paper and to over-dispersion in his patent work).
• Pair implementation: Reproduce the flat-vs-hierarchical comparison with flatnav vs. hnswlib/FAISS HNSW on the 92k MS MARCO embeddings (recall@10 vs. QPS, memory); then measure hubness (k-occurrence skew) of BGE-small vs. SPECTER2 embeddings on the scientific-article set and test whether high-hubness embeddings navigate faster.
• Discussion hooks: Is hubness a bug of the embedding model or a feature the index exploits? Should we train embeddings for retrievability (graph navigability) rather than only for relevance?

12. ACORN: Performant and Predicate-Agnostic Search Over Vector Embeddings and Structured Data (Patel et al., SIGMOD 2024)

• Link: https://arxiv.org/abs/2403.04871 (arXiv:2403.04871); https://dl.acm.org/doi/10.1145/3654923 · Code: https://github.com/stanford-futuredata/ACORN (FAISS-based C++)
• Tag: recent · Difficulty: 3 · Compute: laptop
• Summary: Filtered vector search (embedding similarity plus metadata predicates) breaks HNSW because filtering disconnects the graph. ACORN builds a denser HNSW with predicate-agnostic neighbour expansion during search, achieving 2-10x higher QPS than prior methods on existing benchmarks, over 30x on new ones and over 1000x at 25M vectors. Context: the filtered track of the Big-ANN NeurIPS'23 competition (Simhadri et al., https://arxiv.org/abs/2409.17424).
• Why it matters: Every real system (patent search by IPC class and date, scholarly search by year/venue) is filtered search; the students' FAISS HNSW knowledge shows exactly why pre- and post-filtering fail.
• Pair implementation: Build metadata (year, venue/journal) for the scientific-article set and NFCorpus, embed with BGE-small, and compare pre-filtering, post-filtering, and ACORN (or a Python reimplementation of predicate-agnostic expansion over hnswlib) on recall@10 vs. QPS as filter selectivity varies from 50% to 0.1%.
• Discussion hooks: Should temporal or categorical constraints live in the index, in the query, or in the embedding itself (the instructor's "timelines on embeddings")? Is hybrid structured+vector search a database problem or an IR problem?

Framing resources for the instructor's 30-min intro

• Bruch, Foundations of Vector Retrieval (monograph, 2024): https://arxiv.org/abs/2401.09350 — chapters on quantisation, graph indexes and sparse MIPS.
• ECIR 2026 half-day tutorial "Neural Lexical Search with Learned Sparse Retrieval" (Yates, Lassance, Rulli, Yang, MacAvaney, Singh, Nguyen, Lei): https://link.springer.com/chapter/10.1007/978-3-032-21321-1_5 ; plus the unifying LSR framework paper (Nguyen, MacAvaney, Yates, ECIR 2023): https://arxiv.org/abs/2303.13416.
• Lin, "Operational Advice for Dense and Sparse Retrievers: HNSW, Flat, or Inverted Indexes?" (2024): https://arxiv.org/abs/2409.06464 and "Vector Search with OpenAI Embeddings: Lucene Is All You Need" (2023): https://arxiv.org/abs/2308.14963 — the cost-per-query, "do you need a vector DB?" view.
• Leaderboards and industry adoption: https://ann-benchmarks.com/ ; Big-ANN NeurIPS'23 results https://arxiv.org/abs/2409.17424 ; Elastic ELSER https://www.elastic.co/search-labs/blog/elastic-learned-sparse-encoder-elser-retrieval-performance ; Vespa long-context ColBERT with int8 token vectors https://blog.vespa.ai/announcing-long-context-colbert-in-vespa/ ; Weaviate MUVERA https://weaviate.io/blog/muvera.

Notes

• Web search budget ran out mid-task; all later verification was done by fetching arXiv/GitHub pages directly. Every URL above was fetched or appeared in a search result.
• Near-misses dropped from the main list: DiskANN (Subramanya et al., NeurIPS 2019, https://neurips.cc/virtual/2019/poster/14308) and ScaNN (Guo et al., https://arxiv.org/abs/1908.10396) are canonical but billion-scale/SSD-oriented, hard to reproduce meaningfully on a laptop; TurboQuant (Zandieh et al., 2025, https://arxiv.org/abs/2504.19874) is a strong RaBitQ alternative but is framed around KV-cache; Jina-ColBERT-v2 (Jha et al., MRL 2024, https://arxiv.org/abs/2408.16672) is a solid model release but less teachable than the small-ColBERT report; LACONIC (Xu et al., SIGIR 2026, https://arxiv.org/abs/2601.01684; 8B Llama LSR, 60.2 nDCG@10 MTEB, 74% smaller index than dense) and the Voronoi token-pruning paper (Kankanampati et al., SIGIR 2026, https://arxiv.org/abs/2603.09933) are the most recent follow-ups to entries 3 and 8.
• Code caveats: CSPLADE and Mistral-SPLADE have no public code found; SPLATE has no standalone repo found (the NAVER splade repo may contain it, unverified). WARP index building really wants a GPU.
• Overlaps with other themes: PromptReps overlaps with any "LLM embeddings" theme; the ConstBERT reproduction overlaps with an "evaluation/reproducibility" theme; ColPali-style visual multi-vector belongs elsewhere. "Decoding Dense Embeddings: Sparse Autoencoders for Interpreting and Discretizing Dense Retrieval" (Park et al., EMNLP 2025, https://arxiv.org/abs/2506.00041) sits between this theme and interpretability and connects to the instructor's embedding-geometry interest.
• Licence note: SPLADE-v3 weights are CC-BY-NC-SA-4.0, fine for coursework.

Theme 3: Scholarly document retrieval and representation

Default slot: Session 3 (Nov 20) — Domains.

Scholarly document retrieval and representation — candidate papers

1. Neighborhood Contrastive Learning for Scientific Document Representations with Citation Embeddings — SciNCL (Ostendorff et al., EMNLP 2022)

• Link: https://arxiv.org/abs/2202.06671 (arXiv:2202.06671) · Code: https://github.com/malteos/scincl (model: https://huggingface.co/malteos/scincl)
• Tag: classic · Difficulty: 2 · Compute: Colab
• Summary: SPECTER's hard citation cut-off (cited = positive, not cited = negative) ignores that uncited papers can be near-identical. SciNCL first embeds the S2ORC citation graph (node embeddings), then samples positives from the k-nearest neighbours (k≈20-25) and hard negatives from a controlled band further out (k≈4000), with a margin that avoids positive/negative collisions. A SciBERT fine-tuned this way with a triplet loss beats SPECTER on SciDocs.
• Why it matters / link to the course: Direct successor to SPECTER (known) and the cleanest illustration of "how you mine positives/negatives" for the InfoNCE/hard-negative concepts the students saw but never implemented; it is also the baseline family the instructor's S2Full paper competes with.
• Pair implementation: Reproduce on a small citation graph (e.g., a 20-50k-paper S2ORC/peS2o slice): compute node2vec-style graph embeddings, mine neighbourhood positives/negatives, fine-tune BGE-small or MiniLM with sentence-transformers' MNRL/triplet loss. Apply: compare SPECTER2, SciNCL and the pair's model on the small scientific-article testbed and SciFact (nDCG@10).
• Discussion hooks: Is a citation-graph embedding a "teacher" that leaks graph structure into text, and does that generalise to new (uncited) papers? Why does the sampling margin matter more than the loss function?

2. Multi-Vector Models with Textual Guidance for Fine-Grained Scientific Document Similarity — ASPIRE (Mysore et al., NAACL 2022)

• Link: https://arxiv.org/abs/2111.08366 (arXiv:2111.08366) · Code: https://github.com/allenai/aspire (models allenai/aspire-contextualsentence-multim-compsci, -biomed, plus bi-encoder variants)
• Tag: classic · Difficulty: 3 · Compute: Colab
• Summary: Represents a paper by several sentence-level "aspect" vectors instead of one CLS vector, and trains with a new supervision signal: sentences in full texts that co-cite two papers together describe how they are related. Matching is either single-aspect (fast) or sparse multi-aspect via Optimal Transport (Earth Mover's Distance). Improves faceted similarity on four datasets including CSFCube.
• Why it matters / link to the course: Bridges ColBERT-style multi-vector retrieval (taught conceptually) with scientific-document similarity, and exploits full-text citation contexts, the same resource the instructor mines in S2Full.
• Pair implementation: Reproduce with the released models on CSFCube (small, faceted). Apply: run the aspire bi-encoder vs. SPECTER2 on the scientific-article testbed and SciFact; optionally train a MiniLM multi-vector variant on co-citation sentences extracted from a peS2o sample.
• Discussion hooks: Do single-vector paper embeddings fundamentally conflate background/method/result facets? Is OT matching worth its cost compared to ColBERT's MaxSim?

3. SemCSE: Semantic Contrastive Sentence Embeddings Using LLM-Generated Summaries for Scientific Abstracts (Brinner & Zarriess, arXiv 2025)

• Link: https://arxiv.org/abs/2507.13105 (arXiv:2507.13105) · Code: https://github.com/inas-argumentation/SemCSE
• Tag: recent · Difficulty: 2 · Compute: Colab
• Summary: Unsupervised alternative to citation supervision: Llama-3-8B generates several short summaries of each abstract (prompted continuations such as "A comprehensive summary of our work would be that..."); summaries of the same paper are positives, others negatives, and a SciDeBERTa is trained contrastively. Achieves the second-best overall SciRepEval average (65.76), outperforming domain-specific models, and the authors introduce a benchmark isolating "semantic" from citation-based similarity.
• Why it matters / link to the course: Exactly the "views of the same document" self-supervision the instructor uses for patents (section views) and papers (full-text windows + self-alignment), with LLM paraphrases as the augmentation; a natural foil to SimCSE-dropout over-dispersion.
• Pair implementation: Reproduce at small scale: generate 2-3 summaries per abstract for ~10k SciFact/NFCorpus/testbed documents with Qwen2.5-3B (4-bit, Colab), fine-tune BGE-small with CachedMNRL. Apply: evaluate on the scientific-article testbed and SciFact, and plot alignment/uniformity vs. the SimCSE-dropout baseline.
• Discussion hooks: Are LLM summaries "free" supervision or a distillation of the LLM's own biases? Should citation similarity and semantic similarity be evaluated as two different tasks?

4. LitSearch: A Retrieval Benchmark for Scientific Literature Search (Ajith et al., EMNLP 2024)

• Link: https://arxiv.org/abs/2407.18940 (arXiv:2407.18940) · https://aclanthology.org/2024.emnlp-main.840/ · Code: https://github.com/princeton-nlp/LitSearch
• Tag: recent · Difficulty: 1 · Compute: laptop
• Summary: 597 realistic literature-search questions about recent ML/NLP papers, built from GPT-4-generated questions over inline-citation paragraphs plus author-written questions, all expert-checked. Dense retrievers beat BM25 by 24.8 points absolute recall@5, and Google Search/commercial tools trail the best dense retriever by up to 32 recall points.
• Why it matters / link to the course: A BEIR-style benchmark students can load in an hour, exposing where the spring pipeline (BM25 -> dense -> RRF -> cross-encoder) breaks on real research questions.
• Pair implementation: Reproduce the BM25 vs. dense gap with BGE-small/all-MiniLM. Apply: run the full spring pipeline plus LLM query rewriting (HyDE-style with Qwen2.5-3B) on LitSearch and on the scientific-article testbed; report recall@5/nDCG@10 per query type (inline-citation vs. author-written).
• Discussion hooks: Are GPT-4-generated queries a benchmark of the retriever or of GPT-4's phrasing? Why do commercial engines lose to a 110M-parameter dense model here?

5. DORIS-MAE: Scientific Document Retrieval using Multi-level Aspect-based Queries (Wang et al., NeurIPS 2023 Datasets & Benchmarks)

• Link: https://arxiv.org/abs/2310.04678 (arXiv:2310.04678) · Code: https://github.com/Real-Doris-Mae/Doris-Mae-Dataset
• Tag: recent · Difficulty: 2 · Compute: Colab + small API budget (optional)
• Summary: 100 long, human-written research queries (a paragraph describing a project idea), decomposed into 3-9 aspects and sub-aspects, over 363k CS abstracts. Relevance labels are produced by Anno-GPT, a validated LLM-annotation framework claimed to cut cost 500x; 17 retrieval methods degrade sharply compared with standard datasets.
• Why it matters / link to the course: Concrete case of LLM-as-a-judge (name-dropped) for building graded relevance, and of multi-aspect queries that single-vector retrievers handle poorly.
• Pair implementation: Reproduce a subset of the 17-method table with BM25, MiniLM/BGE and a cross-encoder on a reduced corpus. Apply: aspect decomposition with a small LLM, per-aspect retrieval, RRF fusion; test the same decomposition on the scientific-article testbed and re-judge 50 testbed query-document pairs with an LLM to measure agreement with human labels.
• Discussion hooks: How much of the "difficulty" is query length versus true multi-aspect need? Would you trust Anno-GPT labels to build the next BEIR?

6. OpenScholar: Synthesizing Scientific Literature with Retrieval-augmented LMs (Asai et al., arXiv 2024; Nature 2026)

• Link: https://arxiv.org/abs/2411.14199 (arXiv:2411.14199); journal version https://www.nature.com/articles/s41586-025-10072-4 · Code: https://github.com/AkariAsai/OpenScholar · benchmark https://github.com/AkariAsai/ScholarQABench
• Tag: recent · Difficulty: 2 · Compute: Colab
• Summary: A retrieval-augmented pipeline over 45M open-access papers: dense passage retriever, reranker, an 8B Llama-3.1 generator trained on self-generated feedback loops, and iterative citation-checked refinement. Introduces ScholarQABench (2,967 expert queries, 208 long-form answers across CS, physics, neuroscience, biomedicine); OpenScholar-8B beats GPT-4o by 6.1% and PaperQA2 (Skarlinski et al., arXiv:2409.13740, code https://github.com/Future-House/paper-qa) by 5.5% on correctness, and GPT-4o alone hallucinates citations frequently.
• Why it matters / link to the course: Extends the mini-RAG and "Seven failure points of RAG" material to scientific literature; the released reranker/retriever/8B model are usable in 4-bit on a T4.
• Pair implementation: Reproduce citation-accuracy evaluation on the ScholarQABench SciFact/PubMedQA short-form tasks with OpenScholar-8B vs. a vanilla Qwen/Llama RAG. Apply: build the OpenScholar-style retrieve-rerank-generate-verify loop on NFCorpus/SciFact as the corpus and measure citation precision.
• Discussion hooks: Is "citation accuracy" the right proxy for scientific correctness? Does the self-feedback loop generalise beyond the domains it was trained on?

7. PaSa: An LLM Agent for Comprehensive Academic Paper Search (He et al., ACL 2025)

• Link: https://arxiv.org/abs/2501.10120 (arXiv:2501.10120) · https://aclanthology.org/2025.acl-long.572/ · Code: https://github.com/bytedance/pasa (checkpoints bytedance-research/pasa-7b-crawler, pasa-7b-selector; data CarlanLark/pasa-dataset)
• Tag: recent · Difficulty: 3 · Compute: Colab
• Summary: Two-agent "deep research" search: a Crawler issues queries and expands the citation graph of promising hits; a Selector filters candidates. Both are 7B LLMs trained with RL on AutoScholarQuery (35k fine-grained queries from top AI conferences); evaluated on the human-written RealScholarQuery benchmark, where PaSa substantially outperforms Google, Google Scholar and GPT-4o search baselines on recall.
• Why it matters / link to the course: The agentic counterpart of "Rethinking search" (Metzler et al., name-dropped): retrieval as a sequence of decisions over a citation graph rather than one similarity lookup.
• Pair implementation: Reproduce recall on RealScholarQuery (50 queries) with the released 7B selector in 4-bit plus the Semantic Scholar API as the crawler backend. Apply: a lightweight agent (Qwen2.5-3B, citation-expansion over the testbed's reference lists or over SciFact's corpus) versus the static BM25+dense baseline.
• Discussion hooks: When is citation-graph expansion better than a better embedding? How do we evaluate "comprehensiveness" when the relevant set is unknown?

8. CiteME: Can Language Models Accurately Cite Scientific Claims? (Press et al., NeurIPS 2024 Datasets & Benchmarks)

• Link: https://arxiv.org/abs/2407.12861 (arXiv:2407.12861) · Code: https://github.com/bethgelab/CiteME
• Tag: recent · Difficulty: 2 · Compute: Colab + small API budget (optional)
• Summary: Given a text excerpt from an ML paper with one citation masked, identify the cited paper. Frontier LMs reach only 4.2-18.5% accuracy versus 69.7% for humans; CiteAgent (GPT-4o with search and reading tools) reaches 35.3%. This is local citation recommendation reframed as an LLM benchmark; the classic two-stage neural baseline is Gu et al., ECIR 2022 (HAtten prefetch + SciBERT rerank, arXiv:2112.01206, code https://github.com/nianlonggu/Local-Citation-Recommendation).
• Why it matters / link to the course: Citation contexts are the supervision signal behind SPECTER/SciNCL/S2Full; CiteME asks whether the same signal can be inverted into retrieval at inference time, and directly tests the cross-encoder reranking students already know.
• Pair implementation: Reproduce the Gu et al. pipeline (BM25/MiniLM prefetch + MiniLM cross-encoder rerank) on ACL-200 or on CiteME excerpts against a Semantic Scholar-derived candidate pool. Apply: treat citation contexts extracted from the scientific-article testbed as queries and measure recall@k against cited items.
• Discussion hooks: Why do parametric LMs fail so badly at attribution while small retrievers succeed? Should citation recommendation optimise for the paper the author cited, or for the best paper?

9. SciFact-Open: Towards Open-Domain Scientific Claim Verification (Wadden et al., Findings of EMNLP 2022)

• Link: https://arxiv.org/abs/2210.13777 (arXiv:2210.13777) · https://aclanthology.org/2022.findings-emnlp.347/ · Code: https://github.com/dwadden/scifact-open
• Tag: classic · Difficulty: 2 · Compute: Colab
• Summary: SciFact (the testbed dataset) only verifies claims against a 5k-abstract corpus. SciFact-Open re-annotates claims against a 500k-abstract corpus with pooled system outputs; systems trained on SciFact drop by at least 15 F1, and the new evidence exposes phenomena such as evidence supporting only a special case of the claim.
• Why it matters / link to the course: Turns the testbed's SciFact into a retrieval-at-scale study, showing how BEIR-style closed-corpus numbers can overstate real performance.
• Pair implementation: Reproduce the closed-vs-open recall drop using SciFact as the closed corpus and a larger PubMed/peS2o-derived abstract pool as distractors, with BM25, BGE-small and the MiniLM cross-encoder. Apply: measure how SciFact nDCG@10 in the testbed shifts as distractors are added, and check LLM (Qwen2.5-7B, 4-bit) veracity labels against gold.
• Discussion hooks: Is pooling-based annotation biased toward the systems that built the pool? Does the "special case" evidence problem mean binary relevance is the wrong target for claim verification?

10. A Gold Standard Dataset for the Reviewer Assignment Problem (Stelmakh et al., arXiv 2023)

• Link: https://arxiv.org/abs/2303.16750 (arXiv:2303.16750) · Code: https://github.com/niharshah/goldstandard-reviewer-paper-match
• Tag: recent · Difficulty: 1 · Compute: laptop
• Summary: 477 self-reported expertise scores from 58 researchers on papers they had read, used to evaluate paper-reviewer similarity algorithms (TPMS, ELMo, SPECTER, SPECTER+MFR). Error rates for ordering two papers by relevance to a reviewer range from 12-30% (easy) to 36-43% (hard), with big gaps between methods. Recent follow-ups: Chain-of-Factors (Zhang et al., WWW 2025, arXiv:2310.14483, code https://github.com/yuzhimanhua/CoF) and RATE/LR-bench (Liu et al., arXiv:2601.19637, code https://github.com/Gnociew/RATE-Reviewer-Assignment), which fine-tunes Qwen3-Embedding on keyword profiles and reports 77.41% vs. 75.17% for SPECTER2.
• Why it matters / link to the course: A real-world use of SPECTER-family embeddings where the "query" is a person's publication set; small enough to be fully re-run on a laptop.
• Pair implementation: Reproduce the SPECTER vs. TPMS comparison from the released data. Apply: build reviewer profiles from author publication lists in the scientific-article testbed (or LR-bench) and compare SPECTER2, BGE-small, and a max-pooled multi-paper profile; report pairwise ordering error.
• Discussion hooks: Should expertise be an embedding of a person at all? How should recency and topic drift in a reviewer's publications be weighted?

11. Mapping the changing structure of science through diachronic periodical embeddings (Lyu & Ke, Chaos, Solitons & Fractals 2025)

• Link: https://arxiv.org/abs/2503.23285 (arXiv:2503.23285) · Code: https://github.com/netknowledge/diachronic-p2v (MAG snapshot on Zenodo)
• Tag: recent · Difficulty: 2 · Compute: laptop
• Summary: Builds a separate citation network per decade from MAG (93M papers, 53k periodicals, 1950 onward), runs random walks and word2vec (SGNS, d=100), and studies semantic change of journals across decades via local k-NN overlap and orthogonal-Procrustes global alignment. Finds an overall drift toward specialisation, growing interdisciplinarity of bioscience journals, and detects emerging fields (AIDS, nanotechnology) as new local clusters. Complementary dataset: SciEvo, 2M papers over 30 years for temporal scientometrics (Jin et al., arXiv:2410.09510, https://github.com/Ahren09/SciEvo).
• Why it matters / link to the course: The instructor's "timelines on embeddings": diachronic embedding alignment, neighbourhood stability and drift metrics, applied to scientific corpora rather than words.
• Pair implementation: Reproduce the local/global change measures on a small citation network (a SciEvo slice or the testbed's references) using node2vec/gensim. Apply: embed the testbed's papers by year with BGE-small, align yearly spaces with Procrustes, and report per-topic drift and neighbour-turnover; discuss what this implies for retrievers trained on old corpora.
• Discussion hooks: Is Procrustes alignment measuring science or measuring the embedding's own instability? Should retrieval systems be time-aware (query date vs. document date)?

12. Measuring novelty in science with word embedding (Shibayama et al., PLOS ONE 2021)

• Link: https://doi.org/10.1371/journal.pone.0254034 (https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0254034) · Code: none found
• Tag: classic · Difficulty: 1 · Compute: laptop
• Summary: A paper is novel if it cites semantically distant references: each reference is embedded from its text, pairwise distances among a paper's references are aggregated, and the resulting score is validated against an established bibliometric distance and shown to predict future citations. Recent embedding-based successors: the Relative Neighbor Density novelty algorithm (Wang et al., arXiv:2503.01508, AUROC 0.82 in CS) and an axiomatic benchmark for novelty metrics with released code (Liu & Zhai, arXiv:2604.15145), which finds embedding metrics handle surface but not conceptual redundancy.
• Why it matters / link to the course: Science-of-science use of document embeddings and a direct measure of embedding-space geometry (dispersion of a reference set), which ties in with the alignment/uniformity analysis of S2Full.
• Pair implementation: Reproduce the reference-dispersion score with BGE-small/SPECTER2 on a citation-linked sample (SciEvo or S2ORC slice) and correlate with citation counts. Apply: compute novelty/dispersion for the scientific-article testbed and check whether retrievers systematically under-rank "novel" documents.
• Discussion hooks: Is embedding dispersion novelty, or just interdisciplinarity? Does over-dispersion of an embedding model (the instructor's SimCSE finding) inflate novelty scores?

Framing resources for the instructor's 30-min intro

• SciRepEval / SPECTER2 (Singh et al., EMNLP 2023): https://arxiv.org/abs/2211.13308 and the Ai2 blog https://allenai.org/blog/specter2-adapting-scientific-document-embeddings-to-multiple-fields-and-task-formats-c95686c06567 — includes the critique of SciDocs (tasks too easy, correlated, noisy recommendation split).
• peS2o (Soldaini & Lo, Ai2 2023): https://github.com/allenai/peS2o and https://huggingface.co/datasets/allenai/peS2o — ~40M open-access papers (s2orc full text, s2ag abstracts), the natural source for student slices.
• Ai2 Scholar QA (Singh et al., ACL 2025 demo): https://arxiv.org/abs/2504.10861, code https://github.com/allenai/ai2-scholarqa-lib — a deployed reference architecture to show live in the intro.
• A review on the novelty measurements of academic papers (Zhao & Zhang, Scientometrics 2025): https://arxiv.org/abs/2501.17456 — for the science-of-science slot.

Notes

• CoSAEmb (Singh & Singh, SDP 2024, https://aclanthology.org/2024.sdp-1.27/) trains section-aware aspect embeddings on full papers with triplet loss and margin gradation, closely mirroring the instructor's section-view augmentation; I dropped it only because no code repository was found, but it is worth reading as related work.
• "Pooling and Semantic Shift: The Fundamental Challenges in Long Text Embedding and Retrieval" (Gao et al., arXiv:2603.21437, 2026) argues that pooling over semantically diverse long text causes embedding collapse; relevant theory for the full-text-window design in S2Full, no code found.
• "Still Fresh? Evaluating Temporal Drift in Retrieval Benchmarks" (Kuissi et al., arXiv:2603.04532, 2026, code https://github.com/fresh-stack/driftbench) studies temporal drift but on technical-documentation corpora, not science; a possible add-on to entry 11.
• AutoResearchBench (arXiv:2604.25256, 2026) and ScholarGym (arXiv:2601.21654) are newer deep-research-for-science benchmarks; I did not verify authors, so they are not listed as entries.
• No "science embeddings on ModernBERT" paper with public models surfaced beyond BioClinical ModernBERT (arXiv:2506.10896, clinical domain); the instructor's own S2Full work seems to be the reference here.
• Overlaps: BRIGHT (arXiv:2407.12883) science subsets and long-context encoders belong to the other agent's theme; PaperQA2 and Gu et al. 2022 are folded into entries 6 and 8. The web-search budget ran out near the end, so code links were verified by fetching the repositories directly.

Theme 4: Patent retrieval, patent representation learning, prior-art search

Default slot: Session 3 (Nov 20) — Domains.

1. PaECTER: Patent-level Representation Learning using Citation-informed Transformers (Ghosh et al., arXiv 2024)

• Link: https://arxiv.org/abs/2402.19411 (arXiv:2402.19411) · Code: model https://huggingface.co/mpi-inno-comp/paecter, data https://huggingface.co/datasets/mpi-inno-comp/paecter_dataset
• Tag: recent · Difficulty: 2 · Compute: Colab
• Summary: Fine-tunes Google's BERT for Patents with a triplet loss whose positives are examiner-added citations (the patent analogue of SPECTER's citation signal). The resulting 0.3B-parameter, 1024-dim encoder beats BERT for Patents and general embedders (E5, GTE, BGE) on citation-prediction ranking. It is the de facto open patent encoder and the baseline the instructor's own work compares against.
• Why it matters: Direct transfer of SPECTER/SPECTER2 (spring course) to patents; the citation-contrastive recipe is the same as in the instructor's S2Full work.
• Pair implementation: Reproduce: fine-tune BGE-small/MiniLM with the HF paecter triplets (a 20-50k subset) and show the gain over the base model on held-out citation retrieval. Apply: rerun the same triplet recipe on the scientific-article testbed (citations as positives) and compare with SPECTER2 and the spring bi-encoder on nDCG@10.
• Discussion hooks: Examiner citations encode "prior art" but also examiner habits; are we learning novelty or bureaucracy? Why does a citation-trained model still lose to BM25 out-of-domain (see DAPFAM below)?

2. PatentSBERTa: A Deep NLP based Hybrid Model for Patent Distance and Classification using Augmented SBERT (Bekamiri et al., arXiv 2021)

• Link: https://arxiv.org/abs/2103.11933 (arXiv:2103.11933) · Code: https://github.com/AI-Growth-Lab/PatentSBERTa, https://huggingface.co/AI-Growth-Lab/PatentSBERTa
• Tag: classic · Difficulty: 1 · Compute: laptop
• Summary: Fine-tunes SBERT (RoBERTa-based) on 1.5M patent claims with an augmented-SBERT procedure, then uses plain KNN over embeddings to predict CPC subclasses, reaching F1 > 66% over 663 subclasses. Shows that a good embedding plus nearest neighbours rivals dedicated classifiers, and gives a patent-to-patent distance usable for landscaping.
• Why it matters: The simplest bridge from sentence-transformers (spring) to the patent domain; the instructor's self-supervision paper uses it as a supervised baseline.
• Pair implementation: Reproduce: KNN CPC-subclass classification with the HF model on a 10-20k HUPD or PatenTEB class_text2ipc3 sample versus all-MiniLM. Apply: the same embed-plus-KNN classifier on SciFact/NFCorpus topical labels, and use the embeddings as the dense leg of the baseline pipeline.
• Discussion hooks: Claims are legalese; why do claims-only embeddings work for technical similarity at all? Is "augmented SBERT" just distillation from a cross-encoder?

3. PatentMatch: A Dataset for Matching Patent Claims & Prior Art (Risch et al., arXiv 2020)

• Link: https://arxiv.org/abs/2012.13919 (arXiv:2012.13919) · Code: https://github.com/julian-risch/PatentMatch-FARM, data https://hpi.de/naumann/s/patentmatch
• Tag: classic · Difficulty: 2 · Compute: Colab
• Summary: Builds 6.3M claim/passage pairs from EPO examiner search reports: each claim is linked to the passage of a cited document that anticipates it ("X" citation) or is merely relevant ("A"). Baseline BERT pair-classifiers and a DPR retriever show the task is hard (near-chance for early BERT), making it the standard fine-grained prior-art benchmark.
• Why it matters: Extends DPR and cross-encoders (spring) to passage-level prior-art retrieval; the granularity is the one FiNE-Patents (item 9) builds on.
• Pair implementation: Reproduce: train a MiniLM cross-encoder on 50k PatentMatch pairs and report accuracy/MRR on the test split versus the zero-shot ms-marco cross-encoder. Apply: use it as the reranker in the shared pipeline on SciFact (claim-to-evidence, structurally identical) and compare.
• Discussion hooks: Is claim-to-passage matching an NLI task or a retrieval task? What does a 55% accuracy baseline tell us about label noise from examiners?

4. Patent Language Model Pretraining with ModernBERT (Yousefiramandi & Cooney, arXiv 2025)

• Link: https://arxiv.org/abs/2509.14926 (arXiv:2509.14926) · Code: none found on abstract page
• Tag: recent · Difficulty: 3 · Compute: Colab
• Summary: Pretrains three ModernBERT MLMs on 60M+ patent records (mean ~700 words) with FlashAttention, RoPE and long context. ModernBERT-base-PT beats general ModernBERT on 3/4 classification tasks, is competitive with PatentBERT, and runs about 3x faster at inference.
• Why it matters: The long-context encoder question from the instructor's ModernBERT/S2Full work, applied to patents; a modern successor to BERT for Patents.
• Pair implementation: Reproduce at small scale: continue MLM pretraining of ModernBERT-base on ~50k HUPD abstracts+claims for a few Colab hours, then fine-tune for IPC classification and compare with vanilla ModernBERT. Apply: contrast 512- versus 2048-token inputs as bi-encoder backbones on the scientific-article testbed.
• Discussion hooks: Does domain MLM still pay off when contrastive fine-tuning follows? Is patent language really "long", or is the useful signal in the first 512 tokens?

5. PatenTEB + DAPFAM: a patent embedding benchmark and a cross-domain family-level retrieval dataset (Ayaou et al., arXiv 2025)

• Link: https://arxiv.org/abs/2510.22264 (arXiv:2510.22264); DAPFAM https://arxiv.org/abs/2506.22141 (arXiv:2506.22141) · Code: https://github.com/iliass-y/patenteb, HF user datalyes (12 patembed models, 15 task datasets)
• Tag: recent · Difficulty: 2 · Compute: laptop/Colab
• Summary: PatenTEB is an MTEB-style suite of 15 patent tasks (retrieval, classification, paraphrase, clustering; 2.06M examples) plus the patembed family (67M-344M params, up to 4096 tokens) trained multi-task; patembed-base is SOTA on MTEB BigPatentClustering.v2 (0.494 V-measure). DAPFAM (1,247 query families, 45,336 targets, IPC3-defined IN/OUT partitions) shows out-of-domain recall about 5x lower than in-domain, passage-level beating document-level, and dense models only modestly ahead of BM25 while RRF is a strong cheap fusion.
• Why it matters: Patent counterpart of BEIR/MTEB (spring hands-on); DAPFAM's BM25/dense/RRF findings replicate the spring pipeline in a new domain.
• Pair implementation: Reproduce: run the shared baseline pipeline on DAPFAM IN vs OUT and confirm the domain gap and RRF gain. Apply: submit 3 patembed sizes plus BGE-small to the four testbed sets via mteb, and test whether patent training helps or hurts on SciFact/NFCorpus.
• Discussion hooks: Should prior-art search be evaluated at family level rather than document level? Is a 4096-token encoder worth it if passages beat documents?

6. Benchmarking Patent Embeddings: A Multi-Task Evaluation of 22 Models Across Retrieval, Classification, and Clustering (Yousefiramandi & Cooney, arXiv 2026)

• Link: https://arxiv.org/abs/2605.24297 (arXiv:2605.24297) · Code: "publicly available" per abstract; link not shown on abstract page
• Tag: recent · Difficulty: 2 · Compute: Colab
• Summary: Evaluates 22 embedders (22M to 12B, incl. Qwen3-Embedding, Llama-Nemotron) on 113k WIPO patents and DAPFAM, and compares fine-tuning "recipes". Cross-sectional alignment (different sections of one patent as positives) gives the largest retrieval gain (+7.1% nDCG@10); combining signals helps classification/clustering; fine-tuning on one technology landscape degrades others; title+abstract+claims is the best text view everywhere.
• Why it matters: Independent confirmation of the instructor's section-based self-supervision idea; ties to MTEB methodology and to the alignment/uniformity view of embedding geometry.
• Pair implementation: Reproduce: fine-tune BGE-small with section-pair positives (abstract vs claims) on 20k HUPD patents using MultipleNegativesRankingLoss and show the nDCG@10 gain on DAPFAM. Apply: the same title-vs-abstract self-alignment on the scientific-article testbed, and measure alignment/uniformity before and after.
• Discussion hooks: Why do 7B+ LLM embedders not dominate 100M encoders here? Is section alignment "free supervision" or does it just learn document identity?

7. Enhancing Patent Retrieval Using Automated Patent Summarization (Kamateri et al., PatentSemTech@SIGIR 2025)

• Link: https://arxiv.org/abs/2507.16371 (arXiv:2507.16371) · Code: none found
• Tag: recent · Difficulty: 1 · Compute: Colab + small API budget (or local Qwen 1.5-3B)
• Summary: Prior-art queries are whole patents, far too long for encoders; the paper generates extractive and abstractive summaries of the query patent and uses them as queries on three patent benchmarks, finding summary-based queries significantly outperform full-section queries.
• Why it matters: The patent version of the query-side generation family name-dropped in spring (HyDE, Query2Doc, doc2query); a concrete answer to the long-document problem.
• Pair implementation: Reproduce: on DAPFAM or a PatentMatch subset, compare full-abstract, first-claim, extractive (TextRank) and LLM-abstractive queries with BM25 and BGE-small. Apply: LLM-summarised versus raw queries on ArguAna (whose queries are long arguments) and the MS MARCO subset.
• Discussion hooks: Where does summarisation stop being query formulation and become HyDE-style hallucination? Should the corpus side be summarised too?

8. Is It Novel and Why? Fine-Grained Patent Novelty Prediction Based on Passage Retrieval (Knappich et al., SIGIR 2026)

• Link: https://arxiv.org/abs/2605.02392 (arXiv:2605.02392) · Code: dataset/code release announced; link not shown on abstract page
• Tag: recent · Difficulty: 3 · Compute: Colab + small API budget
• Summary: Recasts novelty assessment from binary claim classification into feature-level retrieval plus reasoning: 3,658 first claims annotated at feature level from European Search Opinions; LLM workflows decompose the claim into features, retrieve supporting passages in the cited prior art, and aggregate. LLM workflows beat embedding baselines on passage retrieval and novel-feature identification and resist the spurious correlations that fool trained classifiers.
• Why it matters: RAG (spring) turned into an examiner-style pipeline; complements Ikoma & Mitamura 2025 (arXiv:2502.06316) and PANORAMA (arXiv:2510.24774).
• Pair implementation: Reproduce: feature decomposition with Qwen2.5-7B (4-bit) plus BGE passage retrieval on a 300-claim FiNE-Patents sample versus a dense-only baseline. Apply: the same claim-to-feature-to-evidence workflow on SciFact (claim verification with rationales), reporting retrieval recall of gold rationales.
• Discussion hooks: Is "novelty" a retrieval problem or an entailment problem? How would you audit an LLM examiner for the priority-date rule?

9. Can Large Language Models Generate High-quality Patent Claims? (Jiang et al., Findings of NAACL 2025)

• Link: https://arxiv.org/abs/2406.19465 (arXiv:2406.19465) · Code: none found on abstract page
• Tag: recent · Difficulty: 2 · Compute: Colab + small API budget
• Summary: Builds a claim-generation dataset and evaluates general and patent-specific LLMs. Generating from the full description beats generating from the abstract; patent-specific LLMs underperform GPT-4; models handle independent claims but fail on dependent ones; fine-tuning improves feature completeness. Human evaluation stresses that outputs still need legal revision.
• Why it matters: Generation as a lens on legalese versus technical description; connects to the instructor group's PatentEval error typology (NAACL 2024) and to LLM-as-a-judge.
• Pair implementation: Reproduce: LoRA-fine-tune Qwen2.5-1.5B on 2k HUPD description-to-claims pairs and score with the paper's metrics and an LLM judge. Apply: use generated claims as doc2query-style expansions for the patent corpus, then test the same expansion trick on NFCorpus.
• Discussion hooks: If claim generation is unreliable, is it still useful as a retrieval signal? Which metric captures "legally valid"?

10. Large Language Models, Encoder Architectures and Hybrid Approaches for Patent Classification (Emer et al., arXiv 2026)

• Link: https://arxiv.org/abs/2601.23200 (arXiv:2601.23200) · Code: none found
• Tag: recent · Difficulty: 2 · Compute: Colab
• Summary: Systematic comparison for hierarchical multi-label CPC classification: a fine-tuned Qwen3.5-9B versus a task-adapted BERT-for-Patents. The encoder wins while using one to two orders of magnitude less inference energy; a hybrid routes only high-uncertainty patents to the LLM; cross-cutting CPC section Y is where both fail, with downstream effects on technology mapping.
• Why it matters: The "do we need an LLM?" question with an energy/cost axis; pairs naturally with PatentSBERTa (item 2).
• Pair implementation: Reproduce: BERT-for-Patents (or MiniLM) fine-tune versus Qwen2.5-3B (4-bit, LoRA) for IPC3 on 20k HUPD patents, measuring F1 and energy/latency with codecarbon; implement the uncertainty router. Apply: same encoder-vs-LLM-vs-hybrid comparison for NFCorpus topic labels.
• Discussion hooks: Should uncertainty routing be the default deployment pattern? Does hierarchical CPC deserve hierarchical losses?

11. Anticipating Innovation Using Large Language Models (Fenoaltea et al., arXiv 2026) — timelines/trend slot

• Link: https://arxiv.org/abs/2605.04875 (arXiv:2605.04875) · Code: none found on abstract page
• Tag: recent · Difficulty: 3 · Compute: Colab
• Summary: TechToken adds IPC codes as tokens to a transformer fine-tuned on patent text, so each technology gets a contextual embedding. Rising context similarity between two code embeddings predicts their first co-occurrence in a patent, in some cases decades ahead, and the model beats SOTA on patent benchmarks.
• Why it matters: Exactly "timelines on embeddings": diachronic embedding geometry as a forecasting device, linking to the instructor's temporal-analysis interest and to classic text-based innovation indices (Kelly et al. 2021).
• Pair implementation: Reproduce: on HUPD 2004-2018 abstracts, train per-year IPC-code embeddings (a cheap variant: mean of MiniLM abstract embeddings per code and year) and test whether similarity growth predicts new code pairs after 2015. Apply: the same year-sliced embedding drift on the scientific-article testbed or NFCorpus by MeSH topic.
• Discussion hooks: Is "convergence in embedding space" causal or an artefact of vocabulary drift? How would you validate a forecast without hindsight bias?

12. PAP2PAT: Benchmarking Outline-Guided Long-Text Patent Generation with Patent-Paper Pairs (Knappich et al., Findings of ACL 2025)

• Link: https://aclanthology.org/2025.findings-acl.496/ (arXiv:2410.07009) · Code: https://github.com/boschresearch/Pap2Pat
• Tag: recent · Difficulty: 2 · Compute: laptop/Colab
• Summary: 1.8k paper/patent pairs (same invention, both texts) plus document outlines, mined with lab-practice heuristics; the generation task shows open LLMs reuse paper content but lack technical depth and become repetitive, with fine-tuning raising hallucination. The pair data itself is a rare scholarly-to-patent linkage resource.
• Why it matters: The bridge between the instructor's two corpora (scientific papers and patents); a paper-to-patent retrieval task is SPECTER meets PaECTER.
• Pair implementation: Reproduce: the paper's retrieval-style analysis at small scale: embed the 1.8k papers and patents with SPECTER2, PaECTER, BGE, and compute paper-to-patent MRR; test whether patent encoders handle paper prose. Apply: cross-register retrieval on the scientific-article testbed (abstract-only versus full title+abstract queries).
• Discussion hooks: How different are the "same invention" embeddings across registers, and what does that say about alignment/uniformity? Could paper-to-patent links serve as free contrastive supervision?

Framing resources for the instructor's 30-min intro

• Jiang & Goetz, "Natural Language Processing in the Patent Domain: A Survey", Artificial Intelligence Review 58:214 (2025), https://arxiv.org/abs/2403.04105 — nine analysis and four generation tasks; best single map.
• Shomee et al., "A Survey on Patent Analysis: From NLP to Multimodal AI", ACL 2025, https://arxiv.org/abs/2404.08668 — lifecycle taxonomy, multimodal angle.
• Google "BERT for Patents" white paper (Srebrovic & Yonamine 2020) https://services.google.com/fh/files/blogs/bert_for_patents_white_paper.pdf and the Google Patents Public Datasets repo https://github.com/google/patents-public-data/blob/master/models/BERT%20for%20Patents.md — the origin of the encoder lineage and BigQuery data access.
• Kelly, Papanikolaou, Seru, Taddy, "Measuring Technological Innovation over the Long Run", AER: Insights 2021, https://www.aeaweb.org/articles?id=10.1257/aeri.20190499 — canonical backward/forward text-similarity innovation index since 1840; the pre-embedding ancestor of item 11.
• Awesome-LLM4Patents list https://github.com/QiYao-Wang/Awesome-LLM4Patents; CLEF-IP 2011 overview (Piroi et al.) https://ceur-ws.org/Vol-1177/CLEF2011wn-CLEF-IP-PiroiEt2011.pdf; HUPD (Suzgun et al., NeurIPS 2023 D&B) https://arxiv.org/abs/2207.04043 / https://huggingface.co/datasets/HUPD/hupd as the sampling source for implementations.

Notes

• The shared testbed contains no patent data, so every "reproduce" step draws a small sample from HUPD, DAPFAM, PatentMatch or the PatenTEB HF datasets (all public), while the "apply" step ports the method to SciFact/NFCorpus/ArguAna/the scientific-article set.
• Instructor-group papers seen but not proposed as topics: Zuo, Gerdes et al., "Patent Representation Learning via Self-supervision" (arXiv:2511.10657); "Sparse Coverage: Semantic Center Representations for Patent Prior-Art Retrieval" (CORIA-TALN 2026, arXiv:2608.16918, CLEF-IP 2013); PatentEval (NAACL 2024, https://aclanthology.org/2024.naacl-long.147/). Items 6 and 12 are the natural external counterparts.
• Near-misses (verified, dropped for space): Ikoma & Mitamura 2025 novelty (arXiv:2502.06316, no code); PANORAMA examination trails (arXiv:2510.24774, HF LG-AI-Research/PANORAMA); PatentEdits (arXiv:2411.13477); PEDANTIC definiteness (arXiv:2505.21342); PatentGPT (arXiv:2404.18255); Ascione & Sterzi embedding comparison with interference ground truth (arXiv:2403.16630); Althammer et al. ECIR 2021 BERT-PLI reproducibility on CLEF-IP (arXiv:2012.11405, code) — a good template if you want a "reproducibility" slot; Helmers et al. PLOS ONE 2019 full-text prior art (arXiv:1901.03136); BIGPATENT (ACL 2019, https://aclanthology.org/P19-1212/) only as data.
• Trend-slot alternates: Filippi-Mazzola et al., PLOS ONE 2023, patent similarity 1976-2021 (arXiv:2212.06046); Lyu & Ke 2025 diachronic periodical embeddings (arXiv:2503.23285, scholarly not patent); Sternfeld et al. 2025 LLM triple graphs over arXiv+USPTO (arXiv:2510.25370).
• Gaps: no strong recent paper specifically on multilingual patent retrieval or on priority-date constraints surfaced (arXiv searches returned only 2002-era work); CLEF-IP (EN/FR/DE) and Zhang et al. 2025 (arXiv:2508.17782, evaluation by language/jurisdiction) are the best hooks. Weak-evidence papers dropped: FullRecall (five test cases), Ding et al. LLM-RAG (arXiv:2508.14064), PHAGE (arXiv:2605.10073, no code).
• Overlaps: item 7 with the query-expansion theme (HyDE/doc2query), items 8-9 with LLM/RAG themes, item 5 with a benchmarking theme, items 1/12 with a scholarly-document theme.

Theme 5: Domain adaptation and synthetic training data

Default slot: Session 3 (Nov 20) — Domains.

Domain adaptation of retrievers & synthetic training data for IR — candidate papers

All URLs below were seen in search results or fetched directly during this session.

1. GPL: Generative Pseudo Labeling for Unsupervised Domain Adaptation of Dense Retrieval (Wang et al., NAACL 2022)

• Link: https://arxiv.org/abs/2112.07577 (arXiv:2112.07577) · ACL: https://aclanthology.org/2022.naacl-main.168/ · Code: https://github.com/ukplab/gpl
• Tag: classic · Difficulty: 1 · Compute: Colab
• Summary: A dense retriever trained on MS MARCO loses a lot on specialised corpora. GPL generates queries for target-domain passages with a T5 query generator (doc2query-style), mines negatives with the source-domain retriever, and uses a cross-encoder to produce soft margin scores that the bi-encoder is trained to reproduce (MarginMSE). On six domain-specific BEIR datasets it beats the zero-shot state of the art by up to 9.3 nDCG@10 and clearly outperforms plain query generation (GenQ) and TSDAE-style pre-training.
• Why it matters / link to the course: Directly extends doc2query-T5 + cross-encoders that students already know into a training recipe; it is the reference answer to "MS MARCO -> scientific domain gap". Wang, Reimers, Gurevych are the SBERT team, so the recipe is native to sentence-transformers.
• Pair implementation: Reproduce the GPL pipeline on SciFact/NFCorpus with BGE-small or MiniLM as the student, doc2query/msmarco-t5-base-v1 as generator and ms-marco-MiniLM-L-6-v2 as teacher (a few thousand generated queries is enough on a T4). Report zero-shot vs GenQ vs GPL nDCG@10 on all three BEIR testbed datasets; ablate the pseudo-label source (BM25 negatives vs dense negatives).
• Discussion hooks: Is the cross-encoder teacher the real "domain knowledge" here, or just a denoiser of bad generated queries? What happens to GPL when the teacher itself is out of domain (e.g. patents)?

2. InPars: Data Augmentation for Information Retrieval using Large Language Models (Bonifacio et al., 2022; short version at SIGIR 2022)

• Link: https://arxiv.org/abs/2202.05144 (arXiv:2202.05144) · Code: https://github.com/zetaalphavector/inpars
• Tag: classic · Difficulty: 1 · Compute: Colab (open LLM) or Colab+small API budget
• Summary: Prompt a large LM (GPT-3 in v1) with three in-context (doc, query) examples to write queries for target-corpus documents, keep the top-k pairs by LM likelihood, and fine-tune a reranker on them. Rerankers trained only on the synthetic data beat BM25 and self-supervised retrievers, and mixing synthetic with MS MARCO data improves zero-shot transfer. Follow-ups: InPars-v2 (open generator + reranker filtering, https://arxiv.org/abs/2301.01820), InPars-Light (Boytsov et al. 2023, https://arxiv.org/abs/2301.02998, code https://github.com/searchivarius/inpars_light — 30M-param MiniLM reranker, +7-30% over BM25 with BLOOM) and InPars+ (Krastev et al. 2025, https://arxiv.org/abs/2508.13930, code https://github.com/danilotpnta/IR2-project — CPO-tuned generator and DSPy-optimised CoT prompts, evaluated on SciFact).
• Why it matters / link to the course: The LLM-side twin of doc2query; the ranking-by-likelihood filter is the first "false positive" filter students will meet.
• Pair implementation: Re-implement InPars-Light: generate ~5k queries for SciFact/NFCorpus with Qwen2.5-3B/7B-Instruct (4-bit) or Flan-T5, filter by log-probability, fine-tune the course's MiniLM cross-encoder, and compare against the off-the-shelf ms-marco-MiniLM-L-6-v2 in the class pipeline (BM25 -> rerank). Bonus: reproduce InPars+ prompt-optimisation on SciFact.
• Discussion hooks: The 2022 result relied on GPT-3; does the method survive with a 3B open model (see Gwon et al. below)? Is likelihood filtering selecting "easy" queries that make evaluation optimistic?

3. Promptagator: Few-shot Dense Retrieval From 8 Examples (Dai et al., ICLR 2023)

• Link: https://arxiv.org/abs/2209.11755 (arXiv:2209.11755) · OpenReview: https://openreview.net/pdf?id=gmL46YMpu2J · Code: none found (Google-internal; Gwon et al. 2025 is the open reproduction)
• Tag: classic · Difficulty: 2 · Compute: Colab
• Summary: Each BEIR task gets 8 hand-written examples; a 137B LLM is prompted to generate task-specific queries for the corpus, then round-trip consistency filtering (keep a query only if a retriever trained on the synthetic data retrieves its source passage in the top-k) cleans the data, and a dual-encoder plus reranker are trained. Promptagator++ beats retrievers trained on MS MARCO on most BEIR tasks with zero human relevance labels. The companion "Study on LLMs for Promptagator-Style Dense Retriever Training" (Gwon, Jedidi, Lin, CIKM 2025 short, https://arxiv.org/abs/2510.02241) shows open LLMs as small as 3B are effective query generators.
• Why it matters / link to the course: Introduces the idea that query intent differs per domain (claims for SciFact, arguments for ArguAna) — precisely the shift students saw when MS MARCO models underperform on scientific retrieval. Consistency filtering reappears in nearly every 2024-26 paper.
• Pair implementation: Reproduce Gwon et al.: write 8 examples per testbed dataset, generate queries with Qwen2.5-3B/7B, apply round-trip filtering with BGE-small, fine-tune BGE-small with MultipleNegativesRankingLoss, evaluate on SciFact/ArguAna/NFCorpus. Ablate "with vs without the 8 examples" and "with vs without consistency filtering".
• Discussion hooks: Promptagator uses 8 examples but a 137B model; GPL uses zero examples but a 220M T5 — which is the cheaper form of supervision? Does round-trip filtering just remove hard-but-valid queries?

4. Remining Hard Negatives for Generative Pseudo Labeled Domain Adaptation (Yuksel, Rau & Kamps, 2025)

• Link: https://arxiv.org/abs/2501.14434 (arXiv:2501.14434) · Code: none found
• Tag: recent · Difficulty: 2 · Compute: Colab
• Summary: GPL mines hard negatives once with the source-domain model and never refreshes them, although the student steadily adapts to the target domain. R-GPL periodically re-mines negatives with the model being adapted; this improves on 13/14 BEIR and 9/12 LoTTE datasets. Two companion papers from the same group analyse why adaptation works: "On Correlating Factors for Domain Adaptation Performance" (https://arxiv.org/abs/2501.14466: the type distribution of generated queries matters most) and an interpretability analysis (https://arxiv.org/abs/2501.14459).
• Why it matters / link to the course: A clean, incremental extension of GPL (#1); connects to the spring lecture on hard negatives (explained but never implemented) and to the instructor's interest in how embedding geometry drifts during adaptation.
• Pair implementation: Start from the GPL pipeline above and add re-mining every N steps on NFCorpus/SciFact; plot nDCG@10 vs re-mining frequency and measure how much the negative sets overlap between source-model mining and adapted-model mining. If pair #1 also exists, share the GPL code and compare.
• Discussion hooks: Re-mining with an improving model increases the false-negative risk — where is the sweet spot? Should we re-mine the positives too (cf. Gecko)?

5. Teaching Dense Retrieval Models to Specialize with Listwise Distillation and LLM Data Augmentation (Tamber et al., 2025)

• Link: https://arxiv.org/abs/2502.19712 (arXiv:2502.19712) · Code: https://github.com/manveertamber/enhancing_domain_adaptation
• Tag: recent · Difficulty: 2 · Compute: Colab
• Summary: Fine-tuning a strong general embedder (e.g. BGE) on in-domain data with InfoNCE, even with mined hard negatives, can reduce effectiveness on BEIR-style datasets. Replacing the loss by listwise KL-distillation from a cross-encoder (RankT5-3B) and augmenting with diverse Llama-3.1-8B queries (questions, claims, titles, keywords, web-style) yields consistent gains, and synthetic queries match human-written ones as training data; the cross-encoder teacher becomes the bottleneck.
• Why it matters / link to the course: Sharpens the InfoNCE/in-batch-negatives lecture with a negative result; the diverse query types echo Promptagator; the teacher-bottleneck finding motivates LLM rerankers (other theme).
• Pair implementation: Reproduce the "InfoNCE hurts" phenomenon: fine-tune BGE-small on SciFact/NFCorpus with (a) InfoNCE + BM25 negatives and (b) listwise KL from ms-marco-MiniLM-L-6-v2 (or a BGE-reranker-base) scores, using generated queries; compare with zero-shot BGE-small and with the class's RRF+rerank pipeline.
• Discussion hooks: If distillation from a reranker is the best way to specialise a bi-encoder, is the bi-encoder just a cache of the reranker? Why does contrastive fine-tuning destroy general-purpose knowledge?

6. Evaluating the Effectiveness and Scalability of LLM-Based Data Augmentation for Retrieval (Chitale et al., EMNLP 2025)

• Link: https://arxiv.org/abs/2509.16442 (arXiv:2509.16442) · Code: https://aka.ms/DAGR (code + augmented datasets)
• Tag: recent · Difficulty: 2 · Compute: Colab
• Summary: A systematic study (100+ configurations) of LLM query augmentation for compact dual encoders. Augmentation helps, but returns diminish beyond a certain scale, small LLMs generate data as useful as large ones, and gains are largest for weakly pre-trained encoders. The paper offers a practical recipe for choosing augmentation budget vs model size.
• Why it matters / link to the course: The "how much synthetic data do I actually need?" question students will face on the testbed; complements #2/#3 with an empirical scaling view.
• Pair implementation: Reproduce the scaling curve on one testbed dataset: generate 500 / 2k / 8k queries with Qwen2.5-1.5B and 7B, fine-tune MiniLM-L6 and BGE-small, plot nDCG@10 vs number of synthetic pairs and generator size. Use the released augmented data to sanity-check.
• Discussion hooks: If a 1.5B generator is as good as a 70B one, what is the generator actually contributing — language or knowledge? Should augmentation budget be spent on more queries or on better filtering?

7. NV-Retriever: Improving text embedding models with effective hard-negative mining (Moreira et al., 2024)

• Link: https://arxiv.org/abs/2407.15831 (arXiv:2407.15831) · Code: model on Hugging Face (nvidia/NV-Retriever-v1); no training code found
• Tag: recent · Difficulty: 2 · Compute: Colab
• Summary: Mined "hard" negatives are often unlabeled positives. NV-Retriever proposes positive-aware mining: discard candidates whose teacher score exceeds the positive's score minus a margin (TopK-MarginPos) or above a percentage of it (TopK-PercPos, best at 95%). With e5-mistral-7b-instruct as mining teacher, NV-Retriever-v1 reached 60.9 on MTEB Retrieval and placed first when published (July 2024).
• Why it matters / link to the course: The canonical false-negative filter; the same trick underlies the synthetic-negative debate in #9 and Gecko (#8). Relevant to the instructor's citation-contrastive training where co-cited papers are frequent false negatives.
• Pair implementation: Fine-tune BGE-small on MS MARCO-subset pairs (or GPL-generated pairs) with negatives mined by BGE-small itself, comparing naive top-k, top-k with random offset, and TopK-PercPos filtering using the cross-encoder as scorer; report BEIR testbed nDCG@10 and the fraction of negatives removed.
• Discussion hooks: A 95% threshold assumes a well-calibrated teacher — what if the teacher is worse than the student in the target domain? Is "false negative" a data-labelling problem or a graded-relevance problem?

8. Gecko: Versatile Text Embeddings Distilled from Large Language Models (Lee et al., 2024)

• Link: https://arxiv.org/abs/2403.20327 (arXiv:2403.20327) · Code: none found (model behind the Google API)
• Tag: recent · Difficulty: 2 · Compute: Colab+small API budget (or a local 7B LLM)
• Summary: Two-step LLM distillation: an LLM generates task-diverse (query, task) pairs for sampled passages, then the same LLM relabels the top retrieved candidates, choosing a new positive and hard negatives (the original passage is often not the best positive). A 256-dim Gecko beats all 768-dim MTEB entries; the 768-dim version scores 66.31, competing with 7x larger models.
• Why it matters / link to the course: Closes the loop between synthetic query generation (#2, #3) and hard-negative selection (#7): the LLM acts as both generator and judge (ties to LLM-as-a-judge).
• Pair implementation: Implement the relabeling step on SciFact: generate queries with an open LLM, retrieve top-20 with BGE-small, ask the LLM to pick the best positive and 2 hard negatives, and fine-tune BGE-small on the relabeled triples vs the original (doc, query) pairs. Measure how often the LLM re-assigns the positive.
• Discussion hooks: When the LLM changes the positive, is the training signal "more correct" or just "more LLM-like"? Does LLM relabeling reintroduce the domain bias we wanted to escape?

9. Don't Retrieve, Generate: Prompting LLMs for Synthetic Training Data in Dense Retrieval (Sinha, 2025)

• Link: https://arxiv.org/abs/2504.21015 (arXiv:2504.21015, v4 May 2026) · Code: none found
• Tag: recent · Difficulty: 1 · Compute: Colab
• Summary: Instead of mining negatives from the corpus, four LLMs (4B-30B) are prompted to write hard negatives; DistilBERT retrievers trained on them are evaluated on ten BEIR datasets. The generative pipeline consistently underperforms BM25- and cross-encoder-mined negatives, and the 14B model beats the 30B one. Contrast with SyNeg (Li et al., 2024, https://arxiv.org/abs/2412.17250), which reports gains with multi-attribute self-reflection prompts plus hybrid sampling of synthetic and mined negatives.
• Why it matters / link to the course: The needed counter-evidence on synthetic data; ties to #7 (what makes a negative "hard" but true) and to the instructor's over-dispersion finding that augmentation choice reshapes embedding geometry.
• Pair implementation: Replicate the head-to-head on SciFact/NFCorpus with MiniLM/BGE-small: BM25 negatives vs cross-encoder negatives vs Qwen-generated negatives (plain prompt and SyNeg-style prompt); additionally measure alignment/uniformity of the resulting embeddings to see whether generated negatives cause over-dispersion.
• Discussion hooks: Are generated negatives too easy (off-distribution) or too hard (paraphrased positives)? Should hard negatives ever come from outside the corpus you will search?

10. MedCPT: Contrastive Pre-trained Transformers with Large-scale PubMed Search Logs for Zero-shot Biomedical Information Retrieval (Jin et al., Bioinformatics 2023)

• Link: https://arxiv.org/abs/2307.00589 (arXiv:2307.00589) · journal: https://academic.oup.com/bioinformatics/article/39/11/btad651/7335842 · Code: https://github.com/ncbi/MedCPT
• Tag: recent (2023) · Difficulty: 1 · Compute: laptop
• Summary: A domain-specific retriever + reranker pair trained contrastively on 255M PubMed query-article click pairs (in-batch negatives for the bi-encoder, retriever-mined negatives for the cross-encoder). MedCPT sets state of the art on six biomedical IR tasks and beats GPT-3-sized cpt-text-XL. Pair it with FinMTEB (Tang & Yang, EMNLP 2025, https://arxiv.org/abs/2502.10990, code https://github.com/yixuantt/FinMTEB), which shows general MTEB scores poorly predict finance performance and that Fin-E5 trained on persona-based synthetic data outperforms general models.
• Why it matters / link to the course: The "just train in-domain" baseline every adaptation method must beat; the PubMedBERT-sized models run on a laptop; NFCorpus and SciFact are exactly its domain. Connects to SPECTER2/SciBERT and the instructor's scholarly embeddings.
• Pair implementation: Evaluate MedCPT retriever and reranker vs MiniLM/BGE-small and the class cross-encoder on NFCorpus, SciFact and the scientific-article dataset (in-domain vs out-of-domain, ArguAna as control); then run a small FinMTEB-style probe: does MTEB rank predict rank on the testbed? Optionally adapt BGE-small with GPL and see how close it gets to MedCPT.
• Discussion hooks: Click logs are cheap but noisy relevance — is a search-log retriever a domain model or a popularity model? Given FinMTEB's finding, should every domain have its own MTEB?

11. Pseudo Relevance Feedback is Enough to Close the Gap Between Small and Large Dense Retrieval Models (Li, Wang, Koopman & Zuccon, 2025)

• Link: https://arxiv.org/abs/2503.14887 (arXiv:2503.14887) · Code: none found
• Tag: recent · Difficulty: 2 · Compute: Colab
• Summary: PromptPRF: extract query-independent features (entities, keywords, summaries) from top-ranked documents offline with an LLM, then feed them to a small LLM-based dense retriever via prompting, no training required; on TREC DL and BEIR it consistently improves and lets small models approach larger ones. Related: LLM-VPRF (same group, https://arxiv.org/abs/2504.01448) extends vector PRF to LLM embedders; the classics are ANCE-PRF (Yu et al., CIKM 2021, https://arxiv.org/abs/2108.13454) and ColBERT-PRF (Wang et al., ACM TOIS, https://dl.acm.org/doi/10.1145/3572405); PLAID-PRF (Wang, MacAvaney & Macdonald, SIGIR 2026, https://arxiv.org/abs/2607.18626) brings it to PLAID (+4.3% nDCG@10).
• Why it matters / link to the course: Test-time adaptation without any training data: the Rocchio idea from classical IR ported to dense retrieval; complements the training-time methods above.
• Pair implementation: Implement vector PRF (Rocchio on embeddings: q' = alphaq + betamean(top-k doc embeddings)) for BGE-small on all three testbed datasets, then PromptPRF-style: extract keywords/summaries from top-3 docs with a small LLM and re-encode the expanded query; sweep k and alpha; compare with HyDE-style generation (other theme).
• Discussion hooks: PRF helps when the first pass is decent — does it amplify domain-shift errors when it is not? Is PRF a form of domain adaptation or just query expansion?

12. More Than Efficiency: Embedding Compression Improves Domain Adaptation in Dense Retrieval (Zuo & Khashabi, 2026)

• Link: https://arxiv.org/abs/2601.13525 (arXiv:2601.13525; oral at SURGeLLM workshop, ACL 2026) · Code: none found
• Tag: recent · Difficulty: 1 · Compute: laptop
• Summary: Fitting PCA on the target corpus and projecting only the query embeddings improves nDCG@10 in 75.4% of 9 retrievers x 14 MTEB datasets, with no retraining or labels. The claim: compression removes non-discriminative directions and preserves domain-specific ones, so a purely geometric operation acts as domain adaptation. Related test-time work: TTT-Embed (Chen & Wu, 2026, https://arxiv.org/abs/2608.12569, optimises a query vector at test time, up to +8.36 nDCG@10, works with closed APIs) and DART (Liu & Li, KnowFM@ACL 2026, https://arxiv.org/abs/2606.01070, test-time gradient updates of a bilinear scorer, +2.1% nDCG@10 on six BEIR sets).
• Why it matters / link to the course: Squarely about embedding-space geometry (alignment/uniformity, anisotropy) — the instructor's own research angle — and relates to Matryoshka/dimension truncation from the spring course.
• Pair implementation: Reproduce on the testbed with MiniLM, BGE-small/base and one 7B embedder (frozen): PCA fitted on each corpus, applied to queries only / docs only / both, at several dimensions; compare with MRL truncation and with whitening; report nDCG@10 and alignment/uniformity before and after. All numpy + FAISS Flat, runs on a laptop.
• Discussion hooks: If PCA on queries helps, what is broken about the query encoder's geometry after MS MARCO training? Is this domain adaptation or just anisotropy removal, and how would you tell?

Framing resources for the instructor's 30-min intro

• Sentence-Transformers "Domain Adaptation" docs (TSDAE + GPL, with runnable scripts): https://sbert.net/examples/sentence_transformer/domain_adaptation/README.html — TSDAE paper (Wang, Reimers, Gurevych, Findings of EMNLP 2021): https://arxiv.org/abs/2104.06979.
• Negative Sampling Techniques in Information Retrieval: A Survey (Wischounig, Abdallah & Jatowt, Findings of EACL 2026): https://arxiv.org/abs/2603.18005 — 35 papers, random/mined/synthetic taxonomy incl. LLM-generated negatives; ideal map for papers 4-9.
• Large Language Models for Information Retrieval: A Survey (Zhu et al., ACM TOIS; v5 Sept 2025): https://arxiv.org/abs/2308.07107 — sections on LLM-generated training data and query rewriting.
• Resources for Brewing BEIR (Kamalloo et al., SIGIR 2024 resource paper): https://arxiv.org/abs/2306.07471 — Pyserini reproductions of BM25/TAS-B/Contriever/uniCOIL/SPLADE and the official BEIR leaderboard; good for a slide on "zero-shot spread across domains". Optionally How to Train Your DRAGON (Lin et al., 2023, https://arxiv.org/abs/2302.07452, code https://github.com/facebookresearch/dpr-scale/tree/main/dragon) as the "diverse supervision" reference and a strong zero-shot BERT-base baseline checkpoint.

Notes

• Near-misses dropped or folded in: UDAPDR (Saad-Falcon et al., EMNLP 2023, https://arxiv.org/abs/2303.00807; +5.2 nDCG@10 on BEIR over zero-shot ColBERTv2) is a strong classic but ColBERTv2-centric and heavy for students; no code link on the arXiv page. Li & Gaussier (COLING 2024, https://arxiv.org/abs/2403.08970) is a GPL-like pseudo-labeling method with a T5-3B labeler (French/Grenoble connection) — a fine alternative to #4. Pal et al., "Parameter-Efficient Sparse Retrievers and Rerankers using Adapters" (ECIR 2023, https://arxiv.org/abs/2303.13220) covers adapters for domain adaptation of SPLADE (2% of params); I found no dedicated 2024-26 "LoRA for bi-encoders in domain adaptation" paper with a verified URL, so PEFT is best covered as an implementation option inside #5/#10 (LoRA via PEFT on BGE-base is trivial in sentence-transformers). InPars-Light and Gwon et al. are folded into #2 and #3.
• Not verified this session (search budget exhausted), hence not listed: coCondenser, RetroMAE, ICT, Spider, SPECTER2 adapters paper (SciRepEval). "A Little Human Data Goes A Long Way" (Ashok & May, ACL 2025, https://arxiv.org/abs/2410.13098) is about fact verification/QA, not IR, but its "last 10% of human data matters" finding is a useful intro slide.
• Venue caveats: InPars' SIGIR 2022 short version and DRAGON's venue were not confirmed from the arXiv pages (no comments field); cite the arXiv versions. The Tamber et al. paper may have a later conference version (a Springer chapter "Contrastive Learning Falls Short..." appeared in search results, unverified).
• Overlaps with other themes: #8 (Gecko) touches E5-Mistral/LLM2Vec synthetic-data work; #9 relates to the CoRNStack code hard-negative paper; #11 borders HyDE/Query2Doc (query rewriting theme) and ColBERT/PLAID; #12 borders Matryoshka/embedding-compression. Pairs #1 and #4 should share the GPL pipeline; #7, #8 and #9 form a natural "what is a good negative?" session.

Theme 6: Multilingual and cross-lingual retrieval; temporal aspects and diachronic embeddings

Default slot: Session 3 (Nov 20) — Domains.

Part A — Multilingual and cross-lingual retrieval

1. M3-Embedding (BGE-M3): Multi-Linguality, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillation (Chen et al., Findings of ACL 2024)

• Link: https://arxiv.org/abs/2402.03216 (arXiv:2402.03216) · https://aclanthology.org/2024.findings-acl.137/ · Code: https://huggingface.co/BAAI/bge-m3 (FlagEmbedding)
• Tag: recent · Difficulty: 2 · Compute: Colab
• Summary: One XLM-R-based model (568M) produces dense, sparse (lexical weights) and multi-vector (ColBERT-style) representations for 100+ languages and up to 8,192 tokens. The three heads are trained jointly with "self-knowledge distillation": the fused score of all three serves as teacher for each head. Strong results on MIRACL and MKQA; combining the three heads beats each alone.
• Why it matters: Unifies three spring-course concepts (bi-encoder, SPLADE-style sparse, ColBERT late interaction) in a single encoder; the de-facto open multilingual baseline in 2024-26.
• Pair implementation: Reproduce the dense/sparse/multi-vector ablation and their fusion on SciFact and NFCorpus (nDCG@10); compare the model's own fusion with the class RRF pipeline. Then translate SciFact queries to French (opus-mt / NLLB-600M) and measure how much each head loses cross-lingually.
• Discussion hooks: Is one model for three signals better than three specialised models? Does the multi-vector head really earn its storage cost on a 5k-doc corpus?

2. jina-embeddings-v3: Multilingual Embeddings With Task LoRA (Sturua et al., arXiv 2024)

• Link: https://arxiv.org/abs/2409.10173 (arXiv:2409.10173) · Code: https://huggingface.co/collections/jinaai/jina-embeddings-v3
• Tag: recent · Difficulty: 2 · Compute: Colab
• Summary: A 570M multilingual encoder with five task-specific LoRA adapters (query, passage, clustering, classification, text-matching), Matryoshka training (1024 down to 32 dims) and 8k context. Outperforms multilingual-e5-large-instruct on multilingual MTEB tasks and OpenAI/Cohere embeddings on English tasks.
• Why it matters: Directly extends the spring course's Matryoshka (MRL) and asymmetric query/passage encoding; a clean illustration of "one backbone, many adapters".
• Pair implementation: Reproduce the MRL curve (nDCG@10 vs. dimensions 32-1024) on SciFact/ArguAna and the adapter ablation (query/passage adapters vs. none vs. text-matching adapter). Apply on the MS MARCO subset with FAISS to quantify the index-size/quality trade-off.
• Discussion hooks: Are task adapters a principled fix or a patch for a badly-shared embedding space? How low can dimension go before cross-lingual neighbours collapse?

3. MIRACL: A Multilingual Retrieval Dataset Covering 18 Diverse Languages (Zhang et al., TACL 2023)

• Link: https://aclanthology.org/2023.tacl-1.63/ · https://arxiv.org/abs/2210.09984 (arXiv:2210.09984) · Code: https://github.com/project-miracl/miracl
• Tag: classic · Difficulty: 1 · Compute: laptop / Colab
• Summary: 78k queries and 726k human relevance judgments over Wikipedia in 18 languages (including French), with BM25, mDPR and hybrid baselines. Shows that hybrid sparse+dense wins almost everywhere and that dense models trained on English MS MARCO transfer unevenly. Companion resource: mMARCO (Bonifacio et al. 2021, https://arxiv.org/abs/2108.13897), MS MARCO machine-translated into 13 languages, which showed translation quality correlates with retrieval effectiveness.
• Why it matters: The multilingual counterpart of BEIR (used hands-on in spring); the standard evaluation set behind BGE-M3, mE5, jina-v3.
• Pair implementation: Reproduce BM25 vs. multilingual-e5-small vs. hybrid on MIRACL-fr dev (small corpus slice). Apply: build "SciFact-fr" by translating queries only (mMARCO recipe) and report the cross-lingual drop for the class pipeline.
• Discussion hooks: Wikipedia-only relevance: does MIRACL measure retrieval or entity memorisation? Should French IR evaluation rely on machine-translated MS MARCO?

4. Evaluating Large Language Models for Cross-Lingual Retrieval (Zuo et al., Findings of EMNLP 2025)

• Link: https://arxiv.org/abs/2509.14749 (arXiv:2509.14749) · Code: none found
• Tag: recent · Difficulty: 2 · Compute: Colab (+ optional small API budget)
• Summary: Systematic study of the "translate-then-retrieve vs. joint multilingual embedding" question in the LLM era, at passage and document level. Multilingual bi-encoders as first stage beat BM25+MT; LLM rerankers (pointwise, pairwise, listwise) "fall severely short" when applied directly cross-lingually, but the benefit of MT shrinks as the reranker gets stronger.
• Why it matters: Extends the spring cross-encoder/reranking stage and the name-dropped "LLM rerankers" to the cross-lingual case; the same question Questel faces for multilingual patent search.
• Pair implementation: Reproduce the main grid on MIRACL-fr (or French-query SciFact): {BM25+MT, mE5-small} x {no rerank, MiniLM cross-encoder, Qwen2.5-3B pointwise reranker in 4-bit}, with and without query translation. Apply the best configuration on NFCorpus.
• Discussion hooks: If a 3B LLM reranker already understands French, why translate at all? Where does bias against non-English documents enter the pipeline?

5. Language Bias in Information Retrieval: The Nature of the Beast and Mitigation Methods (Yang, Jiang & Baldwin, EMNLP 2024 MRL workshop; arXiv 2025)

• Link: https://arxiv.org/abs/2509.06195 (arXiv:2509.06195) · Code: none found (dataset Multi-EuP: https://arxiv.org/abs/2311.01870)
• Tag: recent · Difficulty: 2 · Compute: Colab
• Summary: Asks whether semantically identical queries in different languages retrieve the same documents from a multilingual collection (European Parliament, 24 languages). BM25 and mBERT/XLM-R DPR rankers show strong language preference; the authors introduce LaKDA, a loss regularising query representations across languages, which narrows the gap.
• Why it matters: Fairness angle on the InfoNCE training students learned; connects to the instructor's embedding-geometry work (language forms sub-clusters that dominate nearest-neighbour search).
• Pair implementation: Build a bilingual corpus by translating half of NFCorpus into French; measure language preference of BGE-M3 / mE5 (share of French docs in top-10 for English vs. French queries, rank correlation). Apply a small LaKDA-style fine-tune of paraphrase-multilingual-MiniLM and report the change.
• Discussion hooks: Is "language-invariant" retrieval even desirable for a French user? Can one measure bias without parallel documents?

6. Leveraging LLMs for Synthesizing Training Data Across Many Languages in Multilingual Dense Retrieval — SWIM-IR (Thakur et al., NAACL 2024)

• Link: https://aclanthology.org/2024.naacl-long.426/ · https://arxiv.org/abs/2311.05800 (arXiv:2311.05800) · Code: https://github.com/google-research-datasets/swim-ir
• Tag: recent · Difficulty: 2 · Compute: Colab + small API budget (or local 3B LLM)
• Summary: Generates 28M synthetic query-passage pairs in 33 languages with "summarise-then-ask" prompting (PaLM 2), then fine-tunes multilingual dense retrievers (SWIM-X). Without any human labels, SWIM-X is competitive with mContriever on MIRACL, XOR-Retrieve and XTREME-UP, including very low-resource languages.
• Why it matters: The multilingual/low-resource realisation of the name-dropped InPars/Promptagator idea; directly reuses in-batch-negative training from spring.
• Pair implementation: Reproduce SAP prompting with Qwen2.5-3B (or a few euros of API) to generate French queries for ~2k NFCorpus abstracts; fine-tune multilingual-e5-small with MultipleNegativesRankingLoss; evaluate on French-translated NFCorpus queries vs. zero-shot. Ablate summary-first vs. direct query generation.
• Discussion hooks: Do synthetic queries encode the LLM's cultural/linguistic biases into the retriever? Where is the floor language-wise for this trick?

Part B — Temporal retrieval, evolving corpora, diachronic embeddings

7. FreshLLMs: Refreshing Large Language Models with Search Engine Augmentation (Vu et al., arXiv 2023)

• Link: https://arxiv.org/abs/2310.03214 (arXiv:2310.03214) · Code: https://github.com/freshllms/freshqa (dataset refreshed weekly)
• Tag: classic · Difficulty: 1 · Compute: Colab + small API budget
• Summary: FreshQA: 600 questions split by answer volatility (never/slow/fast-changing, false-premise). All LLMs fail on fast-changing and false-premise questions; FreshPrompt (retrieved snippets with dates, ordered so the freshest is last) fixes much of it and beats Self-Ask and Perplexity. Number and order of evidences matter.
• Why it matters: The canonical "RAG as a cure for staleness" paper, extending the spring mini-RAG and Barnett's failure points with a temporal failure mode.
• Pair implementation: Reproduce the volatility gap with a local 3-8B model on the current FreshQA sheet, using Wikipedia API results as evidence (free); test evidence ordering and date-tagging. Apply to the scientific testbed by adding "as of " constraints and checking whether the reader respects document dates.
• Discussion hooks: Is freshness a retrieval problem or a reader problem? How should nDCG account for temporal validity of a relevant document?

8. TempRetriever: Fusion-based Temporal Dense Passage Retrieval for Time-Sensitive Questions (Abdallah et al., WSDM 2026)

• Link: https://arxiv.org/abs/2502.21024 (arXiv:2502.21024) · Code: none found (datasets ArchivalQA, ChroniclingAmericaQA are public)
• Tag: recent · Difficulty: 2 · Compute: Colab
• Summary: Extends DPR by embedding the query date and document timestamp and fusing them with the text embeddings, plus time-based hard negatives (same topic, wrong period). Gains of +6.6 / +9.6 points Top-1 over DPR on ArchivalQA / ChroniclingAmericaQA. Simpler training-free cousin: TempRALM (Gade & Jetcheva 2024, https://arxiv.org/abs/2401.13222) adds a temporal-proximity score to the retriever score; Matryoshka variant: TMRL (Huynh et al. 2026, https://arxiv.org/abs/2601.05549).
• Why it matters: Puts a temporal signal into the bi-encoder students already fine-tune; hard negatives from time rather than lexical overlap.
• Pair implementation: Reproduce on a 20k-passage ChroniclingAmericaQA slice with MiniLM/BGE-small: text-only vs. date-fusion vs. +time negatives. Apply on the scientific-article testbed (publication years available): does date fusion help queries phrased with a period ("early work on...")? Compare with TempRALM's post-hoc score.
• Discussion hooks: Should time live in the embedding or in a separate filter/score? What happens to documents whose timestamp is unknown or wrong?

9. Query Drift Compensation: Enabling Compatibility in Continual Learning of Retrieval Embedding Models (Goswami et al., CoLLAs 2025)

• Link: https://arxiv.org/abs/2506.00037 (arXiv:2506.00037) · Code: https://github.com/dipamgoswami/QDC
• Tag: recent · Difficulty: 2 · Compute: Colab
• Summary: When a retriever is fine-tuned on newly arriving data, its query embeddings drift and become incompatible with the already-indexed corpus, forcing re-indexing. The paper builds a continual-learning benchmark for embedding models, uses embedding distillation for stability, and projects new-model query embeddings back into the old space at retrieval time, avoiding re-indexing. Companion: "Still Fresh? Evaluating Temporal Drift in Retrieval Benchmarks" (Kuissi, Subrahmanyan, Thakur & Lin 2026, https://arxiv.org/abs/2603.04532) compares FreshStack snapshots one year apart and finds relevant docs move but model rankings stay stable; DSI++ (Mehta et al., EMNLP 2023, https://arxiv.org/abs/2212.09744) is the generative-retrieval version of the same problem.
• Why it matters: Practical face of "evolving corpora": the FAISS index students built in spring becomes stale the moment the encoder is updated.
• Pair implementation: Split the MS MARCO subset (or SciFact+NFCorpus+ArguAna as sequential domains) into 3 sessions; fine-tune BGE-small sequentially and measure old-index compatibility with/without distillation and QDC. Apply to the class pipeline: how much nDCG is recovered without re-encoding the 92k passages?
• Discussion hooks: Is backward compatibility worth the accuracy ceiling it imposes? Should benchmarks be re-annotated yearly, as "Still Fresh?" suggests is not necessary?

10. Time is Encoded in the Weights of Finetuned Language Models (Nylund, Gururangan & Smith, ACL 2024)

• Link: https://aclanthology.org/2024.acl-long.141/ · https://arxiv.org/abs/2312.13401 (arXiv:2312.13401) · Code: none verified (repo URL in paper; not reachable during this check)
• Tag: recent · Difficulty: 2 · Compute: Colab
• Summary: Fine-tune a model on text from one year, subtract the pre-trained weights: the resulting "time vector" improves performance on that year, adjacent years' vectors are close in weight space, and interpolating between them yields models for intervening or future years without training. Holds across tasks, domains, sizes and time scales.
• Why it matters: The literal "timeline on embeddings": temporal generalisation of LMs seen as geometry in parameter space; complements the instructor's alignment/uniformity view of embedding geometry.
• Pair implementation: Reproduce with GPT-2-small (or T5-small) on arXiv cs abstracts from 2012/2016/2020/2024: time-vector cosine matrix, perplexity on held-out years, interpolation to a skipped year. Apply the idea to a bi-encoder: fine-tune MiniLM per period on the scientific testbed and test whether a "retrieval time vector" transfers.
• Discussion hooks: Does a retriever have a "date"? Could one ship a single index plus interpolated encoders instead of re-training?

11. Diachronic Word Embeddings Reveal Statistical Laws of Semantic Change — HistWords (Hamilton, Leskovec & Jurafsky, ACL 2016)

• Link: https://aclanthology.org/P16-1141/ · https://arxiv.org/abs/1605.09096 (arXiv:1605.09096) · Code: https://github.com/williamleif/histwords (embeddings 1800-2000 for English, French, German, Chinese)
• Tag: classic · Difficulty: 1 · Compute: laptop
• Summary: Trains word2vec/SVD/PPMI embeddings per decade, aligns them with orthogonal Procrustes, and validates against attested semantic shifts. Proposes two laws: frequent words change slower (conformity) and polysemous words change faster (innovation). Modern follow-ups: SemEval-2020 Task 1 (Schlechtweg et al. 2020, https://arxiv.org/abs/2007.11464), the gold standard for change detection in EN/DE/LA/SV, and Periti & Tahmasebi's systematic comparison of contextualised embeddings (NAACL 2024, https://arxiv.org/abs/2402.12011), where XL-LEXEME beats BERT variants and matches GPT-4.
• Why it matters: The origin of "embeddings as measurement instrument for time"; students see that alignment of embedding spaces (spring: Procrustes-free contrastive spaces) is the crux of any diachronic analysis.
• Pair implementation: Reproduce the Procrustes pipeline and the conformity law on a corpus with dates (arXiv cs.CL abstracts by 5-year bins, or HistWords vectors). Evaluate on SemEval-2020 English (Spearman with gold change scores); then compare with a contextual method (XL-LEXEME or BERT clustering) as in Periti & Tahmasebi. Optional split for a 13th pair: contextualised LSC as its own topic.
• Discussion hooks: Do the "laws" survive with contextual embeddings and LLM-era corpora? Is semantic change in science (terms like "attention", "transformer") the same phenomenon as in everyday language?

12. Unsupervised word embeddings capture latent knowledge from materials science literature (Tshitoyan et al., Nature 2019)

• Link: https://www.nature.com/articles/s41586-019-1335-8 · Code: https://github.com/materialsintelligence/mat2vec
• Tag: classic · Difficulty: 2 · Compute: laptop / Colab
• Summary: word2vec trained on 3.3M materials-science abstracts encodes the periodic table and structure-property relations without supervision. Crucially, embeddings trained on abstracts up to year T rank candidate compounds by similarity to "thermoelectric"; many top candidates are reported as thermoelectrics only in later years, i.e. the embedding space anticipates discoveries. Follow-up: Sourati & Evans, "Accelerating science with human-aware AI" (2023, https://arxiv.org/abs/2306.01495), adds the author network and reaches up to 400% better prediction; a scientometric version for topic evolution is Harikandeh et al., Scientometrics 2023 (doi:10.1007/s11192-023-04642-4).
• Why it matters: This is the paper for "embeddings as a tool to trace how a field or technology evolves"; it connects to the instructor's scholarly/patent embedding work and to Questel's prior-art and trend use cases (TechToken, covered by another theme, is its LLM-era patent descendant).
• Pair implementation: Reproduce the backtest at small scale: train word2vec (and, for contrast, average SPECTER2/ModernBERT document vectors) on abstracts of one subfield year by year (e.g. arXiv cs.IR or the scientific testbed), and measure precision@k of "concepts near X in year T" that co-occur with X after T. Apply: plot centroid trajectories of subtopics over years on the testbed as an embedding timeline of the field.
• Discussion hooks: Prediction or hindsight bias? Does a contrastively trained document encoder (SPECTER-style) smooth away exactly the temporal signal word2vec preserves?

Framing resources for the instructor's 30-min intro

• Goworek, Macmillan-Scott & Özyiğit, "Bridging Language Gaps: Advances in CLIR with Multilingual LLMs" (2025 survey, https://arxiv.org/abs/2510.00908): the translation-to-embedding shift in one read.
• Ciancone et al., "Extending the Massive Text Embedding Benchmark to French" (MTEB-French, 2024, https://arxiv.org/abs/2405.20468; code/leaderboard https://github.com/Lyon-NLP/mteb-french): the French model landscape; also a possible alternate paper for a pair.
• Periti & Montanelli, "Lexical Semantic Change through LLMs: a Survey" (ACM CSUR 2024, https://dl.acm.org/doi/10.1145/3672393) and Kutuzov et al., "Diachronic word embeddings and semantic shifts: a survey" (COLING 2018, https://aclanthology.org/C18-1117.pdf).
• Piryani et al., "It's High Time: A Survey of Temporal Question Answering" (ACL 2026, https://arxiv.org/abs/2505.20243); and for the "history of embeddings" timeline slide, Nguyen, Sai & Nguyen, "Tracing the Evolution of Word Embedding Techniques in NLP" (2026 bibliometric study, https://arxiv.org/abs/2603.13271).

Notes

• Verified alternates (Part A): Multilingual E5 technical report (Wang et al. 2024, https://arxiv.org/abs/2402.05672), the natural baseline for 1-6; mGTE (Zhang et al. 2024, https://arxiv.org/abs/2407.19669) and jina-reranker-v3 (Wang, Li & Xiao 2025, https://arxiv.org/abs/2509.25085) for multilingual reranking; bge-reranker-v2-m3 has no dedicated paper (its model card cites BGE-M3 and arXiv:2312.15503) and jina-reranker-v2 is blog-only, so I did not list them as papers. Translate-Distill (Yang et al., ECIR 2024, https://arxiv.org/abs/2401.04810) is a good CLIR training paper but needs MT of training data at scale. Chirkova et al. (Naver Labs Europe), "RAG in multilingual settings" (2024, https://arxiv.org/abs/2407.01463) is a French-lab option if the instructor wants an mRAG topic. CIRAL (Adeyemi et al., SIGIR 2024, https://dl.acm.org/doi/10.1145/3626772.3657884) covers African-language CLIR but its 2.5M-passage corpus is too large for laptops.
• Tokenizer/script effects: I found no strong retrieval-specific paper; the closest verified work is Xhelili, Liu & Schütze, "Breaking the Script Barrier..." (Findings of EMNLP 2024, https://arxiv.org/abs/2406.19759), which is about PLM alignment rather than IR. I left it out.
• Verified alternates (Part B): TempRALM (2024) and TMRL (2026) are folded into entry 8; "Still Fresh?" (2026) and DSI++ (2023) into entry 9. "RAG or Learning?" (Liu, Cao & Li 2026, https://arxiv.org/abs/2604.05096) is a fresh knowledge-drift benchmark but LLM-centric. Re3 (Cao et al. 2025) was withdrawn by its authors, dropped. "Test of Time" (Fatemi et al. 2024, https://arxiv.org/abs/2406.09170) is pure temporal reasoning, no retrieval, dropped. (Chat)GPT v BERT (Periti et al., Findings of EACL 2024, https://arxiv.org/abs/2401.14040, code https://github.com/FrancescoPeriti/ChatGPTvBERT), Dynamic Contextualized Word Embeddings (Hofmann et al., ACL 2021, https://arxiv.org/abs/2010.12684), Yao et al. "Dynamic Word Embeddings for Evolving Semantic Discovery" (WSDM 2018, https://arxiv.org/abs/1703.00607) and DHPLT (Fedorova et al., LChange'26, https://arxiv.org/abs/2602.11968, diachronic corpora in 41 languages) are all valid substitutes for entry 11.
• Uncertainties: FreshLLMs is cited here as arXiv 2023 only (a later ACL venue is likely but I did not verify it); jina-embeddings-v3 likewise as arXiv. Code for Zuo et al., Yang et al. and TempRetriever was not found on the abstract pages. The time-vectors GitHub URL I tried returned 404. ArchivalQA derives from the NYT corpus; ChroniclingAmericaQA is the license-safe choice for entry 8.
• Overlaps with other themes: entry 2's Matryoshka angle touches the embedding-model theme; entry 9's FreshStack link touches RAG evaluation; entry 12 is the classic ancestor of TechToken (patent trend forecasting), which another agent covers; MMTEB, ModernBERT and EuroBERT were deliberately not listed.
• Process note: the shared WebSearch budget ran out mid-task; remaining verification was done with direct page fetches (arXiv abstracts, ACL Anthology, dblp, GitHub, Semantic Scholar API). Every paper above has a URL I actually fetched or saw in a result list.

Theme 7: LLM rerankers, LLM query rewriting, reasoning-intensive retrieval

Default slot: Session 4 (Nov 27) — LLMs in the loop.

Twelve candidates (3 classics, 9 recent), grouped as rerankers (1-6), query expansion (7-10), reasoning-intensive retrieval (11-12). Every URL below was seen in a search result or fetched page.

A. LLM rerankers

1. Is ChatGPT Good at Search? Investigating LLMs as Re-Ranking Agents — "RankGPT" (Sun et al., EMNLP 2023, outstanding paper)

• Link: https://arxiv.org/abs/2304.09542 (arXiv:2304.09542) · https://aclanthology.org/2023.emnlp-main.923/ · Code: https://github.com/sunnweiwei/rankgpt
• Tag: classic · Difficulty: 1 · Compute: Colab (+ optional few-euro API budget)
• Summary: Asks an LLM to output a permutation of candidate passage IDs (listwise), using a sliding window (e.g. 20 docs, stride 10) to rerank top-100. GPT-4 matches or beats supervised cross-encoders on TREC DL/BEIR; the authors introduce NovelEval to rule out contamination and distill GPT-3.5 permutations into a 440M model that beats a 3B supervised reranker on BEIR.
• Why it matters: Direct successor to the spring cross-encoder stage (Nogueira & Cho); the distillation result is the bridge from "API toy" to a deployable reranker.
• Pair implementation: Reproduce listwise sliding-window reranking of BM25 top-50 on SciFact/NFCorpus with an open 7B model in 4-bit (Qwen2.5-7B-Instruct or the open RankZephyr checkpoint via rank_llm); compare nDCG@10 against ms-marco-MiniLM-L-6-v2. Then distil: train MiniLM cross-encoder on LLM-produced permutations for the MS MARCO subset (RankNet loss) and check whether it beats the original.
• Discussion hooks: Is a permutation over 20 IDs really "ranking", or a very expensive pairwise comparator? How much of RankGPT's gain is memorised MS MARCO?

2. A Setwise Approach for Effective and Highly Efficient Zero-shot Ranking with LLMs (Zhuang et al., SIGIR 2024)

• Link: https://arxiv.org/abs/2310.09497 (arXiv:2310.09497) · https://dl.acm.org/doi/10.1145/3626772.3657813 · Code: https://github.com/ielab/llm-rankers
• Tag: recent · Difficulty: 2 · Compute: laptop (Flan-T5-large) / Colab
• Summary: The cleanest head-to-head of the four prompting paradigms: pointwise (yes/no or query-likelihood), pairwise (PRP, Qin et al., arXiv:2306.17563, with bubble/heap sort), listwise (RankGPT), and the proposed setwise prompt ("which of these k passages is most relevant?") that reads the answer from the logits of a single decoding step. Setwise reaches pairwise-level effectiveness with far fewer LLM calls and tokens.
• Why it matters: The perfect "taxonomy" paper for the seminar; the repo implements all four paradigms with Flan-T5 so students can measure the cost/effectiveness Pareto front themselves.
• Pair implementation: Reproduce the pointwise/pairwise/setwise/listwise comparison with Flan-T5-large (and one Qwen2.5-1.5B/3B) on SciFact and NFCorpus top-30, logging #inferences, tokens and latency. Add TourRank (Chen et al., arXiv:2406.11678, WWW 2025, code https://github.com/chenyiqun/TourRank) as a fifth, tournament-style strategy and test robustness to initial ordering.
• Discussion hooks: Why do off-the-shelf LLMs "not understand" listwise prompts but handle "pick the best of 5"? Is sorting with an LLM comparator a solved problem or a cost trap?

3. Fine-Tuning LLaMA for Multi-Stage Text Retrieval — RepLLaMA / RankLLaMA (Ma et al., SIGIR 2024)

• Link: https://arxiv.org/abs/2310.08319 (arXiv:2310.08319) · https://dl.acm.org/doi/10.1145/3626772.3657951 · Code: https://github.com/texttron/tevatron (examples/example_repllama_vllm.md); checkpoints on HuggingFace
• Tag: recent · Difficulty: 3 · Compute: Colab (LoRA on 0.5-1.5B)
• Summary: Instead of prompting, fine-tune LLaMA-2 as (a) a bi-encoder using the EOS-token embedding with contrastive loss (RepLLaMA) and (b) a pointwise reranker with a scalar head (RankLLaMA), on MS MARCO. The pipeline beats RankGPT-4 by 0.4 / 7.3 nDCG@10 on DL19/DL20 and transfers zero-shot to BEIR; long-context LLMs also handle full documents without segmentation.
• Why it matters: Same InfoNCE/in-batch-negatives recipe as DPR and the instructor's citation-contrastive training, but with a decoder LLM as the encoder; it is the template for all "LLM embedder" models (E5-Mistral, NV-Embed, ...).
• Pair implementation: Reproduce at small scale: LoRA-fine-tune Qwen2.5-0.5B (or 1.5B) as a pointwise RankLLaMA on the 92k-passage MS MARCO subset, and as a RepLLaMA-style EOS-embedding bi-encoder with sentence-transformers MNRL loss; evaluate on SciFact/NFCorpus vs BGE-small and MiniLM cross-encoder.
• Discussion hooks: Is a decoder with a scalar head just a big cross-encoder? What does the "last token embedding" geometry look like (alignment/uniformity) versus a mean-pooled BERT?

4. Attention in LLMs Yields Efficient Zero-Shot Re-Rankers — In-Context Reranking (Chen et al., ICLR 2025)

• Link: https://arxiv.org/abs/2410.02642 (arXiv:2410.02642) · https://openreview.net/forum?id=yzloNYH3QN · Code: https://github.com/OSU-NLP-Group/In-Context-Reranking
• Tag: recent · Difficulty: 2 · Compute: Colab (3B fp16 or 7-8B in 4-bit, eager attention)
• Summary: No generation at all: put the documents and the query in one prompt, aggregate the attention mass the query tokens place on each document, and subtract the same signal computed with a content-free query to calibrate. Two forward passes (O(1)) rerank N documents; ICR beats RankGPT with the same open-weight LLM on BEIR and on multi-hop tasks, cutting latency by more than 60%.
• Why it matters: Turns the LLM into a "late-interaction" scorer reminiscent of ColBERT's MaxSim, but over attention instead of embeddings; a bridge between the reranking and interpretability literatures.
• Pair implementation: Reproduce ICR with Qwen2.5-3B/Llama-3.2-3B (fp16) or a 4-bit 7B on SciFact and NFCorpus top-20; ablate the calibration step and the choice of layers/heads. Compare nDCG@10 and wall-clock against listwise generation (paper 1) and the MiniLM cross-encoder.
• Discussion hooks: Is attention a relevance signal or an artifact of position bias? Why does calibration help so much, and does it transfer to non-instruction-tuned models?

5. Rank1: Test-Time Compute for Reranking in Information Retrieval (Weller et al., COLM 2025)

• Link: https://arxiv.org/abs/2502.18418 (arXiv:2502.18418) · Code: https://github.com/orionw/rank1 (models 0.5B-32B, AWQ quantized 7B+; 600k+ R1 traces)
• Tag: recent · Difficulty: 2 · Compute: Colab (Rank1-0.5B/1.5B/3B; 7B AWQ)
• Summary: First reranker trained to "think": DeepSeek-R1 reasoning traces for 600k+ MS MARCO query-passage pairs are distilled into Qwen2.5 models (0.5B-32B) that emit a chain-of-thought and then a relevance token. Rank1 is state-of-the-art on reasoning (BRIGHT) and instruction-following benchmarks, generalises out-of-distribution, and yields human-readable explanations; quantized versions keep most of the gains.
• Why it matters: Extends the cross-encoder with test-time compute, the same lever driving reasoning LLMs; the explanations are directly useful for the instructor's scholarly/patent retrieval settings, where relevance must be justified.
• Pair implementation: Run Rank1-0.5B/1.5B/3B (pointwise, top-20) on SciFact, NFCorpus and the scientific-article set; measure nDCG@10 and tokens generated per pair vs MiniLM cross-encoder. Then reproduce the contrarian finding of Jedidi et al. 2025, "Don't Overthink Passage Reranking" (arXiv:2505.16886; see also Lu et al., arXiv:2510.08985): force the answer without the reasoning block and compare.
• Discussion hooks: If disabling reasoning sometimes helps (polarised scores lose partial relevance), what exactly did the distillation teach? Should a reranker be graded on nDCG or on the quality of its explanation?

6. ReasonRank: Empowering Passage Ranking with Strong Reasoning Ability (Liu et al., ACL 2026)

• Link: https://arxiv.org/abs/2508.07050 (arXiv:2508.07050) · https://aclanthology.org/2026.acl-long.1005/ · Code: https://github.com/8421BCD/ReasonRank (7B/32B Qwen2.5, 8B Qwen3; 13k training set on HF)
• Tag: recent · Difficulty: 3 · Compute: Colab (7B 4-bit inference; API budget for tiny data synthesis)
• Summary: A listwise reasoning reranker: synthesises 13k reasoning-intensive training examples with DeepSeek-R1 (with a self-consistency filter), then trains in two stages, cold-start SFT followed by GRPO with a "multi-view ranking reward" tailored to sliding-window listwise ranking. ReasonRank-32B reached 40.8 nDCG@10 on BRIGHT (SOTA at the time) with lower latency than pointwise reasoning rerankers such as Rank1.
• Why it matters: Combines RankGPT's listwise format with RL, as in Rank-R1 (Zhuang et al., arXiv:2503.06034, code in ielab/llm-rankers/Rank-R1) and Rank-K (Yang et al., arXiv:2505.14432); a compact tour of how rerankers are trained in 2025-26.
• Pair implementation: Run ReasonRank-7B (4-bit) vs RankZephyr-7B on BRIGHT (2 subsets) and on SciFact; then reproduce the data side at toy scale: synthesise ~200 reasoning-labelled listwise examples with a small API budget, SFT Qwen2.5-0.5B with LoRA, and see whether any of the gain appears on NFCorpus.
• Discussion hooks: Is the reward (nDCG on windows) teaching reasoning or teaching output format? Where is the cross-over point at which a 7B "thinking" reranker beats a 100x cheaper cross-encoder in real systems?

B. LLM query expansion and rewriting

7. Precise Zero-Shot Dense Retrieval without Relevance Labels — HyDE (Gao et al., ACL 2023)

• Link: https://arxiv.org/abs/2212.10496 (arXiv:2212.10496) · https://aclanthology.org/2023.acl-long.99/ · Code: https://github.com/texttron/hyde
• Tag: classic · Difficulty: 1 · Compute: laptop + small API budget (or local 1-3B LLM)
• Summary: Have InstructGPT write a hypothetical answer document, embed it with an unsupervised encoder (Contriever), and search with that embedding (averaged with the query's); the dense bottleneck filters out hallucinated detail. Zero-shot HyDE strongly outperforms Contriever and is comparable to fine-tuned retrievers on web search, QA, fact verification and non-English queries.
• Why it matters: The canonical "generation-augmented retrieval" idea, name-dropped in spring; directly reusable in the baseline pipeline (BGE-small instead of Contriever).
• Pair implementation: Reproduce HyDE on SciFact/NFCorpus/ArguAna with all-MiniLM/BGE-small as encoder and Qwen2.5-3B or a few euros of API calls as generator; ablate number of sampled documents and query/doc weighting. Apply to the scientific-article dataset: does a hypothetical abstract help more than a hypothetical answer?
• Discussion hooks: Why does HyDE hurt ArguAna-type tasks (queries are already documents)? Is HyDE dead now that instruction-tuned embedders exist, or just cheaper than training one?

8. Query2doc: Query Expansion with Large Language Models (Wang et al., EMNLP 2023)

• Link: https://arxiv.org/abs/2303.07678 (arXiv:2303.07678) · https://aclanthology.org/2023.emnlp-main.585/ · Code: none official found
• Tag: classic · Difficulty: 1 · Compute: laptop + small API budget
• Summary: Few-shot prompt an LLM for a pseudo-document, then concatenate it to the query (repeating the query n times to control term weighting) for BM25, or feed it to a dense retriever at training and test time. Improves BM25 by 3-15% on MS MARCO and TREC DL and helps fine-tuned dense retrievers in and out of domain.
• Why it matters: The sparse-retrieval twin of HyDE and the LLM successor of doc2query-T5 from the spring course; ideal for the patent setting where BM25 remains dominant.
• Pair implementation: Reproduce Query2doc-BM25 with rank_bm25 on SciFact/NFCorpus/MS MARCO subset (vary query repetition n and generator size: Qwen2.5-1.5B/7B vs API); compare against HyDE (paper 7) and RRF of expanded+unexpanded runs. Apply on ArguAna to show the failure case.
• Discussion hooks: Expansion helps BM25 but what about SPLADE-style learned sparse? Is the LLM adding knowledge or just vocabulary?

9. When do Generative Query and Document Expansions Fail? A Comprehensive Study Across Methods, Retrievers, and Datasets (Weller et al., Findings of EACL 2024)

• Link: https://arxiv.org/abs/2309.08541 (arXiv:2309.08541) · https://aclanthology.org/2024.findings-eacl.134/ · Code: https://github.com/orionw/LM-expansions
• Tag: recent · Difficulty: 2 · Compute: Colab
• Summary: Systematic study of LM query and document expansions (HyDE, Query2doc, doc2query-style) across sparse and dense rankers on 12 datasets: expansions help weak rankers (BM25, small encoders) but hurt strong ones, and help mostly when there is a distribution shift (e.g. very long queries). The takeaway is a decision rule for when to pay for expansion.
• Why it matters: The critical companion to papers 7-8 and a model of good empirical IR methodology; connects to the spring discussion of doc2query-T5 and BEIR zero-shot transfer.
• Pair implementation: Reproduce the "weak vs strong ranker" interaction on the shared testbed: BM25, MiniLM, BGE-small/base, and the MiniLM cross-encoder, with and without HyDE/Query2doc expansions on SciFact, NFCorpus, ArguAna. Extend with Abe et al. (SIGIR 2025 short, arXiv:2505.12694, code https://github.com/aken12/LLM-based-QE-fails): bucket queries by LLM familiarity/ambiguity.
• Discussion hooks: If expansion only helps weak retrievers, is it obsolete for production systems? How would you detect an "unfamiliar" query before expanding it?

10. Corpus-Steered Query Expansion with Large Language Models — CSQE (Lei et al., EACL 2024 short)

• Link: https://arxiv.org/abs/2402.18031 (arXiv:2402.18031) · https://aclanthology.org/2024.eacl-short.34/ · Code: https://github.com/Yibin-Lei/CSQE
• Tag: recent · Difficulty: 2 · Compute: Colab + small API budget (or local 7B 4-bit)
• Summary: Fixes the hallucination/outdated-knowledge problem of Query2doc by grounding expansion in the corpus: an initial BM25 run, the LLM judges which retrieved documents are relevant and extracts their pivotal sentences, and these sentences (plus optional LLM-generated text) expand the query. Training-free; strongest exactly on queries where the LLM lacks knowledge.
• Why it matters: The LLM reincarnation of pseudo-relevance feedback (RM3), alongside Generative Relevance Feedback (Mackie et al., SIGIR 2023, arXiv:2304.13157; MAP +5-19% over RM3) and LameR (Shen et al., Findings of ACL 2024, arXiv:2304.14233). Very relevant to patent prior-art search, where the corpus vocabulary, not world knowledge, is what matters.
• Pair implementation: Reproduce CSQE with rank_bm25 + Qwen2.5-7B (4-bit) on SciFact/NFCorpus/MS MARCO subset; compare RM3 (Pyserini), Query2doc, GRF-style generation-only, and CSQE. Apply to the scientific-article dataset and analyse on which queries corpus-steering beats pure generation.
• Discussion hooks: Is CSQE just RM3 with an LLM as the relevance oracle? Does grounding kill the "knowledge injection" benefit of Query2doc?

C. Reasoning-intensive and instruction-following retrieval

11. BRIGHT: A Realistic and Challenging Benchmark for Reasoning-Intensive Retrieval (Su et al., ICLR 2025)

• Link: https://arxiv.org/abs/2407.12883 (arXiv:2407.12883) · https://iclr.cc/virtual/2025/poster/27702 · Code/Data: https://github.com/xlang-ai/BRIGHT · leaderboard https://brightbenchmark.github.io/
• Tag: recent · Difficulty: 2 · Compute: Colab
• Summary: 1,384 real queries from StackExchange, LeetCode, maths and theorem-based tasks (12 subsets) where relevance requires reasoning about principles rather than lexical/semantic overlap. The best MTEB retriever (SFR-Embedding-Mistral, 59.0 on MTEB) drops to 18.3 nDCG@10; LLM reasoning-augmented query rewriting adds up to 12.2 points, and reranking with LLMs helps further. The leaderboard has since moved from ~18 to 66.9 (April 2026).
• Why it matters: Name-dropped in spring; it is the "BEIR of 2025" and the yardstick for papers 5, 6, 12. Connects to the instructor's interest in scholarly retrieval (theorem/paper subsets).
• Pair implementation: Reproduce the baseline table on 3 small subsets (e.g. biology, economics, theorem-questions) with BM25, BGE-small, and the spring pipeline; then reproduce the "reason-then-retrieve" gain using Qwen2.5-7B to rewrite queries. Apply BRIGHT-style reasoning rewrites to ArguAna and the scientific-article set to see whether they help on non-reasoning tasks.
• Discussion hooks: Is BRIGHT measuring retrieval or the LLM's ability to solve the question first? How much of the leaderboard progress is test-set-specific tuning?

12. ReasonIR: Training Retrievers for Reasoning Tasks (Shao et al., 2025)

• Link: https://arxiv.org/abs/2504.20595 (arXiv:2504.20595) · Code: https://github.com/facebookresearch/ReasonIR (ReasonIR-8B, data and synthesis code)
• Tag: recent · Difficulty: 3 · Compute: Colab (8B 4-bit inference; small-scale synthesis + BGE-small fine-tuning)
• Summary: Trains a bi-encoder from Llama-3.1-8B on synthetic data built for reasoning: varied-length queries and "hard" queries whose relevant documents are only reachable by reasoning, with hard negatives. ReasonIR-8B reaches 29.9 nDCG@10 on BRIGHT (36.9 with a reranker), and as a RAG retriever improves MMLU by 6.4% and GPQA by 22.6% over closed-book.
• Why it matters: The dense-retrieval answer to BRIGHT and the data-centric complement to reasoning rerankers; the synthetic-hard-query recipe is contrastive training with clever negatives, close to the instructor's citation-contrastive work. DIVER (Sun et al., arXiv:2508.07995, code https://github.com/AQ-MedAI/Diver, Diver-Retriever-4B) packages the same idea into a full expansion + retrieval + reranking pipeline.
• Pair implementation: Evaluate ReasonIR-8B (4-bit) vs BGE-base on two BRIGHT subsets and SciFact. Then reproduce the recipe at toy scale: generate ~1k "hard" synthetic queries for NFCorpus documents with Qwen2.5-7B, fine-tune BGE-small with MNRL and hard negatives, and test on BRIGHT subsets and the shared testbed.
• Discussion hooks: Can a single vector ever encode "reasoning-relevance", or is this only pushing the reasoning into the training data? Why does the reranker still add 7 points on top?

Framing resources for the instructor's 30-min intro

1. Zhu et al., "Large Language Models for Information Retrieval: A Survey", ACM TOIS (arXiv:2308.07107, v4 Sept 2025): https://arxiv.org/abs/2308.07107 ; paper list at https://github.com/RUC-NLPIR/LLM4IR-Survey. Covers query rewriter / retriever / reranker / reader taxonomy.
2. SIGIR 2025 tutorial "R2LLMs: Retrieval and Ranking with LLMs" (Zhuang, Ma et al.): https://dl.acm.org/doi/10.1145/3726302.3731689 ; SIGIR-AP 2025 edition https://dl.acm.org/doi/10.1145/3767695.3769522. Slides walk through pointwise/pairwise/setwise/listwise and reasoning rerankers.
3. BRIGHT leaderboard https://brightbenchmark.github.io/ (short/long-doc tracks; 18 -> 66.9 nDCG@10 in two years) as the "timeline" slide.
4. Tooling: rank_llm (RankVicuna/RankZephyr/FIRST, https://github.com/castorini/rank_llm) and ielab/llm-rankers; plus the SIGIR 2025 reproducibility paper "Beyond Reproducibility: Advancing Zero-shot LLM Reranking Efficiency" https://dl.acm.org/doi/10.1145/3726302.3730323 for the efficiency picture. Song & Zheng, "A Survey of Query Optimization in LLMs" (arXiv:2412.17558) for expansion/decomposition/disambiguation taxonomy.

Notes

• Instruction-following retrieval (briefly, likely covered elsewhere): FollowIR (Weller et al., NAACL 2025, arXiv:2403.15246, https://aclanthology.org/2025.naacl-long.597/) and Promptriever (Weller et al., 2024, arXiv:2409.11136, code https://github.com/orionw/promptriever; +14.3 p-MRR on FollowIR, prompting a bi-encoder like an LLM). A cheap pair task would be prompting Promptriever with patent-style negative constraints on ArguAna; I did not give it a full slot.
• Folded rather than dropped: RankZephyr (Pradeep et al., arXiv:2312.02724) / RankVicuna (arXiv:2309.15088) appear as open-source stand-ins for RankGPT; FIRST (Reddy et al., EMNLP 2024, arXiv:2406.15657, code https://github.com/gangiswag/llm-reranker, 50% faster via first-token logits) would make a good alternate for paper 1 if two pairs want listwise reranking. TourRank, PRP, Rank-R1, Rank-K, GRF, LameR, DIVER, and the two "reasoning does not help" papers are embedded in the entries above.
• Dropped: RAG-Fusion (Rackauckas, arXiv:2402.03367) exists but is a thin industry report; multi-query + RRF is better taught as an extension task inside papers 7-8. Rewrite-Retrieve-Read (Ma et al., EMNLP 2023, arXiv:2305.14283) is RAG-oriented and probably belongs to a RAG theme.
• Compute caveats: ICR needs raw attention weights (eager attention), which limits students to ~3B fp16 or 4-bit 7B on a T4; ReasonRank/ReasonIR training is out of reach, so those entries are inference + toy-scale reproduction of the data recipe.
• Uncertain: TourRank's venue is inferred from ACM DOI 10.1145/3696410.3714863 (WWW 2025 proceedings prefix); Query2doc has no official code that I could confirm; ReasonIR's conference venue is not stated on arXiv.
• Overlap with other themes: RepLLaMA (LLM embedders), BRIGHT (benchmarks), Promptriever/FollowIR (instruction-following), Rank1's explanations (LLM-as-a-judge).

Theme 8: Generative retrieval, learning to rank, click models, distillation (the classics the spring course skipped)

Default slot: Session 4 (Nov 27) — LLMs in the loop.

Candidate papers — Generative retrieval, LTR, click models, distillation

1. Transformer Memory as a Differentiable Search Index (Tay et al., NeurIPS 2022)

• Link: https://arxiv.org/abs/2202.06991 (arXiv:2202.06991) · Code: no official release; the DSI-QG repo (https://github.com/ArvinZhuang/DSI-QG) contains a plain DSI trainer
• Tag: classic · Difficulty: 2 · Compute: Colab
• Summary: One T5 does both indexing (doc text → docid) and retrieval (query → docid), trained multi-task; retrieval is beam search over docids. Docid variants: atomic, naive string, semantic string (hierarchical k-means over BERT embeddings). On NQ320K, DSI-XXL with semantic docids reaches Hits@1 40.4 vs 24.3 for a T5 dual encoder and 11.6 for BM25; on NQ10K the gap is 2x. Indexing-only ("zero-shot") DSI already beats BM25.
• Why it matters: The concrete realisation of Metzler et al. 2021 "Rethinking search" (name-dropped in spring); the paradigm the whole GR block orbits, and the antithesis of the BM25 → bi-encoder → FAISS pipeline students know.
• Pair implementation: Reproduce DSI on NQ10K with T5-small/base on Colab (semantic vs naive string docids). Apply on SciFact: build semantic docids by hierarchical k-means over BGE-small embeddings, index with doc2query-T5 pseudo-queries (spring tool), report nDCG@10 vs BM25/BGE.
• Discussion hooks: Is a model that memorises the corpus an index or a language model? What happens when 100 new documents arrive tomorrow?

2. How Does Generative Retrieval Scale to Millions of Passages? (Pradeep et al., EMNLP 2023)

• Link: https://aclanthology.org/2023.emnlp-main.83/ (arXiv:2305.11841) · Code: none found (use the DSI-QG repo above)
• Companion: DSI-QG, Zhuang et al. 2022, https://arxiv.org/abs/2206.10128 (arXiv:2206.10128)
• Tag: classic · Difficulty: 2 · Compute: Colab
• Summary: First systematic scaling study of GR, from 100k passages up to the full 8.8M MS MARCO, with T5 up to 11B. Findings: synthetic queries as the indexing representation are the single essential ingredient (confirming DSI-QG); proposed architectural tweaks do not pay off once compute is accounted for; naively scaling parameters does not close the gap; GR is competitive with dual encoders on small corpora but far behind at 8.8M.
• Why it matters: A model negative-result paper that teaches how to design a scaling ablation; links directly to doc2query-T5 from the spring course.
• Pair implementation: Reproduce the scaling curve at laptop scale on the MS MARCO subset: index 1k / 10k / 92k passages with T5-small (atomic vs string docids) using doc2query pseudo-queries; plot MRR@10 vs corpus size against BM25 and MiniLM. Ablate 1/5/10 synthetic queries per doc.
• Discussion hooks: If GR needs pseudo-queries from a model that already ranks (doc2query), what is GR adding? Should GR target small closed corpora (a patent family, a lab's papers) rather than the web?

3. From RankNet to LambdaRank to LambdaMART: An Overview (Burges, MSR-TR-2010-82, 2010)

• Link: https://www.microsoft.com/en-us/research/publication/from-ranknet-to-lambdarank-to-lambdamart-an-overview/ (PDF: https://www.microsoft.com/en-us/research/wp-content/uploads/2016/02/MSR-TR-2010-82.pdf) · Code: LightGBM lambdarank / rank_xendcg objectives (https://lightgbm.readthedocs.io/en/latest/Parameters.html)
• Companion: Liu 2009 survey, pointwise/pairwise/listwise, https://dl.acm.org/doi/10.1561/1500000016
• Tag: classic · Difficulty: 1 · Compute: laptop
• Summary: RankNet = pairwise cross-entropy on score differences; its gradients ("lambdas") can be rescaled by |ΔnDCG| of swapping the pair, giving LambdaRank, which optimises a non-smooth metric directly; LambdaMART plugs the lambdas into gradient-boosted trees. An ensemble of LambdaMARTs won the 2010 Yahoo! LTR Challenge and it is still the default production ranker.
• Why it matters: LTR was only name-dropped; it is the natural fourth stage of the spring pipeline (features = BM25, dense cosine, CE score, RRF rank → LambdaMART) and the technology behind patent-search engines like the instructor's employer.
• Pair implementation: Reproduce LambdaMART on MSLR-WEB10K fold 1 with LightGBM (136 features, nDCG@10), comparing pointwise regression, lambdarank, rank_xendcg. Apply: feature-based hybrid LTR on the MS MARCO subset (train 700 / test 300 queries) with BM25, MiniLM/BGE, CE and RRF features; does learned fusion beat RRF? Show feature importances.
• Discussion hooks: Why did a 2010 GBDT survive the transformer wave? Is RRF just LTR without training data?

4. Are Neural Rankers still Outperformed by Gradient Boosted Decision Trees? (Qin et al., ICLR 2021)

• Link: https://openreview.net/forum?id=Ut1vF_q_vC ; abstract also at https://research.google/pubs/are-neural-rankers-still-outperformed-by-gradient-boosted-decision-trees/ · Code: none found (authors are the TF-Ranking team)
• Tag: classic · Difficulty: 2 · Compute: laptop/Colab
• Summary: Audits 2017-2020 neural LTR papers and shows most are inferior "by a large margin" to the best public GBDT on feature-based benchmarks; identifies causes (no feature transformation, no augmentation, weak listwise losses) and proposes a unified framework of counter-strategies that reaches parity with the tree baseline while beating published neural rankers.
• Why it matters: A methodology lesson on baselines in the "are we really making progress?" genre; pairs naturally with #3.
• Pair implementation: Reproduce the gap on MSLR-WEB10K: LightGBM vs a plain MLP with listwise softmax loss vs MLP + log1p feature transform + Gaussian-noise augmentation; how much of the gap closes? Apply both rankers to the hybrid-feature testbed of #3 (coordinate the pairs so tables are comparable).
• Discussion hooks: Is neural LTR worth it when the features are already neural (dense and CE scores)? Are 2010 feature datasets still representative?

5. Unbiased Learning-to-Rank with Biased Feedback (Joachims et al., WSDM 2017)

• Link: https://arxiv.org/abs/1608.04468 (arXiv:1608.04468); https://dl.acm.org/doi/10.1145/3018661.3018699 · Code: Propensity SVM-rank, https://www.cs.cornell.edu/people/tj/svm_light/svm_proprank.html
• Companions: Craswell et al. WSDM 2008 cascade model, https://dl.acm.org/doi/10.1145/1341531.1341545; Chuklin, Markov, de Rijke 2015 Click Models for Web Search (https://books.google.com/books/about/Click_Models_for_Web_Search.html?id=0_5PCgAAQBAJ) with PyClick (https://github.com/markovi/PyClick)
• Tag: classic · Difficulty: 2 · Compute: laptop
• Summary: Clicks are relevance observed with propensity P(examined | rank). Inverse-propensity-scored empirical risk is an unbiased estimator of the full-information risk; click models play the role of the propensity estimator (estimated with a swap intervention). Derives Propensity SVM-rank and shows it beats naive click training in simulation and in a live search experiment.
• Why it matters: The counterfactual reasoning that also underlies offline evaluation from logs; connects to interleaving and to the gap between expert nDCG (spring) and implicit feedback.
• Pair implementation: Simulate clicks on BM25 rankings of the MS MARCO subset (qrels as relevance, PyClick cascade/PBM with tunable position bias), then train LightGBM lambdarank on naive clicks vs IPS-weighted clicks vs full labels; plot nDCG@10 vs bias severity and click volume. Bonus: estimate propensities from a simulated swap experiment.
• Discussion hooks: If ULTR gains vanish on real logs (#11), what does that say about our simulators? What is the "click" in scholarly or patent search: downloads, citations, saved-to-library?

6. Efficiently Teaching an Effective Dense Retriever with Balanced Topic Aware Sampling (Hofstätter et al., SIGIR 2021)

• Link: https://arxiv.org/abs/2104.06967 (arXiv:2104.06967) · Code: https://github.com/sebastian-hofstaetter/tas-balanced-dense-retrieval (model distilbert-dot-tas_b-b256-msmarco)
• Companions: Margin-MSE cross-architecture distillation, Hofstätter et al. 2020, https://arxiv.org/abs/2010.02666; RocketQAv2, Ren et al. EMNLP 2021, https://arxiv.org/abs/2110.07367 (code https://github.com/PaddlePaddle/RocketQA)
• Tag: classic · Difficulty: 2 · Compute: Colab
• Summary: Distils a cross-encoder (plus ColBERT) teacher into a DistilBERT bi-encoder with Margin-MSE (student margin regresses onto teacher margin); composes batches from one query-topic cluster so in-batch negatives are informative, and balances pairs by teacher margin. Trains on a single consumer GPU in under 48 h with batch size 32; on TREC-DL 2020 it beats BM25 by 44% and the previous best dense model by 5% nDCG@10.
• Why it matters: Turns the spring's "InfoNCE and hard negatives (explained, never implemented)" into an implemented recipe; the mechanism behind the sentence-transformers msmarco-*-tas-b models; a possible replacement for citation supervision in the instructor's contrastive training work.
• Pair implementation: Distil the spring CE (ms-marco-MiniLM-L-6-v2) into a MiniLM-L6 bi-encoder on ~50k MS MARCO triples with Margin-MSE vs InfoNCE, random vs topic-aware batches (k-means on query embeddings). Evaluate zero-shot on SciFact/NFCorpus/ArguAna against off-the-shelf all-MiniLM-L6-v2.
• Discussion hooks: Does the student inherit the teacher's ceiling and biases? Would a ColBERT teacher suit a bi-encoder student better than a CE?

7. Recommender Systems with Generative Retrieval — TIGER (Rajput et al., NeurIPS 2023)

• Link: https://arxiv.org/abs/2305.05065 (arXiv:2305.05065); https://proceedings.neurips.cc/paper_files/paper/2023/file/20dcab0f14046a5c6b02b61da9f13229-Paper-Conference.pdf · Code: no official; unofficial https://github.com/NonameUntitled/tiger
• Recent companions: Liu et al., KDD 2026, https://arxiv.org/abs/2509.25522 (SID-based GR saturates with scale; LLM-as-recommender scales better, up to +20%); Chen et al. (Pinterest) 2026, https://arxiv.org/abs/2606.00324 (a prefix-memory encoder for semantic IDs)
• Tag: classic · Difficulty: 2 · Compute: Colab
• Summary: Items → content embedding (Sentence-T5) → RQ-VAE → a short coarse-to-fine "Semantic ID"; a T5 trained on user histories generates the next item's SID. Beats sequential recommenders on Amazon Reviews and handles cold-start items. The GR variant that industry actually adopted.
• Why it matters: The successful cousin of DSI; semantic IDs are DSI's semantic docids made learnable, and RQ-VAE is the residual quantisation students saw in FAISS PQ — an embedding-geometry question in disguise.
• Pair implementation: Reproduce TIGER on Amazon Beauty with a tiny T5 on Colab (Recall@10 vs SASRec). Apply on the testbed: RQ-VAE semantic IDs from BGE-small embeddings of SciFact docs (3-4 codebooks × 256), train T5-small query → SID; compare with #1's k-means docids; inspect codebook collisions and what each prefix "means".
• Discussion hooks: Why do semantic IDs work for recommendation where DSI struggles for search? Is a semantic ID just product quantisation with a decoder?

8. Generative Retrieval as Multi-Vector Dense Retrieval (Wu et al., SIGIR 2024)

• Link: https://arxiv.org/abs/2404.00684 (arXiv:2404.00684); https://dl.acm.org/doi/10.1145/3626772.3657697 · Code: https://github.com/Furyton/GR-as-MVDR
• Sequel: Constrained Auto-Regressive Decoding Constrains Generative Retrieval, Wu et al., SIGIR 2025, https://arxiv.org/abs/2504.09935 (code https://github.com/Furyton/constrained-generation-in-gr)
• Tag: recent · Difficulty: 3 · Compute: Colab
• Summary: Shows GR's score is a sum of products of query-side decoder states and docid-token embeddings through an alignment matrix, i.e. formally an instance of ColBERT-style multi-vector retrieval, but aligned document→query instead of query→document. On NQ320K and MS MARCO (T5-SEAL) both paradigms favour exact token matches; in reranking the MVDR direction gets 61.3 Recall@1 vs 47.4 for GR's. The 2025 sequel derives a KL lower bound on the error caused by trie constraints and shows beam search over marginals is suboptimal out of domain.
• Why it matters: Unifies two spring concepts (ColBERT late interaction and DSI) in one equation; an ideal theory talk.
• Pair implementation: With the released code, train a T5-small SEAL-style GR on a NQ subset or SciFact, extract alignment matrices and measure exact-match rates; compare with a MiniLM late-interaction model on SciFact nDCG@10. Demo: alignment heatmaps for live queries.
• Discussion hooks: If GR is MVDR with the worse alignment direction, what remains of "the model is the index"? Does the constrained-decoding bound doom GR on BEIR-style OOD evaluation?

9. Does Generative Retrieval Overcome the Limitations of Dense Retrieval? (Zhang et al., arXiv 2025)

• Link: https://arxiv.org/abs/2509.22116 (arXiv:2509.22116) · Code: none found
• Tag: recent · Difficulty: 3 · Compute: Colab
• Summary: Theory: GR is globally normalised MLE with the corpus in the parameters; DR is a locally normalised contrastive objective with a low-rank bilinear score, so GR should degrade more slowly with corpus size and scale with parameters rather than embedding dimension. Experiments with Qwen3-0.6B GR (length-6 RQ codebook docids or title docids) vs Qwen3-Embedding-0.6B (DPR- and ColBERT-style) on NQ and MS MARCO: growing the corpus costs DR 6.9% Hit@1 but GR only 3.3%; scaling GR gives ~5% vs ~1% for DR; yet GR does not universally win.
• Why it matters: The 2025 state of "GR with LLMs", with a crisp theoretical claim; speaks to the instructor's embedding-geometry interests (is dimension the bottleneck?) and to Matryoshka from spring.
• Pair implementation: Reproduce the corpus-size/dimension ablation on the MS MARCO subset: DR = BGE-small truncated to 64/128/384 dims (MRL-style) vs GR = Qwen2.5/3-0.5B with LoRA generating RQ docids from doc2query pseudo-queries, corpus 5k → 92k; MRR@10 curves. OOD check on SciFact.
• Discussion hooks: Is "global normalisation" a real advantage or a bigger softmax? How does the low-rank argument square with 4096-d LLM embeddings?

10. Generative Retrieval Overcomes Limitations of Dense Retrieval but Struggles with Identifier Ambiguity (Bracher & Vakulenko, arXiv 2026, work in progress)

• Link: https://arxiv.org/abs/2604.05764 (arXiv:2604.05764) · Code: https://anonymous.4open.science/r/GR-on-LIMIT/
• Companion: Takacs, Bracher, Vakulenko 2026, failure taxonomy + web debugging tool for SEAL/MINDER, https://arxiv.org/abs/2606.17721
• Tag: recent · Difficulty: 1-2 · Compute: laptop
• Summary: On LIMIT (50k one-sentence docs, 1k queries; the synthetic benchmark built to expose theoretical limits of embedding retrieval), n-gram GR (SEAL, MINDER; public BART-large NQ checkpoints, no extra training) reaches Recall@2 0.92-0.95 vs <0.03 for 7-8B embedding models and 0.86 for BM25. Adding 46 semantically reversed hard negatives (LIMIT-H) drops BM25 to 0.25 and GR to ~0.60: FM-index n-gram identifiers become ambiguous.
• Why it matters: Ties GR to the embedding-limits debate and is a model of what a pair can do in four weeks: small, cheap, fully reproducible.
• Pair implementation: Reproduce LIMIT and LIMIT-H numbers with the public checkpoints (FM-index builds on CPU), then run zero-shot SEAL on SciFact/NFCorpus vs BM25/BGE and classify failures with the Takacs et al. taxonomy. Demo the debugging tool live.
• Discussion hooks: Is n-gram GR just BM25 with a neural query rewriter? Are synthetic benchmarks like LIMIT informative or adversarial toys?

11. Unbiased Learning to Rank Meets Reality: Lessons from Baidu's Large-Scale Search Dataset (Hager et al., SIGIR 2024)

• Link: https://arxiv.org/abs/2404.02543 (arXiv:2404.02543); https://dl.acm.org/doi/10.1145/3626772.3657892 · Code: https://github.com/philipphager/ultr-reproducibility (HF datasets philipphager/baidu-ultr_uva-mlm-ctr: precomputed 768-d embeddings, LTR features, positions, clicks, expert 0-4 labels; ~35 GB total, partitioned)
• Companions: Baidu-ULTR, Zou et al. NeurIPS 2022 D&B, https://arxiv.org/abs/2207.03051 (1.2B sessions, 7,008 annotated queries); DualIPW, Yu et al. WWW 2025, https://arxiv.org/abs/2502.11414; "Can LLM Annotations Replace User Clicks for LTR?", Yu et al. CIKM 2026, https://arxiv.org/abs/2511.06635 (code https://github.com/Trustworthy-Information-Access/LLMAnn_Click)
• Tag: recent · Difficulty: 2 · Compute: Colab
• Summary: Reproducibility study of the standard ULTR toolbox (naive pointwise, regression-EM, dual learning algorithm, pairwise debiasing, two-tower) on the first billion-click real dataset (WSDM Cup 2023 / NTCIR ULTRE-2): every method robustly improves click prediction, none consistently improves nDCG on expert annotations, and results swing with the loss and features chosen. A sober counterpoint to years of simulation-only ULTR.
• Why it matters: The empirical answer to #5; teaches the difference between "unbiased w.r.t. clicks" and "better ranking".
• Pair implementation: Download one partition, train the small feed-forward rankers with naive vs DLA vs pairwise-debias losses, report expert nDCG@10 and click log-likelihood. Run the same code on #5's simulated testbed clicks (coordinate the pairs) to stage "simulation vs reality" in one table.
• Discussion hooks: If IPS does not help on real logs, is the position-bias model wrong or is relevance not what clicks measure? Should the next Baidu-ULTR be LLM-annotated (CIKM 2026 companion)?

12. Beyond Hard Negatives: The Importance of Score Distribution in Knowledge Distillation for Dense Retrieval (Jang et al., SIGIR 2026)

• Link: https://arxiv.org/abs/2604.04734 (arXiv:2604.04734); https://doi.org/10.1145/3805712.3809843 · Code: none found
• Companions: CADET, Tamber et al. 2025, https://arxiv.org/abs/2505.19274 (CE listwise distillation + multi-type synthetic queries beats InfoNCE fine-tuning; code https://github.com/manveertamber/cadet-dense-retrieval); Morand et al. CIKM 2026, https://arxiv.org/abs/2603.03010 (Margin-MSE and InfoNCE best for CE distillation; negatives matter as much as the loss; code https://github.com/xpmir/cross-encoders); Rank-DistiLLM, Schlatt et al. ECIR 2025, https://arxiv.org/abs/2405.07920
• Tag: recent · Difficulty: 2 · Compute: Colab
• Summary: With a Qwen3-Reranker-8B teacher and BERT-base/DistilBERT students on MS MARCO (KL or Margin-MSE, K=8 candidates per query), which candidates you distil on matters: top-K (hard negatives only) collapses the teacher's score distribution, while stratified sampling across the score spectrum preserves its variance and entropy. MS MARCO dev MRR@10 0.313 (stratified) vs 0.295 (random) vs 0.255 (top-K); BEIR-13 nDCG@10 0.365 vs 0.326 / 0.334.
• Why it matters: The 2026 update of #6 (TAS-B's balanced margins rediscovered with LLM teachers); invites an alignment/uniformity analysis of what stratified distillation does to the student's embedding geometry — the instructor's toolkit.
• Pair implementation: Precompute ms-marco-MiniLM CE (or Qwen3-Reranker-0.6B) scores for BM25+BGE top-100 on 5-10k MS MARCO queries; distil MiniLM-L6 students with top-K vs random vs stratified sampling (K=8, Margin-MSE); evaluate on the MS MARCO subset and SciFact/NFCorpus; plot alignment/uniformity and score histograms.
• Discussion hooks: Are hard negatives overrated? Does distilling a whole score distribution transfer the teacher's calibration or its biases?

Framing resources for the instructor's 30-min intro

1. Lin, Nogueira, Yates, Pretrained Transformers for Text Ranking: BERT and Beyond (2021), https://arxiv.org/abs/2010.06467 — multi-stage ranking, LTR lineage, distillation; frames "reranking vs learned representations".
2. Li et al., "From Matching to Generation: A Survey on Generative Information Retrieval" (TOIS 2025), https://arxiv.org/abs/2404.14851, paper list at https://github.com/RUC-NLPIR/GenIR-Survey — docid taxonomy, training, incremental indexing, generative recommendation.
3. WSDM 2024 tutorial "Unbiased Learning to Rank: On Recent Advances and Practical Applications" (Gupta, Hager, Huang, Vardasbi, Oosterhuis), https://sites.google.com/view/wsdm-2024-tutorial-ultr/ ; Chuklin/Markov/de Rijke SIGIR 2015 click-models tutorial, https://irlab.science.uva.nl/wp-content/papercite-data/pdf/chuklin-introduction-2015.pdf
4. Online evaluation and LTR data: Chapelle, Joachims, Radlinski, Yue, "Large-scale validation and analysis of interleaved search evaluation" (TOIS 2012), https://dl.acm.org/doi/10.1145/2094072.2094078 ; MSLR-WEB10K/30K (10k/30k queries, 136 features, 5-grade labels), https://www.microsoft.com/en-us/research/project/mslr/

Notes

• Qin et al. 2021: the OpenReview PDF is behind a verification wall this session, so I confirmed only the abstract and authors (OpenReview + Google Research). From memory the benchmarks are MSLR-Web30K, Yahoo! and Istella and the model is called DASALC — verify before assigning.
• DSI and Pradeep 2023 have no official code; the verified DSI-QG repo (mt5-base, 8xV100 in the paper) is the practical base and must be scaled to T5-small. SEAL/MINDER (#10) need an FM-index build (sdsl-lite): installation is the main risk. #10 and its companion are "work in progress" preprints from Vienna University of Economics and Business, not peer-reviewed.
• Dropped for compute: RIPOR (Zeng et al. WWW 2024, https://arxiv.org/abs/2311.09134, code https://github.com/HansiZeng/RIPOR), PAG (SIGIR 2024, https://arxiv.org/abs/2404.14600) and its SIGIR 2026 reproduction "Lost in Decoding?" (Mekonnen et al., https://arxiv.org/abs/2604.23396) all train on full MS MARCO; Self-Retrieval (Tang et al. NeurIPS 2024, https://arxiv.org/abs/2403.00801) needs 7B fine-tuning; "Vectorizing the Trie" (2026, https://arxiv.org/abs/2602.22647) is accelerator engineering. Good intro material for the SOTA line.
• Other verified companions not given entries: NCI (Wang et al. NeurIPS 2022, https://arxiv.org/abs/2206.02743), SEAL (Bevilacqua et al. NeurIPS 2022, https://arxiv.org/abs/2204.10628), GenRet (Sun et al. NeurIPS 2023, https://arxiv.org/abs/2304.04171). GENRE was not re-verified this session.
• Overlaps: #6/#12 border the answerai-colbert-small distillation and Rank1/RankGPT themes; I kept the bi-encoder-student side only and list Rank-DistiLLM as a companion. #10 touches LIMIT, which an embedding-limits theme may also cover. TIGER (#7) is recsys rather than IR proper.
• Interleaving got no dedicated entry (no teachable recent paper found within the search budget); it is covered via #5, #11 and framing item 4. The search budget ran out at the end, so 2026 LLM-based GR beyond the papers listed was not exhaustively surveyed.

Theme 9: RAG architectures and the generation side

Default slot: Session 5 (Dec 4) — RAG and agents.

1. Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection (Asai et al., ICLR 2024)

• Link: https://arxiv.org/abs/2310.11511 (arXiv:2310.11511) · Code: https://github.com/AkariAsai/self-rag (models selfrag/selfrag_llama2_7b, _13b on HF)
• Tag: classic · Difficulty: 3 · Compute: Colab
• Summary: Standard RAG retrieves a fixed number of passages unconditionally, which can hurt when they are irrelevant. Self-RAG trains one LM to emit reflection tokens (Retrieve?, IsREL, IsSUP, IsUSE) so it decides on demand whether to retrieve, then critiques each passage and its own output; decoding uses these tokens as a segment-level beam score. A 7B/13B Llama-2 beats ChatGPT and Llama2-chat on PopQA, fact verification and ASQA citation precision/recall.
• Why it matters: The canonical "adaptive RAG" paper (name-dropped in spring); reflection tokens turn the generator into its own relevance judge, i.e. a cross-encoder folded into the LLM.
• Pair implementation: Reproduce on a PopQA/ASQA subset with selfrag_llama2_7b in 4-bit, retrieval on/off. Then treat IsREL/IsSUP token probabilities as a passage scorer and evaluate it as a reranker on SciFact/NFCorpus (nDCG@10) against the spring cross-encoder baseline.
• Discussion hooks: Is a critic that was trained on GPT-4 labels really "self"-reflection? Does the adaptive-retrieval decision generalise outside Wikipedia-style questions?

2. Corrective Retrieval Augmented Generation (CRAG) (Yan et al., arXiv 2024)

• Link: https://arxiv.org/abs/2401.15884 (arXiv:2401.15884) · Code: https://github.com/HuskyInSalt/CRAG
• Tag: recent · Difficulty: 2 · Compute: Colab
• Summary: A lightweight T5-based retrieval evaluator scores retrieved documents and triggers one of three actions: Correct (keep, but decompose-then-recompose into key strips), Incorrect (discard and fall back to web search), or Ambiguous (both). Plug-and-play on top of RAG or Self-RAG; improves both on PopQA, Bio, PubHealth, ARC.
• Why it matters: Simplest instance of "retrieve, evaluate, correct"; siblings for the same slot are Adaptive-RAG (Jeong et al., NAACL 2024, https://aclanthology.org/2024.naacl-long.389/, code https://github.com/starsuzi/Adaptive-RAG: a T5 classifier routes no/single/multi-step retrieval) and FLARE (Jiang et al., EMNLP 2023, https://aclanthology.org/2023.emnlp-main.495/, code https://github.com/jzbjyb/FLARE: retrieve when next-sentence token confidence drops).
• Pair implementation: Reproduce the evaluator's gain on a PopQA subset with Qwen2.5-3B as reader. Apply: train/fine-tune a small evaluator (MiniLM cross-encoder) on SciFact judgments and measure how often "Incorrect" fires and whether decompose-recompose improves claim verification accuracy vs plain top-k.
• Discussion hooks: CRAG's evaluator is just a relevance classifier; how is this different from thresholding the reranker students already have? What should "web search fallback" mean in a closed corpus like patents?

3. RAPTOR: Recursive Abstractive Processing for Tree-Organized Retrieval (Sarthi et al., ICLR 2024)

• Link: https://openreview.net/forum?id=GN921JHCRw (arXiv:2401.18059) · Code: https://github.com/parthsarthi03/raptor (pluggable SBERT embeddings, Llama/Mistral summarisers)
• Tag: recent (2024 classic) · Difficulty: 2 · Compute: Colab (small API optional)
• Summary: Chunks are embedded, soft-clustered (UMAP + GMM), each cluster summarised by an LLM, and the process recurses to build a tree; retrieval searches the "collapsed tree" of leaves and summaries together. With GPT-4 it improves QuALITY by 20 absolute points; consistent gains on NarrativeQA and QASPER with SBERT + UnifiedQA too.
• Why it matters: First widely used hierarchical index; directly relevant to the instructor's full-text scientific corpus (S2Full) where questions span whole papers. Note the 2025 rebuttal: Laitenberger, Manning & Liu, "Stronger Baselines for RAG with Long-Context LMs" (EMNLP 2025, https://arxiv.org/abs/2506.03989) find a simple order-preserving retrieve-then-read (DOS RAG) matches RAPTOR.
• Pair implementation: Reproduce collapsed-tree vs leaf-only retrieval on a QuALITY subset with MiniLM embeddings and Qwen2.5-1.5B summaries. Apply: build a tree over NFCorpus or the scientific-article set, report nDCG@10 for leaf-only vs collapsed tree vs DOS-RAG baseline, plus indexing cost.
• Discussion hooks: Summaries are LLM hallucination surfaces: what happens to attribution when the retrieved unit never existed in the corpus? Are trees worth it once context windows reach 128k?

4. From Local to Global: A Graph RAG Approach to Query-Focused Summarization (Edge et al., Microsoft, arXiv 2024)

• Link: https://arxiv.org/abs/2404.16130 (arXiv:2404.16130) · Code: https://github.com/microsoft/graphrag (maintenance mode; warns indexing is expensive)
• Tag: recent · Difficulty: 3 · Compute: Colab + small API budget
• Summary: Global "sensemaking" questions ("what are the main themes?") are query-focused summarisation, not retrieval. GraphRAG extracts an entity-relation graph with an LLM, detects Leiden communities, pre-writes community summaries, and answers global queries by map-reduce over them. On ~1M-token corpora it beats naive RAG on comprehensiveness and diversity (LLM-judged), at large indexing cost.
• Why it matters: Defines the "local vs global query" distinction; patent landscaping at Questel is exactly a global query. LightRAG (Guo et al., EMNLP 2025, https://arxiv.org/abs/2410.05779, code https://github.com/HKUDS/LightRAG) is the cheap, incremental alternative with dual-level (entity/theme) retrieval and Ollama support.
• Pair implementation: Index ~500 SciFact or NFCorpus abstracts with LightRAG + local Qwen2.5-7B (or a few euros of API), reproduce naive vs local vs global on 20 hand-written global questions with an LLM judge. Apply: entity-level retrieval on SciFact, nDCG@10 vs the dense baseline (expect a loss: that is the point).
• Discussion hooks: Head-to-head LLM-judge wins on "diversity" are easy to game; what would a real evaluation of global answers look like? When does a graph beat a good embedding space?

5. From RAG to Memory: Non-Parametric Continual Learning for LLMs (HippoRAG 2) (Gutiérrez et al., ICML 2025)

• Link: https://arxiv.org/abs/2502.14802 (arXiv:2502.14802); HippoRAG 1: NeurIPS 2024, https://arxiv.org/abs/2405.14831 · Code: https://github.com/OSU-NLP-Group/HippoRAG (vLLM/local LLM support)
• Tag: recent · Difficulty: 3 · Compute: Colab + small API budget
• Summary: Hippocampal-memory analogy: an LLM does OpenIE over passages to build a knowledge graph (phrase + passage nodes); at query time seed nodes are found by embedding similarity and Personalized PageRank spreads activation to rank passages in a single step. HippoRAG 2 adds passage nodes and LLM "recognition memory" filtering of seeds; +7% on associative (multi-hop) tasks over NV-Embed-v2 while not losing on simple factual QA.
• Why it matters: The first graph-RAG that clearly beats strong dense retrievers on multi-hop; PPR over a document graph maps naturally onto citation graphs (the instructor's S2Full).
• Pair implementation: Reproduce on a 1k-passage MuSiQue/2Wiki subset with released code and Qwen2.5-7B for OpenIE (or a small API budget). Apply on SciFact/NFCorpus: recall@10 of PPR retrieval vs BGE-small; optionally replace OpenIE edges with citation links on the scientific-article set.
• Discussion hooks: Is "memory" just retrieval with a graph prior? Which failure does PPR fix that a reranker cannot?

6. Dense X Retrieval: What Retrieval Granularity Should We Use? (Chen et al., EMNLP 2024)

• Link: https://aclanthology.org/2024.emnlp-main.845/ (arXiv:2312.06648) · Code: https://github.com/chentong0/factoid-wiki (chentong00/propositionizer-wiki-flan-t5-large)
• Tag: recent · Difficulty: 1 · Compute: laptop / Colab
• Summary: Indexes Wikipedia as propositions (atomic, self-contained factoids generated by a Flan-T5) rather than passages or sentences. Proposition retrieval improves recall@5 for all six tested dense retrievers on five QA sets and gives better downstream QA at a fixed token budget, mostly by helping on long-tail entities.
• Why it matters: Extends the spring "chunking basics" into a controlled granularity study; the natural home for the chunking-context cluster: Anthropic's Contextual Retrieval (LLM-written chunk headers, 49% fewer top-20 failures, https://www.anthropic.com/engineering/contextual-retrieval), late chunking (Günther et al., arXiv:2409.04701, https://arxiv.org/abs/2409.04701), the negative result "Is Semantic Chunking Worth the Computational Cost?" (Qu et al., arXiv:2410.13070, https://arxiv.org/abs/2410.13070) and the ECIR 2025 workshop comparison "Reconstructing Context" (Merola & Singh, https://arxiv.org/abs/2504.19754). Links to the instructor's random full-text windows in S2Full.
• Pair implementation: Propositionise SciFact/NFCorpus with the released model; index passage vs sentence vs proposition with BGE-small and compare nDCG@10 / recall@5. Add two rows: contextual headers written by Qwen2.5-1.5B, and late chunking with jina-embeddings-v2-small.
• Discussion hooks: Propositions multiply index size 6x; who pays? Is chunk-level context a retrieval fix or a symptom of weak encoders (cf. long-context encoders like ModernBERT)?

7. Lost in the Middle: How Language Models Use Long Contexts (Liu et al., TACL 2024)

• Link: https://aclanthology.org/2024.tacl-1.9/ (arXiv:2307.03172) · Code: none verified (authors released data/prompts)
• Tag: classic · Difficulty: 1 · Compute: Colab
• Summary: Multi-document QA where the gold passage's position is varied among 10-30 distractors: accuracy is a U-curve, highest when the answer is first or last and dropping below the closed-book baseline in the middle, even for "long-context" models. Same effect on synthetic key-value retrieval.
• Why it matters: The empirical reason reranking order matters after retrieval, and the starting point of the whole "RAG vs long context" debate.
• Pair implementation: Reproduce the U-curve with Qwen2.5-3B/7B (and a Qwen3 model) on the paper's NQ-open setup at 10 and 20 documents. Apply: for the SciFact RAG reader, compare relevance order vs "best at both ends" vs original document order and measure answer accuracy and citation precision.
• Discussion hooks: Do 2025-2026 models still show the curve, and if not, does it change anything about reranking? Is positional bias an architecture or a training-data artefact?

8. Retrieval Augmented Generation or Long-Context LLMs? A Comprehensive Study and Hybrid Approach (Li et al., EMNLP 2024 industry)

• Link: https://arxiv.org/abs/2407.16833 (arXiv:2407.16833) · Code: none found
• Tag: recent · Difficulty: 2 · Compute: Colab
• Summary: Benchmarks RAG vs full long-context (LC) prompting across nine datasets and three LLMs: LC wins on average, but RAG is far cheaper and the two agree on >60% of queries. Self-Route lets the model answer from RAG if it can, else say "unanswerable" and fall back to LC, keeping LC-level accuracy at up to 65% less cost. A failure analysis (multi-step, general queries, long complex queries) explains where RAG loses.
• Why it matters: The clearest statement of the cost/accuracy trade-off; pairs with OP-RAG (Yu et al., https://arxiv.org/abs/2409.01666: keep retrieved chunks in document order, accuracy vs k is an inverted U), DOS RAG (see #3), CAG (Chan et al., WWW 2025, https://arxiv.org/abs/2412.15605, code https://github.com/hhhuang/CAG: preload the corpus KV cache, no retrieval) and LongRAG (Jiang, Ma & Chen, https://arxiv.org/abs/2406.15319: 4k-token retrieval units).
• Pair implementation: With Qwen2.5-7B (32k) on a QASPER/NarrativeQA subset, reproduce RAG vs LC vs Self-Route and report the accuracy-vs-tokens curve. Apply on the testbed: vary k from 1 to 30 retrieved passages, relevance order vs OP-RAG order, on SciFact claim verification; optionally a CAG variant that preloads all NFCorpus docs matching a topic.
• Discussion hooks: Is "RAG vs LC" a false dichotomy once retrieval just decides what enters the KV cache? Does the 2024 verdict survive 1M-token contexts and prefix caching?

9. RECOMP: Improving Retrieval-Augmented LMs with Compression and Selective Augmentation (Xu, Shi & Choi, ICLR 2024)

• Link: https://arxiv.org/abs/2310.04408 (arXiv:2310.04408) · Code: https://github.com/carriex/recomp (Contriever-based extractive and T5-large abstractive compressors on HF)
• Tag: recent · Difficulty: 2 · Compute: Colab
• Summary: Retrieved documents are compressed before being prepended: an extractive compressor (dual encoder selecting sentences) or an abstractive one (T5 trained by distilling GPT-3.5 summaries filtered for end-task gain). Both can return an empty string ("selective augmentation"). Reaches ~6% of the original tokens with minimal loss on NQ/TQA/HotpotQA.
• Why it matters: Context compression is the quiet workhorse of production RAG; contrast with token-level LLMLingua (Jiang et al., EMNLP 2023, https://arxiv.org/abs/2310.05736, up to 20x) and embedding-as-token xRAG (Cheng et al., NeurIPS 2024, https://arxiv.org/abs/2405.13792). Ties to the instructor's interest in what a single embedding carries.
• Pair implementation: Reproduce compression rate vs EM on an NQ subset with the released compressors and Qwen2.5-3B as reader. Apply: compress top-10 SciFact/NFCorpus passages with RECOMP-extractive vs LLMLingua-2 and report tokens-in vs claim-verification accuracy; the extractive compressor is essentially a sentence-level reranker students can train on MS MARCO.
• Discussion hooks: If the extractive compressor is a bi-encoder over sentences, is compression just retrieval at a second granularity? Who audits what the abstractive compressor dropped?

10. The Power of Noise: Redefining Retrieval for RAG Systems (Cuconasu et al., SIGIR 2024)

• Link: https://arxiv.org/abs/2401.14887 (arXiv:2401.14887) · Code: https://github.com/florin-git/The-Power-of-Noise
• Tag: recent · Difficulty: 1 · Compute: laptop / Colab
• Summary: Controlled study on NQ with Llama-2-7B, MPT-7B, Phi-2, Falcon-7B of what enters the reader: related-but-wrong distractors hurt, while random documents surprisingly help, and gold-position effects mirror Lost in the Middle. Suggests the retriever's objective should not be pure relevance.
• Why it matters: The IR community's entry into the generation side; and a fresh reproducibility hook: "The Powerless Noise" (Mazuryk et al., SIGIR 2026 reproducibility track, arXiv:2607.03615, https://arxiv.org/abs/2607.03615) reproduces the effect under the original settings but shows it appears, weakens or vanishes with small changes in model, prompt and output constraints, partly due to truncated/malformed generations. Related lens: "Sufficient Context" (Joren et al., https://arxiv.org/abs/2411.06037).
• Pair implementation: Reproduce the random-document gain with Phi-2/Qwen2.5-1.5B on the repo's NQ subset, then apply the 2026 paper's perturbations (prompt, max tokens, model). Apply on SciFact: reader accuracy with gold + k random vs gold + k hard-negative passages from the BM25/dense runs.
• Discussion hooks: Should retrievers optimise reader accuracy rather than nDCG? How many "surprising LLM findings" are decoding artefacts?

11. Enabling Large Language Models to Generate Text with Citations (ALCE) (Gao et al., EMNLP 2023)

• Link: https://aclanthology.org/2023.emnlp-main.398/ (arXiv:2305.14627) · Code: https://github.com/princeton-nlp/ALCE
• Tag: classic · Difficulty: 2 · Compute: Colab
• Summary: Benchmark (ASQA, QAMPARI, ELI5) and automatic metrics for cited generation: fluency, correctness, and citation recall/precision judged by an NLI model. Compares retrieve-then-cite prompting, summarisation/snippet inputs, interactive retrieval and post-hoc citation; even ChatGPT gets only ~50% citation recall on ELI5, and retrieval quality is the bottleneck.
• Why it matters: The evaluation standard used by Self-RAG and most later work; attribution is exactly what patent examiners and scholarly search need (Questel). Precursor: Attributed QA (Bohnet et al., 2022, https://arxiv.org/abs/2212.08037). A 2026 pointer for the discussion: MCite-RL, citation-enhanced agentic RL (arXiv:2608.21808, seen in the arXiv listing, not vetted).
• Pair implementation: Reproduce citation recall/precision on an ASQA subset with Qwen2.5-7B using a DeBERTa NLI judge instead of TRUE-T5-XXL. Apply: SciFact claims already have gold evidence sentences, so "generate a verdict with [n] citations" gives an exact citation-precision score against gold rationales, comparable across retrievers (BM25 vs dense vs reranked).
• Discussion hooks: Is a citation that supports the sentence but was not what the model "used" a valid citation? Can NLI-based metrics be trusted on scientific text?

12. Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learning (Jin et al., arXiv 2025)

• Link: https://arxiv.org/abs/2503.09516 (arXiv:2503.09516) · Code: https://github.com/PeterGriffinJin/Search-R1 (Qwen2.5-3B/7B checkpoints: https://huggingface.co/collections/PeterJinGo/search-r1-67d1a021202731cb065740f5)
• Tag: recent · Difficulty: 3 · Compute: Colab (inference only)
• Summary: Extends DeepSeek-R1-style RL so the model interleaves <think>, <search> and <information> blocks with a live retriever; retrieved tokens are loss-masked and reward is outcome-only (EM). +41% (7B) and +20% (3B) over RAG baselines on seven QA sets, with multi-turn search emerging from training.
• Why it matters: The template for 2025-2026 "reasoning-augmented retrieval"; RAG-R1 (arXiv:2507.02962, https://arxiv.org/abs/2507.02962) adds multi-query parallelism; surveys below give the landscape. Closes the loop with the spring mini-RAG: the LLM is now the query formulator.
• Pair implementation: No training: run the released Search-R1-3B with a local BM25/FAISS retriever over the testbed on NQ/HotpotQA subsets and compare to prompt-only iterative retrieval (IRCoT-style) with Qwen3-4B in thinking mode. Apply: count searches per query and recall@10 of the union of issued queries on SciFact vs the single-query baseline (FlashRAG has both pipelines).
• Discussion hooks: Is the gain from reasoning, or from issuing several queries (query expansion by another name, cf. HyDE/Query2Doc)? Should retrievers now be trained against an RL agent's queries rather than human ones?

Framing resources for the instructor's 30-min intro

• Gao et al., "Retrieval-Augmented Generation for Large Language Models: A Survey" (arXiv:2312.10997, https://arxiv.org/abs/2312.10997): the Naive / Advanced / Modular RAG taxonomy everyone cites.
• Li et al., "Towards Agentic RAG with Deep Reasoning: A Survey of RAG-Reasoning Systems in LLMs" (arXiv:2507.09477, https://arxiv.org/abs/2507.09477): reasoning-enhanced RAG vs RAG-enhanced reasoning vs synergised systems; complements Singh et al., "Agentic RAG: A Survey" (arXiv:2501.09136, https://arxiv.org/abs/2501.09136).
• FlashRAG toolkit (Jin et al., WWW 2025 resource; https://arxiv.org/abs/2405.13576, https://github.com/RUC-NLPIR/FlashRAG): 23 RAG methods + 7 reasoning methods (Self-RAG, RECOMP, FLARE, Adaptive-RAG, Search-R1, IRCoT) with FAISS/BM25 backends and 3B-7B Qwen configs; a good common scaffold for all pairs.
• Anthropic, "Contextual Retrieval" engineering post (https://www.anthropic.com/engineering/contextual-retrieval): compact, quantified story of hybrid BM25 + embeddings + reranking that bridges the spring pipeline to this seminar.

Notes

• Near-misses dropped for slot count: Astute RAG (Wang et al., ACL 2025, arXiv:2410.07176, https://arxiv.org/abs/2410.07176; knowledge-conflict resolution, no official code found; could replace #10 if noise vs conflicts is preferred); RankRAG (Yu et al., NVIDIA, arXiv:2407.02485, https://arxiv.org/abs/2407.02485; needs 8B-70B instruction tuning, no code); LC-Boost "Are Long-LLMs a Necessity?" (Qian et al., arXiv:2405.15318); Ovadia et al., "Fine-Tuning or Retrieval?" (arXiv:2312.05934, https://arxiv.org/abs/2312.05934) as a one-slide aside.
• Venues I could not confirm from a fetched page: Search-R1's conference (listed on arXiv only as "31 pages"), Sufficient Context (arXiv only). All other venues were read on arXiv comments lines, ACL Anthology, OpenReview or the official repos.
• Fresh 2026 arXiv titles seen only in listings (unvetted, for a "what's next" slide): AgenticRag-R1 (2608.29622), PAGE-RAG (2608.29753), ISO-RAG (2609.00513), LitTraceQA (2608.07370), MCite-RL (2608.21808).
• Overlaps with other themes: late chunking and contextual retrieval also belong to the long-context-encoder theme (mentioned only, inside #6); Search-R1/RAG-R1 overlap with an "LLM-as-retriever / agentic search" theme; the instructor's ModernBERT work is the natural bridge for #6-#8.
• Compute caveat: GraphRAG (#4) and HippoRAG 2 (#5) need an LLM over every document at indexing time; pairs must subsample to a few hundred documents or spend a few euros of API budget, which is why both are rated Colab + small API.

Theme 10: Agentic search, deep-research agents, RL-trained search agents

Default slot: Session 5 (Dec 4) — RAG and agents.

Legend for the small-scale flag: [1-3B OK] = the core result can be reproduced on Colab with a 1-3B open model over a local BM25/dense index of a BEIR corpus; [partial] = only the inference/analysis half is feasible, training is not.

1. ReAct: Synergizing Reasoning and Acting in Language Models (Yao et al., ICLR 2023)

• Link: https://arxiv.org/abs/2210.03629 (arXiv:2210.03629) · Code: https://react-lm.github.io
• Tag: classic · Difficulty: 1 · Compute: laptop / Colab [1-3B OK]
• Summary: The canonical "Thought → Action → Observation" loop: the LLM interleaves free-text reasoning with tool calls (search[entity], lookup[string]) against a Wikipedia API. On HotpotQA and FEVER, ReAct beats chain-of-thought alone by grounding reasoning in retrieved text and reduces hallucinated facts; the trace is human-readable and debuggable. Every 2025 search agent (Search-R1, WebDancer, Open Deep Research) is a trained or scaled ReAct.
• Why it matters / link to the course: Turns the single-shot retrieve-then-read mini-RAG (Lewis 2020) students built into an iterative loop; directly addresses Barnett's "missing content" and "wrong specificity" failure points.
• Pair implementation: Implement ReAct from scratch (no framework) with Qwen2.5-3B-Instruct and a search(query) tool backed by BM25 over SciFact / NFCorpus; reproduce the CoT-vs-ReAct gap on 200 HotpotQA questions (Wikipedia slice) and report nDCG@10 of the union of retrieved documents on the testbed vs. one-shot BM25.
• Discussion hooks: Is a 3B model "reasoning" or pattern-matching the few-shot trace? Why did ReAct's own ablation show CoT+self-consistency beating ReAct on HotpotQA when internal knowledge suffices?

2. Interleaving Retrieval with Chain-of-Thought Reasoning for Knowledge-Intensive Multi-Step Questions (IRCoT; Trivedi et al., ACL 2023)

• Link: https://arxiv.org/abs/2212.10509 (arXiv:2212.10509) · Code: https://github.com/StonyBrookNLP/ircot
• Tag: classic · Difficulty: 1 · Compute: Colab [1-3B OK]
• Summary: Each generated CoT sentence becomes the next retrieval query, and retrieved paragraphs condition the next sentence, so retrieval and reasoning guide each other. On HotpotQA, 2WikiMultiHopQA, MuSiQue and IIRC, IRCoT with BM25 improves recall by up to 21 points and QA by up to 15 points over one-step retrieval, and works with Flan-T5-XL (3B) as well as GPT-3. Also introduces the now-standard "multi-hop QA + BM25 over Wikipedia" testbed.
• Why it matters / link to the course: The IR-side view of agentic search: the agent is a query-reformulation policy. Extends the query-expansion / doc2query ideas with LLM-generated queries; the benchmark trio (HotpotQA, 2Wiki, MuSiQue) is what all RL papers below use.
• Pair implementation: Reproduce the "recall@k of gold paragraphs vs. number of retrieval steps" curve with Qwen2.5-1.5B/3B and rank_bm25 on a 2Wiki subset (FlashRAG has IRCoT ready-made). Then run IRCoT-style multi-step retrieval on NFCorpus and measure whether multi-query retrieval + RRF beats single-query BM25→BGE-small in nDCG@10.
• Discussion hooks: Why does IRCoT help recall more than answer accuracy? Is "one query per reasoning sentence" a reasonable model of how a patent examiner searches?

3. Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learning (Jin et al., arXiv 2025)

• Link: https://arxiv.org/abs/2503.09516 (arXiv:2503.09516) · Code: https://github.com/PeterGriffinJin/Search-R1 (checkpoints: HF collection PeterJinGo/search-r1)
• Tag: recent · Difficulty: 3 · Compute: Colab [partial: inference yes, RL only toy-scale]
• Summary: Trains Qwen2.5-3B/7B with PPO/GRPO to emit <search> queries inside its reasoning against a local e5+FAISS Wikipedia index, using only an exact-match outcome reward; retrieved tokens are masked from the loss for stability. Gains of +41% (7B) and +20% (3B) over RAG baselines across seven QA datasets (NQ, HotpotQA, 2Wiki, MuSiQue, Bamboogle...). The companion "Empirical Study" (Jin et al. 2025, arXiv:2505.15117) shows format rewards help, intermediate retrieval rewards barely do, and the choice of search engine during training changes dynamics.
• Why it matters / link to the course: The retriever is frozen; what gets trained is the querier. Contrasts with InfoNCE-trained bi-encoders: reward is downstream answer correctness, not relevance labels.
• Pair implementation: Run the released Search-R1-3B checkpoint with a local BM25 or BGE-small index (replace the e5 server) on 300 HotpotQA questions and reproduce the multi-turn-search vs. RAG gap; count searches per question. Apply the same policy on SciFact (claim → search → verdict) and compare retrieval recall with the spring baseline. Optional stretch: a 100-step GRPO run with Qwen2.5-0.5B + LoRA (TRL) to show format-reward emergence.
• Discussion hooks: If the reward is just exact match, why does the model learn to search at all? Does masking retrieved tokens mean the model never learns to read critically?

4. s3: You Don't Need That Much Data to Train a Search Agent via RL (Jiang et al., EMNLP 2025)

• Link: https://arxiv.org/abs/2505.14146 (arXiv:2505.14146) · Code: https://github.com/pat-jj/s3
• Tag: recent · Difficulty: 2 · Compute: Colab [partial: reward + evaluation yes, searcher RL no]
• Summary: Decouples a trainable searcher from a frozen black-box generator and rewards the searcher with "Gain Beyond RAG" (GBR): generation accuracy with its retrieved context minus accuracy with naive top-k RAG. With only 2.4k training examples it beats Search-R1 trained on 70x more data on six general and five medical QA benchmarks, and the searcher transfers across generators.
• Why it matters / link to the course: GBR is a utility-based relevance signal: it asks whether a document changes the answer, not whether it is topically relevant (nDCG). Natural bridge to the cross-encoder reranking students know, and to the instructor's interest in what "relevance" means for full-text scientific retrieval.
• Pair implementation: Reproduce GBR measurement (no training): for Qwen2.5-3B as generator over NFCorpus/SciFact, compute answer accuracy with BM25 top-k vs. with oracle/BGE/cross-encoder-reranked contexts, and show that nDCG@10 and GBR rank retrievers differently. Run the released s3 searcher checkpoint (7B, 4-bit) on the testbed if VRAM allows.
• Discussion hooks: Should we replace nDCG with generation-utility metrics for RAG retrievers? What happens to GBR when the generator already knows the answer?

5. ZeroSearch: Incentivize the Search Capability of LLMs without Searching (Sun et al., arXiv 2025)

• Link: https://arxiv.org/abs/2505.04588 (arXiv:2505.04588) · Code: https://github.com/Alibaba-NLP/ZeroSearch
• Tag: recent · Difficulty: 2 · Compute: Colab [partial: simulated-retriever SFT yes, RL no]
• Summary: Replaces the search engine during RL with a lightly fine-tuned LLM that generates documents for a query, either helpful or deliberately noisy, and applies a curriculum that degrades document quality over training. A 7B simulated engine matches Google-based training and a 14B surpasses it, at zero API cost; simulation models in 3B/7B/14B and REINFORCE/GRPO/PPO scripts are released.
• Why it matters / link to the course: The LLM-as-corpus idea is the mirror image of doc2query / InPars / Promptagator (synthetic queries for documents): here the model synthesizes documents for queries. Raises the question of what a "retrieval environment" needs to be.
• Pair implementation: LoRA-fine-tune Qwen2.5-1.5B into a document simulator on NFCorpus (query → relevant/noisy passage pairs from qrels); then, with a frozen Qwen2.5-3B reader, compare answer accuracy on held-out NFCorpus/SciFact questions using (a) real BM25 passages, (b) simulated passages, (c) the curriculum mix. Report how far the simulated corpus is from the real one in embedding space (alignment/uniformity, per the instructor's geometry work).
• Discussion hooks: Is a policy trained on hallucinated documents learning to search or learning to ignore search? Could a patent corpus ever be simulated this way?

6. Search-o1: Agentic Search-Enhanced Large Reasoning Models (Li et al., EMNLP 2025)

• Link: https://arxiv.org/abs/2501.05366 (arXiv:2501.05366); ACL Anthology https://aclanthology.org/2025.emnlp-main.276/ · Code: https://github.com/RUC-NLPIR/Search-o1
• Tag: recent · Difficulty: 2 · Compute: Colab [1-3B OK with a small "thinking" model]
• Summary: Training-free: a long-reasoning model (QwQ-32B) is prompted to emit search queries when it detects a knowledge gap mid-chain; a separate "Reason-in-Documents" step compresses the retrieved pages into a short note before injecting it back, so the reasoning chain is not flooded. Strong gains on GPQA/math/code and six open-domain QA sets over both vanilla reasoning and standard agentic RAG. WebThinker (Li et al., NeurIPS 2025, arXiv:2504.21776, same group) extends it to browsing and report drafting.
• Why it matters / link to the course: The Reason-in-Documents module is a generative reranker/compressor, the natural successor of the cross-encoder in the spring pipeline; it also operationalises Barnett's "wrong format / incomplete context" failures.
• Pair implementation: Reimplement the loop with Qwen3-1.7B/4B in thinking mode (or DeepSeek-R1-Distill-Qwen-1.5B) and BM25 over SciFact; reproduce the ablation "with vs. without Reason-in-Documents" on 200 SciFact claims (verdict accuracy + tokens consumed). Compare the compressor against ms-marco-MiniLM reranking as a context filter.
• Discussion hooks: Is compression by the same model that reasons a source of confirmation bias? When the chain is 5k tokens long, where should retrieved evidence be inserted?

7. SimpleDeepSearcher: Deep Information Seeking via Web-Powered Reasoning Trajectory Synthesis (Sun et al., Findings of EMNLP 2025)

• Link: https://arxiv.org/abs/2505.16834 (arXiv:2505.16834); https://aclanthology.org/2025.findings-emnlp.739/ · Code: https://github.com/RUCAIBox/SimpleDeepSearcher
• Tag: recent · Difficulty: 2 · Compute: Colab [1-3B OK]
• Summary: Argues the RL papers are data-inefficient: synthesise search trajectories in a live web environment, filter them by multi-criteria curation (diversity, quality, correct answer, minimal turns), and do plain SFT on 871 examples. This beats RL-trained baselines on five benchmarks, positioning SFT-on-curated-trajectories as a cheaper alternative to Search-R1-style training. Related: R1-Searcher (Song et al. 2025, arXiv:2503.05592) and ReSearch (Chen et al., NeurIPS 2025, arXiv:2503.19470) are the RL-only counterparts at 7B/32B.
• Why it matters / link to the course: Distillation of search behaviour, akin to GPL/InPars distilling relevance; the data-curation criteria are a nice object of study.
• Pair implementation: LoRA-SFT Qwen2.5-1.5B/3B on the released 871 trajectories (30-60 min on T4), then evaluate against the untuned model on 2Wiki/MuSiQue subsets with local BM25 retrieval instead of the web. Apply the tuned agent to NFCorpus questions and report search count and nDCG@10 of the aggregated retrieved set.
• Discussion hooks: Does "learning to search" transfer from Wikipedia trajectories to a medical corpus? Is 871 examples enough because the base model already knows how to search from pre-training?

8. WebSailor: Navigating Super-human Reasoning for Web Agent (Li et al., arXiv 2025)

• Link: https://arxiv.org/abs/2507.02592 (arXiv:2507.02592) · Code: https://github.com/Alibaba-NLP/WebAgent (also hosts WebDancer, arXiv:2505.22648, and the Tongyi DeepResearch 30B-A3B report, arXiv:2510.24701)
• Tag: recent · Difficulty: 3 · Compute: Colab + small API budget [partial: data-synthesis idea only]
• Summary: Claims proprietary deep-research agents win on BrowseComp because they were trained on tasks with extreme uncertainty; WebSailor synthesises such tasks (SailorFog-QA) by random walks over a knowledge graph plus deliberate information obfuscation, then does rejection-sampling fine-tuning and an RL variant (DUPO) that re-samples hard rollouts. First open agents to approach proprietary BrowseComp/BrowseComp-zh scores. WebDancer is the ReAct-based SFT→RL pipeline it builds on.
• Why it matters / link to the course: Shows that the bottleneck moved from model to training-task construction; the obfuscation trick is a recipe for building hard test collections, relevant to prior-art search where the query deliberately hides the invention's vocabulary.
• Pair implementation: Reproduce the data-synthesis stage at small scale: build a citation graph over SciFact/NFCorpus documents, generate 100 obfuscated multi-hop questions with Mistral/Qwen, and measure how a ReAct agent (Qwen2.5-3B + BM25) degrades with obfuscation level. Demo a released WebAgent checkpoint if a small variant fits in 16 GB VRAM.
• Discussion hooks: Are BrowseComp-style "needle" questions a good proxy for real research? Is obfuscation just training the model to guess?

9. BrowseComp-Plus: A More Fair and Transparent Evaluation Benchmark of Deep-Research Agent (Chen et al., arXiv 2025) — with BrowseComp (Wei et al., 2025) as context

• Link: https://arxiv.org/abs/2508.06600 (arXiv:2508.06600); BrowseComp: https://arxiv.org/abs/2504.12516 (arXiv:2504.12516) · Code: https://github.com/texttron/BrowseComp-Plus (BM25 and Qwen3-Embedding indexes provided)
• Tag: recent · Difficulty: 2 · Compute: Colab [1-3B OK on the fixed corpus]
• Summary: OpenAI's BrowseComp (1,266 hard, short-answer browsing questions; GPT-4o 0.6%, Deep Research 51.5%) is unreproducible because the live web changes and the search API is a black box. BrowseComp-Plus (Waterloo, Jimmy Lin's group) freezes a ~100K-document corpus with human-verified evidence and mined hard negatives so that the retriever and the agent can be varied independently; scores range from 3.86% (Search-R1) to 70.1% (GPT-5 + Qwen3-Embedding-8B) and retrieval recall is reported alongside accuracy.
• Why it matters / link to the course: This is BEIR methodology (fixed corpus, qrels, Cranfield-style comparability) applied to agents; it lets students ask "how much of deep-research accuracy is retrieval quality?".
• Pair implementation: Download the BM25 index, run a ReAct/Search-o1-style agent with Qwen3-4B (4-bit) on 100 questions and reproduce the retriever ablation by swapping BM25 for BGE-small (encode 100K docs on T4, ~15 min) and for BM25+BGE RRF; report accuracy, evidence recall and search count per question.
• Discussion hooks: Does a stronger retriever help a weak agent more than a weak retriever hurts a strong one? Should agent benchmarks ever use the live web?

10. DeepResearch Bench: A Comprehensive Benchmark for Deep Research Agents (Du et al., arXiv 2025)

• Link: https://arxiv.org/abs/2506.11763 (arXiv:2506.11763) · Code: https://github.com/Ayanami0730/deep_research_bench
• Tag: recent · Difficulty: 2 · Compute: Colab + small API budget
• Summary: 100 PhD-level research tasks across 22 fields derived from 96K real user queries; evaluates reports, not short answers, with RACE (reference-based, adaptive per-task criteria, LLM-judged) and FACT (citation-level support and effective-citation counts). Gemini-2.5-Pro Deep Research and Claude/OpenAI systems lead; judge scores correlate strongly with humans. Companion survey: Huang et al. 2025 (arXiv:2506.18096); open replication: HF Open Deep Research (55% GAIA, code-agent design, https://huggingface.co/blog/open-deep-research).
• Why it matters / link to the course: Moves evaluation from nDCG/EM to LLM-as-a-judge over long outputs (name-dropped in spring); the "literature review" tasks are exactly scholarly retrieval + synthesis, close to the instructor's S2Full work.
• Pair implementation: Run HF Open Deep Research (smolagents) with an open 7-8B model or a few euros of Mistral API on 5-10 DeepResearch Bench tasks whose sources are restricted to the local scientific-article testbed; reproduce FACT (citation support checked by a local cross-encoder/NLI) and a reduced RACE with an open judge. Compare against a plain RAG report.
• Discussion hooks: Do LLM judges reward length and citation count rather than truth (Search Arena, Miroyan et al., ICLR 2026, found users prefer more citations regardless of support)? What would a nDCG for reports look like?

11. Over-Searching in Search-Augmented Large Language Models (Xie et al., EACL 2026)

• Link: https://arxiv.org/abs/2601.05503 (arXiv:2601.05503) · Code: none found (OverSearchQA dataset announced as released)
• Tag: recent · Difficulty: 1-2 · Compute: Colab [1-3B OK]
• Summary: Search-augmented LLMs invoke search when it cannot help: retrieval improves answerable questions but hurts unanswerable ones, and the effect worsens with reasoning models and over multi-turn dialogue. Proposes Tokens-Per-Correctness as an efficiency metric, releases OverSearchQA, and tests query-level and retrieval-level mitigations. Companions: "Search Wisely" (Wu et al., EMNLP 2025, arXiv:2505.17281) rewards confident search decisions with β-GRPO (+4 EM); "Diagnosing Search Behavior and Failure Modes in Long-Horizon Search Agents" (Liu et al., arXiv:2608.01913) finds agents retrieve useful evidence early then keep searching, and calls for evidence-sufficiency stopping criteria.
• Why it matters / link to the course: The stopping problem is IR's old "when has the examiner found enough prior art?" question in agent form; connects to CRAG/Self-RAG adaptive retrieval and to cost/latency, a real concern at Questel.
• Pair implementation: With Qwen2.5-1.5B/3B and Qwen3-1.7B (thinking), run a ReAct agent on a mix of answerable SciFact/NFCorpus questions and unanswerable ones (questions whose evidence was removed from the index); reproduce the accuracy-vs-searches and TPC curves and test two stopping rules (confidence threshold, evidence-sufficiency check by a cross-encoder).
• Discussion hooks: Is "no relevant document exists" a retrieval result or a reasoning result? Should the search engine tell the agent to stop?

12. Beyond Semantic Similarity: Rethinking Retrieval for Agentic Search via Direct Corpus Interaction (Li et al., arXiv 2026)

• Link: https://arxiv.org/abs/2605.05242 (arXiv:2605.05242) · Code: https://github.com/DCI-Agent/DCI-Agent-Lite
• Tag: recent · Difficulty: 2 · Compute: Colab [1-3B feasible, 4-8B recommended for tool use]
• Summary: Instead of a retrieval API, the agent gets terminal tools (grep/rg, find, bash pipelines, file read) over the raw corpus and iteratively refines exact-lexical hypotheses. On BRIGHT + BEIR, DCI-Agent (Claude Sonnet 4.6) reaches 68.5 average nDCG@10 vs. 47.0 for the best reranking baseline (ReasonRank-32B), and a cheap GPT-5.4-nano variant gets 56.7; strong on BrowseComp-Plus and multi-hop QA. Thesis: retrieval quality depends on the resolution of the interface between model and corpus. Related perspectives: Interact-RAG (Hui et al., arXiv:2510.27566), "LLM-Oriented IR: A Denoising-First Perspective" (Dai et al., SIGIR 2026, arXiv:2605.00505), "Agentic Information Retrieval" (Zhang et al., arXiv:2410.09713).
• Why it matters / link to the course: The most direct "search engine for agents" paper: it revisits Metzler et al. 2021 from the opposite direction (no model-as-index, but model-as-librarian over a raw corpus), and revives exact lexical matching (BM25's strength) that dense retrieval gave up; patent search with IPC codes and chemical names is a textbook case.
• Pair implementation: Dump SciFact and NFCorpus as text files, give Qwen3-4B/8B (4-bit) grep/rg/read tools with a 10-call budget, and reproduce the nDCG@10 comparison against BM25, BGE-small, and the spring BM25→BGE→cross-encoder pipeline; log tool-call distributions (paper: 33% grep, 62% bash). Test on the scientific-article testbed for the instructor's graded-relevance nDCG@10.
• Discussion hooks: Is DCI just BM25 with a smart user, and if so what does that say about learned retrievers? Does it scale past 100K documents, or is the index coming back through the back door?

Framing resources for the instructor's 30-min intro

1. Xi et al., "A Survey of LLM-based Deep Search Agents: Paradigm, Optimization, Evaluation, and Challenges" (arXiv:2508.05668) https://arxiv.org/abs/2508.05668 with the curated list https://github.com/YunjiaXi/Awesome-Search-Agent-Papers (taxonomy: iterative retrieval → tuning-free → SFT → RL). Best single map of the theme.
2. Huang et al., "Deep Research Agents: A Systematic Examination and Roadmap" (arXiv:2506.18096) https://arxiv.org/abs/2506.18096 and Zhang W. et al., "From Web Search towards Agentic Deep Research" (arXiv:2506.18959) https://arxiv.org/abs/2506.18959 for the "web search → deep research" narrative and test-time scaling plots.
3. Jin et al., "An Empirical Study on RL for Reasoning-Search Interleaved LLM Agents" (arXiv:2505.15117) https://arxiv.org/abs/2505.15117 for three slides of what actually matters in training (format reward, model scale, retriever choice). Broader RL surveys: Li W. et al. arXiv:2509.06733; Lin et al. arXiv:2510.16724.
4. Perspective bookends: Metzler et al. 2021 (SIGIR Forum, arXiv:2105.02274) https://arxiv.org/abs/2105.02274; Zhang et al. 2024 "Agentic Information Retrieval" https://arxiv.org/abs/2410.09713; Dai et al. SIGIR 2026 "LLM-Oriented IR: A Denoising-First Perspective" https://arxiv.org/abs/2605.00505.
5. Practical backbone for pairs: FlashRAG (Jin et al., WWW 2025 resource track, arXiv:2405.13576) https://github.com/RUC-NLPIR/FlashRAG implements IRCoT, FLARE, Self-Ask, Iter-RetGen, Self-RAG and the reasoning agents Search-R1, R1-Searcher, SimpleDeepSearcher with e5/BM25/SPLADE over Wikipedia or MS MARCO; HF Open Deep Research blog https://huggingface.co/blog/open-deep-research for a live demo.

Notes

• Verification caveats. All URLs above were seen in search results or fetched. Venues I could not confirm from the abstract pages and therefore left as "arXiv 2025": Search-R1 (widely cited as COLM 2025), ZeroSearch, DeepResearcher, WebSailor, BrowseComp-Plus; GAIA (Mialon et al., arXiv:2311.12983) is commonly cited as ICLR 2024 but the abs page did not state it. Search-R1's README example trains Llama-3.2-3B; the paper reports Qwen2.5-3B/7B. The ReSearch GitHub link redirects to https://github.com/Agent-RL/ReCall (successor project). The Over-Searching paper page did not show a code URL.
• Near-misses dropped or folded in: R1-Searcher (RUCAIBox, Reinforce++, 7B/8B only) and ReSearch (NeurIPS 2025, 7B/32B) are near-duplicates of Search-R1 for a 25-min talk, offered as alternates; DeepResearcher (GAIR, arXiv:2504.03160) needs live-web API budgets; WebDancer, WebThinker, Tongyi DeepResearch report folded into 6 and 8; FRAMES (Krishna et al., NAACL 2025, arXiv:2409.12941; 824 questions over 2-15 Wikipedia pages) is the best multi-hop evaluation set to add to any pair's testbed but not a presentation topic; Search Arena (ICLR 2026, arXiv:2506.05334); Towards AI Search Paradigm (Baidu, arXiv:2506.17188) is a 23-author system description, hard to reproduce; "As We May Search" (Zerhoudi et al., ICTIR 2026, arXiv:2606.29652, local-first search) and GEO (Aggarwal et al., KDD 2024, arXiv:2311.09735, content optimised for generative engines) are interesting sidebars for the "search for machines" discussion. Toolformer (Schick et al. 2023, arXiv:2302.04761) and FLARE (Jiang et al., EMNLP 2023, arXiv:2305.06983) are context only.
• Overlaps with other themes: FLARE/Self-RAG/CRAG with the RAG theme; BRIGHT (used by DCI) with reasoning-intensive retrieval; LLM-as-a-judge (DeepResearch Bench) with evaluation; ZeroSearch with synthetic-data themes (InPars/Promptagator); Search-o1's Reason-in-Documents with LLM rerankers.
• Balance: 2 classics, 10 recent (2025-2026); 7 of 12 fully reproducible with 1-3B models on Colab (1, 2, 6, 7, 9, 11, 12), 4 partially (3, 4, 5, 8), 1 needs a small API budget (10).

Theme 11: Evaluation of retrieval and RAG in the LLM era

Default slot: Session 6 (Dec 11) — Trust.

Twelve entries (three classics, nine 2024–2026), ordered LLM-judges → critiques → synthetic collections → TREC RAG → RAG frameworks → statistics → QPP → benchmark hygiene. Every link below was seen in a search result or fetched page.

1. Perspectives on Large Language Models for Relevance Judgment (Faggioli et al., ICTIR 2023)

• Link: https://arxiv.org/abs/2304.09161 (arXiv:2304.09161) · ACM: https://dl.acm.org/doi/10.1145/3578337.3605136 · Code: none found
• Tag: classic · Difficulty: 1 · Compute: laptop
• Summary: The paper that opened the debate. Eleven authors (Faggioli, Dietz, Clarke, Demartini, Hagen, Hauff, Kando, Kanoulas, Potthast, Stein, Wachsmuth) define a human–machine collaboration spectrum for relevance judging (fully manual → human verification → AI assistance → fully automatic), report pilot experiments comparing ChatGPT-era LLM labels with trained TREC assessors, and deliberately present both the optimistic and the sceptical position. Agreement with humans is only moderate, and the authors flag the risk that a system optimised by an LLM is then judged by the same LLM.
• Why it matters: Extends the spring-course qrels/nDCG material by asking where the qrels come from; every other paper in this theme cites it.
• Pair implementation: Reproduce "fully automatic" judging with a small open model (Qwen2.5-3B/7B-Instruct 4-bit via transformers/llama.cpp) on SciFact and NFCorpus: label the top-20 of the baseline BM25→MiniLM→cross-encoder pipeline, compute Cohen's κ against the human qrels, then compare system rankings (Kendall τ of nDCG@10 across ≥6 runs) under human vs LLM qrels. Place the result on the collaboration spectrum by also trying the "human verifies LLM" mode on 100 pairs.
• Discussion hooks: If κ with humans is 0.3 but system-ranking τ is 0.9, which number should we care about? Is "relevance" even a property an LLM can observe without the user?

2. Large Language Models Can Accurately Predict Searcher Preferences (Thomas et al., SIGIR 2024) + UMBRELA (Upadhyay et al., 2024) + "Does UMBRELA Work on Other LLMs?" (Farzi & Dietz, SIGIR 2025)

• Link: https://arxiv.org/abs/2309.10621 (arXiv:2309.10621); ACM https://dl.acm.org/doi/10.1145/3626772.3657707 · UMBRELA https://arxiv.org/abs/2406.06519 (arXiv:2406.06519) · Farzi & Dietz https://arxiv.org/pdf/2507.09483 (arXiv:2507.09483) · Code: https://github.com/castorini/umbrela
• Tag: recent · Difficulty: 2 · Compute: Colab (+ optional few € of API for a GPT-4o reference)
• Summary: Thomas et al. (Bing) calibrate LLM prompts against first-party user-preference data and show GPT-4 labels are as accurate as human labellers, better than third-party crowd workers "for a fraction of the cost", pick the same hardest queries/best runs, and train better rankers; prompt wording (even paraphrases) changes accuracy. UMBRELA is the open-source reproduction with GPT-4o on TREC DL 2019–2023 (high rank correlation with human qrels) and became the official TREC 2024 RAG judge. Farzi & Dietz re-run UMBRELA across models: DeepSeek-V3 ≈ GPT-4o, Llama-3.3-70B slightly lower, and quality "further degrades with smaller LLMs".
• Why it matters: This is the LLM-as-a-judge idea name-dropped in spring, made concrete; the UMBRELA toolkit has local-HF and ensemble judges, so it plugs into the shared testbed.
• Pair implementation: Reproduce the Farzi & Dietz size ladder on SciFact/NFCorpus with UMBRELA's prompt (Qwen2.5 1.5B/3B/7B, Llama-3.x-8B), measuring per-label κ and leaderboard τ vs human qrels; then test Thomas et al.'s claim that prompt features (role, aspects, few-shot) matter by ablating the DNA prompt with the best small model.
• Discussion hooks: Bing calibrates against clicks, which academic collections lack: what is our "gold"? At what model size does an LLM judge stop being useful, and is that size stable across domains (biomedical NFCorpus vs claims in SciFact)?

3. LLM-based Relevance Assessment Still Can't Replace Human Relevance Assessment (Clarke & Dietz, 2024/2026) — with Soboroff 2024 and "LLM-Evaluation Tropes"

• Link: https://arxiv.org/abs/2412.17156 (arXiv:2412.17156, revised Jan 2026) · Soboroff https://arxiv.org/abs/2409.15133 (arXiv:2409.15133) · Tropes https://arxiv.org/abs/2504.19076 (arXiv:2504.19076) · Alaofi et al. https://arxiv.org/abs/2501.17969 (arXiv:2501.17969, SIGIR-AP 2024) · Code: none found
• Tag: recent · Difficulty: 2 · Compute: laptop
• Summary: Clarke & Dietz answer the TREC 2024 RAG claim (Upadhyay et al., arXiv:2411.08275) that UMBRELA can replace assessors, with three arguments: the evidence is weaker than claimed; a deliberately constructed system gets inflated scores without better retrieval; and a Kendall-τ analysis of hypothetical circularity shows LLM judges "closely mimic" LLM re-rankers ("the inherent narcissism of LLMs"). Soboroff's short paper adds the logical point: an LLM judge defines the ceiling, so nothing better than that LLM can be measured. The "Tropes" paper (Dietz, Zendel, Bailey, Clarke, Cotterill, Dalton, Hasibi, Sanderson, Craswell) catalogues recurring validity failures and proposes guardrails. Alaofi et al. show LLM labellers over-label relevance and are fooled by keyword stuffing and injected instructions.
• Why it matters: The counterweight to entry 2; connects to LLM rerankers (name-dropped in spring) and to why Questel-style professional search still needs human qrels.
• Pair implementation: Reproduce the "gaming" argument on SciFact: (a) keyword-stuff or prompt-inject the top-10 passages and measure how a small-LLM judge's labels shift (Alaofi); (b) circularity — rerank with an LLM (Qwen2.5-7B pointwise) and judge with the same model vs a different family, reporting τ against human qrels and the inflation of the self-judged run.
• Discussion hooks: Is circularity a bug or just the new "pooling bias"? Should TREC publish LLM qrels at all if they cap what future systems can prove?

4. Neural Retrievers Are Biased Towards LLM-Generated Content (Dai et al., KDD 2024) — with Balog, Metzler & Qin, SIGIR 2025

• Link: https://arxiv.org/abs/2310.20501 (arXiv:2310.20501); ACM https://dl.acm.org/doi/abs/10.1145/3637528.3671882 · Balog et al. https://arxiv.org/abs/2503.19092 (arXiv:2503.19092) · Code: none verified
• Tag: recent · Difficulty: 1 · Compute: laptop
• Summary: Dai et al. rewrite corpora with LLMs while preserving meaning and show dense retrievers and cross-encoder rerankers rank the LLM-written version higher ("source bias"), with a text-compression explanation: LLM text has more focused semantics and less noise. Balog, Metzler & Qin extend the question to the judge side and give the first empirical evidence that LLM judges favour LLM-based rankers, while (contrary to earlier reports) finding no bias against AI-generated content in their preliminary study.
• Why it matters: Directly relevant to the instructor's work on embedding geometry: source bias is a geometric artefact of contrastive encoders. Also the missing piece of the circularity story in entry 3.
• Pair implementation: Reproduce source bias on NFCorpus/SciFact: paraphrase 500 relevant docs with Qwen2.5-3B, index human+LLM versions together with MiniLM/BGE-small and the MiniLM cross-encoder, and measure rank shift and "relative Δ" as in the paper; then check whether a small LLM judge grades the paraphrased version higher (Balog's question).
• Discussion hooks: If LLM text is "easier to match", is source bias a bias or a genuine quality signal? What happens to test collections once half the web is model-written?

5. Synthetic Test Collections for Retrieval Evaluation (Rahmani et al., SIGIR 2024) + SynDL (Rahmani et al., WWW 2025)

• Link: https://arxiv.org/abs/2405.07767 (arXiv:2405.07767) · SynDL https://arxiv.org/abs/2408.16312 (arXiv:2408.16312), ACM https://dl.acm.org/doi/10.1145/3701716.3715311 · Code/data: https://github.com/rahmanidashti/SynDL; LLMJudge label collection https://llm4eval.github.io/LLMJudge-benchmark/ (paper arXiv:2502.13908)
• Tag: recent · Difficulty: 2 · Compute: Colab
• Summary: The SIGIR paper asks whether both queries and judgments can be synthetic and finds that LLM-built collections rank systems reliably (with caveats about bias toward LLM-based runs). SynDL scales this to 1,988 queries and 637,063 query–passage labels over TREC DL, with high system-ordering agreement with every DL year. "Judging the Judges" releases ~40 LLM relabelings of TREC DL 2023 from the LLMJudge challenge so the community can study systematic bias and ensembles.
• Why it matters: Extends BEIR/MTEB hands-on work: students build their own mini-benchmark instead of consuming one — the InPars/Promptagator idea applied to evaluation rather than training.
• Pair implementation: Build "SynSciFact": generate 100 synthetic claims/queries from SciFact abstracts with a 7B model, judge pooled top-k with the same or a different small LLM, and compare system rankings (τ) against the real SciFact qrels for ≥6 baseline runs; then test the cross-model ensemble idea from LLMJudge (majority vote of two small judges).
• Discussion hooks: Do synthetic queries reflect real information needs or the generator's priors? Should a synthetic collection ever be used to report results, or only to develop?

6. The Great Nugget Recall: Automating Fact Extraction and RAG Evaluation with LLMs (Pradeep et al., SIGIR 2025) — TREC RAG track evaluation

• Link: https://arxiv.org/pdf/2504.15068 (arXiv:2504.15068); ACM https://doi.org/10.1145/3726302.3730090 · AutoNuggetizer report arXiv:2411.09607 · Ragnarök https://arxiv.org/abs/2406.16828 (arXiv:2406.16828, ECIR 2025) · Support evaluation https://arxiv.org/abs/2504.15205 (arXiv:2504.15205) · TREC 2025 overview https://arxiv.org/abs/2603.09891 (arXiv:2603.09891) · Code: https://github.com/castorini/nuggetizer (pip, OpenAI/vLLM backends), https://github.com/castorini/ragnarok
• Tag: recent · Difficulty: 2 · Compute: Colab + small API budget (or local vLLM 7B)
• Summary: Revives the 2003 TREC QA nugget methodology: an LLM extracts atomic facts ("nuggets") per topic, labels them vital/okay, and assigns them to system answers; scores correlate strongly with manual nugget evaluation (21 topics × 45 runs in the initial report) and with ~7K Search Arena human preferences. The companion support paper shows human–GPT-4o agreement of 56% (from scratch) rising to 72% with post-editing, and an independent human agrees better with GPT-4o than with another human. TREC 2025 adds a Relevance Judgment task and narrative queries over MS MARCO v2.1 (>150 submissions).
• Why it matters: The state-of-the-art answer to "how do we evaluate the mini-RAG we built in spring" beyond BLEU/ROUGE; the AutoNuggetizer pipeline is reusable on any corpus.
• Pair implementation: Reproduce the nuggetizer on 30 NFCorpus or MS MARCO-subset queries: create nuggets from the human-relevant passages, generate answers with two RAG variants (BM25 vs hybrid+rerank, Qwen2.5-7B generator), auto-assign nuggets, and check whether nugget recall separates the two systems consistently with a 3-person manual assignment on 10 topics.
• Discussion hooks: Nuggets reward completeness; how do we avoid rewarding verbosity? Is "support" (citation) accuracy a retrieval metric or a generation metric?

7. RAGAS: Automated Evaluation of Retrieval Augmented Generation (Es et al., 2023) — contrasted with ARES (Saad-Falcon et al., NAACL 2024)

• Link: https://arxiv.org/abs/2309.15217 (arXiv:2309.15217) · ARES https://aclanthology.org/2024.naacl-long.20/ (arXiv:2311.09476) · Code: https://github.com/explodinggradients/ragas ; https://github.com/stanford-futuredata/ARES
• Tag: classic · Difficulty: 1 · Compute: laptop (RAGAS) / Colab (ARES)
• Summary: RAGAS defines reference-free, LLM-computed metrics — faithfulness (fraction of answer statements entailed by the context), answer relevance (via question regeneration) and context relevance — and has become the de-facto industry library (now context precision/recall, noise sensitivity, LiteLLM backends). ARES instead fine-tunes lightweight classifier judges on synthetic queries and corrects them with prediction-powered inference using a few hundred human labels, giving confidence intervals. Together they represent the two design philosophies: zero-shot prompting vs trained judges with statistical calibration.
• Why it matters: Directly evaluates the spring mini-RAG and the "seven failure points" (Barnett 2024) with quantitative metrics; the ARES/PPI idea foreshadows the statistics entries.
• Pair implementation: Reproduce the RAGAS metric definitions from scratch with a local judge (Qwen2.5-7B) rather than the library, run them on 100 MS MARCO-subset questions for two RAG variants, and check agreement with the human relevance labels for context relevance and with a manual 50-item faithfulness annotation. Note ARES's own docs recommend an A100, so treat it as a conceptual contrast (or run its PPI step only).
• Discussion hooks: Are reference-free metrics measuring quality, or the judge's own preferences (entry 3)? Why did the simpler, uncalibrated tool win adoption?

8. Evaluating Retrieval Quality in Retrieval-Augmented Generation — eRAG (Salemi & Zamani, SIGIR 2024)

• Link: https://arxiv.org/abs/2404.13781 (arXiv:2404.13781); ACM https://dl.acm.org/doi/10.1145/3626772.3657957 · Code: https://github.com/alirezasalemi7/eRAG (pip install erag)
• Tag: recent · Difficulty: 2 · Compute: Colab
• Summary: Shows that classic query–document relevance labels correlate poorly with downstream RAG utility, and proposes eRAG: feed each retrieved document individually to the generator, score the output with the task metric, and use that score as the document's relevance label, then aggregate with any IR measure. Kendall's τ with end-to-end RAG performance improves by 0.168–0.494 over baselines, with up to 50× less GPU memory than end-to-end evaluation.
• Why it matters: Redefines "relevance" as utility for the downstream LLM — a direct bridge from the spring IR metrics to RAG; model-agnostic library.
• Pair implementation: Reproduce on the MS MARCO subset (1k queries with answers): compute eRAG labels with Qwen2.5-1.5B/3B for the top-10 of BM25, MiniLM and the hybrid pipeline, then compare (a) human-qrel nDCG@10, (b) eRAG nDCG@10 and (c) actual end-to-end answer quality, reporting the three pairwise τ values as in the paper.
• Discussion hooks: If relevance is generator-specific, is an IR test collection still reusable? Does eRAG penalise documents that are only useful in combination?

9. RAGChecker: A Fine-grained Framework for Diagnosing Retrieval-Augmented Generation (Ru et al., NeurIPS 2024 D&B)

• Link: https://arxiv.org/abs/2408.08067v1 (arXiv:2408.08067); NeurIPS https://proceedings.neurips.cc/paper_files/paper/2024/hash/27245589131d17368cccdfa990cbf16e-Abstract-Datasets_and_Benchmarks_Track.html · Code: https://github.com/amazon-science/RAGChecker (pip install ragchecker)
• Tag: recent · Difficulty: 2 · Compute: Colab (+ API or local model for claim checking)
• Summary: Extracts claims from response and gold answer and runs claim-level entailment against contexts to produce overall precision/recall/F1, retriever metrics (claim recall, context precision) and generator metrics (context utilisation, noise sensitivity in relevant/irrelevant context, hallucination, self-knowledge, faithfulness). Meta-evaluation shows better correlation with human judgments than RAGAS-style metrics; the diagnostic finding is a recall/noise trade-off: better retrievers make generators more noise-sensitive.
• Why it matters: Turns Barnett's qualitative "failure points" into a measurable diagnosis; claim-level checking is the same machinery as nugget evaluation (entry 6).
• Pair implementation: Reproduce the recall/noise-sensitivity trade-off on NFCorpus: vary k (3, 5, 10) and retriever (BM25 vs hybrid+rerank) with a 7B generator, compute RAGChecker's retriever/generator metrics with a local extractor/checker (RefChecker supports open models), and plot noise sensitivity against claim recall.
• Discussion hooks: Eight metrics per system — which one would you optimise a product against? Do claim extractors agree with each other enough to make the numbers reproducible?

10. Statistical Significance Testing in IR: An Empirical Analysis of Type I, Type II and Type III Errors (Urbano, De Lima & Hanjalic, SIGIR 2019) — applied to LLM qrels (Otero, Parapar & Barreiro, SIGIR 2025)

• Link: https://arxiv.org/abs/1905.11096 (arXiv:1905.11096) · Otero et al. https://arxiv.org/html/2411.13212v1 (arXiv:2411.13212, "Limitations of Automatic Relevance Assessments with LLMs for Fair and Reliable Retrieval Evaluation") · McKechnie et al. arXiv:2507.07924 · Code: https://github.com/julian-urbano/sigir2019-statistical
• Tag: classic · Difficulty: 2 · Compute: laptop
• Summary: Urbano et al. simulate IR experiments from TREC data (>500M p-values) to measure actual Type I/II/III error rates of the t-test, Wilcoxon, sign, bootstrap and permutation tests; the paired t-test (Wilcoxon close behind) is the most reliable, and the sign and bootstrap-shift tests should be avoided. Otero et al. bring this to LLM judges: LLM-generated qrels are "unfair at ranking top-performing systems" and produce an "exceedingly high rate of false positives" for significance, even when τ looks fine. McKechnie, McDonald & Macdonald propose balanced accuracy over Type I+II errors as a single discriminative-power score for assessment methods.
• Why it matters: Students learned nDCG but never tested significance; this is the "how many queries / which test" toolkit, and it shows that the entry-2 τ correlations hide a statistical failure mode.
• Pair implementation: Reproduce Urbano's Type I analysis at small scale: split SciFact/NFCorpus queries into random halves for the same system, count false "significant" differences per test over 1,000 splits; then, à la Otero, redo the pairwise system comparisons with LLM qrels from entry 1/2 and report false-positive/negative rates relative to human qrels (plus effect sizes and Bonferroni/Holm correction across pairs).
• Discussion hooks: With 300 queries and κ = 0.4, how many "significant" findings in a seminar are real? Should a paper report τ or the Type I rate of its automatic judge?

11. Query Performance Prediction Using Relevance Judgments Generated by LLMs — QPP-GenRE (Meng et al., ACM TOIS 2025)

• Link: https://arxiv.org/abs/2404.01012 (arXiv:2404.01012); ACM https://dl.acm.org/doi/10.1145/3736402 · Code: https://github.com/ChuanMeng/QPP-GenRE
• Tag: recent · Difficulty: 2 · Compute: Colab
• Summary: Decomposes QPP into predicting the relevance of each item in the ranked list with a (QLoRA-fine-tuned, 4-bit) Llama-3-8B, then computes any IR measure from those pseudo-labels; achieves state-of-the-art Pearson/Kendall correlation with true RR@10 and nDCG@10 for BM25 and ANCE on TREC DL 19–22 and iKAT, and the fine-tuned judge agrees well with human assessors. It also explains why a query is predicted hard (which documents are judged non-relevant).
• Why it matters: Unifies LLM judging (entries 1–2) and query difficulty; QPP is exactly what a patent-search or scholarly-search UI needs to decide when to reformulate or route to a human.
• Pair implementation: Reproduce zero-shot QPP-GenRE with Qwen2.5-7B on SciFact and the scientific-article dataset: predict nDCG@10 per query for the baseline pipeline, correlate with actual nDCG@10 (Pearson/Kendall), and compare against classic post-retrieval predictors (score standard deviation, NQC). Optional: QLoRA-fine-tune a 1.5B judge on NFCorpus train qrels and see whether it beats zero-shot 7B.
• Discussion hooks: If the judge is good enough for QPP, is it good enough for evaluation — or does QPP tolerate noise that evaluation cannot? Can QPP-GenRE detect out-of-domain queries where the retriever is confidently wrong?

12. Resources for Brewing BEIR: Reproducible Reference Models and an Official Leaderboard (Kamalloo et al., 2023) — with Pyserini / ir_datasets / ir_measures and "Maintaining MTEB"

• Link: https://arxiv.org/abs/2306.07471 (arXiv:2306.07471) · Pyserini https://dl.acm.org/doi/10.1145/3404835.3463238 (SIGIR 2021) · ir_datasets https://arxiv.org/pdf/2103.02280 (SIGIR 2021) · ir-measures https://link.springer.com/chapter/10.1007/978-3-030-99739-7_38 (ECIR 2022) · Maintaining MTEB https://arxiv.org/abs/2506.21182 (arXiv:2506.21182) · Code: https://github.com/beir-cellar/beir , castorini/pyserini, allenai/ir_datasets, terrierteam/ir_measures
• Tag: recent (2023) · Difficulty: 1 · Compute: laptop
• Summary: Kamalloo et al. diagnose why BEIR numbers in papers do not match: no reference implementations, inconsistent preprocessing and metric code; they ship reproducible dense/sparse baselines and a self-service leaderboard. Pyserini (reproducible BM25/dense first stage), ir_datasets (uniform dataset access) and ir_measures (one evaluation API over trec_eval and others) are the tooling that makes such hygiene possible. "Maintaining MTEB" (Chung, Kerboua, Kardos, Solomatin, Enevoldsen) describes the CI, dataset-integrity checks and the generalisability/zero-shot disclosure that flags models trained on benchmark training splits.
• Why it matters: The students' spring baseline uses rank_bm25; this entry explains why their BM25 may differ from the BEIR leaderboard's, and what training-data leakage means for the MTEB scores they read.
• Pair implementation: Reproduce BEIR reference BM25 and BGE-small numbers on SciFact/NFCorpus/ArguAna with Pyserini + ir_measures, and diff against rank_bm25 + the spring evaluation code (tokenisation, k1/b, title handling, tie-breaking). Then audit the testbed for leakage: check whether SciFact/NFCorpus training splits appear in the disclosed training data of two MTEB models and report the "zero-shot" status.
• Discussion hooks: How much of a 1-point nDCG gain is implementation noise? Should leaderboards hide models that trained on benchmark training splits, or only label them?

Framing resources for the instructor's 30-min intro

• Rahmani et al., Report on the 1st Workshop on LLM for Evaluation in IR (LLM4Eval 2024), https://arxiv.org/abs/2408.05388 — panel positions and the LLMJudge challenge; good map of who believes what.
• Dietz et al., Principles and Guidelines for the Use of LLM Judges, ICTIR 2025, https://dl.acm.org/doi/10.1145/3731120.3744588 — the "how to do it responsibly" checklist that pairs with the Tropes paper (arXiv:2504.19076).
• Sakai, Statistical Reform in Information Retrieval?, SIGIR Forum 2014, https://sigir.org/files/forum/2014J/2014J_sigirforum_Article_TetsuyaSakai.pdf, and Fuhr, Some Common Mistakes in IR Evaluation, SIGIR Forum 2017, https://dl.acm.org/doi/10.1145/3190580.3190586 — effect sizes, CIs, MRR/MAP pitfalls; Sakai's reply "On Fuhr's Guideline" http://www.sigir.org/wp-content/uploads/2020/06/p14.pdf.
• TREC 2025 RAG track overview, https://arxiv.org/abs/2603.09891 — the live picture of nugget, support and RJ evaluation at scale.

Notes

• The web-search budget ran out after the last batch; all remaining checks were done with direct fetches. RAGAS's venue is given as arXiv 2023 because I could not re-verify the EACL 2024 demo publication in this session.
• Near-misses dropped or folded in: MacAvaney & Soldaini, One-Shot Labeling for Automatic Relevance Estimation (SIGIR 2023, arXiv:2302.11266, code autoqrels) — a nice non-LLM alternative for entry 1 if a pair wants a DuoT5-based judge; ALCE (Gao et al., EMNLP 2023, arXiv:2305.14627) for citation accuracy — folded into entry 6's support evaluation; Arabzadeh & Clarke, Benchmarking LLM-based Relevance Judgment Methods (arXiv:2504.12558) — a good extra reading for entry 2.
• "Judging the Judges" states 42 label sets from eight teams on arXiv but 39 from seven teams on the project page; I wrote "~40".
• Overlaps with other themes: entry 12 touches MTEB engineering (MMTEB and "The Powerless Noise" are covered elsewhere — I cite only the "Maintaining MTEB" paper and the BEIR resources); entry 4 (source bias) borders on dense-retrieval themes; entry 11 (QPP) could be claimed by a query-understanding theme.
• Compute caveats: ARES's own docs recommend an A100, so I positioned it as a contrast within entry 7 rather than a reproduction target; nuggetizer and RAGChecker work with local vLLM/open models but are cheapest with a few euros of GPT-4o-mini calls for the reference run.

Theme 12: Security, robustness and bias of neural retrieval and RAG

Default slot: Session 6 (Dec 11) — Trust.

Ground rules for the whole theme (state them in the intro session). Every attack below is reproduced only against a local copy of a BEIR corpus (SciFact / NFCorpus / ArguAna / the MS MARCO subset) served by the students' own spring-course pipeline (BM25 + MiniLM/BGE-small + cross-encoder) and a local generator (Qwen2.5-3B/7B or Mistral-7B in 4-bit). No attacks on live systems, no poisoned corpora uploaded to public hubs, no real personal data (use synthetic PII for the privacy papers). Under these rules all twelve papers are ethically unproblematic and technically feasible on a laptop or a free Colab T4.

1. Poisoning Retrieval Corpora by Injecting Adversarial Passages (Zhong et al., EMNLP 2023)

• Link: https://arxiv.org/abs/2310.19156 (arXiv:2310.19156) · https://aclanthology.org/2023.emnlp-main.849/ · Code: https://github.com/princeton-nlp/corpus-poisoning
• Tag: classic · Difficulty: 2 · Compute: laptop (MiniLM) / Colab (Contriever)
• Summary: First corpus-poisoning attack on dense retrievers: HotFlip-style discrete token flips optimise a handful of passages to maximise similarity with a set of training queries. Injected into a large corpus, they get retrieved for unseen queries and even other domains: 50 passages optimised on NQ mislead >94% of queries on financial/forum BEIR sets. The ECIR 2025 reproducibility paper by Li, Eustratiadis & Kanoulas (https://arxiv.org/abs/2501.04802, code https://github.com/liyongkang123/hotflip_corpus_poisoning) shows the attack weakens against newer retrievers, and Su et al. (ACL Findings 2025, https://arxiv.org/abs/2406.05087) improve the search (AGGD).
• Why it matters / link to the course: Turns the DPR/SBERT embedding geometry students already know into an attack surface; the "universal passage" is literally a point placed near a query cluster, which connects to the instructor's alignment/uniformity analyses.
• Pair implementation: Reproduce HotFlip poisoning of all-MiniLM-L6-v2 or BGE-small on NFCorpus (3.6k docs), measuring top-k attack success vs number of injected passages. Then test transfer: passages optimised on NFCorpus queries injected into SciFact, and whether BM25, RRF fusion and the cross-encoder reranker filter them out.
• Discussion hooks: Why does hybrid BM25+dense fusion (which we teach as an effectiveness trick) act as a defence? Is the attack still meaningful when a reranker sits downstream?

2. PoisonedRAG: Knowledge Corruption Attacks to RAG of LLMs (Zou et al., USENIX Security 2025)

• Link: https://arxiv.org/abs/2402.07867 (arXiv:2402.07867) · https://www.usenix.org/conference/usenixsecurity25/presentation/zou-poisonedrag · Code: https://github.com/sleeepeer/PoisonedRAG
• Tag: recent · Difficulty: 2 · Compute: Colab
• Summary: Formalises "knowledge corruption" as an optimisation problem: a malicious text must (a) be retrieved for a target question and (b) make the LLM output a target answer. Black-box variant simply prepends the question to an LLM-written false passage; white-box uses gradients. Five texts per target question yield ~90% attack success in a multi-million-text database. Trigger-based variants generalise this: BadRAG (Xue et al. 2024, https://arxiv.org/abs/2406.00083) and Phantom (Chaudhari et al. 2024, https://arxiv.org/abs/2405.20485) craft passages retrieved only when a trigger appears in the query. Zhang et al. (2025, https://arxiv.org/abs/2505.18543) benchmark 13 such attacks and 7 defences.
• Why it matters / link to the course: Direct sequel to the spring mini-RAG and to Barnett's "seven failure points"; makes the retrieval stage the security-critical component.
• Pair implementation: Reproduce the black-box attack on the local MS MARCO subset with a local Qwen2.5-3B/7B generator (5 poisoned passages for 50 target queries, ASR and clean accuracy). Apply: vary retriever (BM25 / BGE-small / RRF / +cross-encoder) and k, then add a Phantom-style trigger word and measure how selective the retrieval backdoor is.
• Discussion hooks: The black-box attack needs no model access at all: is RAG over web-scale corpora inherently indefensible? Does "citing sources" help the user or only the attacker's credibility?

3. Machine Against the RAG: Jamming RAG with Blocker Documents (Shafran et al., USENIX Security 2025)

• Link: https://arxiv.org/abs/2406.05870 (arXiv:2406.05870) · https://www.usenix.org/conference/usenixsecurity25/presentation/shafran · Code: https://github.com/avitalsh/jamming_attack
• Tag: recent · Difficulty: 2 · Compute: Colab
• Summary: A denial-of-service twist: a single "blocker" document, once retrieved, makes the RAG system refuse or fail to answer a specific query. Compares several generation methods including a black-box optimisation; shows that standard safety alignment makes models easier to jam (they refuse readily) and evaluates simple defences.
• Why it matters / link to the course: Same retrieval mechanics as #2 but a different goal; nicely pairs with the "answer refusal" failure point and with the certified defence in #5.
• Pair implementation: Reproduce jamming on SciFact-as-QA (claims as queries, one blocker per claim) with a local 3-7B model; report jamming rate vs k and vs generator. Apply: test whether the spring cross-encoder reranker or a perplexity filter demotes blockers, and compare with the source-bias finding (#8) that fluent LLM text is promoted.
• Discussion hooks: Is refusal a safety feature or an availability vulnerability? Who is liable when a competitor jams answers about your product?

4. Not what you've signed up for: Indirect Prompt Injection (Greshake et al., AISec @ CCS 2023)

• Link: https://arxiv.org/abs/2302.12173 (arXiv:2302.12173) · Code: https://github.com/greshake/llm-security
• Tag: classic · Difficulty: 1 · Compute: laptop
• Summary: Introduces indirect prompt injection: instructions hidden in retrieved content (web pages, emails, documents) hijack an LLM-integrated app, enabling data exfiltration, manipulation and "wormable" injections; demonstrated on Bing Chat and synthetic GPT-4 apps. The 2026 follow-up by Chang et al. (USENIX Security 2026, https://arxiv.org/abs/2601.07072) fixes the hard step that early work skipped: guaranteeing the injected document is actually retrieved, via a trigger fragment plus attack fragment, with near-100% retrieval across 8 embedding models. Defence lines to cite: Spotlighting (arXiv:2403.14720), StruQ, SecAlign (https://arxiv.org/pdf/2410.05451), and Google's "Lessons from defending Gemini" (https://arxiv.org/pdf/2505.14534).
• Why it matters / link to the course: Every RAG demo in the spring course was an injection target; this is the paper that names the problem.
• Pair implementation: Build a local RAG over NFCorpus with Qwen2.5-7B-Instruct 4-bit; inject instruction-bearing documents (including a Chang-style trigger fragment) and measure retrieval rate and hijack rate. Apply Spotlighting-style delimiting and a "retrieved text is data" system prompt; report the residual success rate. Local models only, never a hosted assistant.
• Discussion hooks: Is prompt injection a retrieval problem (keep bad docs out) or a generation problem (make models ignore them)? Can it ever be "solved" while models read natural language?

5. Certifiably Robust RAG against Retrieval Corruption (Xiang et al., arXiv 2024, v2 2026)

• Link: https://arxiv.org/abs/2405.15556 (arXiv:2405.15556) · Code: https://github.com/inspire-group/RobustRAG
• Tag: recent · Difficulty: 3 · Compute: Colab
• Summary: RobustRAG is the first defence with certified robustness: retrieved passages are isolated into disjoint groups, the LLM answers each group separately, and answers are securely aggregated (keyword voting or decoding-level aggregation). This yields provable lower bounds on answer quality against an adaptive attacker injecting a bounded number of passages; evaluated on RealtimeQA, NQ and BioGen with Mistral-7B, Llama-7B and GPT-3.5. TrustRAG (Zhou et al. 2025, https://arxiv.org/abs/2501.00879, code https://github.com/HuichiZhou/TrustRAG) is the cheap training-free counterpart: k-means on retrieved embeddings plus cosine/ROUGE filters; RAGForensics (Zhang et al., WWW 2025, https://arxiv.org/abs/2504.21668) does post-hoc traceback.
• Why it matters / link to the course: Converts "RRF-style aggregation" into a security argument; the isolation idea is a direct answer to #2 and #3.
• Pair implementation: Reproduce keyword aggregation with Qwen2.5-3B (or Mistral-7B 4-bit) on the MS MARCO subset under PoisonedRAG poisoning from #2, reporting clean vs certified accuracy for 1-3 injected passages. Apply TrustRAG's clustering filter on the same poisoned index and compare cost/benefit.
• Discussion hooks: Certification assumes the attacker controls at most m of k passages: is that assumption realistic on the open web? What do we lose (multi-hop reasoning?) when passages cannot see each other?

6. The Good and The Bad: Exploring Privacy Issues in RAG (Zeng et al., ACL Findings 2024)

• Link: https://arxiv.org/abs/2402.16893 (arXiv:2402.16893) · https://aclanthology.org/2024.findings-acl.267/ · Code: https://github.com/phycholosogy/RAG-privacy
• Tag: recent · Difficulty: 1-2 · Compute: Colab
• Summary: Shows with simple prompts that a RAG system can be made to regurgitate verbatim sentences from its retrieval store (untargeted) or specific private facts (targeted), while paradoxically reducing leakage of the LLM's own training data. Evaluates naive defences (summarisation, thresholds). The SaTML 2026 SoK on RAG privacy (https://arxiv.org/abs/2601.03979) gives a taxonomy for the intro.
• Why it matters / link to the course: The vector store students built in spring is a data-exfiltration channel; complements the Vec2Text embedding-inversion paper covered by another theme (Morris et al., EMNLP 2023, https://arxiv.org/abs/2310.06816), which attacks the embeddings rather than the generator.
• Pair implementation: Reproduce untargeted extraction on NFCorpus with a local Qwen2.5-7B generator (fraction of retrieved sentences reproduced verbatim). Apply targeted extraction on a synthetic "patient notes" corpus with fake PII inserted into SciFact-sized chunks; compare leakage with and without summarisation-before-generation. No real personal data.
• Discussion hooks: If RAG lowers training-data memorisation leaks, is a private RAG store safer than fine-tuning? Who owns the sentences a RAG system quotes?

7. Is My Data in Your Retrieval Database? Membership Inference against RAG (Anderson et al., ICISSP 2025)

• Link: https://arxiv.org/abs/2405.20446 (arXiv:2405.20446) · https://www.scitepress.org/PublishedPapers/2025/131083/ · Code: none found
• Tag: recent · Difficulty: 1 · Compute: Colab
• Summary: A prompt-based membership inference attack: ask the RAG system whether a candidate passage appears in its context and read the answer (black-box) or token probabilities (gray-box); works across two datasets and several generators, with a template-instruction defence that helps only sometimes. The stronger WWW 2025 follow-up, Mask-based MIA (Liu, Zhang & Long, https://arxiv.org/abs/2410.20142), masks words in the candidate document and infers membership from mask-prediction accuracy.
• Why it matters / link to the course: Membership inference is the standard privacy audit in ML; this transfers it from model weights to the index, i.e. to the retrieval step students understand best.
• Pair implementation: Build two SciFact indexes (members vs held-out docs), run Anderson's prompt attack and Liu's mask attack with a local 3-7B model, report AUC. Apply: vary k, similarity threshold and chunk size and see which retrieval hyperparameters leak most.
• Discussion hooks: Is membership leakage from a retrieval index even avoidable, given that retrieval is designed to surface exact matches? Does GDPR "right to be forgotten" make sense for a vector index?

8. Neural Retrievers are Biased Towards LLM-Generated Content (Dai et al., KDD 2024)

• Link: https://arxiv.org/abs/2310.20501 (arXiv:2310.20501) · https://dl.acm.org/doi/abs/10.1145/3637528.3671882 · Code: https://github.com/KID-22/Source-Bias
• Tag: recent · Difficulty: 1 · Compute: laptop (+ small local LLM or a few euros of API for rewriting)
• Summary: Defines "source bias": when a corpus contains human-written documents and LLM rewrites carrying the same information, neural retrievers and rerankers rank the LLM versions higher, while BM25 does not; attributed via a text-compression analysis to LLM text having more focused semantics. Cocktail (Dai et al., ACL Findings 2024, https://arxiv.org/abs/2405.16546, https://github.com/KID-22/Cocktail) packages 16 mixed-source datasets and finds a ranking-quality vs bias trade-off. Xion & Nejdl (ECIR 2026, https://arxiv.org/abs/2602.10833) show the bias is training-induced (MS MARCO fine-tuning), and "Spiral of Silence" (Chen et al., ACL 2024, https://arxiv.org/abs/2404.10496) simulates the long-run feedback loop; "Retrieval Collapses When AI Pollutes the Web" (Yu et al., WWW 2026, https://arxiv.org/abs/2602.16136) reports 67% pool contamination becoming >80% exposure contamination.
• Why it matters / link to the course: Directly relevant to the instructor's contrastive training work: which training objective/data creates the bias? Also the bridge from security to the ecosystem-level "timeline" question of corpora filling with AI text.
• Pair implementation: Reproduce source bias on SciFact or NFCorpus by generating LLM rewrites of all docs (Qwen2.5-7B locally or Cocktail's released corpora if the dataset is included) and computing the relative nDCG@10 gap human vs LLM for BM25, MiniLM, BGE-small and the cross-encoder. Apply a 3-round Spiral-of-Silence loop: RAG answers are re-inserted into the corpus and the human-doc share in the top-10 is tracked.
• Discussion hooks: Is "preferring cleaner text" a bias or good relevance modelling? What happens to citation-based scientific retrieval when abstracts are increasingly LLM-polished?

9. Perplexity Trap: PLM-Based Retrievers Overrate Low-Perplexity Documents (Wang et al., ICLR 2025)

• Link: https://arxiv.org/abs/2503.08684 (arXiv:2503.08684) · Code: https://github.com/WhyDwelledOnAi/Perplexity-Trap
• Tag: recent · Difficulty: 2 · Compute: Colab
• Summary: Explains #8 causally: PLM-based retrievers learn document perplexity as a relevance feature, so low-perplexity (LLM-written) text is overrated. Proposes Causal Diagnosis and Correction (CDC), an inference-time debiasing that estimates the perplexity effect and subtracts it from the relevance score, reducing source bias with little effectiveness loss.
• Why it matters / link to the course: A clean example of probing what a contrastively-trained encoder actually encodes, in the spirit of the instructor's embedding-geometry analyses; ties into Wataoka et al.'s finding that LLM judges also favour low-perplexity text (NeurIPS 2024 SafeGenAI workshop, https://arxiv.org/abs/2410.21819).
• Pair implementation: Reproduce the perplexity-relevance correlation on the mixed corpus from #8 using GPT-2/Qwen2.5-0.5B perplexities and BGE-small scores; fit CDC and report bias reduction vs nDCG@10. Apply CDC to the cross-encoder reranker stage of the spring pipeline.
• Discussion hooks: If perplexity is a useful quality prior for the web, should we remove it at all? Could the same trap explain SPECTER-style scholarly retrievers preferring well-edited abstracts?

10. LLM Evaluators Recognize and Favor Their Own Generations (Panickssery et al., NeurIPS 2024)

• Link: https://arxiv.org/abs/2404.13076 (arXiv:2404.13076) · https://proceedings.neurips.cc/paper_files/paper/2024/hash/7f1f0218e45f5414c79c0679633e47bc-Abstract-Conference.html · Code: none found
• Tag: recent · Difficulty: 1 · Compute: Colab + small API budget (or local models only)
• Summary: GPT-4, GPT-3.5 and Llama-2 rate their own summaries above those of other models and humans, and the paper shows self-recognition ability is linked to the strength of that self-preference. For IR the counterpoint is Chen et al. (ACL Findings 2025, https://arxiv.org/abs/2410.20833): simulating pointwise LLM reranking and RAG reading on NQ/MARCO/TriviaQA, they find no significant self-preference in fact-centric tasks, where factual accuracy dominates.
• Why it matters / link to the course: LLM-as-a-judge and LLM rerankers were only name-dropped in spring; this is the bias literature students need before trusting either for evaluation.
• Pair implementation: Reproduce the pairwise self-preference test with two local models (Qwen2.5-7B vs Llama-3.x-8B) judging each other's passage rewrites from #8. Apply as reranker: LLM pointwise relevance scores on the human/LLM mixed SciFact corpus, testing whether the judge favours its own rewrites beyond the generic source bias of #8.
• Discussion hooks: If self-preference vanishes for factual tasks, is LLM-judged nDCG safe for our scientific-article testbed? Should benchmarks disclose which model generated their documents?

11. Fairness of Exposure in Rankings (Singh & Joachims, KDD 2018)

• Link: https://arxiv.org/abs/1802.07281 (arXiv:1802.07281) · https://doi.org/10.1145/3219819.3220088 · Code: none found (many re-implementations; see FairDiverse below)
• Tag: classic · Difficulty: 2 · Compute: laptop
• Summary: Reframes fair ranking as allocation of exposure (position-based attention) rather than relevance: a linear program over doubly-stochastic ranking matrices maximises expected utility under constraints such as demographic parity, disparate treatment or disparate impact of exposure between groups, then samples rankings via Birkhoff-von Neumann decomposition. Canonical reference behind the TREC Fair Ranking track (2022 overview: Ekstrand et al., https://arxiv.org/abs/2302.05558, Wikipedia-editor task) and the Zehlike, Yang & Stoyanovich survey (ACM CSUR 2022, https://arxiv.org/abs/2103.14000). The FairDiverse toolkit (Xu et al. 2025, https://arxiv.org/abs/2502.11883, https://github.com/XuChen0427/FairDiverse) implements 28 fairness/diversity algorithms; Wang et al. (NAACL 2024, https://arxiv.org/abs/2404.03192) test whether LLM rankers are fair.
• Why it matters / link to the course: Extends nDCG's position-discount model students know into a normative tool; on the shared testbed it connects naturally to #8 (exposure for human vs LLM-written documents) and to patent/scholarly search where exposure decides who gets cited.
• Pair implementation: Reproduce the LP with cvxpy/scipy on toy examples, then apply on the scientific-article testbed (or SciFact) with groups defined by publication year (old vs recent) or source (human vs LLM rewrite from #8): compute exposure disparity of the spring pipeline and post-process for parity, reporting the nDCG@10 cost. Optionally run one FairDiverse baseline.
• Discussion hooks: Which grouping deserves fairness in scientific search: authors, institutions, years, or languages? Is exposure parity between human and AI-generated documents desirable or absurd?

12. Typos that Broke the RAG's Back: Genetic Attack on RAG (Cho et al., EMNLP Findings 2024)

• Link: https://arxiv.org/abs/2404.13948 (arXiv:2404.13948) · https://aclanthology.org/2024.findings-emnlp.161/ · Code: https://github.com/zomss/GARAG
• Tag: recent · Difficulty: 2 · Compute: Colab
• Summary: GARAG is a black-box genetic algorithm that applies low-level perturbations (typos, character swaps) to retrieved documents to simultaneously degrade the retriever and the reader; RAG turns out fragile to noise that is common in real documents (OCR, forums). The defence side is Zhuang & Zuccon's CharacterBERT + Self-Teaching for typo-robust dense retrievers (SIGIR 2022, https://arxiv.org/abs/2204.00716, https://github.com/ielab/CharacterBERT-DR, with a real-typo MS MARCO query set), and the query-side benchmark is Penha, Câmara & Hauff's query-variation generators (ECIR 2022, https://arxiv.org/abs/2111.13057, https://github.com/Guzpenha/query_variation_generators), which found ~20% average effectiveness drops.
• Why it matters / link to the course: Robustness without an adversary: the same BEIR pipeline, scored under realistic noise; Self-Teaching is a small contrastive/distillation fine-tune of the kind the instructor's work uses.
• Pair implementation: Reproduce GARAG on NFCorpus with BGE-small + Qwen2.5-3B (attack success vs number of edits). Apply the defence: fine-tune MiniLM/BGE-small on Colab with Self-Teaching-style typo augmentation, and re-evaluate on Penha-style query variants of the MS MARCO subset and on the GARAG-perturbed documents.
• Discussion hooks: Should robustness to typos be a benchmark metric alongside nDCG? Are subword tokenizers the real culprit, and would ModernBERT-style long-context encoders behave differently?

Framing resources for the instructor's 30-min intro

• Tran et al., "Retrieved But Not Reliable: A Survey on Attacks and Defenses in RAG", EMNLP Findings 2026, https://arxiv.org/abs/2608.24977 : threat models over corpus/retriever/generator; attacks by accuracy/privacy/fairness objective; pipeline-stage defences. The most current single map of #1-#7.
• Liu, Zhang, Guo, de Rijke et al., "Robust Neural Information Retrieval: An Adversarial and Out-of-distribution Perspective", https://arxiv.org/abs/2407.06992, with the SIGIR 2024 tutorial https://arxiv.org/abs/2406.08891 : the pre-LLM IR view of adversarial and typo/OOD robustness (#1, #12).
• Dai et al., "Bias and Unfairness in IR Systems: New Challenges in the LLM Era", KDD 2024 tutorial, https://arxiv.org/abs/2404.11457 and paper list https://github.com/KID-22/LLM-IR-Bias-Fairness-Survey : source bias, fairness and evaluation bias in one framework (#8-#11).
• Curated lists and benchmarks: https://github.com/Arstanley/Awesome-Trustworthy-RAG ; SafeRAG benchmark (https://arxiv.org/abs/2501.18636, attack tasks: silver noise, inter-context conflict, soft ad, white DoS).

Notes

• Venue of RobustRAG (#5): arXiv v2 dated April 2026 and an OpenReview page exist, but I could not load OpenReview to confirm the conference; cite as arXiv until checked.
• Anderson et al. (#7) and Panickssery et al. (#10): no official code found; both are simple prompt protocols that a pair can re-implement in an afternoon.
• Cocktail (#8): I confirmed it contains 16 datasets but not which BEIR sets; check whether SciFact/NFCorpus are included before promising students pre-generated rewrites.
• Near-misses dropped: BadRAG and Phantom (no code found, folded into #2); Su et al. AGGD (folded into #1); TrustRAG (folded into #5, a good fallback if RobustRAG proves too heavy); Xion & Nejdl 2026 and Yu et al. 2026 (folded into #8; either could replace Spiral of Silence if the instructor wants a 2026 paper); Wataoka et al. self-preference (folded into #9); Chen et al. ACL-F 2025 (folded into #10).
• Overlaps with other themes: Vec2Text (privacy, another agent) is referenced in #6 only; LLM-as-a-judge/rerankers (#10) may overlap with an evaluation or LLM-reranking theme; the source-bias cluster (#8-#9) touches any theme on synthetic data / doc2query-style augmentation, since training on LLM-generated positives is exactly what Xion & Nejdl identify as bias-inducing.
• Ethics: none of the twelve requires interaction with a live system; #4 and #6 must stay on local models and synthetic data; poisoned indexes should be kept off shared repositories.

Theme 13: Multimodal and visually rich document retrieval, document parsing, code and tables

Default slot: Session 6 (Dec 11) — Trust.

Shared-testbed note for this theme. The spring testbed is text-only, so I suggest the instructor fix one extra visual testbed for these pairs: ~30 arXiv PDFs (ideally the papers behind the small scientific-article dataset), rendered with pdf2image at ~150 dpi → ~300–500 page images, plus ~50 questions written by the students (with page-level relevance). Every visual-retrieval pair then reports nDCG@5 on the same pages, and text-side pairs still use SciFact/NFCorpus. T4 fit is stated per paper.

1. Learning Transferable Visual Models From Natural Language Supervision — CLIP (Radford et al., ICML 2021)

• Link: https://arxiv.org/abs/2103.00020 (arXiv:2103.00020) · Code: https://github.com/openai/CLIP (open_clip on HF)
• Tag: classic · Difficulty: 1 · Compute: laptop (ViT-B/32 runs on CPU)
• Summary: Trains an image encoder and a text encoder jointly with a symmetric InfoNCE loss on 400M web image–caption pairs; the shared space gives zero-shot classification and image↔︎text retrieval without task-specific training. The follow-up SigLIP (Zhai et al., ICCV 2023, arXiv:2303.15343) replaces softmax-InfoNCE with a pairwise sigmoid loss that removes the need for a global normalisation over the batch and scales better; Long-CLIP (Zhang et al., ECCV 2024, arXiv:2403.15378) fixes the 77-token text limit.
• Why it matters: This is literally the InfoNCE/in-batch-negatives objective from the spring course applied across modalities, and every model below (DSE, ColPali, GME, jina-v4) is a descendant. The modality gap in CLIP space is a nice embedding-geometry topic (alignment/uniformity, as in the instructor's S2Full paper).
• Pair implementation: Reproduce zero-shot image–caption retrieval with open_clip ViT-B/32; then fine-tune it with both InfoNCE and sigmoid loss on a few thousand figure/table–caption pairs from SciMMIR (Wu et al., ACL Findings 2024, arXiv:2401.13478, https://github.com/Wusiwei0410/SciMMIR) and report Recall@k for figure retrieval on scientific papers; measure alignment/uniformity before and after.
• Discussion hooks: Why do image and text embeddings still occupy separate cones after contrastive training? Does sigmoid loss change that? Is a caption a good proxy for a scientific figure at all?

2. CodeSearchNet Challenge: Evaluating the State of Semantic Code Search (Husain et al., arXiv 2019)

• Link: https://arxiv.org/abs/1909.09436 (arXiv:1909.09436) · Code: https://github.com/github/CodeSearchNet ; data on HF code-search-net/code_search_net
• Tag: classic · Difficulty: 1 · Compute: laptop / Colab
• Summary: Releases a corpus of ~6M functions in six languages (2M with docstring-derived "queries") and a challenge set of 99 natural-language queries with ~4k expert relevance judgements, then benchmarks bag-of-words, RNN, CNN and self-attention bi-encoders trained with the docstring↔︎code contrastive objective. Establishes the NL→code retrieval task and its evaluation (MRR on docstring pairs, nDCG on expert annotations) still used by CoIR.
• Why it matters: It is DPR-style dual-encoder retrieval with a "free" supervision signal (docstrings) — the same trick as citation supervision in SPECTER/S2Full, which makes it the natural bridge to the instructor's self-supervised-views idea.
• Pair implementation: Fine-tune MiniLM or ModernBERT-base with MultipleNegativesRankingLoss on ~50k Python docstring–code pairs and reproduce the MRR gap between BM25 and a trained bi-encoder; apply the same recipe to the spring testbed (SciFact) to show how much code-trained encoders transfer to text.
• Discussion hooks: Docstrings are written by the same person who wrote the code — how much of "semantic code search" is lexical overlap in disguise? What is the code analogue of the BEIR zero-shot gap?

3. Unifying Multimodal Retrieval via Document Screenshot Embedding — DSE (Ma et al., EMNLP 2024)

• Link: https://arxiv.org/abs/2406.11251 (arXiv:2406.11251); https://aclanthology.org/2024.emnlp-main.373/ · Code: https://github.com/texttron/tevatron (DSE recipes); models Tevatron/dse-phi3-docmatix-v2 etc. on HF
• Tag: recent (canonical) · Difficulty: 2 · Compute: Colab (Phi-3-vision ~4B fits a T4 in fp16 for inference; single-vector index is tiny)
• Summary: Treats the screenshot of a page as the universal document format: a VLM encodes the image into a single dense vector, trained with contrastive loss on Wikipedia screenshots (Wiki-SS) and slides (SlideVQA). Bypasses OCR/parsing entirely and beats OCR-then-text retrieval by >15 nDCG@10 on the mixed-modality slide task while staying competitive on pure text.
• Why it matters: DPR (spring) with the encoder swapped for a VLM; the cleanest single-vector counterpart to ColPali, so a pair presenting DSE and a pair presenting ColPali can debate single- vs multi-vector on the same pages.
• Pair implementation: Reproduce the "screenshot beats OCR text" result on a SlideVQA subset with the released DSE checkpoint; apply DSE vs BGE-small-on-Docling-text on the shared PDF testbed.
• Discussion hooks: Is losing the text token stream a feature or a bug for RAG citations? How does image resolution trade off against retrieval quality?

4. ColPali: Efficient Document Retrieval with Vision Language Models (Faysse et al., ICLR 2025)

• Link: https://arxiv.org/abs/2407.01449 (arXiv:2407.01449) · Code: https://github.com/illuin-tech/colpali ; models https://huggingface.co/vidore (ColQwen2 = vidore/colqwen2-v1.0, Qwen2-VL-2B + LoRA)
• Tag: recent (canonical) · Difficulty: 2 · Compute: Colab. T4 fit: yes — ColQwen2 (~2.2B) is ~4.5 GB in fp16 (T4 has no native bf16; use fp16 or fp32 ≈ 9 GB), ~1–3 s/page, so 300–500 pages index in 10–25 min; multi-vector index ≈ 768 patches × 128 d ≈ 0.4 MB/page fp32 → a few hundred MB, fine in RAM with numpy MaxSim. ColPali itself (PaliGemma-3B, ~6 GB fp16) also fits but is slower and licence-gated.
• Summary: Feeds page images to a VLM, keeps all patch-token embeddings (projected to 128 d), and scores with ColBERT-style late interaction (MaxSim) against query tokens. Introduces ViDoRe (10 page-retrieval tasks) and shows ColPali beats the best OCR+captioning+BGE-M3 pipeline by a wide margin (81.3 vs ~67 nDCG@5) with far simpler indexing.
• Why it matters: Makes the ColBERT/PLAID material from the spring hands-on, and is name-dropped in the spring course; ideal for the patent-drawing/figure retrieval problems at Questel.
• Pair implementation: Reproduce ColQwen2's ViDoRe numbers on 2 tasks (e.g. DocVQA, TabFQuAD); on the shared PDF testbed compare ColQwen2 (MaxSim) vs mean-pooled single vector from the same model vs BM25 on Docling text, and measure index size/latency. Optional: token pooling (cluster patch vectors) to shrink the index.
• Discussion hooks: Is the gain from vision or from the 2B-parameter backbone (the text baseline used a 0.5B encoder)? How would you deploy 700 vectors/page for a million-page patent corpus?

5. ViDoRe Benchmark V2 / V3 (Macé et al., arXiv 2025; Loison et al., arXiv 2026)

• Link: https://arxiv.org/abs/2505.17166 (arXiv:2505.17166); https://arxiv.org/abs/2601.08620 (arXiv:2601.08620) · Code: https://github.com/illuin-tech/vidore-benchmark ; leaderboard https://huggingface.co/spaces/vidore/vidore-leaderboard
• Tag: recent · Difficulty: 1–2 · Compute: Colab (evaluation only)
• Summary: V2 argues V1 is saturated (top models >90 nDCG@5, "essentially eliminating meaningful differentiation") and builds four multilingual tasks with blind contextual queries, long cross-document queries and human-in-the-loop generation. V3 (26k pages, 3,099 queries, 6 languages, 10 domains) evaluates full RAG pipelines and finds visual retrievers beat text-only pipelines, late interaction and textual reranking help, but systems still fail on non-textual elements and visual grounding.
• Why it matters: The BEIR/MTEB lesson from the spring — benchmark saturation, query realism, annotation cost — replayed for visual retrieval; also a fine "timeline" object (V1 2024 → V3 2026).
• Pair implementation: Re-run 3–4 open models (ColQwen2, DSE, GME-2B, BGE-M3 on OCR text) on one V2 and one V3 subset and reproduce the V1→V2 drop; then build a "V2-style" hard query set for the shared PDF testbed and show which pairs' methods survive it.
• Discussion hooks: When a benchmark is saturated in a year, what does that say about the benchmark vs the models? Are synthetic-then-human-validated queries measuring what real users ask?

6. SERVAL: Surprisingly Effective Zero-Shot Visual Document Retrieval Powered by Large Vision and Language Models (Nguyen et al., EMNLP 2025)

• Link: https://aclanthology.org/2025.emnlp-main.1568/ · Code: none found in the paper page
• Tag: recent ("ColPali is not enough") · Difficulty: 2 · Compute: Colab (Qwen2-VL-2B or similar for captioning; any text encoder)
• Summary: Instead of contrastively training a VLM, generate a rich textual description of each page with an off-the-shelf VLM and embed it with a strong text encoder. Zero-shot, this reaches 63.4 nDCG@5 on ViDoRe-v2, above the best specialised multi-vector visual encoder, and is comparable on MIRACL-VISION multilingual.
• Why it matters: A doc2query/HyDE-style "generate then embed" trick (spring concepts) turned against ColPali; directly relevant to the instructor's question of what text representation of a document you should actually embed.
• Pair implementation: Reproduce SERVAL on one ViDoRe-v2 task with Qwen2-VL-2B captions + BGE-base; on the shared PDF testbed compare captions vs Docling text vs ColQwen2 patches, and combine them with RRF.
• Discussion hooks: Is the VLM just doing better OCR-plus-summarisation? What is lost when a page becomes 300 words of description — and does that matter more for tables than for prose?

7. VisRAG: Vision-based Retrieval-augmented Generation on Multi-modality Documents (Yu et al., arXiv 2024)

• Link: https://arxiv.org/abs/2410.10594 (arXiv:2410.10594) · Code: https://github.com/openbmb/visrag
• Tag: recent · Difficulty: 2 · Compute: Colab (VisRAG-Ret on MiniCPM-V-2 fits T4; use Qwen2-VL-2B-Instruct as generator instead of the 8B model)
• Summary: End-to-end vision RAG: a VLM-based retriever indexes page images and a VLM generator reads the retrieved images directly, so no parsing stage exists anywhere. Reports 20–40% end-to-end gains over text-based RAG on DocVQA-style tasks, with separate analyses of retrieval and generation stages. Related: M3DocRAG (Cho et al., arXiv:2411.04952 — ColPali + Qwen2-VL on 40k-page open-domain corpora) and the very recent PIXELRAG (Wang et al., June 2026, arXiv:2606.28344, https://github.com/StarTrail-org/PixelRAG — 30M Wikipedia web screenshots, up to +18.1% accuracy over text RAG, 3× token savings).
• Why it matters: Extends the spring mini-RAG and Barnett's "seven failure points" — the parsing failure point simply disappears, but new ones (page granularity, resolution) appear.
• Pair implementation: Reproduce the retrieval-stage gain on a SlideVQA/ArxivQA subset; on the shared PDF testbed build both pipelines (Docling+BGE+Qwen2.5-1.5B vs VisRAG-style ColQwen2+Qwen2-VL-2B) and score answers with a small LLM-as-judge budget.
• Discussion hooks: If the generator sees pixels, what does the "citation" look like? How do you chunk a page image?

8. GME: Improving Universal Multimodal Retrieval by Multimodal LLMs (Zhang et al., CVPR 2025)

• Link: https://arxiv.org/abs/2412.16855 (arXiv:2412.16855) · Code: models https://huggingface.co/collections/Alibaba-NLP/gme-models (gme-Qwen2-VL-2B-Instruct)
• Tag: recent · Difficulty: 2 · Compute: Colab. T4 fit: yes, same footprint as ColQwen2 (2B, fp16), single vector so index is trivial.
• Summary: One MLLM-based dense retriever for text→text, text→image, image→image and "fused" (text+image) queries and candidates, trained with synthetic fused-modal data to fix the imbalance of existing training sets; introduces the UMRB benchmark and reports SOTA across the board. The companion approach VLM2Vec (Jiang et al., ICLR 2025, arXiv:2410.05160, https://github.com/TIGER-AI-Lab/VLM2Vec) shows the generic recipe "any VLM + instruction + contrastive loss → embedder" and the MMEB benchmark.
• Why it matters: The "one encoder for everything" claim, evaluated with MTEB-style tooling students already know; the modality-gap analysis links to embedding-geometry work.
• Pair implementation: Run gme-2B on SciFact/NFCorpus (text mode) vs BGE-small to test the "universal" claim on the spring testbed; then run it on the shared PDF testbed and on SciMMIR figure retrieval, and plot the text/image modality gap in its space.
• Discussion hooks: Does a 2B universal model beat a 33M text specialist on text? Who pays the 60× inference cost?

9. jina-embeddings-v4: Universal Embeddings for Multimodal Multilingual Retrieval (Günther et al., arXiv 2025)

• Link: https://arxiv.org/abs/2506.18902 (arXiv:2506.18902) · Code: https://huggingface.co/jinaai/jina-embeddings-v4
• Tag: recent · Difficulty: 2 · Compute: Colab. T4 fit: 3.8B ≈ 7.6 GB fp16, inference at batch 1–2 works but ~2× slower than ColQwen2; fine for a few hundred pages.
• Summary: A 3.8B Qwen2.5-VL-based model that outputs both a single vector and ColBERT-style multi-vectors, with task-specific LoRA adapters (retrieval, STS, code), and a Matryoshka-truncatable dimension. Claims SOTA on text, code and visual-document retrieval and introduces the Jina-VDR benchmark for visually rich retrieval.
• Why it matters: Bundles four spring concepts (bi-encoder, ColBERT, MRL, adapters) into one artefact, and covers text, code and images — a natural "state of 2025" model for the instructor's embedding-timeline intro.
• Pair implementation: On the shared PDF testbed, compare v4 single-vector vs multi-vector vs truncated-MRL dims (128/256/…); on SciFact, compare its text mode to BGE-base; on CoIR's CodeSearchNet subset, its code adapter.
• Discussion hooks: Do LoRA "task adapters" contradict the universal-embedding dream? At what dimension does Matryoshka truncation start hurting late interaction?

10. OCR Hinders RAG: Evaluating the Cascading Impact of OCR on Retrieval-Augmented Generation (Zhang et al., ICCV 2025)

• Link: https://arxiv.org/abs/2412.02592 (arXiv:2412.02592) · Code: https://github.com/opendatalab/OHR-Bench
• Tag: recent · Difficulty: 2 · Compute: laptop for Docling (CPU, ~1–5 s/page); Colab for Nougat/MinerU (Nougat-base 350M runs a few s/page on T4). Tools: Docling (Auer et al., arXiv:2408.09869, https://github.com/docling-project/docling), MinerU (Wang et al., arXiv:2409.18839, https://github.com/opendatalab/MinerU), Nougat OCR-free (Blecher et al., arXiv:2308.13418, https://github.com/facebookresearch/nougat); parser benchmark OmniDocBench (Ouyang et al., CVPR 2025, arXiv:2412.07626).
• Summary: OHR-Bench: 8,561 document images from seven RAG domains with 8,498 Q&A pairs, plus perturbed ground-truth text to isolate two noise types (semantic vs formatting). Every OCR/parsing pipeline degrades retrieval and generation, and the authors conclude "none is competent" for building high-quality RAG knowledge bases; VLM pipelines are evaluated as an alternative. A 2026 follow-up on financial PDFs (El Bachyr et al., arXiv:2604.12047) studies parser × chunking interactions.
• Why it matters: Puts numbers on the spring "chunking basics" and Barnett's failure points; for the instructor's S2Full corpus, PDF-to-text quality is a hidden variable of the full-text-window training signal.
• Pair implementation: Reproduce the semantic- vs formatting-noise degradation curve on an OHR-Bench subset with BGE-small; on the shared PDF testbed run PyMuPDF vs Docling vs Nougat, index with the spring pipeline, and report nDCG deltas per parser and per element type (tables vs prose).
• Discussion hooks: Is parsing quality or chunking quality the bigger lever? Should we stop parsing (DSE/ColPali) or parse better?

11. TARGET: Benchmarking Table Retrieval for Generative Tasks (Ji et al., arXiv 2025) — with TableRAG (Chen et al., NeurIPS 2024)

• Link: https://arxiv.org/abs/2505.11545 (arXiv:2505.11545); TableRAG https://arxiv.org/abs/2410.04739 (arXiv:2410.04739) · Code: https://target-benchmark.github.io ; TableRAG code via HF paper page
• Tag: recent · Difficulty: 2 · Compute: laptop (embedding retrievers) / Colab+small API for the TableRAG LLM part
• Summary: TARGET asks how well retrievers find the right table for question answering, fact verification and text-to-SQL over data lakes: dense embeddings clearly beat BM25 on tables, but performance is highly sensitive to missing metadata (titles, headers) and varies a lot across datasets. TableRAG tackles the other scale problem — a single million-token table — via query expansion plus schema and cell retrieval before the LM, giving SOTA on Arcade/BIRD-SQL-derived benchmarks. (Text-to-SQL classic to cite: BIRD, Li et al., NeurIPS 2023, arXiv:2305.03111.)
• Why it matters: Extends the spring BM25-vs-dense comparison to structured data and shows serialisation choices (how you turn a table into text) are a form of "document representation" — the same question as sections-as-views in the patent paper.
• Pair implementation: Reproduce BM25 vs BGE-small on one TARGET task with 2–3 table serialisations (markdown, header-only, row sample); apply the best serialisation to tables extracted by Docling from the shared PDF testbed and evaluate table-level retrieval.
• Discussion hooks: Should a table be embedded as one document, as rows, or as its schema? Why does dropping headers hurt dense models more than BM25 — or does it?

12. CoIR: A Comprehensive Benchmark for Code Information Retrieval Models (Li et al., arXiv 2024/25) — with CoRNStack (Suresh et al., ICLR 2025)

• Link: https://arxiv.org/abs/2407.02883 (arXiv:2407.02883); CoRNStack https://arxiv.org/abs/2412.01007 (arXiv:2412.01007) · Code: https://github.com/CoIR-team/coir ; Nomic Embed Code / CoRNStack data on HF (nomic-ai); CodeXEmbed models Salesforce/SFR-Embedding-Code-400M_R (Liu et al., arXiv:2411.12644); jina-code-embeddings (Kryvosheieva et al., arXiv:2508.21290)
• Tag: recent · Difficulty: 2 · Compute: laptop/Colab with the 400M–1.5B models (7B models inference-only in 4-bit)
• Summary: CoIR gathers 10 datasets, 8 retrieval tasks (text→code, code→code, code→text, hybrid), 14 languages and >2M snippets in a BEIR/MTEB-compatible pip package, showing even SOTA general embedders underperform on code. CoRNStack builds a large contrastive code corpus with consistency filtering and curriculum hard-negative mining, and its models/rerankers top CoIR and localise functions in real GitHub repos.
• Why it matters: CoIR is BEIR for code (students know the tooling); CoRNStack is the practical version of the hard-negative story explained-but-never-implemented in the spring, and connects to the instructor's interest in cleaning contrastive supervision (S2Full citation pairs, patent section views).
• Pair implementation: Evaluate BGE-small vs SFR-Embedding-Code-400M vs jina-code on 3 CoIR tasks; then fine-tune MiniLM/ModernBERT on 50k CoRNStack pairs with and without mined hard negatives and report the CoIR gain; sanity-check on SciFact that code fine-tuning did not wreck text retrieval.
• Discussion hooks: Which hard negatives are false negatives in code (near-duplicate functions)? Should code and text share one embedding model?

Framing resources for the instructor's 30-min intro

• ColPali explainer blog by the first author (figures reusable for slides): https://huggingface.co/blog/manu/colpali
• ViDoRe leaderboard (V1/V2/V3, live ranking of visual retrievers): https://huggingface.co/spaces/vidore/vidore-leaderboard
• Survey: "Ask in Any Modality: A Comprehensive Survey on Multimodal RAG" (Abootorabi et al., 2025, arXiv:2502.08826; https://github.com/llm-lab-org/Multimodal-RAG-Survey)
• Survey: "Survey on Question Answering over Visually Rich Documents: Methods, Challenges, and Trends" (Barboule, Piwowarski, Chabot, 2025, arXiv:2501.02235) — a French (Sorbonne/Orange) survey, easy to cite for a Paris audience.

Notes

• Venues not verified from the arXiv comment field: VisRAG and M3DocRAG (I believe ICLR 2025 and NAACL 2025 Findings respectively, but list them as arXiv 2024); CoIR (arXiv v3 June 2025). CLIP's ICML 2021 venue is from memory, not the fetched page. All arXiv IDs, titles and first authors were confirmed from fetched abstract pages or search hits.
• Search budget: the session's WebSearch quota ran out mid-task; remaining verification used WebFetch on arXiv/ACL/HF pages, so coverage of very recent (mid-2026) critiques of ColPali is thinner than I would like. Two near-misses I saw but did not fully vet: "Evaluating Modern RAG: Textual, Multimodal, Dense, and Late Interaction Pipelines" (Kuru & Keskin, Aug 2026, arXiv:2608.23176 — a practitioner-oriented comparison, no code) and "Spatially-Grounded Document Retrieval via Patch-to-Region Relevance Propagation" (Georgiou, Dec 2025, arXiv:2512.02660, https://github.com/athrael-soju/Snappy — uses ColPali patch scores to select OCR regions, 28.8% context-token reduction); either could replace SERVAL as the "limits of ColPali" slot.
• Dropped: Marker (no paper, GitHub tool only); MinerU2.5 (arXiv:2509.22186, heavy VLM); ChartQA (2022, QA rather than retrieval); Spider 2.0 text-to-SQL (too far from IR for one slot); LlamaParse/commercial parser comparisons (blog posts only).
• Overlaps with other themes: SERVAL overlaps with generative query/document expansion (doc2query/HyDE theme); ColPali/jina-v4 overlap with late-interaction/efficiency themes; CoRNStack overlaps with a hard-negative-mining/training-data theme; jina-v4 overlaps with Matryoshka/universal-embedding themes. If those themes are assigned elsewhere, swap in M3DocRAG or PIXELRAG here.
• Colab T4 summary: ColQwen2 (2B), GME-2B, DSE-Phi3 (4B) and Qwen2-VL-2B all fit a 16 GB T4 in fp16 for inference/indexing of a few hundred pages (10–30 min); jina-v4 (3.8B) and ColPali (3B) fit with batch 1–2. Fine-tuning any of these on a T4 is only realistic with LoRA on ≤2B models and a few hundred pairs; the 7B code embedders are inference-only in 4-bit.

Addendum: papers added after the Gemini / Google Scholar pass (Sept 2026)

Five papers named in the external critique that the thirteen searches had missed or only folded into other entries. All verified on arXiv; the two from the instructor's own group are listed for completeness.

A1. LongEmbed: Extending Embedding Models for Long Context Retrieval (Zhu et al., EMNLP 2024)

• Link: https://arxiv.org/abs/2404.12096 (arXiv:2404.12096) · Code: https://github.com/dwzhu-pku/LongEmbed
• Tag: recent · Difficulty: 2 · Compute: laptop / Colab
• Summary: Benchmarks embedding models on long-document retrieval (LongEmbed: two synthetic and four real tasks up to 32k tokens) and studies training-free context extension: position interpolation, RoPE-based methods, parallel context windows. RoPE-based models extend far better than absolute-position models; the released E5-RoPE and E5-Mistral variants extend to 32k.
• Why it matters / link to the course: The natural companion of ModernBERT (Theme 1) and of the instructor's full-text-window training: what happens to a single vector as the input grows. Fits one "long-context encoders" talk together with ModernBERT.
• Pair implementation: Reproduce the context-extension comparison for a 512-token encoder (BGE-small, E5-base) on two LongEmbed tasks; then embed full-text scientific papers (a handful from arXiv or the S2Full corpus) whole vs chunked vs windowed and evaluate on the scientific-article testbed.
• Discussion hooks: Does a longer window help retrieval or only hurt less? Is the ceiling in the position encoding or in mean pooling?

A2. Found in the Middle: Permutation Self-Consistency Improves Listwise Ranking in LLMs (Tang et al., NAACL 2024)

• Link: https://arxiv.org/abs/2310.07712 (arXiv:2310.07712) · Code: https://github.com/castorini/perm-sc
• Tag: recent · Difficulty: 1 · Compute: Colab
• Summary: Listwise LLM rankers (RankGPT-style) are sensitive to the order in which candidates are presented. Shuffle the candidate list several times, rank each permutation, and aggregate the rankings (Kemeny-style); this removes the positional bias and improves nDCG on TREC DL and BEIR for GPT-3.5/4 and open models.
• Why it matters / link to the course: The reranking counterpart of Lost in the Middle (Theme 9) and a cheap fix for RankGPT (Theme 7); shows RRF-style aggregation used inside a single model.
• Pair implementation: Reproduce the order sensitivity of a listwise 7B reranker (4-bit) on SciFact top-20 by running five shuffles; aggregate and report the gain against the single-pass ranking and the MiniLM cross-encoder; measure the cost multiplier.
• Discussion hooks: Is position bias an artefact of instruction tuning? If five passes are needed, is a cross-encoder simply cheaper?

A3. Autoregressive Search Engines: Generating Substrings as Document Identifiers — SEAL (Bevilacqua et al., NeurIPS 2022)

• Link: https://arxiv.org/abs/2204.10628 (arXiv:2204.10628) · Code: https://github.com/facebookresearch/SEAL
• Tag: classic · Difficulty: 2 · Compute: laptop (FM-index) / Colab
• Summary: Instead of learning docids, generate n-grams that occur in the corpus, constrained by an FM-index so every generated string exists somewhere; documents are scored by the n-grams generated for them. Competitive with DPR on NQ and KILT with a BART-large model and no docid design. In Theme 8 it appears inside the LIMIT-based entry; it deserves its own slot as the most reproducible generative-retrieval method (public checkpoints, CPU index).
• Why it matters / link to the course: Generative retrieval that stays lexical: a bridge between BM25, doc2query and DSI (Theme 8).
• Pair implementation: Build the FM-index over SciFact/NFCorpus, run the public BART checkpoint zero-shot, and compare with BM25 and BGE-small; then analyse which generated n-grams drive hits and misses.
• Discussion hooks: Is SEAL a neural query rewriter on top of an exact-match index? What does its failure on LIMIT-H say about n-gram identifiers?

A4. Citation-Driven Multi-View Training for Patent Embeddings: QaECTER and Sophia-Bench (Djemmal, Zuo, Gerdes & Guiller, FUSION 2026)

• Link: https://arxiv.org/abs/2604.22897 (arXiv:2604.22897) · Code/model: QaECTER is deployed in Questel products; Sophia-Bench (10,000 queries, 75,000 documents, ten years, eight IPC sections, twelve jurisdictions) announced for release
• Tag: recent · Difficulty: 2 · Compute: Colab
• Summary: Citation-driven multi-view training of a patent encoder, reported to outperform general-purpose and patent-specific retrievers, including models 23× larger, across query types, technology domains and jurisdictions on the Sophia-Bench benchmark.
• Why it matters / link to the course: The instructor's group's own answer to PaECTER (Theme 4) and the companion of the patent self-supervision paper; the cross-jurisdiction and temporal stratification of the benchmark is exactly the "priority-date" gap the other searches could not fill.
• Pair implementation: Only if the instructor agrees and the benchmark is released: reproduce the multi-view recipe at small scale on HUPD with BGE-small and compare with PaECTER on DAPFAM; otherwise use the paper as the discussant's reference for the PaECTER slot.
• Discussion hooks: How much of the gain is the multi-view objective versus the citation data Questel holds? Can a benchmark that is internal be a benchmark?

A5. Sparse Coverage: Semantic Center Representations for Patent Prior-Art Retrieval (Zuo, Gerdes et al., CORIA-TALN 2026)

• Link: https://arxiv.org/abs/2608.16918 (arXiv:2608.16918) · Code: see paper
• Tag: recent · Difficulty: 2 · Compute: Colab
• Summary: Represents a patent by a small set of semantic "centres" rather than one vector, targeting prior-art retrieval on CLEF-IP 2013; a multi-vector view of the section-based self-supervision idea.
• Why it matters / link to the course: Joins Theme 2 (multi-vector, ConstBERT's fixed number of vectors) with Theme 4 (patents); the instructor's own group.
• Pair implementation: Only with the instructor's agreement: reproduce the centre construction on a HUPD sample with BGE-small and compare with single-vector and ConstBERT-style representations on DAPFAM.
• Discussion hooks: How many centres does a patent need? Is this ConstBERT with an interpretable pooling?

Framing resource added

• Magdy & Jones, "PRES: a score metric for evaluating recall-oriented information retrieval applications", SIGIR 2010, https://dblp.org/rec/conf/sigir/MagdyJ10.html (PDF at https://doras.dcu.ie/16180/): the recall-oriented metric patent search uses instead of nDCG; used in the session 1 patent primer.