Case study · 2026-04-22 · v0.8.0 / 90%
Vitruvius
Encoder archaeology for dense retrieval.
Six encoders. Three BEIR subsets. Four architecture families. Latency, accuracy, head pruning, position-shuffle sensitivity — all measured with the same from-scratch harness. The goal: find where the transformer's full bidirectional attention is earning its cost, and where it's wasted compute.
At a glance
- Phases
- 1 → 8 / 9
- Encoders
- 6
- Bench cells
- 18 + 54 + 9
- Training
- 500K MS MARCO
- Hardware
- A100-SXM4-80GB
- Seed
- 1729
- Status
- Phase 8 landed
The question
Dense retrieval systems rely on deep encoders to produce embeddings for similarity search. In production, those encoders operate under hard latency constraints — and under pressure, the transformer quietly becomes a bottleneck. The standard response is distillation: smaller transformers that trade accuracy for speed. But that accepts a framing I wanted to push on.
What if the transformer is the wrong inductive bias for retrieval in the first place?
Retrieval is not language modeling. Document encoding is offline and query encoding is latency-critical. The output is a single fixed vector, not a token sequence. The task may be more bag-of-concepts than sequence-order-dependent. Compressing meaning into a dot-product-comparable vector is fundamentally different from predicting the next word.
Vitruvius empirically compares transformer, SSM (Mamba), recurrent (LSTM), and convolutional (CNN) encoders on standard retrieval benchmarks, measures both accuracy AND latency, and asks where on the Pareto frontier the non-transformer architectures actually sit.
Phase 5 · v0.5.0 · 2026-04-19
The Pareto frontier.
Averaged across the three BEIR subsets (NFCorpus, SciFact, FiQA). X-axis log-scaled. Hover a point for numbers.
Not parameter-matched, not equal FLOPs, in-domain evaluation only, max_seq=128. The from-scratch encoders were trained on a single pass on 500K MS MARCO triplets — not tuned, not mined for hard negatives. What the plot does show: with that identical training budget, a 6.4M-parameter BiLSTM sits on the frontier next to MiniLM.
Phase 3 · v0.3.0
Step one: did we validate against the leaderboard?
Before any alternative architecture, the harness has to agree with published results. The 3×3 sweep — three pre-trained transformers against three BEIR subsets — lands 8/9 cells in band against approximate leaderboard references (±0.03).
| Encoder | nfcorpus | scifact | fiqa | sim |
|---|---|---|---|---|
minilm-l6-v2 | 0.3165 | 0.6451 | 0.3687 | cosine |
bert-base | 0.3169 | 0.6082 | 0.3229 | dot |
gte-small | 0.3492 | 0.7269 | 0.3937 | cosine |
bert-base × scifact landed at 0.6082 vs reference 0.68 (Δ = −0.072) — flagged, not massaged. pytrec_eval agrees bit-exact; the dataset is well-behaved on the other two encoders. It is an out-of-domain transfer gap of msmarco-bert-base-dot-v5 specifically on scientific-claim retrieval. Documented, not swept under the rug.
Phase 3.5 · v0.3.5
Where quadratic attention becomes visible
Query encoding latency was profiled on real BEIR queries (not synthetic fixed-length strings), CUDA-event timed, 100 measured passes after 10 warmup. At batch 1, latency is essentially flat across datasets within each encoder — kernel-launch overhead dominates sequence variance.
At batch 32, the story changes. bert-base × scifact jumps to 24.7 ms — 3.4× its batch-1 time and 2.1× its nfcorpus batch-32 time. That is the O(n²) attention cost on longer scientific documents becoming visible. It is precisely the gap a linear-time SSM should close, and the motivation for Phase 5.
| Encoder | bs=1 | bs=32 · nfcorpus | bs=32 · scifact | bs=32 · fiqa |
|---|---|---|---|---|
minilm-l6-v2 | 4.18–4.30 | 6.06 | 8.60 | 7.19 |
bert-base | 7.23–7.26 | 11.64 | 24.73 | 17.85 |
gte-small | 7.60–7.65 | 9.77 | 11.68 | 11.07 |
Phase 5 · v0.5.0
Training the alternatives from random init
Three non-transformer bi-encoders trained from random initialization on 500K MS MARCO triplets with identical hyperparameters: InfoNCE τ=0.05, AdamW lr=1e-4 wd=0.01, linear-warmup + cosine decay, AMP fp16, batch 64, 3 epochs, 19,452 steps. Then plugged into the same bench + latency harness as the pre-trained transformers.
Phase 4 was absorbed here. The only pre-trained Mamba bi-encoder checkpoint turned out to be a cross-encoder scanner, not usable as a drop-in replacement. Kill-switch fired; Mamba got rolled into Phase 5 as a from-scratch bi-encoder trained on the same 500K triplets as LSTM/CNN — genuinely apples-to-apples.
| Encoder | Params | Best val loss | Wall-clock | Peak GPU (MB) |
|---|---|---|---|---|
lstm-retriever | 6.41M | 0.4719 | 10m 16s | 561 |
conv-retriever | 4.92M | 0.7962 | 16m 25s | 189 |
mamba-retriever-fs | 23.74M | 0.1807 | 49m 34s | 2239 |
Mamba trained with num_workers=0 (the only difference) — a DataLoader worker segfault rooted in Triton+fork state inheritance.
conv-retriever × fiqa = 0.037 is below the §5.8 red-flag threshold of 0.05 — flagged as a measured finding, not a training bug. Max receptive field of 7 tokens (stacked kernels 3/5/7) cannot span FiQA's longer financial documents. The training-loss signal already previewed this (CNN final val 0.80 vs LSTM 0.47 vs Mamba 0.18). pytrec_eval agrees bit-exact.
Phase 6 · v0.6.0
What fails, and whose failure is whose
Phase 6 is the first analysis-heavy phase — no new encoders, no new bench runs, no GPU. Purely downstream of the 18 bench JSONs preserved by Phases 3 and 5. A 7,626-row query-level dataframe. Per-query failure grids. Spearman rank agreement across encoders. 470 hand-labelled failing queries across an eight-category taxonomy.
14% → 90.4%
Zero-nDCG@10 rates span from gte-small × scifact (14%) to conv-retriever × fiqa (90.4%). Transformer baseline is not zero even on scifact.
38 / 91 / 235
Queries where any pre-trained transformer succeeds but all three from-scratch fail. 30–36% of the two harder sets is where pre-training's value concentrates.
85 / 32 / 152
Queries where all 6 encoders fail (nDCG@10 < 0.1). Bounds what any dense-retrieval architecture change can recover.
127 vs 19
Unique wins: 127 transformer, 19 from-scratch. The "route-hard-queries-to-big-model, easy-ones-to-small" deployment idea is not supported by the data.
Phase 7 · v0.7.0
Attention heads: how many are actually earning their cost?
Per-head ablation plus cumulative-pruning-curve characterization of MiniLM (72 heads), BERT-base (144 heads), and GTE-small (144 heads) across NFCorpus, SciFact, FiQA. Zero-shot (no retraining). Eight cells complete, one (gte-small × fiqa) deferred for budget.
The measured answer: a majority of attention heads in these retrieval transformers can be pruned at < 5% nDCG@10 drop, zero-shot. Per-dataset detail in experiments/phase7/SUMMARY.md.
transformers 5.x silently drops head_mask. The argument was removed from BertSelfAttention.forward's named params but still accepted via **kwargs — no error, no warning, just silently ignored. The first sweep (~3h of pod compute) produced valid-looking JSONs with every per-head delta_nDCG@10 exactly 0.0. Caught only when inspecting output post-hoc.
Fix: pin transformers<5.0, load models with attn_implementation="eager". New discipline: before any multi-hour sweep, run one cell and verify the independent variable actually moves the measurement. Two lines, two seconds, three hours saved next time.
Phase 8 · v0.8.0 · 2026-04-22
Position sensitivity — the surprise.
Content-token permutation after tokenization (special tokens pinned, padding untouched, seeded). 54 bench cells: 6 encoders × 3 BEIR subsets × 3 shuffle modes. Sensitivity = (baseline − shuffled) / baseline. 0 = invariant. 1 = total collapse.
Family-level sensitivity · docs-shuffled
recurrent
0.166
Lowest — the surprise
transformer
0.211
Not uniquely robust
ssm
0.243
Within transformer range
convolutional
0.314
Most sensitive
The surprise
Handoff §8.4 predicted LSTM sensitivity around 0.8+. Measured: 0.166. The lowest of all four families.
The likely cause: masked mean-pool over bidirectional hidden states aggregates bag-of-concepts information heavily. For retrieval specifically (graded nDCG@10 on content-word-rich BEIR subsets), the cumulative-state component outweighs the strict-order component. The LSTM may have learned to be position-tolerant as a retrieval shortcut on the MS MARCO contrastive signal.
CNN is the most sensitive at 0.314.
Expected "high but bounded" — in practice kernel receptive fields (3, 5, 7) depend on token adjacency and global max+mean pooling does not fully recover.
Transformer ≈ SSM (0.21 vs 0.24).
The "bag-of-concepts argument for transformers" still holds — they retain ~79% of retrieval quality on shuffled docs. But they are not uniquely robust.
FiQA spikes everyone.
All 6 encoders show highest sensitivity on FiQA (0.24–0.49). NFCorpus and SciFact stay ≤0.27. Financial questions carry information in discourse structure that bag-of-concepts can't recover.
Methodology — the discipline under the measurements
The project's metric pipeline is from-scratch (gate IR-2): graded-gain nDCG with gain = 2^rel − 1, discount log2(i+1), iDCG over full qrels. pytrec_eval runs alongside as a cross-check, not a replacement. |Δ| has been ≤ 2e-3 on every one of the 18 bench cells; Recall@k is bit-exact on 7/9.
CUDA timing uses torch.cuda.Event, not time.perf_counter(). FAISS IndexFlatIP, top-k = 100, batch size 128. Seed 1729 everywhere — Ramanujan's taxicab number. Each cell writes a per-query-results block (schema §5.7) so downstream analyses like Phase 6 can join on query_id without re-running the pod.
Two disciplines emerged from costly lessons. After the bert-base checkpoint mis-targeting (dot-trained being L2-normalized, −0.08 nDCG): the encoder declares its similarity, the harness reads it. Forgetting to declare fails at class-creation time. After the transformers 5.x silent head-mask drop: one-cell sanity check before any multi-hour sweep, verifying the independent variable actually moves the measurement.
What remains · Phase 9
Phase 9 — the 100% milestone — is a 5-to-7-page opinionated technical note, arXiv-ready. Not a survey. Not a system paper. A short document that says: here is what we measured, here is what changed our mind, here is what we now believe about alternative encoders for dense retrieval, and here is exactly where we were wrong.
The query_level_sensitivity.parquet artifact (22,878 rows across 6 encoders × ~300–650 queries × 3 modes × 3 datasets) sits ready for the cross-phase join: Phase 6's failure taxonomy joined on query_id against Phase 8's per-query shuffle delta. That join runs locally and does not need pod time. It answers questions like: are `LEN-LONG` queries more position-sensitive? are `PARAPHRASE` failures position-insensitive?
Stack
Version chronology
0.8.0 · Phase 8 — 90%
Token-position shuffle ablation. 54 cells. Family sensitivity table. Three surprises: LSTM lowest, CNN highest, Transformer ≈ SSM.
0.7.0 · Phase 7 — 80%
Per-head ablation + cumulative pruning curves. transformers 5.x bug runbook filed. Eager-attention tradeoff documented.
0.6.0 · Phase 6 — 70%
Per-query failure taxonomy. 470 labelled queries, eight categories, cross-encoder wins/losses sets.
0.5.0 · Phase 5 — 55% (absorbs Phase 4)
From-scratch LSTM + CNN + Mamba bi-encoders on 500K MS MARCO. First 6-point Pareto plot. Pareto-optimal subset identified.
0.3.5 · Phase 3.5 — 30%
Latency profile across three transformer encoders × three BEIR subsets × batch 1/8/32. O(n²) attention cost becomes visible.
0.3.0 · Phase 3 — 20%
3×3 encoder × BEIR validation. 8/9 cells in-band vs leaderboard references. bert-base × scifact flagged as measured out-of-domain gap.
0.2.0 · Phase 2 — 10%
First pod run. MiniLM × NFCorpus reproduced at nDCG@10 = 0.3165. Harness validated before scaling.
0.1.0 · Phase 1
CPU-runnable scaffold. Registry, metrics from scratch, FAISS wrapper, CLI. macOS libomp workaround.