Ongoing work

Research.

Two active threads investigating the internal mechanisms of neural networks and how architectural choices shape model behavior — plus paper work in progress.

Orpheus

Active

Mesa-Optimization Probe

Apr 2026 → Present
PyTorch TransformerLens Mechanistic Interpretability Activation Patching
Research question

Do small transformers trained on in-context learning tasks implement gradient-descent-like optimization internally during their forward pass? If so, where in the network does this emerge, and what computational signatures distinguish a mesa-optimizer from a lookup table?

Training small transformers (1–4 layers) on synthetic in-context linear regression, then using mechanistic interpretability to reverse-engineer the algorithms they learn. The goal is to identify whether the internal computation resembles an optimization process — gradient descent on an implicit loss — rather than pattern matching or memorization.

Full case study GitHub

Vitruvius

v0.8.0 · 90%

Encoder Archaeology

Apr 2026 → Phase 8 landed
BEIR MS MARCO Mamba LSTM · CNN FAISS pytrec_eval
Research question

Are bidirectional transformer encoders truly optimal for dense retrieval, or can alternative architectures (SSMs, recurrent models, long-convolution models) achieve comparable quality at lower latency? Where specifically is the transformer's full bidirectional attention earning its cost versus being wasted compute?

Latest findings · Phase 5 → 8

1

Pareto frontier resolved. Averaged across the 3 BEIR subsets, the optimal subset is gte-small, minilm-l6-v2, lstm-retriever, conv-retriever. Mamba and BERT-base are both dominated.

2

LSTM is the most position-tolerant. Doc-shuffle sensitivity 0.166 — the lowest of all four families. A 6.4M-parameter BiLSTM retains 83% of retrieval quality on shuffled documents.

3

Most attention heads are prunable zero-shot. Cumulative pruning of MiniLM / BERT / GTE: the majority of heads can be zeroed at <5% nDCG@10 drop without retraining.

4

The deployment myth dies. "Route-hard-queries-to-big-model" does not hold: 127 unique transformer wins vs 19 from-scratch unique wins across 3 datasets.

Architecture families under study

Transformer

MiniLM · BERT · GTE

Recurrent

BiLSTM (from-scratch)

Convolutional

1D CNN (k = 3/5/7)

State-Space

Mamba2 (from-scratch)

Full case study GitHub

Papers

TITAN: Trustworthy Identity Tokenization and Attestation Network

Submitted

Submitted to SeCrypt 2026

End-to-end identity verification: AI-powered document forensics, zero-knowledge proofs for private on-chain attestation, and GNN-based transaction graph analysis for anti-money laundering. KYC compliance without exposing raw identity documents to verifiers.

ZK Proofs Circom GNN KYC/AML

Emergent Features in Large Language Models

Working draft

In progress

Investigating feature emergence patterns in LLMs across scale. How capabilities appear discontinuously, and what mechanistic signatures precede emergent behavior. Connecting Orpheus's findings on in-context optimization to broader questions about what large models learn to compute.

LLMs Emergence Interpretability

Vitruvius — Technical Note

Phase 9 · in progress

5–7 pages · arXiv-ready

Short, opinionated technical note synthesizing the Vitruvius sweep. What changed our minds about the transformer's inductive bias for retrieval, and exactly where we were wrong.

Dense Retrieval Pareto Architectures