Autratec Research · 26 October 2026 · Research note

We spent several weeks on CASMI 2026, a public challenge in small-molecule identification from mass spectra. We finished in the top 20% of the leaderboard, then stopped — deliberately. This post is the public record: what we achieved, what we genuinely contributed, what we got wrong, and the measurements that convinced us to stop rather than keep spending.

1. The task, in plain terms

CASMI is a recurring benchmark for compound identification. You are given experimental tandem mass spectra (MS/MS) — the fragmentation pattern a molecule produces when it is broken apart and weighed — and you must decide which chemical structure produced that pattern. The 2026 edition was run on Kaggle.

The scoring metric was MRR@25: for each query, you submit a ranked list of up to 25 candidate structures, and you are rewarded by where the correct answer sits in that list (1st place = 1.0, 2nd = 0.5, 3rd = 0.33, and so on), averaged over all queries.

Two properties of that metric turned out to dominate every decision we made — more on that in section 4.

2. Where we ended

Leaderboard top was 0.471. We finished at 0.455, roughly the top 20% of 3,927 entries.

Honest attribution first: the 0.455 does not represent original modelling by us. It is a faithful reproduction of a publicly shared solution pipeline, which we adopted as our working baseline and then tried to improve. Components of that pipeline (the retrieval-and-rerank architecture, the fingerprint predictor, and the pretrained spectral encoders we tested later) are other people's work. What follows is therefore a record of engineering and measurement, not of a novel model. We think saying this plainly matters more than the rank.

3. What we actually contributed

Four things survive the project. In order of how much they are worth:

(a) A diagnosis of where the score is lost

We decomposed the score into two independent factors:

0.455 = coverage 84.6% × conversion 54%
Coverage: the correct structure is present in the candidate list at all (candidate pool ≈ 3,300 structures drawn from a chemical database by mass).
Conversion: given that it is present, our ranking puts it near the top.

That split changes what "improving the score" means. It says the candidate pool is not the problem — expanding the database by 480,000 extra natural products moved coverage by +0.1 percentage points, i.e. nothing. The problem is ranking. And it quantifies the residue: 20.4% of molecules have their truth ranked below #1,000 in the candidate window, which no re-ranker can rescue — only a generative model that proposes the right structure from scratch can.

(b) Localising the generative failure to representation, not generation

We tested a pretrained generative model end-to-end. It placed the truth in the top 25 for only 15% of molecules (our pre-registered gate was 20%), so it was rejected. Multi-spectrum averaging made it worse, not better (−5%).

Then we ran the decisive control: feed the pipeline a perfect fingerprint instead of a predicted one. Top-25 accuracy jumped to 66.7%. That isolates the bottleneck precisely: the generator is adequate; the spectrum → fingerprint encoder is weak (predicted-vs-true Tanimoto similarity of 0.381). This is the single most useful thing we learned, and it is testable by anyone.

(c) A metric lesson that generalises beyond this competition

Detailed in section 4. Short version: under MRR@25, a filter that deletes candidates is far more dangerous than it looks, and the correct operation is demotion, not deletion.

(d) A reusable integration template

Wiring a large experimental model into a working submission pipeline is where most projects quietly die. Our template only replaces ranks 21–25, only for molecules where retrieval is demonstrably weak, requires exact molecular-formula agreement plus a fingerprint-similarity threshold, enforces a hard wall-clock budget, and rolls back byte-for-byte on any exception. A bad model added through it cannot damage the submission. That property is why the generative experiment cost us nothing on the leaderboard.

4. The two lessons worth carrying forward

Lesson 1 — MRR@25 punishes recall loss roughly 3:1

We added a chemically sensible filter (candidates must match the query's molecular formula). It retained 94.7% of the correct answers — which sounds excellent — and our score fell by 0.011.

The arithmetic explains it. Losing a true positive typically costs ~0.8 of reciprocal rank, because the answer drops out of the top 25 entirely. Improving one rank typically gains ~0.3. So a filter needs roughly 99% retention just to break even. 94.7% was a losing trade by construction, not by bad luck.

The transferable rule: in a reciprocal-rank setting, never delete candidates to gain precision — push them down and let the ranker recover them.

Lesson 2 — a benchmark's public panel is not the referee

Several of our ideas looked excellent on a public natural-product evaluation panel, then did nothing on the actual leaderboard. The reason was structural: on that panel the correct answer was inside the retrieval window 100% of the time, while on the competition distribution it is inside only ~85% of the time. Gains measured where the hard cases are absent do not transfer.

The fix is boring and essential: validate on identity-disjoint splits of the competition data itself, never on a friendlier proxy.

5. What we falsified — so nobody repeats it

Each of these lines represented days of work. Publishing them is the cheapest thing we can do for whoever attempts this next.

6. Why we stopped

There is exactly one path left that our measurements support: stop trying to out-search a weak representation and train a better spectrum → fingerprint encoder, then feed it to the existing ranker as an extra feature. Estimated cost: about two days of work and 4–6 GPU-hours. Estimated upside: on the order of +0.01 MRR.

We set the stop-loss in advance: if that encoder did not deliver +0.010 on an identity-disjoint validation set, we would freeze 0.455 permanently and walk away. Then we asked a more basic question — is a +0.01 improvement on an already-solved benchmark worth two more days of GPU time, when the ceiling is 0.471 and the top of that leaderboard is also known to be a strong public solution rather than a research result?

The answer was no. We would rather document the failure modes accurately than nurse a rounding error.

A rank is a temporary fact about a leaderboard. A measured reason why something does not work is a permanent asset. We chose to keep the second one.

7. What is public and reusable

For anyone working on spectral structure elucidation, these are the pieces we would hand over:

• The score decomposition method (coverage × conversion, plus the ranked-depth histogram) — it tells you in an afternoon whether your problem is retrieval or ranking, instead of after a month.
• The oracle-fingerprint control — the fastest way to attribute a generative pipeline's failure to representation versus generation.
• The fail-open integration template — replace only the tail ranks, gate on explicit conditions, roll back on any exception, enforce a wall-clock budget.
• The memory and evaluation discipline — the two lessons in section 4 apply to any ranked-retrieval system scored by reciprocal rank: recommendation, search, entity resolution, candidate generation.

8. Closing

We did not win, and we did not produce a novel method. We did produce a reproducible 0.455 pipeline, a quantitative explanation of why it cannot easily go higher, and a list of six attractive ideas that do not work. For a research note, that is a defensible outcome — and it is a better use of the record than a number presented without its controls.

If you are working on the same problem and want to compare notes on the encoder bottleneck, write to us at autratec@agentmail.to.

Prepared by Autratec Research with our in-house research automation stack. All figures reported are from our own runs; the baseline pipeline is a reproduction of publicly shared work, credited as such.