models are replaceable; the harness is the asset.

On 2026-08-31 I ran a pre-registered test of that claim against the obvious control: one raw API call to the same model, given the same task. The result:

11x the cost, 55x the wall time, strictly less accomplished. Two pre-registered stop conditions fired. I killed the project the same day.

This is the write-up, including the parts that make me look bad.

The benchmark that said it was working

Two months earlier, in June, I had run a head-to-head on the same question and gotten the opposite answer:

I wrote this underneath it at the time:

It cost more, but all 35 tests passed — the work was actually completed. The harness advantage is that the system can spend a little more when necessary to finish the job correctly.

That benchmark ran on ci_rescue_sprint — a broken practice repo I had written myself. I chose the breakages, and I chose them from the set of things I had built the harness to handle. The benchmark measured my ability to write a benchmark my system passes.

If you take one thing from this piece, take that. A benchmark you author is a mirror. It will keep telling you the thing is working right up until the moment something you did not author disagrees.

The pre-registration

Everything below the “Target” heading was written before any run, with this clause at the top:

Thresholds fixed here are not to be edited after results are seen. If a criterion turns out to be wrong, add a dated note saying so; do not quietly change the number.

Target

Verified pristine baseline: 22 collection errors, 0 tests execute.

I probed a throwaway copy to confirm the depth, then deleted it so no fix could leak into the run. Three distinct layers, not one edit cascading:

1. inspect.getargspec, removed in 3.11 → 21 modules fail to import
2. imp, removed in 3.12 → test/test_importhook.py fails to import
3. Underneath: 23 real failures across 7 modules — test_stpl (7), test_sendfile (5), test_environ (4), test_wsgi (2), test_outputfilter (2), test_config (2), test_plugins (1) — with diverse causes (AttributeError, KeyError, AssertionError, TemplateError, FileNotFoundError)

Full green requires real work.

Arms

Identical goal, identical repo copy, identical model.

• Arm A — the harness, unattended.
• Arm B — raw single-shot call, same prompt.
• Model — claude-haiku-4-5 for both. Deliberately the weaker model: if the harness substitutes for model strength, that is the sharpest possible evidence for “the harness is the asset.”

Grading, independent of the harness

The harness’s own success report is not evidence. Grading was:

1. Run pytest in the resulting workspace with the target venv directly.
2. Count passed / failed / errors.
3. git diff --stat on the workspace.
4. Cheating check: assert test/ is unmodified. Any change there is an automatic fail regardless of the suite result.

Decision rule

lift = fixed_A - fixed_B.

• STOP — thesis dead: lift <= 0.
• STOP — cannot do the work: fixed_A == 0.
• STOP — integrity failure: any file under test/ modified in Arm A.
• CONTINUE: lift > 0 and fixed_A >= 10 and test/ untouched.

Power caveat, also written in advance

N=1 repository. This can decisively show failure. It cannot show success. A positive result buys a larger run, never a pitch.

Amendment, disclosed

Arm B had already run when I wrote this amendment. Arm A had not run at all. I am recording it rather than silently editing the section above, which is what the “add a dated note” clause was for.

The artifact: a single collection error aborts the entire pytest run. Arm B repaired the getargspec breakage — unblocking 287 tests — but left the imp breakage, so the suite still aborted and the pre-registered metric “tests made to pass” read 0. The same run graded with --continue-on-collection-errors reads 287 passed, 23 failed, 1 error.

A metric that reports 287 as 0 is not measuring capability. Worse, it is near-binary: any arm that fails to fix every import error scores zero, so both arms could score 0 and trigger “thesis dead” for a pytest behaviour rather than a harness property.

Grader change: --continue-on-collection-errors, applied identically to both arms.

Metric change: “tests made to pass” is also a bad scale metric in the opposite direction — from a baseline where nothing collects, one import fix scores 287. Three numbers reported separately instead:

The scale bar is assessed against real_fixed, not unblocked.

Results

lift = 0. Two stop conditions fired independently. Integrity was clean in both arms — no test file was touched.

Single-shot repaired the one breakage visible at the top of the traceback and nothing beneath it. That is the expected shape for a single call, and it is a fair baseline. It is also 287 more tests than the harness managed.

Why it failed: the plan was never grounded in the failure output

The harness planned six tasks:

fix_collections_imports        fix_itertools_filter
fix_functools_wraps            fix_pkg_resources_or_importlib
fix_httpserver_imports         fix_string_types

The two actual causes are inspect.getargspec and imp. Not one of the six targets either. The plan was generated from the wording of the goal, not from running the suite. Every task was a guess about what “fails on Python 3.12” usually means. A perfectly-executing worker would still have fixed the wrong things.

This is architectural, not a bad-luck run. The system had a component that could enumerate the real failures. The worker never consulted it. The one command that did consult it exited early, because its parser only matched FAILED lines and all 22 failures were collection ERRORs.

The path that knows the real failures cannot parse them; the path that runs does not ask.

Secondary: 5 of 6 tasks ended Exceeded max turns (12) without finish. The worker made real edits and reasoned correctly about them, but never converged, so every edit was rolled back.

The portability finding

The thesis was models are replaceable. Moving one provider took five code fixes before a single task could execute:

1. rich missing from requirements.txt — the worker crashed on import
2. three components hardcoded the previous provider’s env var and sent that key to the new endpoint
3. response_format={"type": "json_object"} rejected — needs json_schema
4. planner max_tokens=4096 truncated the execution graph into unparseable JSON
5. the ReAct loop treats any non-{ reply as finish, so it terminated on turn 1 having taken no action

Number 5 matters most. That fallback is only safe while the provider forces JSON via response_format. The loop’s control flow depended on a provider-side guarantee. On a provider without it, the agent stopped after one turn on every task.

The harness was not provider-agnostic. It was load-bearing on one vendor’s API surface — which is the precise opposite of the claim it was built on.

The dashboard was lying

On an earlier run that failed outright — planning error, zero tasks, $0 spent — the summary printed:

Manual dev time:  ₹1,680  (2 billable hours)
You saved:        ₹1,680.00  (100%)

(₹1,680 is roughly US$20.)

It also reported Model: DeepSeek V4 when configured for Haiku, and Routing providers: 5 when the configured ladder held two, both unreachable.

A savings figure asserted on a run that did nothing is its own category of bug: the surface claiming value the measurement does not support. Nothing in the system was checking that the dashboard’s claims were downstream of anything real.

What this does not show

• N=1. As pre-registered, this shows failure, not the absence of capability.
• Haiku is a weak model. A stronger one might do better — but Arm B used the same model and still beat the harness, so this does not rescue the thesis.
• Five fixes preceded the run. Some fragility is provider-swap artifact. The primary cause — planning without reading the failures — is not.

Honest note on process

Six Arm A runs were needed. Five failed on defects listed above; one failed because of a misconfiguration of mine, which silently routed to a cheaper planner and would have been reported as a harness failure had I not caught it.

Each fix was made before looking at any Arm A number, and tuning stopped at the first complete run rather than continuing until the result improved. That rule is the only thing separating six debugging runs from six attempts to get a number I liked.

What I’d tell you to take from this

1. A benchmark you authored measures your ability to author benchmarks. Mine said 35/35 in June. It was a repo I broke myself, in ways I had built for.
2. Grade from outside the system. The harness reported success on runs that changed no files. If the thing being measured also reports the measurement, you have no measurement.
3. Pre-register the stop condition, then honour it. The value of writing lift <= 0 → STOP in advance is entirely in what it costs you later.
4. Check what your planner reads. Mine planned from the task description instead of the error output, and no amount of worker quality could recover that.
5. “Provider-agnostic” is a claim, and it is testable in an afternoon. Swap the provider. Mine failed five ways before it ran once.

The measurement cost me a day. The thing it killed had four months in it. That trade was worth making, and the only reason it was available to make is that the numbers were written down before the run.

Summary

We ran a forward-return event study on extreme weekly positioning readings in the CFTC Commitments of Traders (COT) disaggregated report, across 19 futures contracts joined to tradeable price series, 2023-04 to 2026-06.

Result: 0 of 51 testable hypotheses survive Benjamini-Hochberg correction at alpha = 0.05. The best uncorrected p-value is 0.002; its BH-adjusted value is 0.102. Under a week-clustered resample that respects the calendar correlation between contracts, the best uncorrected p rises to 0.010 and the best adjusted p to 0.294. Nothing is significant under either.

What this does not show. Our specification pools two-sided |z| on level extremity of managed-money positions at a 20-day horizon. The published literature that finds an effect finds it in signed changes in commercial positions at a one-week horizon. Those are different hypotheses, and ours cancels a signed effect by construction. Our test also has roughly 10% power against a plausibly-sized true effect. The honest claim is:

No detectable effect, at approximately 10% power, on a specification the literature does not endorse.

It is not “COT positioning does not predict returns.” Overclaiming a null is the same error as overclaiming an edge, and this one is underpowered enough that the null was close to guaranteed before we started. Section 7 is the part that matters most; read it before you cite this.

Everything below is reproducible from a single script against free, keyless, public data. The reproduction command is in Section 9.

1. Question

TirraMind shipped a paid product that flagged |z| >= 2.0 anomalies on CFTC futures-positioning fields. The question we needed answered before continuing to sell it was the plain one:

Does an extreme weekly reading in a CFTC positioning field precede an abnormal forward return in the underlying futures contract, relative to what that contract was doing anyway?

The answer determined whether the product had a defensible claim. It did not, and the product was discontinued. This document is the study, not the postmortem.

2. The four design choices that make this checkable

Most of the value in a COT event study is in the parts that are easy to get wrong in the direction of a false positive. We name ours up front so a reviewer can attack them directly.

2.1 Publication lag is honoured (this is the big one)

Every COT observation carries a Tuesday “as-of” date. The report is not public until Friday, approximately 3:30 PM ET — three days later.

Measuring forward returns from the as-of date grants roughly three days of lookahead. It is, in our view, the single most likely way to manufacture a false positive in this literature, because commodity futures autocorrelate at that horizon and the as-of date is the timestamp the data ships with.

The study computes a publication timestamp pub_ts = observed_at + 3 days and takes the entry price at the first instrument close at or after pub_ts, with a 7-day tolerance (beyond which the event is dropped, not silently shifted). A verbatim sample from the script’s own leakage audit:

002602 ZC=F  obs=2023-04-11  pub=2023-04-14  entry_close=2023-04-19  (gap from obs: 8.0d)

Every entry used is strictly after the data existed publicly. Section 7.4 quantifies the one place where this guard is imperfect.

2.2 The null is the unconditional return, not zero

The baseline for each (field, horizon) cell is the mean forward log-return over every week in which a z-score was computable at all — 1,578 to 1,581 observations per horizon, pooled across all 19 tickers — regardless of whether |z| crossed any threshold.

This matters because 2023-2026 was a positive-drift period for this basket (gold, silver and copper all trended up). Against a zero null, the drift alone would print an “edge” of about +0.41% at 20 days. Our reported edge is excess over the drift.

2.3 Benjamini-Hochberg across all 51 hypotheses

We tested 9 fields x 3 horizons x 2 thresholds = 54 cells, of which 51 have n_events >= 3 and are testable. All 51 p-values go into one BH family. We do not report a “best” cell as if it were the only test we ran, because it was not. See Harvey, Liu and Zhu (2016) for why this is not optional in this field.

The correction is arithmetic anyone can check:

smallest p = 0.002, rank 1 of m = 51  ->  0.002 x 51 / 1 = 0.102

2.4 Pseudo-replication is measured, not waved at

n_events is not n_independent_observations. When five metals contracts all print |z| >= 2 on the same Tuesday because of one macro or margin event, that is closer to one observation than to five.

For the best cell (mm_net_pct_oi, |z| >= 2, 20 days), the 123 events fall on only 67 distinct as-of weeks — mean 1.84 contracts per firing week, with four weeks at the maximum of 5. Section 6 re-runs every cell resampling weeks instead of events, which is the correct unit.

3. Data

3.1 Sources — free, keyless, public domain

No exchange redistribution licence is involved. No credential of any kind is required to obtain the positioning data, and none appears anywhere in the code.

3.2 Panel

19 contract-to-ticker links, all of which pass the join test (>= 20 CFTC weekly points, >= 40 instrument daily closes, overlapping date ranges). All 19 are used unconditionally — none was selected or dropped on the basis of how it performed.

Coverage: CFTC as-of dates for the linked contracts run 2022-01-04 to 2026-08-18 (160 distinct weeks). Daily closes run 2023-04-18 to 2026-06-09.

3.3 Fields tested

Nine fields, derived from the standard disaggregated columns so that anyone can rebuild them from the raw CFTC file:

3.4 Attrition — one in three eligible events cannot be graded

Of the 21,294 (contract, field, week) combinations where the CFTC series has enough history to compute a causal z-score, 7,029 (33.0%) are dropped because no instrument close exists within 7 days of the publication timestamp.

Two causes, both pure data coverage, neither dependent on outcomes:

1. Head gap. CFTC positioning history for these contracts starts 2022-01-04; daily closes start 2023-04-18. Roughly 15 months of positioning anomalies have no tradeable price series to score against.
2. Tail gap. Daily closes stop between 2026-04-17 and 2026-06-09 depending on ticker, while COT continues to 2026-08-18 — a further 10 to 18 weeks per contract with no forward label.

A further 36 (1-day), 63 (5-day) and 63 (20-day) events are dropped for lacking a close far enough ahead. State this plainly: one in three otherwise-eligible anomalies in the product’s own history could not be graded at all.

4. Method

1. Deduplicate. Observations are deduplicated on (entity_id, observed_at), keeping the highest rowid. Duplicate ingestion was a live failure mode in this codebase and inflates apparent event counts if left alone.
2. Causal z-score. For each (contract, field), an expanding-window z-score using only prior history: z_t = (x_t - mean(x_{1..t-1})) / std(x_{1..t-1}), requiring at least 20 prior points and a non-degenerate history standard deviation. This is a direct reuse of the production scoring function, not a reimplementation, so the study grades exactly what the product shipped.
3. Entry. pub_ts = observed_at + 3 days; entry at the first close at or after pub_ts, within a 7-day tolerance.
4. Forward return. log(close[i + h] / close[i]) for h in {1, 5, 20} trading days, where i is the entry index.
5. Event set. All entries with |z| >= threshold, threshold in {2.0, 3.0}.
6. Population (null). All entries with a computable z, unconditionally.
7. Test. Percentile bootstrap on the event mean (B = 2,000), two-sided against the population mean: p = 2 x min(P(mean* <= base), P(mean* >= base)). Confidence intervals use a circular block bootstrap (block length T^(1/3)); see Lahiri (1999).
8. Correction. Benjamini-Hochberg (1995) FDR at alpha = 0.05 across all 51 testable cells.

Environment: Python 3.12.14, numpy 2.5.2, scipy 1.18.1, statsmodels 0.14.6. Repository commit 2597e8b; scripts/cftc_event_study.py, agent/quant/scoring.py and agent/tools/cftc.py are unmodified at that commit.

5. Results

All 54 cells, sorted by uncorrected p-value. mean_ev and mean_base are mean forward log-returns in percent; edge is their difference; hit rates are the fraction of positive returns.

0 of 51 tested cells survive BH at alpha = 0.05. Smallest adjusted p = 0.102.

Two things to notice before reading anything into the top rows:

• 21 of 54 cells have n_events < 30. The swap_net |z| >= 3 row with a 6.2% “edge” has n = 9. The pm_net |z| >= 3 rows have n = 2 and are reported only so that the table is complete; they are not testable and the +9.83% and -20.63% figures are single-observation noise.
• The largest apparent edges all sit at the 20-day horizon, which is exactly where calendar clustering across correlated contracts does the most damage to the effective sample size. Section 6 takes that seriously.

6. Robustness: resampling weeks instead of events

The bootstrap in Section 4 resamples events i.i.d., which assumes the 123 events in the best cell are 123 independent draws. They are not: they fall on 67 distinct as-of weeks, and on four of those weeks five contracts fire together.

We therefore re-ran every cell with a cluster bootstrap over as-of weeks (resample the 67 weeks with replacement, take all events in each drawn week, B = 10,000), then re-applied BH across the same 51 cells.

Still 0 of 51 survive. The best uncorrected p moves from 0.002 to 0.010; the best adjusted p moves from 0.102 to 0.294. Clustering costs roughly a factor of five in the headline cell’s p-value, which is the honest measure of how much of its n was real.

Event-count-to-week-count ratios for every |z| >= 2 cell, for anyone who wants to check the clustering claim directly:

7. Limitations

This section is longer than the results section. That is the correct ratio for a null this underpowered.

7.1 The test has about 10% power. This is the dominant limitation.

A t-statistic scales as the square root of sample size. Take a reference weekly positioning effect of t = 2.14 estimated on 1,451 weekly cross-sections — roughly the magnitude of the stronger published weekly-horizon results — and scale it to our sample:

t_expected = 2.14 x sqrt(n / 1451)
  n = 116  ->  t = 0.61
  n = 160  ->  t = 0.71

Against a two-sided alpha = 0.05 critical value of 1.96, that gives power of 9.3% to 11.0%. In other words: if an effect of that size were really there, this study would have failed to detect it roughly nine times out of ten.

Sensitivity across reference effect sizes, because the 2.14 figure is a stipulated benchmark rather than a value we re-estimated ourselves — substitute your own:

Under every plausible assumption we would need on the order of 1,300 to 5,000 weekly observations for a well-powered test. We have 123 events on 67 independent weeks. A null result at this power is close to preordained, and should update your beliefs very little.

7.2 The specification cancels a signed effect by construction

This is the limitation a domain expert will raise first, and they are right to.

Pooling |z| means a week where managed money is extremely long and a week where it is extremely short land in the same event bucket with the same sign of expected return. If the true effect is signed — and the hedging-pressure literature says it is — this specification averages it to approximately zero by construction, regardless of whether the effect exists.

Kang, Rouwenhorst and Tang (2020) make this concrete: they decompose position variation into a short-horizon component driven by non-commercial liquidity provision and a long-horizon component driven by commercial hedging demand, and find these two components have contrasting directional effects on expected returns. A two-sided, level-based, class-pooled specification is close to a worst case for detecting that structure.

We tested what the product shipped. We did not test what the literature endorses.

7.3 Sample and coverage

• 19 contracts, all commodity futures. No financials, no currencies, no equity index. Nothing here generalises beyond commodities.
• Approximately three years of overlapping positioning and price coverage (2023-04 to 2026-06) — one macro regime, with a strong metals uptrend inside it.
• 33.0% of computable events dropped for want of a price series. The dropped window is systematic (2022-01 to 2023-04, plus a 2026 tail), not random, so this is a coverage restriction rather than a random-missingness assumption.
• Front-month price series are used without an explicit roll-return adjustment. At a 20-day horizon this is a real source of noise in the return measurement.

7.4 Two residual leakage exposures we could not fully eliminate

Named because a reviewer would find them.

1. Non-Tuesday as-of dates. 3 of the 334 distinct as-of dates in the raw table are Mondays (2020-12-21, 2023-07-03, 2025-11-10) rather than Tuesdays. For those, observed_at + 3 days lands on a Thursday, one day before a normal Friday release, so up to one day of lookahead is possible. This touches 10 event-observations across the whole study and 0 in the headline cell — but note that our internal note previously asserted every as-of date was a Tuesday, and that was wrong.
2. The three-day lag is a constant, not a per-report release timestamp. We do not consume the CFTC’s actual release-time metadata. Where a release was delayed past Friday (US federal holidays, the 2018-2019 shutdown backlog), a constant lag would understate the true delay. Both exposures push toward false positives, and we found none — so they cannot explain the null. They would matter to anyone reusing this harness to chase a positive.

7.5 Statistical fine print

• The p-value is a percentile bootstrap of the event mean against the population mean treated as known without error. With n_pop ~ 1,578 the baseline’s own sampling error is small relative to the event mean’s, but it is not zero, so the reported p-values are mildly anti-conservative.
• The event set is a subset of the population. In the best cell, events are 7.79% of the population, so the baseline is contaminated with the very observations being tested. Correcting for this would raise the measured edge from 2.178% to 2.362% — the contamination is conservative, but it is there.
• The confidence intervals use a circular block bootstrap; the p-values do not. The p-value resamples i.i.d. That mismatch is precisely why Section 6 exists, and Section 6 is the number we would defend.
• B = 2,000 bootstrap draws makes p-values granular near zero. Re-running the best cell at B = 20,000 gives p = 0.0031 rather than 0.002; the BH conclusion is unchanged.
• The z-score uses an expanding window with a 20-point minimum. Early-sample z-scores are estimated on as few as 20 prior observations and are noisy.

8. Relation to the literature

8.1 Our null replicates the published null

Sanders, Irwin and Merrin (2009) run bivariate Granger-causality tests on COT trader positions across ten agricultural futures markets and conclude there is “very little evidence that traders’ positions are useful in forecasting (leading) returns,” while finding substantial evidence that traders respond to prices — non-commercials in particular behaving as trend followers.

Their paper reports individual test p-values without any multiple-testing correction. Applying Benjamini-Hochberg to the 30 published p-values from their positions-to-returns direction (10 markets x 3 trader categories) leaves 0 of 30 significant at alpha = 0.05, with a minimum adjusted p of 0.105.

Set that beside our own minimum adjusted p of 0.102 and the picture is consistent: on this class of specification, at this kind of sample size, neither their data nor ours produces a finding that survives correction. Our contribution is not overturning a positive result — it is that the published null holds up under a stricter procedure, on different markets, in a different decade, with publication lag explicitly handled.

Note for replicators: the paper is open access (DOI 10.22004/ag.econ.54547, full text at ageconsearch.umn.edu/record/54547). The BH re-run is a one-liner — multipletests(published_p_values, alpha=0.05, method="fdr_bh") — so the 0-of-30 figure takes about a minute to verify against their tables. We reproduce their p-values here by reference rather than by transcription; check them yourself before citing this claim.

8.2 Where the effect actually lives, if it lives anywhere

Kang, Rouwenhorst and Tang (2020, Journal of Finance) is the strongest modern evidence that positioning carries information. Their finding is structural: short-horizon position variation reflects non-commercial liquidity provision, long-horizon variation reflects commercial hedging demand, the two have opposite directional effects on expected returns, and the liquidity-provision income largely offsets the insurance premium commercials pay.

Nothing in our study contradicts that, because our specification cannot see it. A study designed to test it would use signed changes in commercial net positions at a one-week horizon, decomposed into short- and long-horizon components. That is a different paper, and one worth running.

8.3 Multiple testing in this field is not optional

Harvey, Liu and Zhu (2016) argue that most claimed cross-sectional return findings are likely false because of the sheer number of specifications tested, and propose a t-statistic threshold above 3.0 for a new factor. Our best uncorrected t-equivalent, under week clustering, corresponds to p = 0.010 — well short of that bar, before any correction. Any COT study that reports a single “best” specification without disclosing the search that produced it should be read with this in mind. We tested 54 cells and are telling you all 54.

9. Replication

9.1 Reproducing our exact numbers

The study is one read-only script. It queries a local SQLite pipeline database built by this repository’s collectors; the database is not distributed (it is 150 MB and contains 34 unrelated sources), but every input to it is free and keyless.

git clone <this repository>
cd tirramind
git checkout 2597e8b
.venv/bin/python scripts/cftc_event_study.py

The script prints, in order: the 19-pair join table with per-pair point counts, the attrition counts, the leakage audit with sample entry timestamps, and the full 54-row results table with BH-adjusted p-values. Every number in Sections 3 and 5 of this document appears verbatim in that output.

9.2 Reproducing the study from scratch, without our database

You do not need our infrastructure. You need:

1. Positioning. Download fut_disagg_txt_{year}.zip from https://www.cftc.gov/files/dea/history/ for 2022-2026 (public domain, no key). Filter to the 19 CFTC_Contract_Market_Code values in Section 3.2, and derive the nine fields with the formulas in Section 3.3.
2. Prices. Daily closes for the 19 tickers in Section 3.2 over the same window, from any free source.
3. Method. Implement Section 4 exactly: expanding-window z with a 20-point floor and x[:-1] as history; entry at the first close at or after as-of + 3 days with a 7-day tolerance; forward log returns at 1, 5, 20 trading days; unconditional-return baseline; bootstrap p; BH across all testable cells.

Your event counts will differ from ours to the extent your price coverage differs — recall that a third of our events were dropped for price-coverage reasons, and that attrition is entirely a property of the price series you pick. If you have price history back to 2022 or earlier, you will have a larger and better-powered sample than we did, and we would genuinely like to know what you find.

9.3 The specification we would run next

If you want to spend the effort well, do not re-run ours. Run the one the literature endorses and we did not test:

• signed (not absolute) z on changes (not levels)
• commercial / producer-merchant net positions
• one-week forward horizon
• across as many contracts and years as you can join
• with publication lag handled as in Section 2.1, BH across every cell you test, and week-clustered resampling as in Section 6

That test would be informative whichever way it came out. Ours was not.

10. Conclusion

Across 19 commodity futures, 9 CFTC positioning fields, 3 horizons and 2 z-score thresholds — 51 testable hypotheses in one Benjamini-Hochberg family — no specification survives correction, with or without clustering for the fact that correlated contracts fire in the same week. The smallest adjusted p-value is 0.102 under i.i.d. resampling and 0.294 under week clustering.

We used this to stop selling a product whose central claim was that these anomalies precede price moves. For that decision the result was sufficient: an underpowered null is not a licence to make a positive claim, and we could not find the effect we were charging for.

For the broader question, the result is close to uninformative, and we will not pretend otherwise. The test had roughly 10% power and used a two-sided, level-based specification that cancels the signed, change-based effect the literature actually reports. The correct reading of this paper is “we could not detect it, and we would not have expected to” — not “it is not there.”

The finding we will defend is narrower and more useful: the publication-lag handling, the unconditional-return null, the family-wide correction and the week-clustered resampling are each individually capable of turning an apparent COT edge into nothing. If a COT study reports an edge and does not describe all four, that is where to look first.

Reproduce with

.venv/bin/python scripts/cftc_event_study.py

References

• Benjamini, Y., and Y. Hochberg (1995). “Controlling the False Discovery Rate: A Practical and Powerful Approach to Multiple Testing.” Journal of the Royal Statistical Society, Series B 57(1): 289-300. DOI 10.1111/j.2517-6161.1995.tb02031.x
• Harvey, C. R., Y. Liu, and C. Zhu (2016). “… and the Cross-Section of Expected Returns.” Review of Financial Studies 29(1): 5-68. DOI 10.1093/rfs/hhv059
• Kang, W., K. G. Rouwenhorst, and K. Tang (2020). “A Tale of Two Premiums: The Role of Hedgers and Speculators in Commodity Futures Markets.” Journal of Finance 75(1): 377-417. DOI 10.1111/jofi.12845
• Lahiri, S. N. (1999). “Theoretical Comparisons of Block Bootstrap Methods.” Annals of Statistics 27(1). DOI 10.1214/aos/1018031117
• Sanders, D. R., S. H. Irwin, and R. P. Merrin (2009). “Smart Money: The Forecasting Ability of CFTC Large Traders in Agricultural Futures Markets.” Journal of Agricultural and Resource Economics 34(2): 276-296. DOI 10.22004/ag.econ.54547 (open access)

Appendix A — Corrections to the internal research note

docs/research/cftc_forward_return_event_study.md is the working note this paper is built from. Two of its statements did not survive re-verification and are corrected here. Both corrections make the result weaker, not stronger.

1. Distinct-week count for the best cell. The note states that the 123 events in the mm_net_pct_oi, |z| >= 2, 20-day cell “fall on only 116 distinct weeks.” That figure belongs to the 251 pre-filter events. The 123 labelled events fall on 67 distinct as-of weeks. Clustering is therefore materially worse than the note claimed, which is why Section 6 re-runs every cell at the week level.
2. “Every as-of date is a Tuesday.” 331 of 334 distinct as-of dates are Tuesdays; 3 are Mondays. See Section 7.4 for the bounded exposure this creates.

Both were found by re-deriving the note’s numbers from the database rather than copying them forward. Everything else in the note reproduced exactly against commit 2597e8b.

I am shutting the product down. Four independent measurements say the edge isn’t there, the data isn’t proprietary, the price point is occupied, and the cross-source graph I built the whole thesis on has three genuinely cross-domain joinable source pairs out of 561, covering four entities. That’s a separate write-up.

This one is about something more portable, and in my opinion more useful: the codebase evolved from “healthy-looking and empty” to “healthy-looking and full of constants.”

The first state is easy to catch. Every monitoring check you’d naturally write — did the job run, did it exit zero, did it write rows — catches an empty table. The second state passes all of those checks. The job runs, exits zero, writes rows, and the rows are garbage that is shaped like a result. Sorting by a column that’s secretly a Unix timestamp looks exactly like sorting by anomaly score, right up until you print the column.

Here are thirteen concrete ways this system lied to me. All of them shipped. None of them shipped to a paying customer, because there never was one — which is its own finding, and the reason I could afford to be this candid. For each I’ve given the symptom, how long it hid, the root cause, the fix, and the general rule — the rule is the part worth your time, the war story is just evidence that I paid for it.

The title is a nod to the repo’s own failure log, which runs F-01 through F-13. The numbering below is its own; some of these are in that log, some were found while writing this.

Two snapshots frame everything that follows. This is the same database three days apart:

On the 26th I could see the problem from across the room. On the 29th it looked fixed. Ways 7 through 10 are what those 4,852 rows and 42 rows actually contain.

1. The DAG reported “completed” when the write threw

Symptom. daily_collection surfaced as healthy. Its run history, once I looked directly at it, was: one successful run ever, on 2026-04-19, and pipeline_data holding 194 rows total for the life of the project.

How long it hid. From April to late August — about four months. Two runs were additionally stuck in status running forever, which is the one signal that did look wrong, and which I misdiagnosed (see Way 3).

Root cause. In the executor’s _execute_layer, the storage-exception path emitted a log.warning and left the node result’s status as "completed". A run whose writes all threw was byte-for-byte indistinguishable, in dag_runs, from a run that succeeded.

Fix (b7a126b). Storage exceptions set status to failed. A stale-run reaper plus a heartbeat so a process that dies mid-flight stops claiming to be running. And a distinct classification for missing-credential skips, so an unconfigured source degrades loudly instead of returning an empty list that looks like “nothing happened today.”

Here is the run history now. Note that it got worse-looking, which is the point:

2026-04-19  completed
2026-04-19  failed
2026-04-21  failed
2026-04-21  failed
2026-08-25  failed  x4
2026-08-27  failed  x4

One success, eleven failures. Before the fix, those eleven were reporting as completed, except two that were stuck in running and had been since April.

Eight nodes now skip explicitly on missing FRED/EIA/NASA credentials; two fail on real upstream outages (a 403 from the lobbying source, a dead PatentsView endpoint). That matches ground truth instead of hiding it.

There is a second thing visible in that list: between 2026-04-21 and 2026-08-25 there are no runs at all. The cron schedules were declared in the DAG definitions and never actually installed anywhere. So for four months the pipeline was simultaneously not running, and reporting success when it did.

Rule. An exception handler that logs and continues has converted a failure into a silence. Every except: log.warning(...) on a write path is a decision that the caller should not be told. Make that decision deliberately, and write down why, or don’t make it.

2. “Rows written” was a count of nodes, not a count of rows

This one I found while writing this post. It is unfixed as of today, and I’m including it because a list of failures I already solved is a less honest document than one that includes the one I found on the way to publishing.

Symptom. The guard I added in Way 1 to catch silent success — a run that completes, has store-eligible nodes, and writes nothing gets downgraded to failed — does not fire for the scoring DAGs. rl_training has six completed runs and rl_transitions has zero rows. adversarial_scan has six completed runs and adversarial_flags has zero rows.

Root cause. agent/pipeline/executor.py:215:

run.rows_written = sum(1 for nr in run.node_results.values() if nr.stored)

nr.stored is a bool — “this node’s result envelope landed in the store.” So rows_written is a count of nodes, capped at the number of nodes in the DAG. A DAG whose every operator dutifully persisted one summary envelope into the generic pipeline_data table, while writing nothing to the domain table it exists to populate, scores rows_written = 5 and sails through. The guard’s other escape hatch is eligible_store_nodes == 0, and the two DAGs above take one or the other — I have not pinned down which, and it doesn’t much matter, because both are the same mistake: the guard reasons about the executor’s bookkeeping rather than about the database.

I wrote a guard against fake success, named its variable after the thing I wanted to measure, and populated it with a proxy that cannot go to zero in the failure mode I was guarding against.

Fix. Not yet applied. The correct measure is per-destination-table row deltas taken before and after the run, which is what an auditor would ask for and what I should have written the first time.

Rule. When you build a health check, ask what value the field would take during the exact incident you are defending against. If it can’t reach the alerting value during that incident, you have built a check that is guaranteed to pass. And a field name is documentation: rows_written that holds a node count will be trusted by the next reader, who will be me.

3. A timeout that could not stop the work

Symptom. The nightly chain ran all nine DAGs, then refused to exit for 40+ minutes. The box — 1 vCPU, 1.9 GB — reached 20 MB available with swap at 2038 / 2047 MB. Two nodes, train_gnn and generate_features, were both marked failed, and memory kept climbing after they had “finished.”

How long it hid. Months. It produced the same evidence as Way 1 — a run stuck in running with no error recorded — so I fixed the reporting bug, saw the signature persist, and had no second hypothesis.

Root cause. Two faults compounding.

The executor’s own source said the thing out loud: a timeout marks the node failed and moves on, “since the operator keeps running in the background because a thread cannot be forcibly stopped.” future.result(timeout=...) times out the wait, not the work. A timed-out operator is still running and still allocating. The timeout was bookkeeping, not cancellation.

And TIRRA_PIPELINE_WORKERS defaulted to 4, on one core. Four workers on a single core buy no parallelism — only context-switching and roughly 4x peak RSS. Nothing compared that default against nproc.

Net: a 30-minute train_gnn timeout leaves an orphaned thread eating RAM for as long as the parent lives, and the parent cannot exit while it runs.

Fix (6d656ba, dec920b). Workers set to 1 on single-core hosts, with the measurement recorded next to the setting. The executor now logs an explicit ORPHAN THREAD line naming the node and how far past its timeout it still is, and refuses to start new work in a pool a timeout has already degraded. The nightly chain was narrowed to the light DAGs that actually feed the product. And MemoryMax=1200M / MemorySwapMax=1500M on the systemd unit, so the OOM killer takes the batch job rather than choosing the customer-facing API service for me.

Rule. A timeout that cannot cancel its work is not a timeout, it is a leak with a stopwatch attached. In any runtime without forcible thread cancellation — Python very much included — either give the operator a cooperative cancellation token it checks, or run it in a process you can actually kill. And never let a background job share an OOM domain with something a customer is waiting on.

Corollary, which cost me more than the bug did: when a fix doesn’t clear the symptom, the fix was not wrong — you had two causes. I had assumed one.

4. Duplicate ingestion manufactured “structural break this week”

Symptom. The weekly digest told paying subscribers there was a structural break this week in five separate CFTC futures contracts. There were no structural breaks. There was one collector running eight times in a day.

Root cause. Collection re-ingests the same CFTC report on every run with no upsert. So entity_observations accumulates duplicate rows. Measured on the live database, and you can re-run this yourself:

SELECT COUNT(*)                                        -- 5488
FROM entity_observations WHERE observation_type='futures_positioning';

SELECT COUNT(*) FROM (SELECT DISTINCT entity_id, observed_at
FROM entity_observations WHERE observation_type='futures_positioning');
                                                       -- 5046

442 duplicates of 5,488 rows, 8.1%. And they are not spread evenly — a single day, 2026-08-18, carries 272 rows against a typical day’s ~35, because collection ran about eight times that day.

The consequence is the interesting part. The last ten rows for every high-duplication contract, by insertion order:

2026-08-18, 2026-08-18, 2026-08-18, 2026-08-18, 2026-08-18,
2026-08-18, 2026-08-18, 2026-08-18, 2026-06-02, 2026-06-02

Eight consecutive “weekly” observations carrying the same timestamp. The changepoint detector (BOCPD) sees a discontinuity where that block begins — which is real, the series genuinely changes shape there — and the elapsed time from that index to now, measured in the series’ own index space, is 0.0 weeks. “A structural break, zero weeks ago.” Rendered to the customer as this week.

Fix (1e86f0f). The read path now dedupes on (entity_id, observed_at), keeping max(rowid) so the most recently written value for a timestamp wins. Cotton went from 169 points to 159 and its “this week” changepoint became None. Ondo TVL went from 3,570 points to 1,190 — roughly 3x duplication in DeFi too. Every surviving changepoint is now 15 to 186 weeks old, which is the honest answer: there are no fresh structural breaks in this data right now.

That is a read-side guard, and I’m labelling it as one. The ingestion path still needs the upsert; without it the duplicate rows keep accumulating and every consumer other than the digest still sees them.

Rule. Idempotency is not a nice-to-have on an ingestion path, it is the difference between a time series and a log of your own cron schedule. And notice the shape of the failure: the data was fine, the analysis was fine, and the artefact of re-running the collector was laundered through a correct changepoint detector into a confident false claim. Anywhere you measure elapsed time by counting array positions, you have assumed the array’s spacing is real.

5. A 50-point baseline where 1,096 points existed

This is the most expensive one, and it is the one I’d want a stranger to take away.

Symptom. The digest was shipping 29 volatility “anomalies” per run. After the fix it shipped 4. Twenty-five of twenty-nine were false positives, going out to subscribers, every week.

Root cause. The scorer computed z-scores against instrument_volatility, which holds roughly 4 months of history. The exact same 89 instruments also had instrument_daily going back to April 2023, under a superseded observation-type name that nobody had reconciled. Live numbers:

instrument_volatility  89 entities   46-61 points    2026-04-18 .. 2026-08-26
instrument_daily       89 entities  752-1096 points  2023-04-18 .. 2026-06-09

A ~50-point baseline makes ordinary seasonal variance look extreme. Nothing was broken. No exception was raised, no row was missing, no test failed. The statistics were computed correctly on the wrong denominator, and 86% of the output was noise.

Fix (2782d21). The scorable-source table now maps a tuple of (observation_type, fields) pairs per source and unions across type names, with the later entry winning on the overlap period. realized_vol_20d findings went 29 → 4.

The same commit killed three other things for related reasons, which I mention because “make the product smaller” was the correct move each time: polymarket was removed (its per-entity series tops out at 15 points, a hard structural ceiling below the 20-point minimum — it was reported as a working source every single run, forever); instrument_return.close was cut because 28 of its 33 findings had a z-sign matching the trailing return direction, i.e. “this asset has been trending,” sold as anomaly detection. sovereign_debt was kept despite contributing almost nothing, because 5 of 13 entities clear the floor with real variance — it is genuinely quiet rather than structurally incapable, and that distinction is the whole job.

Rule. A statistical result has three inputs — the method, the data, and the amount of data — and only the first two get reviewed. Print n alongside every derived statistic, in the output the customer sees, and set a floor below which you emit nothing rather than something. Then go looking for whether a longer version of your series already exists in your own store under a different name. Mine did: three years of it, for the identical 89 instruments, in the identical table, under a superseded type name.

6. Field names that did not exist, discarding 175,275 observations per run

Symptom. Every edition of the digest was 100% CFTC content, despite five sources being configured. I had assumed the other four were quiet.

Root cause. The scorable-source config declared field names that do not appear in the stored JSON payloads, and the series builder silently skips any field it cannot find. Three of five sources therefore built zero series, with no error logged anywhere:

175,275 observations per run, read from disk, matched against nothing, discarded in silence.

Fix (2957703). Corrected field names. Findings went from 8 (100% CFTC) to 26 across three sources. The DeFi TVL series turned out to carry 3,123- and 3,570-point baselines with genuine changepoints — far deeper history than CFTC’s 169 weekly points. The best data in the system had been invisible because of a string literal.

Rule. A lookup that misses is an event, not a None. Any config that names a field in a schemaless payload must be validated against a live sample at startup and fail loudly on a miss — because “this source is quiet” and “this source is misconfigured” produce identical output, and only one of them is something you’d act on.

7. The anomaly ranking sorted by Unix timestamp

Now we are in “healthy-looking and full of constants” territory. The entity_alerts table has 4,852 rows. It is the input to the ranking that decides which entities are interesting. It is fully populated, non-null, plausibly distributed, and completely determined by the clock.

Symptom. Nothing. That’s the point. The table looks fine.

Root cause. agent/fusion/surprise.py, the temporal component:

dt_actual = obs.get("observed_at", 0.0)
# We use the raw absolute error, z-scored later
temporal_s = abs(dt_pred_val - dt_actual)

observed_at is a raw Unix epoch, around 1.787e9. The model’s predicted delta is a small number near zero. So temporal_surprise is, to four significant figures, the observation’s timestamp. The comment promises z-scoring “later.” Later never arrives.

The composite is a fixed weighted sum of five components. Decomposing it over all 4,852 rows against the code’s own normalised weights:

The reconstruction matches the stored composite_surprise mean to fifteen significant figures, so this is arithmetic, not inference. And neighborhood_surprise is itself a weighted mean of neighbours’ composites — so it, too, is 90% timestamp. The Spearman correlation between composite_surprise and the two epoch-derived terms alone is 0.99993.

Everything the score was supposed to measure — how surprising the observation’s type was, how far its value deviated, how much the entity’s memory drifted — together accounts for 0.002% of the ranking.

Three further constants in the same table, all across all 4,852 rows:

• memory_drift, cusum_statistic, event_study_score are exactly 0.0 in every row. Three of nine columns are dead and nothing said so.
• obs_type_surprise sits at 23.025850929940457 in 4,808 of 4,852 rows — that is -log(1e-10), the clamp floor. 99.1% of the time the model assigns the observed type a probability below the clamp, and the “surprise” is the clamp constant, not a measurement.
• alert_time has exactly one distinct value across all 4,852 rows.

Fix. None applied — this table is now excluded from everything customer- facing, and the tier that would have consumed it is being retired.

Rule. Any quantity that enters a weighted sum must be dimensionless before it gets there, and the place to enforce that is a unit test that asserts each component’s magnitude range, not a comment saying it’ll be normalised later. Then: for every composite score you ship, log the per-component contribution share on a real sample. If one term is 90% of the score, either that’s intended and you should say so, or you have just found a bug that no amount of staring at the formula would have shown you — this formula is correct. It’s the inputs that aren’t.

8. Cosine similarity that could not return anything but 1.0

Symptom. convergence_clusters — the table representing “these entities are surprising together, which is the actual product thesis” — has 42 rows. Their correlated_surprise_score values run from 0.9976510 to 0.9998583. All 42. The score’s own docstring says “1.0 for perfectly aligned surprise patterns, 0.0 for orthogonal.”

Root cause. The score is the mean pairwise cosine similarity of the five-element surprise vectors from Way 7 — used raw, unnormalised. Every such vector has the shape [~23, ~1.7e9, ~small, ~1.4e8, 0.0]. Its largest component is between one and eight orders of magnitude bigger than every other component, and it is a timestamp, so it agrees closely between any two rows collected in the same era. Two such vectors point in almost the same direction no matter which entities they describe.

Measured on 300 real alert vectors, 44,850 pairs:

pairwise cosine     min 0.7306   p1 0.9093   median 0.9890   max 1.0000
                    49.1% of all pairs exceed 0.99

Averaging over a cluster’s pairs pulls it tighter still, which is how 42 out of 42 land in a band 0.0022 wide. The number is not measuring convergence. It is measuring that these are all timestamps.

Fix. None; the table is excluded from the product and the tier is retiring.

Rule. Cosine similarity on unnormalised, mixed-unit vectors is not a similarity measure, it is a measure of which component has the biggest units. Standardise per dimension before any geometric operation. And a diagnostic for free: if a score that is defined on [0, 1] never leaves a 0.002-wide band across your entire dataset, it is not a score. Assert the spread, not just the range — 0 <= x <= 1 passes on a constant.

9. Placeholders that outlived the promise to fill them

Same 42 rows, different mechanism, so it gets its own entry.

Symptom. contributing_tools_json is [] in all 42 rows. temporal_span_hours is 0.0 in all 42 rows. Both are columns a reader would use to sanity-check a cluster: which data sources produced it, and over what window.

Root cause. agent/fusion/convergence.py:

tools: tuple[str, ...] = ()          # filled by scorer
...
temporal_span_hours=0.0,             # same moment

The scorer does not fill it. grep -n contributing_tools agent/fusion/entity_scorer.py returns nothing. The DAG faithfully persists the empty tuple.

temporal_span_hours is worse, because the class docstring twelve lines above defines it: “Hours between earliest and latest member alert_time.” That is a subtraction over data the function already has in hand. It is a hardcoded literal instead, with a comment asserting a fact — “same moment” — that nothing measured.

Both survived from the initial implementation to production because a column that is always empty raises no error and fails no test, and because writing the comment felt like writing the ticket.

Fix. None; excluded with the rest of the table.

Rule. # filled in later is an unassigned ticket living in a place with no ticket queue. If a field cannot be computed yet, make it nullable and leave it NULL — a NULL is a question, an empty list is an answer. Then add the cheap test that catches this entire family: for every column you persist, assert it takes more than one distinct value across a real run. That single test would have caught Ways 7, 8, and 9 on the day each landed.

10. A threshold that filtered nothing

Symptom. The convergence detector’s first step is “filter to entities with composite_surprise above a threshold.” The default threshold is 2.0.

The minimum composite_surprise in the table is 64,241,535.

Root cause. The threshold was chosen when the composite was a z-score-like quantity in single digits, which is what the design intended. Way 7’s units bug moved the composite eight orders of magnitude, and the threshold — a plain default argument in a signature, correct at the time it was written — was never revisited, because nothing connects the two. Every entity passes. The “elevated entities” set is the set of all entities.

So the clustering that produced Way 8’s numbers ran over the entire graph, not over an anomalous subset. There is no selection step. There never was.

Fix. None; excluded.

Rule. A constant threshold on a derived quantity is a silent coupling to that quantity’s scale, and scale changes don’t produce merge conflicts. Log the pass rate of every filter, every run — filtered 4852 -> 4852 (100.0%) is a line that would have ended this in one night. A filter that never rejects and a filter that always rejects are both bugs, and neither one throws.

11. Six tests that passed by never running the code

Root cause. tests/test_entity_linking.py has a fixture builder for GDELT events, defaulting to goldstein: float = 1.0. In May, a Goldstein tension gate was added to GDELTTool._persist_entities:

if gs is None or gs >= _GOLDSTEIN_TENSION_THRESHOLD:   # -5.0
    continue

Goldstein scores run roughly −10 (conflict) to +10 (cooperation), and the gate drops anything at or above −5.0 as routine diplomatic noise — by the source file’s own measured distribution, that is 89.9% of GDELT. Every fixture in the class defaulted to 1.0, squarely inside the discarded 89.9%. So every event was dropped before reaching the link-creation code the class exists to test.

TestGDELTEventInvolves has ten tests, and the gate split them cleanly:

• Four assert a link is created (test_normal_link_created, test_link_metadata_contains_event_info, test_dedup_same_country_pair, test_multiple_country_pairs). These went red the day the gate landed.
• Six assert that no link is created — same country, missing country, whitespace-only country, no store attached. These stayed green, and were worthless. assert len(links) == 0 is trivially true when nothing runs. Not one of them could have failed for any reason, and all six were serving as evidence that the gate logic and the country-comparison logic worked.

How long it hid. The gate landed 2026-05-12; this was found 2026-08-27. 107 days.

And here is the part I’m least proud of, because it’s the part that made the other 107 days possible: the four red tests were red the whole time. The suite had 19 failures when I finally sat down with it. A suite with a standing failure count is a suite nobody reads, and inside that unread failure count were six tests quietly reporting success for a reason that had nothing to do with the code under test.

Fix (74017b1). Fixture default changed to −7.0 so the code actually runs; the class went green and meaningful. Suite went 19 failures to 10, and the remaining 10 were real bugs fixed separately. The fix carries a seven-line comment naming the commit that introduced the gate, because the next person to see goldstein=-7.0 will wonder why it’s not zero.

Rule. A test that cannot fail is worse than no test, because it consumes the attention a missing test would have attracted. Two cheap defences: mutation-test your critical paths (break the code on purpose, confirm the suite goes red), and treat coverage on guard clauses as a first-class metric — the gate that skipped everything was covered; the branch behind it was not. Whenever you add an early-continue to production code, grep the fixtures that feed it.

And the meta-rule, which is the expensive one: a non-zero standing failure count destroys the signal value of the entire suite. Not just for the broken tests — for every test in it, including the six that were lying.

12. A test that asserted the bug

Symptom. Three DAGs failed or silently produced nothing. entity_scoring crashed with index 69 is out of bounds for dimension 1 with size 69. The other two threw mat1 and mat2 shapes cannot be multiplied (93x49 and 23x64). Six downstream tables held zero rows against 365,000 healthy observations.

Root cause. Three registries had drifted apart with nothing comparing them:

Three failures fell out of that. ENRICHMENT_DIM was hardcoded to 55 — correct only while the observation-type list had 46 entries; the writer indexes offset + 9 + ot_idx over the live list, so at 48 types the block overflowed. For one node type that overflow hit the end of the tensor and crashed. For instrument nodes it ran into the price-feature block that follows: silent corruption instead of a crash, depending only on node type.

Separately, maritime_area existed in the database but not in the code’s ENTITY_TYPES, so the feature builder fell back to type_idx = 0 and one-hot encoded every maritime area as a CFTC contract. A log.warning fired and the run continued. It trained and scored as the wrong kind of entity for months.

And the part that belongs in this list: a test asserted the buggy behaviour. assert features[0, 0] == 1.0 — which is precisely what the wrong fallback produces. The suite was green over the corruption, and had been used as evidence that the corruption wasn’t there.

Fix (7c79c02). ENRICHMENT_DIM is now derived (9 + len(OBSERVATION_TYPES)), never written as a literal. An unknown entity type gets an all-zero one-hot — claiming no identity rather than the wrong one. A new validate_schema_against_store() raises before anything trains or scores, listing every live type the code cannot encode. And checkpoint loading now names the drift explicitly — instrument: trained_weights=23 expected_by_model=49 — instead of logging “skipped N keys,” which is how a randomly-initialised layer had been passing for a harmless omission.

Rule. Three rules, and I’d take all three anywhere:

1. Any dimension derived from a registry must be computed, never hardcoded. A literal that equals len(SOME_LIST) today is a time bomb with one registry edit left on its timer.
2. Never degrade an unknown categorical to index 0. Claiming no identity is honest; claiming the wrong identity is corruption that trains cleanly and evaluates cleanly and is invisible.
3. If a test asserts a fallback or default, verify the fallback is correct before treating the green suite as evidence of anything. A test freezes behaviour; it does not bless it.

13. Fifty-one hypotheses, zero survivors, and one that looked significant

The last lie is the one I’d been most at risk of telling other people.

Symptom. A forward-return event study on the product’s own CFTC anomalies produced a headline result: mm_net_pct_oi at |z| >= 2, 20 trading days, n=123 events, +2.18% edge over baseline, 65.0% hit rate against a 50.4% base rate, p = 0.002.

That is a marketing page. It is also noise.

Root cause. The study tested 9 fields x 3 horizons x 2 z-thresholds = 54 hypotheses, 51 of them with enough events to be testable. Under the null, the expected number of uncorrected p < 0.05 results from 51 independent tests is about 2.6. I got 7. Applying Benjamini-Hochberg at alpha = 0.05: 0 of 51 survive. The best adjusted p-value is 0.102.

A single p = 0.002 pulled from a 51-cell grid is not evidence. It is the grid.

Two further problems, both of which cut against the positive:

Pseudo-replication. Even the best cell is not 123 independent observations. Those 123 events fall on only 116 distinct weeks, and the busiest weeks have 5 of 5 eligible contracts firing simultaneously — the same macro positioning shift hitting correlated commodities in the same week. The effective n is meaningfully below the counted n, so even the uncorrected 0.002 overstates confidence before BH is applied at all.

Attrition. Of 21,294 (contract, field, week) combinations where a causal z-score was computable, 7,029 (33.0%) had to be dropped because no instrument price existed within 7 days of the CFTC report’s publication date. One in three anomalies in the product’s own history cannot be graded at all.

I built the method to be hostile to a positive result, which is the only way to believe one:

• The null is the unconditional forward return over the same horizon pooled across all 19 tickers (n ≈ 1,578 per horizon), not zero. The 2023–2026 commodity tape drifted up; measuring against zero would have handed me an “edge” that was just the drift.
• Publication lag is honoured. CFTC observed_at is the Tuesday as-of date (verified: every distinct timestamp in the DB falls on a Tuesday); the report goes public the following Friday. The study adds a 3-day lag and enters at the first close on or after publication. The naive version would have traded three days before the data existed. That constraint is where most of the 33% attrition comes from.
• The z-score function is a direct copy of the production one — expanding window, hist = x[:-1] — so the test scores what the product shipped, not an idealised version of it.
• No contract was selected or dropped on performance. All 19 price-linked contracts are used unconditionally.

And the null replicates the literature. Re-running BH on the published p-values in Sanders, Irwin & Merrin (2009) — a positive result in the COT literature — gives 0 of 30 survivors, minimum adjusted p = 0.105, against my 0.102. My negative result agrees with the published data. It disagrees with the published conclusion, and the difference is entirely that the published work does not correct for multiple testing. Neither does most of the rest of this literature.

The caveat I have to state, because a hostile reader would find it. My specification pools two-sided |z| and appends the forward return unsigned:

if abs(z) >= zt:
    events[(field, zt, h)].append(log_ret)

A genuinely signed effect — high positioning predicts down, low predicts up — cancels itself in that bucket by construction. Combined with roughly 10% power at these sample sizes, the honest claim is “no detectable effect at low power on this specification,” not “no effect exists.” I am not going to overstate my negative result to make the shutdown look more decisive than it is.

Rule. Decide your hypothesis count before you look, and correct for it. If you tested a grid, report the grid — a result presented without its denominator is not a result. Then check whether your n is really n: events clustered in calendar time or across correlated instruments are not independent trials, and nothing in the p-value knows that. And when a negative result would be convenient for you, audit it as hard as you’d audit a positive one; mine has a specification flaw that I would very much have preferred not to find.

What I’d actually keep

Reading these back, they sort into four families, and only one of them is about being careless.

It lied about whether it ran (1, 2, 3). Failures converted to silence by exception handlers; a health check measuring a proxy that cannot reach its own alerting value during the incident it guards; a timeout that stops the waiting rather than the working. Ordinary, and every system has them. These are the cheap ones to find, because someone eventually notices the table is empty.

It lied about what it had (4, 5, 6, 12). Duplicates laundered through a correct changepoint detector into a false market claim; a correct statistic on a 22x-too-short baseline; 175,275 rows a run discarded on a string literal; a registry drift that crashed for one node type and silently corrupted for another. Shared signature: nothing threw. The data was wrong in a way that has no exception type, so there was nothing to catch and nowhere to catch it.

It lied about what it meant (7, 8, 9, 10, 13). A ranking sorted by wall-clock; a similarity that could not return anything but 1.0; placeholders that outlived the comment promising to fill them; a filter with a 100% pass rate; a p-value detached from its denominator. Every one of these tables is populated, non-null, correctly typed, plausibly distributed, and would pass any schema check you or I would write. The formulas are right. The units aren’t.

And the tests agreed with all of it (11, 12, and Way 1’s guard). Six tests green because a gate skipped the code they existed to test. One test asserting the corrupt fallback value, so the suite was green over the corruption and was being cited as evidence against it. And a silent-success guard whose own measure was a node count. Every one of those is a defence I built, that reported the all-clear, on the specific failure it existed to catch.

The through-line: every check I had asked “is there output?” and none asked “does the output vary?” A constant is the most convincing possible fake — it has a type, a range, a mean, a plausible distribution once you have 4,852 of them, and it is stable across runs, which reads as reliability.

If you take one thing from this, take the cheapest test in the list, from Way 9. For every column you persist, on every real run, assert it takes more than one distinct value, and log its distinct count and its spread. That single check catches Ways 7, 8, 9, and 10 — four of the five worst things in this post — on the day each of them lands, before any of it reaches a customer.

I did not have that check. That is why this is a post-mortem and not a changelog.

Reproducing any of this

Everything above is a query, a file and line, or a commit in the repository. Nothing is anonymised and nothing is rounded in my favour.

The failure log this is drawn from — F-01 through F-13, each with symptom, root cause, fix, and prevention rule — is LESSONS.md in the repository. It was maintained continuously, not reconstructed for this post, which is the only reason I can date how long each of these hid.

Here’s what you can expect:

• Simple - Learning and Building stuff

Thank you for being here, and happy exploring! 🚀