Case study
CostProof
Measures whether cloud cost optimisations actually saved money, by treating each fix as a treatment and estimating its effect against a control group.
- Role
- Solo project: design, statistics, engineering
- Timeline
- 2026 – present
- Stack
- Python, SQL (DuckDB), R, IBM watsonx.ai
- Repository
- SpencerCottrell428/costproof
- 80% → 13%False positives on fixes that did nothing, before/after versus this method
- 23% → 91%How often the 95% confidence interval actually contained the truth
- 117Interventions scored against a known true effect, 30 of them exactly zero
- 1.2e-13Largest disagreement with R’s plm package across all 117 panels
01 / The problem
The savings number is usually wrong
When a team fixes a cloud cost problem and the bill goes down, the standard way to credit the fix is before and after: spend was $10K, now it is $7K, so we saved $3K. That number is usually wrong.
Teams do not fix things at random. They fix whatever just spiked, and spikes fall back on their own. Before/after gives the team credit for a drop that would have happened anyway. Flexera’s 2026 State of the Cloud report estimates that 29% of cloud infrastructure spend is wasted, the first rise in five years, and finds that 63% of organisations now run a FinOps team to reduce it. A lot of budget decisions rest on savings numbers like these.
The real question is a counterfactual one: what would this resource have cost if nobody had touched it? CostProof answers that the way an economist would, by treating each fix as a treatment and comparing it against a control group of similar, untouched resources.
02 / The data
Simulated on purpose
Enterprise billing data is confidential, so CostProof runs on a simulated cloud estate: 216,036 billing rows across 400 resources, about $9.2M a year, conforming to FOCUS 1.2, the billing schema that AWS, Azure and Google Cloud publish. The same code runs on a real export.
Simulation is a deliberate choice here, not a shortcut. On real data the true effect of a fix can never be observed, so an estimator can be run but never scored. With a simulator the true effect of every intervention is known, which is the only way to measure how accurate a method is.
The simulator is built to be unfair in the realistic way. Fixes are triggered when a resource breaches its 120-day rolling baseline, with a 95 to 150 day detection lag, so the bias that fools before/after is present in the data. Thirty of the 117 interventions have a true effect of exactly zero: action taken on a resource that had merely spiked.
The limit of this approach: the simulator encodes my assumptions about how cloud spend behaves, so the validation is optimistic to the extent those are wrong.
03 / The process
From billing feed to governed report
- 01
Billing feed
FOCUS 1.2 rows, validated for conformance before anything else runs.
- 02
Warehouse
Bronze, silver and gold SQL layers. Untagged spend is made explicit, not dropped.
- 03
Panel per fix
Donor resources matched on pre-period growth, not on level.
- 04
Four estimators
Before/after, DiD two ways, and synthetic control, all on the same panel.
- 05
Trust gate
A parallel-trends check decides whether each estimate can be believed.
- 06
Review
Findings, caveats and sources, with large or irreversible actions held for a human.
The difference-in-differences estimator is written out by hand, including the cluster-robust standard errors, because that is where applied work tends to go wrong. It agrees with Python’s statsmodels to about 1e-16 and with R to 1e-13. The project has 54 automated tests.
04 / Results
What the numbers showed
Before/after reported a significant saving on 80% of the interventions that did nothing. The textbook fix, difference-in-differences with cluster-robust standard errors, was no better at 83%. Those standard errors assume many treated groups, and here there is one treated resource per panel, so the intervals come out far too narrow.
Bars two and three have the same point estimate. Only the way uncertainty is measured differs. Building the null distribution by permutation instead took coverage from 23% to 91% and false positives from 83% to 13%, while still detecting 91% of the real effects. Synthetic control never raised a false alarm but caught only 48% of real savings.
The trust gate earns its place. Of 117 estimates, 103 pass the parallel-trends check, and those have 95.1% coverage. The 14 that fail have 64.3%. In dollars, against $2,020,441 of true annual savings, before/after misstates the total by $326,700 net. Summing the absolute error per intervention gives $450,968: the total only looks close because errors in opposite directions cancel.
05 / The AI layer
The model never computes a number
A small agent runs the cost review. It calls tools from a registry, and every tool returns its value together with the method used, its caveats and its sources. Findings above $1,000 a year, irreversible actions, spend with no identifiable owner, and any estimate that failed the trust gate are held for human approval. Every run writes an audit record.
IBM watsonx.ai (Granite) only writes the narrative at the end. That is enforced two ways. A deterministic backend can produce the whole report with no model at all, with identical figures. And a grounding guard reads the model’s text back, extracts every figure, and checks it against the tool output the model was shown. A narrative with an unsupported number is rejected and the deterministic version is used instead.
- $970,158Spend with no business-unit tag (72%)
- $212,576Waste in the ten resources the classifier ranked highest (16%)
- $165,406Prepaid capacity never consumed (12%)
The largest finding is not recoverable spend. It is spend nobody owns, which is why no team has a reason to reduce it. The report says so plainly.
06 / What went wrong
Two mistakes worth keeping
The first full validation run gave 7% coverage for every method at once. I assumed the estimator was broken. It was not. My analysis window reached back 84 days while the simulated detection lag was 20 to 75 days, so the “before” period already contained the waste the fix was about to remove. Correcting the specification took the error from 0.60 to 0.20. The lesson: when every method fails together, suspect the data specification, not the methods.
The trust gate originally rejected 98.3% of interventions. It was a statistical test built on the same standard errors that are invalid with one treated unit. Rebuilt on randomisation inference, its pass rate is 88% and the passing set has 95.1% coverage.
Still unsolved: spillovers between resources, teams throttling usage before the recorded action date, and fixes that roll out across many resources over weeks. The waste classifier’s ROC AUC is 0.65. It ranks the top of the queue well and the middle poorly.