Case Study — Personal Project
KWB — Kalshi Weather Trading Bot
An automated trading bot, live in production, that prices a single Kalshi prediction market — what will NYC's high temperature be today? — by blending four weather models into a probability distribution, and bets only when it thinks the market is mispriced.
4models
Independent weather sources blended into one distribution
~570tests
Unit tests, plus mypy --strict and automated lint
19cities
Weather markets the research program has tested
~9/day
Trading cycles, fully serverless on AWS Lambda
What it is
A single-purpose, fully serverless trading bot for one Kalshi market series: the NYC daily-high-temperature buckets, settled off the Central Park station. It blends multiple weather models into a probability distribution, prices every temperature bucket the market offers, and bets only where it sees a real edge. The decision logic is entirely deterministic — there is no LLM in the trading loop. AI was an engineering collaborator for code review and analysis, never a live trader.
How it works
Pull hourly forecasts from four independent weather sources — NWS, ECMWF, GFS, and NOAA HRRR.
Blend them into a single probability distribution over the day's high temperature, widening the uncertainty where the models disagree.
Convert that distribution into a probability for each temperature bucket the market offers.
Compare those probabilities to live market prices and place small, fractional-Kelly-sized bets only where the modeled edge clears a threshold.
Hold to settlement and book realized P&L when the official climate report finalizes the day's high.
What the research found
Realistic fills decide everything
A deep dive mined ten weeks of stored order books for the rules the bot should have traded. Four came back with healthy paper profits — then all four died the moment a resting order only counted as filled when the market actually traded through its price. The most seductive had a 98% paper win rate and was pure adverse selection: your bid gets hit precisely when someone watching the thermometer knows something you don't.
This market prices weather efficiently
The working thesis was that the model knew which way the market was wrong. Tested truly out of sample — on six other cities' markets it had never seen — that information edge was statistically zero and uncorrelated with forecast accuracy. Quadruple-confirmed: you can't out-forecast a crowd reading the same public forecasts.
Even "certain" isn't certain
By late afternoon the day's high looks known — so surely the dying buckets sell at a discount? No: physically impossible buckets are already at 99¢ by 5pm. And the official settlement landed above the hourly observations more than half the time, because the settlement thermometer catches spikes the hourly feed misses.
Pre-registration is the whole game
Every rule was frozen — thresholds, success bar, and kill criteria locked before results were read. That discipline caught one rule "passing" a test it had actually failed in the fitting window, and turned a week of dead ends into cheap insurance instead of slow, real losses.
One idea survived — and it wasn't the model's
A human hunch — stop trying to out-know the market; get paid small amounts to ride what it's already converging toward — passed the same adversarial gauntlet that killed everything else. It provides liquidity rather than forecasts, and contains no weather model at all. Fittingly, it doesn't work in busy NYC; the edge, if it's real, lives in quieter markets.
Built like production infrastructure
- Treated as production trading infrastructure, not a prototype: ~570 unit tests, mypy --strict, and automated linting on every change.
- Fully serverless on AWS — Lambda (Python 3.12), DynamoDB, CloudWatch, and SES — deployed via SAM / CloudFormation on fixed cron cycles.
- Deliberately dependency-light: httpx, pydantic, and numpy, with the needed statistics implemented directly to avoid pulling in scipy and keep the Lambda artifact small.
- Safety designed in from day one: paper-trading by default, post-only limit orders that never cross the spread, and six independent kill switches — drawdown, daily-loss, anomaly, API-failure, manual, and inception-drawdown.
- Disciplined review caught subtle correctness bugs early — including pricing the next day's market against the current day's forecast — long before real capital was at stake.
- New ideas run as paper experiments in parallel with live trading, each behind a pre-committed pass/fail gate — so nothing touches real capital until it has earned the promotion.
The phase that opened — and what it killed
The research pipeline's job was never to find winners. It was to make losing ideas die cheaply, in paper, before they die expensively, in production — and over the summer it did a lot of that job. The most instructive kill was a rule the pipeline itself surfaced: a discount-band pattern in daily-low temperature markets that showed +$39/day in its first season, under realistic fills, across nineteen cities, past every filter that had killed the earlier candidates. It wasn't enough. Pre-registration said freeze the rule, wait for a fresh month of data, and judge it again with no edits in between. Season two came back −$35/day on the identical rule — not decayed, sign-flipped. The tell was the same one the earlier rules had flunked: the probability our resting order fills given the trade loses was 1.00. When you only get filled on the trades about to go against you, a backtest's "edge" is just adverse selection wearing a costume.
A hot streak on the live NYC bot taught the same lesson from the other direction. Through mid-August it ran up 43% in two weeks, and for the first time ever its calibration metric showed the model genuinely beating the market's price. Every instinct said trust the model, size up. The rule said a pre-registered paper experiment decides that, not a streak. Three weeks later the streak had fully round-tripped and the calibration signal had reverted to market-favored. The parameter I didn't touch is the money I didn't lose.
The kills piled up: a heavy-tailed distribution family, a station-bias correction that passed calibration but never P&L (twice), four hindsight-mined order-book rules, and the daily-low pattern. The survivors that earned their way forward: price-shrinkage, a forecast-source switch, and the convergence-harvest strategy the rest of this page is about.
6
Ideas killed this summer — in paper, before they could lose real money
3
Graduated to live trading or continued paper validation
Accounting is part of the experiment
Two bugs from August belong in the case study, because both were the kind that quietly fabricate research conclusions. The first: the settlement sweep re-paginated each city's entire settled history on every run, and as those listings grew the exchange began rate-limiting the later cities in the loop — every day, silently, because each failure was caught, logged as a warning, and skipped. Whole cities stopped settling for weeks; 122 trades sat unaccounted; the experiment's metrics were being computed on a survivor-biased 71% of the data. What caught it wasn't monitoring — it was a habit: the ledger must reconcile to the penny, every check, no exceptions. It didn't, pulling that thread found the bug, a back-fill recovered every missing settlement, and the fleet has run reconciled-exact ever since.
The second: real-money settlement records can arrive after a market publicly shows its result — a race where the bot could have permanently booked a filled order as "unfilled." It surfaced on the pilot's very first settlement day, earned a regression test, and was fixed before it cost a cent. If your accounting can silently drop money, you don't have a P&L — you have an opinion.
The gate
On August 29 the convergence-harvest experiment faced the bars frozen for it back in July, before any data existed: at least 40 realistic fills, positive P&L per day at t ≥ 2 sustained, live-plausible fill rates, and no single day carrying more than 40% of the profit. Variant C — buy the market's own favorite the evening before, where thin books leave a residual discount — passed all four: 201 fills, +$8.05 per trading day, t = 2.28, and it had held above the t = 2 line for eight straight trading days before I judged it. The first full gate pass in the program's history.
The same evaluation failed variant A, its same-day sibling, on the same data — t = 0.56 and falling, half its profit eaten by exactly the adverse-selection pattern above. I take the rejection as evidence the acceptance means something: a gate that only ever says yes isn't a gate.
The final exam was fills
One assumption still stood between the paper verdict and real money: the fill simulator. Every dead strategy that summer died because paper fills flattered it — so the last stage was a pre-registered live pilot. $300, five-contract orders, thirteen days, 252 resolved real-money orders placed side by side with the paper twin on the same markets.
The simulator hadn't been optimistic; it had priced the skeleton correctly all along. Adverse selection — the thing I most expected live trading to worsen — was identical, 1.00 in both worlds, and per-contract economics matched to a tenth of a cent. On September 1 the strategy graduated: the six validated cities now trade at full size, thirteen replication cities stay at pilot size until they clear their own bar, variant A stays in paper, and NYC — where this strategy tests negative, because the deepest book harvests its own discounts — stays excluded. The system is now two live strategies, each confined to the markets where it passed and banned where it failed, plus three paper stacks still running experiments.
What this is, honestly: the graduated strategy projects on the order of $8 a day at current size, and the NYC bot is up about 25% on its starting bankroll after five months of surviving its own author. Nobody retires on this — that was never the claim. The claim is that a solo operator plus AI can run an institutional-grade research program on nights and weekends. Claude stayed what it was in phase one: an engineering collaborator and, increasingly, the program's referee. The strategy code is deterministic Python with no AI in the trading loop — and more than once the most valuable thing Claude did was refuse to let either of us skip a gate we had frozen when we were smarter.
88.1%
Live fill rate across 252 real-money orders
88.1%
Paper twin, same events — identical
0.3pp
Gap on would-win fills, against an allowed 15
+$32
P&L the pilot booked while proving it
Five lessons, earned the expensive way
Beat the price, not the weather
The market was a better forecaster than my model for months. The first honest metric was admitting it.
Pre-register or you'll negotiate with yourself
Freeze the rule, the thresholds, the success bar, and the kill criteria before you look at results. Every bar that later saved money felt overly strict on the day it was frozen.
Fill realism decides verdicts
P(fill | losing) = 1.00 is the universal tell of a phantom edge. Check it before you check profit.
Streaks are not signals
Both the euphoric kind and the miserable kind round-tripped within weeks. Bars moved; feelings didn't.
Reconciliation is research infrastructure
Two of the summer's worst bugs were found by refusing to let a ledger disagree by a dollar.