AI Agents for Data Science Lost to a 76.5% Baseline

AI agents for data science lost to a two-line Titanic baseline, scoring 72.96% in one hour. See why tabular ML breaks coding agents and what to keep.

Run a frontier coding agent on Kaggle's Titanic competition for one hour and the best it manages, out of two attempts, is 72.96% accuracy. The two-line starter baseline most Titanic tutorials open with, predicting survival from passenger sex alone, scores roughly 76.5% on the public leaderboard. The agent lost to the tutorial. Then the human suggested a single estimator, XGBoost, gave the same setup roughly 90 to 100 more minutes, and watched the score climb to 77.27%. The model underneath did not get smarter between those runs. What changed was who supplied the knowledge and who supplied the check.

Get the latest posts and exclusive content delivered to your inbox.

Join 13 readers. No spam. Unsubscribe in one click, anytime.

That exchange, from a self-reported experiment posted to Reddit'sr/MachineLearning, is the cleanest natural experiment I have seen for where AI agents for data science break. Treat it honestly: one user, one setup, best-of-two runs, and an author demonstrating Gear, their own agent-harness product, so this is not a controlled benchmark. The failure mode it exposes still generalizes, because agent competence is verification-shaped. Agents excel where feedback is cheap, local, and executable, meaning unit tests, linters, and compilers, and they stall where the only feedback is a held-out score computed over a search space of feature transforms and estimator choices. That second description fits tabular machine learning almost perfectly.

The right conclusion is that agents can do ML once humans supply the missing verifier and the priors, and an agentic ML workflow should be split along exactly that line.

The one-hour run and the baseline that beat it

Titanic is the hello world of Kaggle: 891 labeled training rows, 418 test rows, a binary target. Because the evacuation put women and children first, sex alone separates outcomes well. A model that predicts survival for female passengers and not for male ones scores about 76.5% on the public leaderboard, which is why the gender-only Titanic baseline accuracy is the sanity floor experienced competitors check before anything else. Every serious pipeline has to clear two lines of pandas.

The experiment paired OpenAI's Codex with Astra on this exact task, with a one-hour cap. Best of two runs: 72.96%. The poster then explicitly suggested XGBoost and allowed roughly 90 to 100 minutes of optimization, reaching 77.27%. A third setup, an evolved data-science harness running Astra, reported 79.90% with candidate runs averaging about 13 minutes.

Three honesty flags before reading anything into those numbers. The results are self-reported, best-of-two, with no human-comparison runs. The author was showcasing their own harness product, and the 79.90% figure was measured locally on the 418 test rows rather than on the public leaderboard, so it is not directly comparable to the other two. The post also never itemizes what the agent tried during its hour, which is the detail that would settle whether it ever considered gradient boosting. Even with those caveats, the ladder from 72.96 to 77.27 to 79.90 has one shape: the underlying model stayed roughly fixed while the workflow around it changed. Explaining that shape is the rest of this piece.

Why coding agent skill is verification-shaped. Watch what a coding agent actually does when it resolves a hard issue. It edits, runs the tests, reads a failure, localizes, edits again. Each cycle takes seconds and returns a verdict that is binary, local, and correct, and the failure message points at a line. Linters and compilers add more channels. Over an hour the agent consumes hundreds of cheap, unambiguous bits of feedback about its own actions. That is the environment in which coding agents look superhuman.

Now count the channels in a Kaggle loop. The leaderboard returns one scalar per submission, computed on labels you cannot inspect, and noisy because the public split is a proxy for the private one. Repeated submissions make it worse rather than better: climb a static leaderboard aggressively and you overfit the proxy, the exact failure the Ladder leaderboard paper was designed to correct.

The agent's substitute is local cross-validation, and on 891 rows it is a poor stand-in. Fold results jitter. A pipeline with accidental leakage looks brilliant in CV and collapses on held-out data. Nothing in a CV number tells the agent which of its twenty ideas helped. Imagine debugging a flaky test with a reviewer who answers once a day with a number between 0 and 1 and no stack trace. Agents do not become incompetent on Titanic; they lose their standard error-correction channel, and everything downstream degrades.

Tabular ML is the worst case for agents

Tabular competitions concentrate every bad property at once. Start with data volume. The Titanic test set has 418 rows, so each row is about a quarter of a percentage point of accuracy, and the entire gap between the agent's 72.96% and the two-line baseline is roughly 15 passengers. In five-fold CV on the training set, each fold holds about 178 rows, so one row flipping moves the fold score by more than half a point. When honest differences between candidate models are two or three points, the agent is searching inside noise.

Then the search space. Wins on tabular problems rarely come from architecture; they come from feature transforms and estimator choice guided by a strong empirical prior, and that prior says gradient-boosted trees. A large 2022 benchmark study by Grinsztajn and colleagues found tree ensembles still outperforming deep learning on typical tabular data, so the decisive knowledge in this domain is which estimator family to reach for, not how to design a novel one.

The prior is written in a million blog posts, and the agent has presumably read most of them. The gap is between knowing and firing. In a code task, the equivalent prior, check for null before dereferencing, gets confirmed or refuted by a test in seconds. On tabular data, nothing in the execution loop tells the agent that a tree ensemble is the right tool for this specific dataset, so untested hypotheses all look equally plausible to the loop. If you want a compact answer to why LLM agents fail at tabular data, it is this: the decisive knowledge is prior-shaped, and priors only fire when something checks them.

The benchmark evidence on AI agents for data science

The benchmark record agrees with the anecdote once you read it through feedback geometry. SWE-bench-style evaluations give an agent a real GitHub issue and score it by whether the repository's own tests pass afterward. Success is defined by executable verification, the same channel agents already exploit, so these benchmarks measure skill inside a verifier-rich environment. Fast-climbing pass rates there tell you little about verifier-poor domains.

The MLE-bench paper from OpenAI moves the evaluation onto Kaggle, running agents through full ML engineering pipelines on competition data. The reported pattern matters more than any single number. Agents do earn Kaggle medals on some competitions, but the medal results sit at the high-compute, many-attempt end of the configuration space, while single short runs, precisely the Codex Kaggle Titanic regime, land far below strong human baselines. The benchmark where agents do ML at a medal level is the benchmark that supplies what the leaderboard withholds: repeated draws, large compute, and a scaffold that encodes process.

Two further results point the same way. AI Agents That Matter argues that agent benchmarks mislead when they report accuracy without cost, skip reproducibility, and draw from narrow distributions, a fair description of one self-reported Titanic run as evidence. And METR's long-task measurements indicate agent reliability falls sharply as task duration grows, which puts an iterative hour-plus experimentation loop in the steep part of that curve. The SWE-bench versus MLE-bench contrast is not a contradiction. It is the same competence measured in two different feedback environments.

The XGBoost moment and the missing prior

The most instructive moment in the run is the intervention itself. Before it, the framing was open-ended. After it, the human said, in effect, use XGBoost, and the score moved from 72.96% to 77.27%, a gain of 4.31 points. In a session full of generated code, the highest-information token was typed by the human.

Before: "Improve the accuracy of this Titanic model."

After: "Train an XGBoost classifier on Pclass, Sex, Fare, and family-size features. Stratified 5-fold CV, tune tree depth and learning rate, then submit."

The second prompt is smaller, not harder. It collapses a search across every estimator family, preprocessing scheme, and feature set into a search across a handful of hyperparameters, inside a feedback loop, cross-validation, that the agent can run locally as often as it likes. Gradient boosting has been the default answer for structured data since the original XGBoost paper made it fast and portable; the human simply transferred that prior. In verification terms, the human acted as the external verifier, naming an acceptance region before the loop began.

The third data point extends the lesson from words to structure. The evolved harness scored 79.90% with candidate runs averaging around 13 minutes, and its instructions read like an external verifier in prose: fix validation folds and seeds, fit preprocessing inside each fold, keep the search bounded, check the submission schema, stop on evidence. Treat the number as directional, since it was measured locally. The pattern is still unmistakable: as the scaffold supplied more of the checking discipline, the same underlying model got more competent.

Which ML work to hand off and which to keep

The rule falls straight out of the mechanism: delegate machine learning tasks to an AI agent wherever feedback is executable and local, and keep whatever involves judgment, priors, or construction of the verifier itself.

Delegate every loop the agent can check. Keep the loop that checks the agent.

Build the verifier before you delegate

The highest-leverage move is writing the missing feedback channel yourself, as code. Three tests change the game:

Baseline regression test. A sex-only model must reproduce roughly 76.5% accuracy; any pipeline that cannot beat it fails before you waste a submission.

Leakage test. Any feature whose computation touches the target, including target encodings fit on all rows, fails the suite.

Fold-stability test. CV scores across seeds and folds must stay within a tolerance you set; if the spread is wider than the gain, the gain is noise.

With those in place, a delegation prompt can look like this:

Estimator priors and the shortlist: a gradient-boosted ensemble, a regularized linear baseline, and for small data a tabular foundation model. Judging whether a CV delta is real, using the fold-noise arithmetic above. Aligning your metric with the private leaderboard and deciding when to stop. And auditing leakage in whatever the agent proposes, because agents will happily build the Titanic classics, such as per-ticket or family-group survival rates computed over all rows. That is the human plus agent data science workflow in one sentence, and it is what human in the loop machine learning should actually mean in practice: the human is the verifier of record.

Where this argument could break

Calibration cuts both ways, so name the conditions that would weaken the thesis. Longer budgets blunt it, since with enough hours an agent can discover tree ensembles empirically. Better scaffolds blunt it, which is what the 79.90% local result already shows. AutoML-grade search over a bounded model space replaces the human prior with brute force. Richer held-out feedback, meaning solid local validation discipline and tooling that makes the leaderboard proxy less noisy, attacks the root cause directly.

The prior itself may also migrate into tools. The TabPFN paper in Nature reports a tabular foundation model that, in its evaluations, outperforms strong baselines including gradient boosting on small datasets, exactly the Titanic regime. If tabular priors ship inside the tools agents already call, the human no longer needs to type XGBoost, and the estimator-selection half of the argument dissolves. The verification-geometry half survives, because someone still has to design the check.

A clean falsification would look like this: a default scaffold, a single short run, matching strong human baselines across many tabular competitions, replicated by someone with no product to demo. Nothing like that is public today, and the MLE-bench record suggests the gap is real but scaffold-dependent. Until such a result appears, the leverage is in building the verifier yourself and handing the agent everything inside it.

Tyler has tested developer tooling for a decade, first as a platform engineer and now as an independent analyst. He reviews models, frameworks, and APIs the way he would want them reviewed before relying on them for real work.

What Multi-Node vLLM Serving Really Costs Per GPU

Multi-node vLLM serving audited: we normalize the 607.8 tok/s two-station GLM-5.3 demo per GPU, price the all-reduce tax, and derive a scale-out rule.

Version Control for AI Coding Agents Starts Above Git

Version control for AI coding agents starts above Git. Ship a worktree-plus-manifest stack this quarter; the requirements spec covers what replaces it.

DiffusionGemma Makes Full Response Latency the Metric

DiffusionGemma teardown: parallel decoding trades first-token latency for full-response latency, drops the KV cache, and breaks streaming and tool calls.