SmugCalls · research

We found a signal worth +3.4 points. It survived out-of-sample testing. It was still garbage.

This is the write-up, including the part where the validation method we trusted was structurally incapable of catching the problem.

What we were looking for

The constraint mattered: raise the hit rate without losing much volume. Anyone can raise a hit rate by calling less. The question was whether the losing calls were identifiable in advance.

Every call writes 19 features to the log at the moment of the decision — convergence counts, buyer profiles, holder concentration, cluster overlap, wallet-age flags. The outcome, peak multiple after the call, lands in a separate table. 3,909 mature calls, 03 July to 13 August 2026. We split by time: fit on the older 60%, test on the newer 40% the search never saw.

The graveyard

Nothing worked. This part is short because the results were uniform.

213 single-feature rules. Best gained 2.7 points on training and 0.8 on the hold-out. Rules gaining at least 3 points on training: zero. 4,961 two-feature combinations. Best gained 3.4 on training, 0.8 on hold-out.

Entry market cap. No gradient — 40.9, 35.2, 39.8, 41.2, 39.7 percent across quintiles on training, and a different order on the hold-out. Noise.

Token age at the moment of the call. This one was worth the trip, because it went against intuition. Calls fired within the first 30 seconds performed worst in both halves: 37.1% and 32.9%, against baselines of 38.1% and 37.3%. Being faster is not a lever here. We had assumed it was.

The tail. Same features, target moved to 10X and 50X. Not a single rule raised the target rate by even half again on the training half. The features that describe the moment of the call know nothing about the tail.

Hour of day. Real, and it survives: 00–08 UTC sits above baseline in every slice, 16–24 UTC below. But acting on it costs 33 to 40% of volume to buy 1.4 to 2.0 points. Rejected on the constraint, not on the evidence.

With 1,564 hold-out calls, a difference of about 3 points is detectable. Nothing reached it. The honest summary at that stage: within the population we already call, these features have been squeezed dry by the gates already in place. Moving the number needs new information, not new thresholds on old information.

The finding

So we went for information that was not in the call features at all: the history of the wallet that created the token — how many tokens that creator had launched before our call, computed causally, no peeking forward. The population split cleanly:

Share of calls that reached 2× entry cap, by who created the token.

One group sat 14 points below baseline and its confidence interval did not touch the others. The rule wrote itself: drop the bot-farm launchers, keep everything else.

Then we validated it, and this is where it got seductive.

• It held on the hold-out — and got stronger. Training: +1.8 points keeping 89% of volume. Hold-out: +3.4 points keeping 80%. An in-sample effect that grows out-of-sample is the opposite of the overfitting signature.
• The dropped group was stable. 25.2% on training, 24.0% on hold-out, and 26.0 / 23.7 / 24.2 across three consecutive time slices whose baselines were 40.0 / 39.4 / 37.0.
• The threshold was not on a knife edge. Cutting at 30, 50, 100 or 200 prior launches all gave +2.3 to +2.5 points. Only past 500 did it decay.
• It was a property, not a blacklist. 138 distinct addresses, the top eight accounting for 30% of the group, 55 addresses appearing exactly once.

We priced it honestly too: the rule reduces the absolute number of winners. On the hold-out it discarded 313 calls containing 75 hits — 12% of all our doubles — to raise the displayed percentage. Higher rate, fewer winners.

So it shipped as a shadow label, not a live gate. Log the flag, cut nothing, decide after a forward test. That decision is the only reason this article is about a near miss.

The tell

The shadow went live and immediately marked 48% of calls as bot-farm launches. The retrospective analysis said 14.6%.

Three times off. Not rounding, not drift — with a true rate of 15%, seeing 32 of 67 is roughly seven standard deviations out. Something was reading a different number.

First guess: market composition shifted. It had not. Second: the live code computed the feature differently from the analysis; we checked 400 calls both ways and they agreed on 396. Third: a mint could have several rows; the mint column is a primary key, one row each.

Then we compared, for a single token, what the caller logged at decision time against what the database says now:

[08-15 08:08:08] SHADOW-FACTORY 774jh4nU np=583    creator=DiwH5biL
now:                                       np=78,977 creator=ARu4n5mF

Same token, same database, same query. Different creator.

The root cause

The ingest writes launches like this:

INSERT INTO launches(mint, created_ts, creator, ...) VALUES(?,?,?,...)
ON CONFLICT(mint) DO UPDATE SET
  creator = COALESCE(NULLIF(excluded.creator,''), launches.creator),
  ...

Every subsequent event for a mint overwrites the creator. Last writer wins. For tokens that keep emitting events, the field drifts to one high-volume address. Which tokens keep emitting events? The ones that stayed alive:

Sixteen times the trading activity, four times the graduation rate.

So the group we labelled "bot farms" was, to a large extent, the set of tokens whose creator field never got overwritten — because nothing else ever happened to them. We had built a feature out of "did this token stay active", which is downstream of the outcome we were trying to predict. It scored 24.5% for the same reason a dead token scores badly: it was dead.

Why the hold-out split could not save us

This is the part worth keeping.

A time-based hold-out protects against overfitting: the risk that a threshold was tuned to noise in the sample. It does that well, and it killed 213 rules and 4,961 pairs for us in the space of an afternoon.

It offers no protection against leakage, because leakage is not a property of the sample. It is a property of the feature. A contaminated field is contaminated in the training half and in the hold-out half equally, so the effect reproduces perfectly. And reproducing perfectly is exactly what we were using as evidence of soundness.

Worse, the contamination strengthened over the period, so the out-of-sample lift came in higher than the in-sample one — and we read that as robustness. The check we trusted most was the one being fooled hardest.

What actually caught it was not a statistical test. It was deploying the feature where it had to be computed at decision time, and noticing the number did not match.

What we do now

The shadow label stays. It reads the creator field at the moment of the call, before any overwriting can happen, which makes it the causally clean version of the feature — the one the retrospective study could not have computed. Roughly 45% of calls carry it. The forward test will answer the original question honestly, for the first time.

The expectation of +3.4 points is gone. The hypothesis is open again from zero. And we added one question to the checklist, ahead of the statistics:

Does this field change after the moment the decision is made?

For creator, the answer was yes. We had never asked.

---

Method notes

• Computed from a local record of every pump.fun launch and trade, written from the chain, not from a third-party API.
• Confidence intervals are Wilson. Time splits are 60/40 by call timestamp, never random. The three-slice check is chronological thirds, not folds.
• Counts as of 15 August 2026.
• Related: how many pump.fun launches actually graduate — the same habit of asking what a number is counting, applied to a metric everyone quotes.