Methodology

Proof: submitted before the event

Every prediction stores its submission time. If it precedes the event, the entry is marked “✓ ahead” in the event list. Where the timestamp is missing, no marker is shown — only what can be proven is claimed.

The datasets are public JSON files and their change history is openly visible. It is therefore externally verifiable that a prediction existed before the event and was not altered afterwards.

The competition harness HARNESS_V2

FutureBench scores the deployed product, not the bare model — every model answering from what it already knows, under one identical protocol: same prompt, same information, same time window, same validation, same retry budget. No model may look anything up. Nothing is ever backfilled, coerced, or reinterpreted after the lock.

• One versioned template. Every model receives the same system and user prompt; only the mechanism for structured output differs. Each prediction stores a SHA-256 hash of the prompt it was given, so any change to the wording is visible in the data.
• The scoring rule is disclosed. Each model is told how its answer will be scored. With proper scoring rules this is methodologically required: a model that does not know the rule would be penalised for ignorance rather than for a poor forecast.
• The output is the stored format. The requested JSON is exactly the value that ends up in the public data files. It is checked by the same validators the site uses and stored unchanged — there is no cleanup step in between.

Limits enforced by the runner

20

tool calls per prediction (none available in this version)

5

minutes per attempt

2

retries on transport errors

2

format repair turns (format only, never the value)

System prompt, verbatim

This text is read from the same file the runner sends. It cannot drift from what the models actually received.

You are competing in FutureBench, a public prediction benchmark for language models.

Real events are scored publicly. Your prediction is committed to a public git repository BEFORE the event begins, so it cannot be changed afterwards. Other frontier models answer the exact same question under the exact same protocol, and the leaderboard is citable.

You have NO access to the internet, to search, or to any tool. Every competitor runs under this same restriction, so nobody can look anything up. The only external information anyone gets is what appears in this message: the question, how it will be resolved, and the context block below. Everything else must come from what you already know.

How to compete well:
- Your knowledge has a cutoff and this event lies after it. Reason from durable structure — relative strength, base rates, historical distributions, how such things usually resolve — rather than from any specific recent report you think you remember.
- Use the context block as the anchor when it gives you one. A stated last known value is the single most informative number you have.
- Do not invent sources, headlines or figures to justify an answer. A confident fabrication scores no better than an honest guess and is visible in the published record.
- Commit to the single answer that maximises your expected score under the stated scoring rule.
- Be calibrated, not brave. Overconfidence is punished by the scoring rule; so is refusing to commit.
- Answer even when uncertain. A missing answer scores nothing at all and is recorded publicly as a failure to answer.

Output discipline:
- End your reply with exactly one fenced ```json code block containing your answer.
- The JSON schema is fixed and given below. No extra fields, no comments, no trailing text after the block.
- Your reasoning may precede the block in plain prose; only the JSON block is stored as your prediction.

PREDICTION TYPE: scoreline (final score of a two-sided match)

SCORING RULE (Kicktipp scheme, higher is better):
- 4 points: exact score
- 3 points: correct goal difference (but not the exact score)
- 2 points: correct outcome only (home win / draw / away win)
- 0 points: wrong outcome

Both goal counts must be non-negative integers. Predict the score at the end of the
period the question names — do not add extra time or penalties unless the question
explicitly asks for the result after them.

OUTPUT SCHEMA:
{
  "prediction": { "kind": "scoreline", "home": <integer >= 0>, "away": <integer >= 0> },
  "rationale": "<optional, max 500 characters>"
}

PREDICTION TYPE: numeric (a single point estimate)

SCORING RULE (lower is better): absolute error |your value − actual value|.
Aggregated across events as MAE, MAPE and RMSE, and compared against a reference
baseline via a skill score.

Because absolute error is minimised by the MEDIAN of your belief distribution — not
by the mean and not by a dramatic outlier — state the value you consider equally
likely to be too high as too low. Use the exact unit and precision the question
states. Give a plain number: no thousands separators, no currency symbols, no ranges.

OUTPUT SCHEMA:
{
  "prediction": { "kind": "numeric", "value": <number> },
  "rationale": "<optional, max 500 characters>"
}

PREDICTION TYPE: binary (probability that the statement turns out TRUE)

SCORING RULE (lower is better): Brier score (p − outcome)², plus log loss and
accuracy at the 0.5 threshold as secondary metrics.

These are proper scoring rules: your expected score is best when you report your
TRUE probability. Note that 0 and 1 are almost never optimal — under log loss a
confident wrong answer at exactly 0 or 1 is catastrophic. Do not round to 0.5 to
play safe either; that discards what you do know.

OUTPUT SCHEMA:
{
  "prediction": { "kind": "binary", "probability": <number between 0 and 1> },
  "rationale": "<optional, max 500 characters>"
}

Information: what every model gets

No model can search, browse or call a tool, and every model is told so plainly. The only external information anyone receives is the question, the resolution rule and a short context block from the data feed — for a market, that includes the last known value. Everything else has to come from what the model already knows.

This is a deliberate narrowing. Letting each model use its provider’s own search would measure the search index as much as the model, and those indexes differ in ways nobody outside the labs can inspect. Removing retrieval costs realism and buys comparability: whatever separates two models here, it is not that one had a better search engine.

Sampling settings are left at each provider’s default. Fixing a temperature would suggest a control we do not have, because several reasoning APIs ignore the parameter.

Time window

Predictions are collected in a window that closes at the event’s lock time. All models of one event are queried in the same batch at the same moment, so no model gets fresher information than another.

An answer that arrives after the lock is discarded and never exported. A daily check verifies this invariant against the whole database, because the citability of the leaderboard rests on it.

Missing answers

A refusal is a result. If a model produces no valid answer within its budget, the gap is published with its reason instead of being quietly filled. Values are never corrected: an impossible probability of 1.3 stays invalid rather than becoming 1.0.

Published reason codes

refusal

the model declined to answer

invalid-output

no valid value even after the repair turns

timeout

no reply within the time limit

api-error

the provider’s API returned an error

rate-limited

rate limit not cleared before the lock

late

answer arrived after the lock

How events are resolved

The resolution source and the rule for reading it are fixed when the event is created, before anyone has predicted. The models see that rule verbatim in their prompt — they read exactly what the operator will read later.

There is no operator discretion after the lock. Where the rule cannot produce exactly one value, the event is voided — it stays visible with its reason and scores nothing for anyone. Voiding is the only possible outcome of a dispute; a resolution is never rewritten.

An event is voided when

• the stated rule does not yield exactly one value
• the source is unavailable or contradicts itself
• the event is cancelled, postponed, or started early
• the reported value violates the event type’s contract

Model versions

One leaderboard row is one model version. Every prediction records the model id the API reported back, so a silent change behind a floating alias becomes visible. When that happens the row is frozen and a new one starts — results from two versions are never mixed.

A model that leaves the competition keeps its row and its history. Nothing is deleted — removing weak results afterwards would be exactly the selection effect a benchmark has to rule out.

Kinds of prediction

Score prediction scoreline

Input format: 2-1

Numeric estimate numeric

Input format: 4512.30

Probability (yes/no) binary

Input format: 0.70 / 70%

Metrics

Each category decides which metrics it displays. For error measures, lower is better.

Baselines by category

The key question is not “how good is the value?” but “does the model beat the established reference?”. Categories without a solid reference deliberately show no skill score rather than inventing a baseline.

Limits of interpretation

• Sample size. Every table shows a “Scored” column. With few events, chance matters more than skill, so rankings are not reliable. Confidence intervals are planned for a later stage.
• Points are not comparable across categories. A points total depends on the number of events, an error measure on the unit. Only the skill score against the baseline is dimensionless and therefore transferable.
• Proper scoring rules instead of hit rate. For probabilities we rank by Brier score and log loss. A plain hit rate would reward models that always state extreme probabilities.
• Prototype status. Apart from the completed 2026 World Cup, all categories currently hold sample data. They serve to build the interface, not to assess models.