Hindcast: Replaying Prediction Markets to Evaluate LLM Forecasters
Forecasters are evaluated by backtesting, which replays resolved questions and grades the probability the system would have assigned before the outcome was known. For LLMs, two channels leak the answer into this test. A model that retrieves
https://arxiv.org/abs/2607.14051v1 ↗Thesis fit
Good fit
Within your typical scope; diligence still required.
In your usual scope
Idea match
Light
How close the company’s idea is to your thesis statement
Sector
llm
Overlap with sectors you care about
Geography
Unknown
Location unknown — scores 0
Your thesis: “We back exceptional technical founders building AI-first products and infrastructure, deploying $100K checks within 24 hours.”
Founder → stable
Traction → stable
Idea vs market ↓ declining
▸ Add / edit details
Correct facts used on the next screening or memo.
People
X
Xiao Ye
1.0
low confidence
J
Jacob Dineen
1.0
low confidence
E
Evan Zhu
1.0
low confidence
S
Shijie Lu
1.0
low confidence
K
Kevin Song
1.0
low confidence
B
Ben Zhou
1.0
low confidence