How this works

Methodology

Version 1.1, 28 Oct 2026 (adds Lean).

This document is the rulebook for every verdict on allin.fan. If a verdict and this document disagree, the verdict is wrong and gets fixed.

1. Scope

• Who is rated: only the four hosts — Chamath Palihapitiya, David Sacks, David Friedberg, Jason Calacanis. Guests are never rated, even when they make the most interesting claims in an episode.
• What is rated:
• Claims: statements of fact about the world that can be checked against a record (numbers, events, quotes, what a law or document says).
• Predictions: statements about the future that are falsifiable and have a deadline (stated or reasonably implied).
• What is not rated:
• Opinions, values and forecasts with no checkable outcome ("this is bad policy", "AI will change everything").
• Figurative language, jokes and obvious hyperbole ("a trillion times better", "everyone in SF is leaving").
• Statements a host is clearly reporting as someone else's view, unless he endorses it as fact.
• Selection: we pick the claims that carry the argument of a segment, not trivia. A wrong number in passing is not selected unless it is the point.

2. Verdicts

3. Calibration rules

These are the judgement rules that decide close calls. They were set by the editor and apply to every row.

1. Rate the point, not the phrasing. Identify what the host is using the claim to argue, and judge whether the evidence supports that.
2. Figurative or hyperbolic speech is never rated Misleading. If it can't reasonably be taken literally, it is either not rated or rated on its literal core.
3. Wrong numbers only count when the number is the point. Example: citing a €43bn figure when the real one is €54bn is not a problem if the argument ("France is cutting tens of billions") stands. A number that flips the conclusion, or is off by an order of magnitude, is.
4. Context errors make it Misleading, not False. A true fact presented so that it implies something false (e.g. "avoided hiring" presented as "cuts") is Misleading.
5. Hedged predictions are scored on the stated lean. "It's a toss-up, but I think X" is scored as a prediction of X, with the hedge noted.
6. Partial outcomes. Where a prediction has several parts or a range, the rule says up front what counts as partial; partial outcomes are rated Misleading and explained in Notes.
7. Same claim, same verdict. If two hosts make the same claim, both get the same verdict, cross-referenced.
8. Conflicting predictions by the same host are both recorded and cross-referenced; each is scored on its own rule.

4. Evidence standard

• Every verdict needs evidence written in plain words (the Evidence column) and at least one source link, except Unverified, which explains what was searched.
• Source priority: primary records (filings, statutes, official statistics, court records, company reports, transcripts) > reputable reporting citing primary records > other reporting. Social posts are used only as evidence of what someone said.
• Links must point to the exact page used, not a homepage.
• Prediction-market odds are context, never evidence of an outcome.
• The quote is checked against the transcript. Each row's quote must appear in the speaker's own turn at the stated timestamp. Transcript speaker labels (automatic diarization) are sometimes wrong; when a line's speaker is in doubt, it is not used.

5. Predictions: how they are resolved

• The rule is written before the outcome is known. Each prediction gets an explicit resolution rule (thresholds, data source, deadline). Vague words ("narrow", "someone like", "leave") are pinned to a measurable definition.
• Rules are signed off by the editor and marked "Rule (signed off )". After sign-off a rule is not changed to fit the result. If it must change (data source discontinued, event cancelled), the change, date and reason are logged.
• No deadline given? We set a reasonable window (usually 1–2 years, stated in Notes) and score against it.
• Void: if the event that a prediction depends on doesn't happen (e.g. no IPO in the window), it is marked void, not False.
• Review cadence: the NextReview date steps 7 days → 1 month → 3 months → 6 months → 12 months → 24 months → 48 months until the deadline.
• The full list of rules is in the predictions section.

6. Lean (alignment with the current administration)

Separate from the verdict, each row records whether the claim lines up with the US administration in office (currently the Trump administration):

Rules: - Lean describes the claim, not the speaker. A Pro-admin claim can be Accurate; Lean is not a verdict and is never used to change one. - Attacks on state or local governments of the opposing party count as Pro-admin; criticism of foreign governments counts as Neutral. - The rule follows whoever is in office: if the administration changes, new rows are tagged against the new one and old rows keep their tag. - What the site reports is the share of each host's political claims by Lean and the accuracy of each group, so readers can see whether a host's claims track the administration's line, and how accurate those claims are.

7. Workflow

1. New episode transcripts are fetched every 6 hours (GitHub Actions, spoken.md).
2. New episodes are rated automatically over the weekend they come out, with the same rules, research and transcript checks as everything else, and published straight away marked Auto-rated · awaiting review. Prediction rules written this way are marked "Rule (auto)". The editor then reviews them: confirmed rows lose the label; anything changed is logged on the corrections page.
3. Older episodes (the backfill) are processed in batches of 5, imported into the Claims tab of the Google Sheet with Status = Draft, and published only after the editor's review.
4. Claims are drafted with quote, timestamp, verdict, evidence and sources, and every quote is checked automatically against the transcript.

8. Corrections

• If a verdict is wrong, it is fixed in the open: the row is updated and the Notes column records what changed, when and why ("Re-rated: …").
• Readers can report errors through the public correction form (a GitHub issue on the site's repository). Reports are checked against the same rules as above, and the outcome is posted on the issue.

9. Conflicts of interest and limits

• Drafting and research are AI-assisted (Claude, made by Anthropic). Many claims on the show concern Anthropic and its competitors. Those rows get extra scrutiny and the editor makes the final call. Where a host has a commercial interest in his claim, it is noted.
• The editor has no financial relationship with the hosts, their funds or their companies.
• Funding. The audit is funded only by reader tips. It takes no money from the hosts, their companies, their investors or their sponsors, and no money from prediction markets or betting sites. If that ever changes, it will be stated here first. ☕ Support the audit
• Market odds. Some open predictions show "What the markets say": live odds from Kalshi, Polymarket and PredictIt on the nearest matching market. They are context only and play no part in the verdict. The links are plain links with no referral codes. The hosts have called Polymarket the show's official marketplace. It is shown because it often lists the only market on a question, and Kalshi and PredictIt are shown next to it whenever they list one too.
• Verdicts reflect the record available at the time of rating and are updated if better evidence appears.

10. Data fields

ID (AIA-YYYYMMDD-NN), Status, Type (Claim / Prediction), Date, Episode, EpisodeTitle, EpisodeURL, Timestamp, Speaker, Quote, Claim, Category, Verdict, Evidence, Sources, TargetDate, NextReview, Notes, Lean.