yearly GDP growth of 2.5%

participation

The causal lost-jobs attribution and two individual-majority assertions are not established here; the majority assertions are access-limited support conclusions, without evidence of opposite majorities.

survey

substantial inter-rater variation

terative quality checks

prompt-sensitivity controls

tructured measurement schemas (Asirvatham, Mokski, and Shleifer 2026), i

we address is therefore:

These findings motivate testing LLMs as supplementary raters in structured evaluation pipelines.

models

Appendix results

. A paired bootstrap places

range restriction and halo effects that distort agreement metrics (Wang et al. 2025).

can go.

policies that restrict reviewer us

progressive prompting

undocumented

IsItCredible

These notes were written by OpenAI Codex at David Reinstein’s request and are not new empirical findings or agreed changes to the research plan.

Related work update · AI-generated, October 23, 2026

A collaboration with The Unjournal (55+ open evaluation packages for global-priorities research). Funding: Survival and Flourishing Fund, Long Term Future Fund, EA Funds.

80% of the curated human consensus issues on averag

November 5, 2026

Choose the right project view

we found that humans agreed as much or more with AI’s judgment than they did with other humans: human editors agreed with other human editors 80% of the time but on average 87% of the time with the AI’s judgment

Claude Opus 4.8 (Anthropic, 2026) at its maximum reasoning setting, under two conditions.

RapidPeer, including 124 scholarly paper submissions and 134 human reviews (10 subm

asks human reviewers to read the entire paper and provide constructive sugges- tions throughout (

ese problems with the human peer review process have motivated scholars to eval- uate large language models (now often called AI) as a replacement. But AI leads to other worries, including hallucinated claims, fabricated sources, and confidently stated errors (Ye et al., 2024); failure to recognize or reward novelty (Shin et al., 2025); biases learned from human training data unfairly favoring authors based on institution, seniority, or fame (Macharla Vasu et al., 2026; Hosseini and Horbach, 2023); excessive leniency (Russo Latona et al., 2024); vulnerability to manipulation (Lin, 2025; Ye et al., 2024); breaches of confidentiality (Naddaf, 2025); and unclear accountability for errors (Hos- seini and Horbach, 2023; Porsdam Mann et al., 2025). The farthest scholars have gone is to conclude that AI should supplement rather than replace human peer reviews (Liang et al., 2024; Farber, 2025; Doskaliuk et al., 2025), including the largest deployment to date, at the AAAI-26 conference, which added one clearly labeled AI review to the hu- man reviews of each of the 22,977 papers sent for full review (Biswas et al., 2026).

ow inter-reviewer agreement (Born- mann, Mutz, and Daniel, 2010; Mutz, Bornmann, and Daniel, 2012);

judged to be of unusually high quality by journal editors and authors.

Comparing LLM and human reviews of social science research using data from Unjournal.org CodeShow All CodeHide All CodeView Source

Follow-up diagnostic · provisional, AI-scored, November 4, 2026

All scores are a provisional first pass. David Reinstein has not yet reviewed them, or the three critiques the model disputes (below)

Follow-up diagnostic · provisional, AI-scored, November 4, 2026

What this page is. An adaptation of the paper’s models and calibration, made by The Unjournal to help readers and evaluators see which assumptions drive the headline result. It is not the paper, and the authors have not reviewed or endorsed it. Read the original for the authors’ own presentation.

Is the team comfortable offering different pay for similar work? Paper-level assignment limits direct comparison, but evaluators on different papers could still compare notes.

Should the other streams be included, for more data?

Are these amounts right, or should there be a third arm with a $225 base?

A: prize-led

Structural sensitivity: would other reasonable model choices change the results?

Top 20 Papers by Human–AI Synthesis ?Choose whether this table ranks by the current AI assessment, the evidence-weighted human aggregate, or the same Human–AI synthesis used on the main dashboard.

economic and inference loops ?The paper sizes the core research feedback loop only. It does not quantitatively calibrate the loops through economic growth or inference efficiency, so the below-threshold result does not establish whether those other loops are above or below their own thresholds. Evidence and qualifications;

the verdict

capability elasticity

The calibrated verdict, that the core loop is below the threshold for self-sustaining acceleration, holds at the printed inputs, but the print shows one cell with no interval, built from inputs of different periods and scopes;

core research loop

wish list of empirical objects

he biggest unknowns to be informed by additional data are

The distinction between "narrow" and "broad" AI capabilities

return has been around 9

issue 7).

this return

"though they appear to be strengthening" (abstract). Holds as a hedged conjecture.

nference-efficiency loop

economic loops

The same sentence read for feedback loops in general,

the core research loop.

(issue 1)

"A back-of-the-envelope calculation suggests that feedback loops are not currently strong enough to generate a" self-sustaining acceleration (abstract), read for the core research loop. Holds in narrower form (issue 1).

It is not an Unjournal evaluation and has not been checked by a person. It is a blind run: no earlier Unjournal or AI assessment of the paper was used. Conflict of interest: it was produced with Claude, made by Anthropic. No author lists an Anthropic affiliation, but the paper's evidence leans on Anthropic sources: the Claude Mythos Preview system card supplies the 4X uplift behind the 9%, an Anthropic piece by Favaro and Clark (2026) supplies the 8X and 64% figures and is the citation for the opening "signs" of a feedback loop, and the threshold is illustrated with Anthropic models (Claude 3.7 Sonnet, Claude Opus 4.7 and 4.8). METR funds the paper and employs the lead author, and the system card names METR, and Epoch AI (a co-author's affiliation), as pre-release testers of Claude Mythos Preview.

he verdict does not extend to the economic and inference loops;

the capability elasticity of 6.5 is not established as the elasticity, though correcting it at the printed 3× rate strengthens the verdict;

The calibrated verdict, that the core loop is below the threshold for self-sustaining acceleration, holds at the printed inputs, but the print shows one cell with no interval, built from inputs of different periods and scopes; algorithmic progress of 7.0–9.3× a year, consistent with the capability path observed over the window and inside the intervals of the paper's own cited sources, would reverse it;

The modelling framework holds

Human input on prioritization (2 sources; 2 themes)

arithmetic

Please choose the questions you think matter most and can usefully address. A few well-chosen checks may be plenty. You’re welcome to question the paper’s emphasis as well as its answers.

Additional review resource: Would other reasonable model choices change the results? A draft menu and one pilot AI analysis requested by The Unjournal (David Reinstein) and drafted b

Potential issues

AI-generated review input, not an Unjournal evaluation. No person has reviewed this content or checked its numbers.

What seems to be causing this disagreement? (choose all that apply)

Under consideration: we may reword O1 (and re-weight Q2) before the next round. Flagging this openly because it would change what we are asking you to defend, and we would rather say so than quietly swap the question.

Track record

Next in line

We have written briefs on 34 candidate papers:

a pool of 200+ vetted evaluators, check conflicts, and pay for careful critique wi

Economists, forecasters, and policy analysts are moving into this area fast. Growth theorists are co-authoring scenario models with frontier labs. Forecasting researchers are eliciting beliefs from economists, AI-company staff, and superforecasters. Policy organizations are producing quantitative estimates of compute smuggling and regulatory delays. Some of this work is already being used: in fiscal scenarios, in commentary on monetary policy, and in legislative fights.

See the shortlist and briefs Get involved

aiming for weeks rather than years.

on AI's economic effects

Publish a versioned baseline. One parameter sheet, one model commit, deterministic seed, machine-readable output, and regression tests. Record the clean-default distribution and component decomposition.

Lognormal

Within-rater" correlations in the 2025 post.

. 28 rows are duplicate

he current form has no real-world question, so the import seems to fill it from global relevance. Any analysis that treats them as two questions overstates how similar they are.

Impossible intervals. Twelve ratings have the midpoint outside the evaluator's own interval or the bounds reversed. Examples: overall 83 with interval [93, 100]; methods 75 with [88, 33]; a journal tier of 4.0 with interval [25, 5] (probably 2.5). The new form already enforces lower ≤ midpoint ≤ upper, so these come from older entry routes. Fifteen intervals have zero width (five of them "open science 100 [100, 100]").

he "Yellow Vests" carbon-tax paper,

Claims and evidence

With a 90% interval

eport reliability in percentile points, not just coefficients

2) replace the open-science percentile with a short checklist;

show evaluators how much experts typically disagree, right next to the sliders

warn on very narrow or impossible intervals

For papers with two human evaluations, we compared how well a model's rating tracks one randomly chosen human, with how well the other human does (rank correlation, averaged over 200 random choices)

g by "Evaluator 1" across papers mixes different people. Section 2.

Since 2025 the "real-world relevance" field almost always equals the "global relevance" field

There are some fixable data problems

On average, "should" (merit) and "will" (prediction) are almost the same number. The 2024 entries were mostly older evaluations transferred from PubPub, so we're checking whether one value was sometimes copied into the other. Section 6.

Relevance's

apley decompositionAdds the predictors to the regression in every possible order and averages how much each one raises R²

Claude Opus 4.6 rates about 8 lower.

There are some signs of leniency too (anonymous evaluators rate about 5 points lower than named ones on the same paper).

Evaluators' 90% intervals are too narrow for the disagreement we see, by a factor of about two. A co-evaluator's best guess lands inside your 90% interval about half the time; if everyone's intervals were realistic it would be about 70%. Nearly a quarter of evaluator pairs give intervals that don't overlap at all.

A single evaluation carries roughly half signal, half noise on overall and methods; much less on the other criteria. Averaging our usual two evaluations makes the paper-level number noticeably more reliable, but for "advancing knowledge" and "open science" even the average is mostly noise.

ntraclass correlationThe share of total rating variance that is due to real differences between papers, rather than to evaluators disagreeing about the same paper. 0.5 means half and half.

Krippendorff's alpha

much less on the other criteria

On the overall rating, the median gap is 9.5 points

Data: The Unjournal's public ratings export (as of 16 October 2026), one 2026 evaluation added by hand from the published evaluation summary, journal outcomes from OpenAlex and our database, and LLM ratings from the LLM evaluation project.

not yet The Unjournal's priorities or decisions.

ut has not yet checked every number or sentence.

numbers mean,

we should change.

realistic

114 evaluations of 59 papers

Main findings in plain terms The data, and problems in it How evaluators use the scales How much do evaluators agree? Are the 90% intervals realistic? The two journal-tier questions What the overall rating is built from Which questions earn their place How LLM ratings compare Possible changes (AI-generated suggestions) Open questions Sources and how this was made

In the 2034 AI-lab guidelines question, two and three labs are tied as the most likely answers (22% each). The crowd spreads probability across several outcomes. This is a long-horizon indicator rather than a near-term welfare outcome.

t gives 1% to approval of Nueva Pescanova’s Canary Islands octopus farm before June 2027.

he crowd

Can someone trace a claim assessment back to the evaluator’s reasoning, and forward to the test of that concern?

sted checkAccuracy overstates forecasting; predictors measured during regrowth; use earlier data.The claim and the evaluator’s reasoning

Blocked and forward tests; back-dated and dropped predictors; each with its accuracy.

Accuracy overstates forecasting

50/100 from each evaluator, with 90% intervals; Evaluation 2’s interval revised from 30–80 to 40–60.

“Low confidence” in 215 Mha, with reasons. Nothing numerical on the accuracy claim.

measure

87.9% accuracy under the paper’s test design. Separately, 215 Mha of regeneration potential.

a research graph

77.8% within the period falls to 71.7% going forward, and accuracy also falls in new regions

There’s no formal power or equivalence test. Neighbouring pixels aren’t independent, so millions of points aren’t millions of independent observations.

Soil characteristics, which the evaluators also listed, stay in every model, including the forward one.

Other predictors could carry the same information, so dropping three doesn’t rule out leakage through others.

And A2 should forecast about as well as it does within its own period, if nothing else changes over time.

Drop productivity, burned area and road density (scored on same-period held-out points)

No change.

Forward test: train on 2000–2012, predict 2012–2024, MapBiomas labels

Result

Same period, but independent MapBiomas labels instead of the paper’s

Spatial blocks, whole Neotropics

eproduces well on random points in Colombia

or an interval for each claim.

“Low confidence in 215 Mha” is a judgment about one claim, in the evaluator’s own words.

Three kinds of number, each with its own uncertainty. None of them converts into another.

Share of sites that actually regrow: 32%

What’s real and what’s made up: the model’s hit rates are the ones Fouilloux reports for Colombia: it correctly flags 85.3% of real regrowth points and correctly rejects 92.0% of non-regrowth points

the model’s hit rates are the ones Fouilloux reports for Colombia: it correctly flags 85.3% of real regrowth points and correctly rejects 92.0% of non-regrowth points. The share of regrowth in the landscape is the part you choose; the real share isn’t given here. Numbers are expected counts per 1,000 sites.

joint synthesis. I

Recast the model as predicting future outcomes from conditions observed now.

Low. Probably a lower bound on biophysical potential, because human factors held past regrowth back.

predictor timing

Why a good test on past data can still miss future regrowth more detail A model can classify past records well and still do worse in new places or later years. Land use changes, and nearby points tend to look alike, so a random test point is often a close neighbour of a training point

(section 4).

I haven’t checked that table myself.

with our evaluation summary,

Williams et al. (2024)

The paper reports

a random-forest m

them

About 10 minutes to read.

research graph

three kinds of uncertainty

Anne’s

Both evaluators rated the paper 50/100 overall

joint report

This page shows how those pieces fit together, as material for Tuesday’s assessment exercise.

One claim, two evaluations, one replication

AI’s economic impacts and governance are priorities for The Unjournal. We’re looking for research where independent scrutiny could inform decisions about economic growth, jobs, development, power, and policy. This collection includes both economic evidence and governance proposals.

METR's time horizons (with its 2026 update) if capability measurement counts as governance evidence; otherwise the liability set (Weil, Trout, Gans on staged access). → METR time horizons, Overcoming Judgment-Proofness (Weil), When Does Regulation by Insurance Work? (Trout), Staged Access and Liability (Gans)

Working shortlist

AI interpretation of this input

Transcript, key equations and sources

Worth asking for code Survey and elicitation data; FRI has released data from earlier tournaments, so a request is reasonable.

attention research

You made 8 Keep/Maybe/Drop picks or notes here before they were folded into the rating scale. 2 of those papers have no rating yet; the rating panel shows your earlier pick and note so you can convert them.

Rate for evaluation priority

October 2026. We are choosing a handful of these papers to commission for public evaluation; nothing is decided yet. Ratings, picks and comments here feed that choice.

Why this area. AI governance

Why this area. AI governance is one of The Unjournal's stated focus areas, but we have published only one evaluation in it so far. Policy is moving fast (pacing proposals, export controls, the EU AI Act, state liability law), and the research feeding these debates is influential yet rarely gets independent expert scrutiny. We focus on governance and the economics of AI policy, not technical AI safety.

Rate this paper for evaluation priority

and it overlaps the FRI paper. Note the lab authorship.

Where the paper sits in the working list: shortlist, getting attention now, alternates, found on the dashboard, or older papers kept because they are still in use. Attention

Candidate papers on AI governance and the economics of AI policy. Working notes, not Unjournal decisions or evaluations.

Why this page exists. We are considering whether an independent evaluation of this paper would be useful. The synthesis score uses provisional weights, and the human sample is small. It is one input to the decision.

Human input and prioritization discussion

can

Latest completed updates Paper scan: Oct 5, 2026 AI scoring: Oct 5, 2026 Crux/PQ matches: Oct 5, 2026 Public feedback aggregate: Oct 5, 2026 Normal cadence Paper discovery, scoring, matching, and dashboard rebuilds normally run Monday and Thursday. Feedback aggregates refresh separately from paper discovery; see the feedback timestamp above. Feedback 25 current ratings across 20 papers. Each person's latest rating per paper counts once; discussion and display names appear only with explicit public-sharing consent. Data context 57 evaluated papers and 129 in the Unjournal pipeline. Earlier reconstructed API cost: at least ~$4.06. Search:

▾ Open details, rate or discuss

(experimental)

Is the crux description wrong? A better question phrasing or PQ mapping? Tell us here — it saves directly to The Unjournal, no GitHub needed.

reviewed

Dark mode Reset filters Share URL Quick-rate: ON + Suggest a crux Download CSV Filters (1) Reset Showing 8 of 281 cruxes Quick-rate mode on. On any row, click ▼▼ (strong downvote) · ▼ · ▲ · ▲▲ (strong upvote) to rate how valuable / decision-relevant the crux is. Votes save instantly (anonymous, no account).

Coverage by cause area — click to filter · amber = legacy AI cluster · green = Unjournal core & in-scope AI

needs review research paper targeted run: AI governance and the economics of AI policy: priority papers

Brookings

by setting concrete safety requirements for continued development and deployment, setting a speed limit on capabilities growth, monitoring high-stakes AI R&D experiments and building the option to shut them down if needed, requiring that AI R&D take place in secure and isolated environments (e.g.,air-gapped networks), establishing international incident-sharing, clarifying how deterrence would apply around an intelligence explosion so as to avoid unexpected escalation, securing international agreements to pace progress if needed without fear of falling behind, and developing verification tools in advance for such agreement

Preliminary evidence suggests that this feedback loop could radically accelerate AI progress, overcoming frictions such as diminishing returns to research labor and hard-to-automate tasks.

ome experts think that AI R&D could be fully automated within the next few years.

Anthropic reported AI systems completing 26% of internal AI R&D work with only high-level supervision, compared to just 1% five months earlier.