Annotate & Comment: Double-click any text to add a Hypothes.is annotation. No account needed to read; quick signup for a free account to post.

Even if you accept WELLBY as the target unit, the measurement layer forces choices:

layer

close to linear

Predictive validity: SWB predicts consequential outcomes systematically

Transformation Sensitivity Demo

Magnitude-sensitive cost-effectiveness: Even if signs are stable, cost-effectiveness ratios rely on magnitudes

Incremental WELLBY Estimate

What "non-identified" means A parameter is "identified" when data + assumptions pin down a unique value. Ordinal responses only tell us which interval a latent value falls into. Many different latent distributions and transformations can generate the same observed category counts, so rankings of means can change across equally admissible representations.

2. Definitions and key concepts

directly via experimental comparison.

Monotonic transformations can reverse conclusions

Bond and Lang (2019) argue that with ordinal response data, comparing "average happiness" between groups is generally not identified without strong assumptions—monotonic transformations can reverse results.[11]

Time structure and discounting Later (t>1)Follow-up (t=1)Baseline (t=0)Later (t>1)Follow-up (t=1)Baseline (t=0)#mermaid-1772847441513{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;fill:#000000;}#mermaid-1772847441513 .error-icon{fill:#552222;}#mermaid-1772847441513 .error-text{fill:#552222;stroke:#552222;}#mermaid-1772847441513 .edge-thickness-normal{stroke-width:2px;}#mermaid-1772847441513 .edge-thickness-thick{stroke-width:3.5px;}#mermaid-1772847441513 .edge-pattern-solid{stroke-dasharray:0;}#mermaid-1772847441513 .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-1772847441513 .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-1772847441513 .marker{fill:#666;stroke:#666;}#mermaid-1772847441513 .marker.cross{stroke:#666;}#mermaid-1772847441513 svg{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;}#mermaid-1772847441513 .actor{stroke:hsl(0, 0%, 83%);fill:#eee;}#mermaid-1772847441513 text.actor>tspan{fill:#333;stroke:none;}#mermaid-1772847441513 .actor-line{stroke:#666;}#mermaid-1772847441513 .messageLine0{stroke-width:1.5;stroke-dasharray:none;stroke:#333;}#mermaid-1772847441513 .messageLine1{stroke-width:1.5;stroke-dasharray:2,2;stroke:#333;}#mermaid-1772847441513 #arrowhead path{fill:#333;stroke:#333;}#mermaid-1772847441513 .sequenceNumber{fill:white;}#mermaid-1772847441513 #sequencenumber{fill:#333;}#mermaid-1772847441513 #crosshead path{fill:#333;stroke:#333;}#mermaid-1772847441513 .messageText{fill:#333;stroke:none;}#mermaid-1772847441513 .labelBox{stroke:hsl(0, 0%, 83%);fill:#eee;}#mermaid-1772847441513 .labelText,#mermaid-1772847441513 .labelText>tspan{fill:#333;stroke:none;}#mermaid-1772847441513 .loopText,#mermaid-1772847441513 .loopText>tspan{fill:#333;stroke:none;}#mermaid-1772847441513 .loopLine{stroke-width:2px;stroke-dasharray:2,2;stroke:hsl(0, 0%, 83%);fill:hsl(0, 0%, 83%);}#mermaid-1772847441513 .note{stroke:#999;fill:#666;}#mermaid-1772847441513 .noteText,#mermaid-1772847441513 .noteText>tspan{fill:#fff;stroke:none;}#mermaid-1772847441513 .activation0{fill:#f4f4f4;stroke:#666;}#mermaid-1772847441513 .activation1{fill:#f4f4f4;stroke:#666;}#mermaid-1772847441513 .activation2{fill:#f4f4f4;stroke:#666;}#mermaid-1772847441513 .actorPopupMenu{position:absolute;}#mermaid-1772847441513 .actorPopupMenuPanel{position:absolute;fill:#eee;box-shadow:0px 8px 16px 0px rgba(0,0,0,0.2);filter:drop-shadow(3px 5px 2px rgb(0 0 0 / 0.4));}#mermaid-1772847441513 .actor-man line{stroke:hsl(0, 0%, 83%);fill:#eee;}#mermaid-1772847441513 .actor-man circle,#mermaid-1772847441513 line{stroke:hsl(0, 0%, 83%);fill:#eee;stroke-width:2px;}#mermaid-1772847441513 :root{--mermaid-font-family:"trebuchet ms",verdana,arial,sans-serif;}Persistence, decay, response shift?

#mermaid-1772847441491{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;fill:#000000;}#mermaid-1772847441491 .error-icon{fill:#552222;}#mermaid-1772847441491 .error-text{fill:#552222;stroke:#552222;}#mermaid-1772847441491 .edge-thickness-normal{stroke-width:2px;}#mermaid-1772847441491 .edge-thickness-thick{stroke-width:3.5px;}#mermaid-1772847441491 .edge-pattern-solid{stroke-dasharray:0;}#mermaid-1772847441491 .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-1772847441491 .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-1772847441491 .marker{fill:#666;stroke:#666;}#mermaid-1772847441491 .marker.cross{stroke:#666;}#mermaid-1772847441491 svg{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;}#mermaid-1772847441491 .label{font-family:"trebuchet ms",verdana,arial,sans-serif;color:#000000;}#mermaid-1772847441491 .cluster-label text{fill:#333;}#mermaid-1772847441491 .cluster-label span,#mermaid-1772847441491 p{color:#333;}#mermaid-1772847441491 .label text,#mermaid-1772847441491 span,#mermaid-1772847441491 p{fill:#000000;color:#000000;}#mermaid-1772847441491 .node rect,#mermaid-1772847441491 .node circle,#mermaid-1772847441491 .node ellipse,#mermaid-1772847441491 .node polygon,#mermaid-1772847441491 .node path{fill:#eee;stroke:#999;stroke-width:1px;}#mermaid-1772847441491 .flowchart-label text{text-anchor:middle;}#mermaid-1772847441491 .node .katex path{fill:#000;stroke:#000;stroke-width:1px;}#mermaid-1772847441491 .node .label{text-align:center;}#mermaid-1772847441491 .node.clickable{cursor:pointer;}#mermaid-1772847441491 .arrowheadPath{fill:#333333;}#mermaid-1772847441491 .edgePath .path{stroke:#666;stroke-width:2.0px;}#mermaid-1772847441491 .flowchart-link{stroke:#666;fill:none;}#mermaid-1772847441491 .edgeLabel{background-color:white;text-align:center;}#mermaid-1772847441491 .edgeLabel rect{opacity:0.5;background-color:white;fill:white;}#mermaid-1772847441491 .labelBkg{background-color:rgba(255, 255, 255, 0.5);}#mermaid-1772847441491 .cluster rect{fill:hsl(0, 0%, 98.9215686275%);stroke:#707070;stroke-width:1px;}#mermaid-1772847441491 .cluster text{fill:#333;}#mermaid-1772847441491 .cluster span,#mermaid-1772847441491 p{color:#333;}#mermaid-1772847441491 div.mermaidTooltip{position:absolute;text-align:center;max-width:200px;padding:2px;font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:12px;background:hsl(-160, 0%, 93.3333333333%);border:1px solid #707070;border-radius:2px;pointer-events:none;z-index:100;}#mermaid-1772847441491 .flowchartTitleText{text-anchor:middle;font-size:18px;fill:#000000;}#mermaid-1772847441491 :root{--mermaid-font-family:"trebuchet ms",verdana,arial,sans-serif;}InterventionStudy designMeasured outcomesLS / DALY / depressionTranslation layermapping, calibrationCommon currencyWELLBY / DALY / $Decision

Cheap calibration methods: Can vignettes, anchoring questions, or other calibration approaches work in low-resource settings without excessive respondent burden?

true mapping

Scale-use heterogeneity mapping: How do shifters vs. stretchers vary across LMIC populations, and can we predict which matters more in a given context?

These questions represent high-value areas for future research that could meaningfully improve the reliability of WELLBY-based comparisons:

Within-person designs where each person serves as their own control

Treat WELLBY estimates as one input among several, not the final answer

8. Practical Recommendations

Effect size (ΔLS points)

Use mapping models (depression → LS), but carry mapping uncertainty explicitly

life satisfaction

mental health weights can be contentious

Multi-item SWB scales

Reduces scale-use bias 30-50%

DALYs and QALYs: Standardized But Narrower

DALY / QALY

Years of Life Lost (YLL) + Years Lived with Disability (YLD

It does not automatically imply that within-study randomized treatment effects are meaningless It implies you should be explicit about what assumptions let you treat reported changes as welfare units

OECD (2024) concludes data remain meaningful for policy despite critiques

Survey response times can help solve identification (Liu & Netzer, AER 2023)

latent distribution

Strong assumptions needed to treat "0.3 points" as "0.3 welfare units"

mean comparisons non-identified

A strong response to skepticism: even if the numbers seem arbitrary, do they behave like a measurement? Kaiser and Oswald show that single numeric feelings responses have strong predictive power—relationships to later "get-me-out-of-here" actions (changing neighborhoods, jobs, partners) tend to be replicable and close to linear in large longitudinal datasets.[10]

They do not solve cross-study comparability—but demonstrate that in at least one setting, SWB is responsive.

Noise inflates uncertainty—credible estimates need adequate sample sizes and good designs

Measurement error attenuates estimated effects (bias toward zero)—small real effects may be undervalued

is meaningful

LS misses physical suffering or other channels.

What breaks: Duration weighting is wrong. Why it might fail: Adaptation effects—people return to baseline. Mitigation: Long-term follow-up data.

ΔLS has ≈ same welfare meaning across people

ΔU(3→4) = ΔU(7→8)

Validity

Test

Summing is invalid;

Ceiling/floor effects: Even with identical reporting functions, bounded scales can cause mechanical differences in responsiveness at high or low baselines.

Comparing across studies/countries: Different instruments, translations, norms, and populations. If the distribution of stretch factors bi differs, "1 point-year" is not the same welfare unit across the evidence base.

instrument

interpersonal noncomparability is less of a threat for estimating an average treatment effect

reporting functions

studies, countries, or populations with different distributions of "stretch factors.

Δui = bi × ΔLSi.

ai

UA ≈ UB

Incremental

A common overstatement is that

comparisons

This second form requires a defined zero point (e.g., death = 0)

Σi Σt δt (LSit(k) − LSit(0))

ΔWELLBY(k) = Σi Σt δt (LSit(k) − LSit(0))

Benjamin et al. 2023, UK Green Book Wellbeing Guidance, Bond & Lang 2019, Haushofer & Shapiro 2016/2018, Kaiser & Oswald 2022)

AI-Generated Content (March 2025): This page was created through iterative prompting of Claude Code (Opus 4.5) and GPT-5.2 Pro, feeding in workshop discussion content and focal papers for our Pivotal Questions initiative (Benjamin et al. 2023, UK Green Book Wellbeing Guidance, Bond & Lang 2019, Haushofer & Shapiro 2016/2018, Kaiser & Oswald 2022). While grounded in these sources, this content requires further human verification. Specific claims, citations, and numerical details should be checked against the original literature before relying on them.

ng "average happiness"

The key critique

biasing

not a corner case.

MH

Most studies measure outcomes at baseline and one or two follow-ups;

cancels

The measurement-to-decision pipeline #mermaid-1772846605552{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;fill:#000000;}#mermaid-1772846605552 .error-icon{fill:#552222;}#mermaid-1772846605552 .error-text{fill:#552222;stroke:#552222;}#mermaid-1772846605552 .edge-thickness-normal{stroke-width:2px;}#mermaid-1772846605552 .edge-thickness-thick{stroke-width:3.5px;}#mermaid-1772846605552 .edge-pattern-solid{stroke-dasharray:0;}#mermaid-1772846605552 .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-1772846605552 .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-1772846605552 .marker{fill:#666;stroke:#666;}#mermaid-1772846605552 .marker.cross{stroke:#666;}#mermaid-1772846605552 svg{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;}#mermaid-1772846605552 .label{font-family:"trebuchet ms",verdana,arial,sans-serif;color:#000000;}#mermaid-1772846605552 .cluster-label text{fill:#333;}#mermaid-1772846605552 .cluster-label span,#mermaid-1772846605552 p{color:#333;}#mermaid-1772846605552 .label text,#mermaid-1772846605552 span,#mermaid-1772846605552 p{fill:#000000;color:#000000;}#mermaid-1772846605552 .node rect,#mermaid-1772846605552 .node circle,#mermaid-1772846605552 .node ellipse,#mermaid-1772846605552 .node polygon,#mermaid-1772846605552 .node path{fill:#eee;stroke:#999;stroke-width:1px;}#mermaid-1772846605552 .flowchart-label text{text-anchor:middle;}#mermaid-1772846605552 .node .katex path{fill:#000;stroke:#000;stroke-width:1px;}#mermaid-1772846605552 .node .label{text-align:center;}#mermaid-1772846605552 .node.clickable{cursor:pointer;}#mermaid-1772846605552 .arrowheadPath{fill:#333333;}#mermaid-1772846605552 .edgePath .path{stroke:#666;stroke-width:2.0px;}#mermaid-1772846605552 .flowchart-link{stroke:#666;fill:none;}#mermaid-1772846605552 .edgeLabel{background-color:white;text-align:center;}#mermaid-1772846605552 .edgeLabel rect{opacity:0.5;background-color:white;fill:white;}#mermaid-1772846605552 .labelBkg{background-color:rgba(255, 255, 255, 0.5);}#mermaid-1772846605552 .cluster rect{fill:hsl(0, 0%, 98.9215686275%);stroke:#707070;stroke-width:1px;}#mermaid-1772846605552 .cluster text{fill:#333;}#mermaid-1772846605552 .cluster span,#mermaid-1772846605552 p{color:#333;}#mermaid-1772846605552 div.mermaidTooltip{position:absolute;text-align:center;max-width:200px;padding:2px;font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:12px;background:hsl(-160, 0%, 93.3333333333%);border:1px solid #707070;border-radius:2px;pointer-events:none;z-index:100;}#mermaid-1772846605552 .flowchartTitleText{text-anchor:middle;font-size:18px;fill:#000000;}#mermaid-1772846605552 :root{--mermaid-font-family:"trebuchet ms",verdana,arial,sans-serif;}

Some influential critiques argue that different monotone transformations can reverse conclusions about "average happiness"

Is "incremental WELLBY" standard terminology? Some literatures talk about WELLBYs as point-years of life satisfaction (UK guidance) and many evaluation contexts are inherently incremental. But "incremental WELLBY" itself is not uniformly a standard term. In this page, we use it as a descriptive label for counterfactual impact calculation, not as established jargon.

WELLBY (unit of account): UK Green Book guidance defines a WELLBY as a one-point change in life satisfaction on a 0-10 scale, per person per year.[3]HM Treasury (2021/2024). Wellbeing Guidance for Appraisal: Supplementary Green Book Guidance.

many workshop annotations objected to undefined symbols and unclear indexing.

The measurement-to-decision pipeline #mermaid-1772845759179{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;fill:#000000;}#mermaid-1772845759179 .error-icon{fill:#552222;}#mermaid-1772845759179 .error-text{fill:#552222;stroke:#552222;}#mermaid-1772845759179 .edge-thickness-normal{stroke-width:2px;}#mermaid-1772845759179 .edge-thickness-thick{stroke-width:3.5px;}#mermaid-1772845759179 .edge-pattern-solid{stroke-dasharray:0;}#mermaid-1772845759179 .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-1772845759179 .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-1772845759179 .marker{fill:#666;stroke:#666;}#mermaid-1772845759179 .marker.cross{stroke:#666;}#mermaid-1772845759179 svg{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;}#mermaid-1772845759179 .label{font-family:"trebuchet ms",verdana,arial,sans-serif;color:#000000;}#mermaid-1772845759179 .cluster-label text{fill:#333;}#mermaid-1772845759179 .cluster-label span,#mermaid-1772845759179 p{color:#333;}#mermaid-1772845759179 .label text,#mermaid-1772845759179 span,#mermaid-1772845759179 p{fill:#000000;color:#000000;}#mermaid-1772845759179 .node rect,#mermaid-1772845759179 .node circle,#mermaid-1772845759179 .node ellipse,#mermaid-1772845759179 .node polygon,#mermaid-1772845759179 .node path{fill:#eee;stroke:#999;stroke-width:1px;}#mermaid-1772845759179 .flowchart-label text{text-anchor:middle;}#mermaid-1772845759179 .node .katex path{fill:#000;stroke:#000;stroke-width:1px;}#mermaid-1772845759179 .node .label{text-align:center;}#mermaid-1772845759179 .node.clickable{cursor:pointer;}#mermaid-1772845759179 .arrowheadPath{fill:#333333;}#mermaid-1772845759179 .edgePath .path{stroke:#666;stroke-width:2.0px;}#mermaid-1772845759179 .flowchart-link{stroke:#666;fill:none;}#mermaid-1772845759179 .edgeLabel{background-color:white;text-align:center;}#mermaid-1772845759179 .edgeLabel rect{opacity:0.5;background-color:white;fill:white;}#mermaid-1772845759179 .labelBkg{background-color:rgba(255, 255, 255, 0.5);}#mermaid-1772845759179 .cluster rect{fill:hsl(0, 0%, 98.9215686275%);stroke:#707070;stroke-width:1px;}#mermaid-1772845759179 .cluster text{fill:#333;}#mermaid-1772845759179 .cluster span,#mermaid-1772845759179 p{color:#333;}#mermaid-1772845759179 div.mermaidTooltip{position:absolute;text-align:center;max-width:200px;padding:2px;font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:12px;background:hsl(-160, 0%, 93.3333333333%);border:1px solid #707070;border-radius:2px;pointer-events:none;z-index:100;}#mermaid-1772845759179 .flowchartTitleText{text-anchor:middle;font-size:18px;fill:#000000;}#mermaid-1772845759179 :root{--mermaid-font-family:"trebuchet ms",verdana,arial,sans-serif;}InterventionStudy designMeasured outcomesLS / DALY / depression scaleTranslation layermapping, calibration, assumptionsCommon currencyWELLBY / DALY / $Decision / deliberation

interventions

This workshop's focal question is not "which countries are happier.

Plant, M. (2025). "A Happy Possibility: Rational Behavior and the Cardinality Thesis." Working paper.

entire

f you compare to mortality-preventing interventions

ΔWELLBY(k) = Σi Σt δt (LSit(k) − LSit(0))

Level-based WELLBYs (for mortality comparisons):

📊 View Aggregated Results See beliefs elicitation summaries and Metaculus question forecasts

discussion. SEGMENT 1 ~11:00 AM ET 25 min Stakeholder Problem Statement & Pivotal Questions Stakeholders present their WELLBY/DALY challenges (~10 min each), then we introduce key PQs for belief elicitation (~5 min) Speakers: Peter Hickman (Coefficient Giving), Matt Lerner (Founders Pledge) Upcoming

SEGMENT 6 15 min Beliefs Elicitation Guided form to state priors on operationalized pivotal questions Self-guided form + Metaculus questions Upcoming

SEGMENT 7 ~2:20 PM ET 30 min Practitioner Panel & Open Discussion Practical implications for funders and researchers Panelists: Matt Lerner (FP), Peter Hickman (CG)

Scale-use heterogeneity findings, calibration methods, and implications for WELLBY use Speakers: Dan Benjamin (UCLA/NBER), Miles Kimball (CU Boulder)

Evaluator Responses & Discussion Evaluation findings, author dialogue, Unjournal process Presenters: Caspar Kaiser, David Reinstein, Valentin Klotzbücher

html`<div style="background: #f8f9fa; padding: 1rem 1.25rem; border-left: 4px solid #3498db; margin-bottom: 1.5rem; font-size: 0.95em; line-height: 1.6;"> <strong>What these numbers represent:</strong> Simulated <strong>production cost per kilogram of cultured chicken</strong> (wet weight, unprocessed) in <strong>${targetYear}</strong>, based on ${stats.n.toLocaleString()} Monte Carlo simulations. This is the cost to produce meat in a bioreactor — not retail price, which would include processing, distribution, and margins. <br><br> <strong>Why it matters:</strong> If production costs reach <strong>~$10/kg</strong> (comparable to conventional chicken), cultured meat could compete at scale. If costs remain <strong>>$50/kg</strong>, the technology may remain niche. These thresholds inform whether animal welfare interventions should prioritize supporting this industry. </div>` RuntimeError: targetYear is not definedOJS Runtime Error (line 804, column 163) targetYear is not defined

Add questions and comments directly to the collaborative notes above, or submit them via the beliefs elicitation form.

decision problems that funders face when comparing interventions measured in different units (WELLBYs vs DALYs)

How funders currently navigate WELLBY vs DALY in cost-effectiveness analysis

If the effectiveness of some programs have already been measured in terms of WELLBYs, while others are measured in terms of DALYs, what method or what "mapping structure or approach" should we use to compare and convert between them?

These are some of the key operationalized questions from our Wellbeing Pivotal Questions project. We want to elicit expert and stakeholder beliefs—before, during, and after reviewing the evidence and key arguments—to see how views evolve and where consensus exists. (All questions are optional.)

📋 Full question specifications: For more detail, context, and the complete set of operationalized questions, see the canonical Wellbeing PQ formulations on Coda →

About You

One way to think about this: Imagine an ideal research team with unlimited resources, time, and data—perhaps even a kind of omniscience where they could perfectly understand the welfare and psychological states of everyone affected. What probability would you assign that this idealized team would ultimately conclude the statement is true?

DALY_01 What is the best numerical conversion factor between WELLBYs and DALYs? If a charity prevents 1 DALY (Disability-Adjusted Life Year), approximately how many WELLBYs does this represent? Current estimates range from 2-15. View on Metaculus

Brief context on the Unjournal evaluation process

Overview Statistics

—a “think first, score second” protocol designed to ground numeric ratings in specific textual evidence

We pass the PDF directly to the model’s native multimodal input rather than extracting text, preserving tables, figures, equations, and layout cues that ad-hoc scraping could mangle. A single API call per paper avoids hand-offs and summary loss from m

Sample and human reference data.

development economics, health policy, environmental economics,

models’ training data may include fragments of these papers or related discussions.

while we block extrinsic priors

fields

evaluation

Discussion

Qualitative coverage varies widely across papers: on some, the LLM captures nearly all consensus human concerns; on others, it misses key critiques or raises issues absent from the expert consensus

Central compression—the tendency for LLMs to pull extreme ratings toward the middle of the scale

respectably

, approaching the ceiling set by inter-rater variability among humans themselves.

(likely Founders Pledge)

Confirmed: Monday, March 16, 2026 · 11am–5pm ET / 4pm–10pm UK · Fully online · ~3.5 hours of live sessions (join only the segments you're interested in) + asynchronous

Your primary role in this conversation (optional)

Which Potential Segments Interest You?

Focal Question (DALY_01) If the impact of one program is measured in WELLBYs and another program impact is measured in DALYs, and we have a reported effect size and standard deviation for each, what is the best numerical conversion or mapping between them? Note: This may be treated as a secondary topic depending on time constraints.

💬 Questions & Comments Submit questions and comments through the form below. Note: Submissions are publicly visible to all participants.

expert

that experts can state their beliefs q

Proposed segments (unfold)

ogether the paper's authors, the evaluators who assessed it,

differences

GPT-5 Pro ranks higher; purple (right) = Claude Opus ranks higher.

Each column shows rank order (1 = highest rated) within that source;

r n_papers_slope-

Rank comparison. Figure 2.2 places human rankings

matched sample

(explaining the slightly different ρ values).

scale relative to humans, producing narrower spread

ge but diverge more at the extremes.

evaluator pairs is no tighter than the LLM-human scatter in panel

Compare panels (b) and (c) directly to see whether LLM-human scatter is tighter than human-human scatter.

evaluator pair (papers with 3 raters contribute 3 points); uses all r n_matched

Figure 2.1:

Results CodeShow All CodeHide All CodeView Source Setup and libraries

) ratings for each paper, revealing inter-rater variability—CIs often span 20–40~points.

In most cases both LLMs fall within the range of human opinions, though several papers show substantial divergence.

Per-paper overview and model comparison. Figure 2.1 presents three complementary views of overall (0–100 percentile) ratings. Panel (a) displays individual human evaluator ratings alongside GPT-5 Pro (orange diamo

evaluator ratings

Per-paper overview and model comparison. Figure 2.1 presents three

We evaluate 6 frontier LLMs against human expert reviews from The Unjournal.

Results CodeShow All CodeHide All CodeView Source

criterion-level ceiling.

If two human evaluators agree at Spearman ρ = 0.55, an LLM achieving ρ = 0.57 against the human mean is performing within human inter-rater range.

severity, topic familiarity, interpretation of the scale)

33 of these were also evaluated by Claude Opus 4.6

six frontier LLMs

Funding for The Unjournal has been provided by the Survival and Flourishing Fund, the Long Term Future Fund, and EA Funds.

The Unjournal setting is particularly well suited for this comparison. It commissions paid expert evaluations using a structured rubric covering seven percentile criteria with 90% credible intervals plus journal-tier predictions, and publishes the resulting packages openly

strain

Our headline finding is that the best-performing model (GPT-5 Pro) matches or exceeds pairwise human inter-rater rank agreement on overall quality,

while the journal-tier predictions provide an external reference point2

reducing classic gatekeeping motives and increasing reviewer effort.

structured measurement schemas (Asirvatham, Mokski, and Shleifer 2026), iterative quality-checking workflows (Zhang and Abernethy 2025), or the kind of prompt-robustness engineering motivated by specification-search concerns (Asher et al. 2026)—should improve further.

can achieve

with no iteration, retrieval augmentation, chain-of-thought scaffolding, or multi-step agentic loop.

deliberately

deliberately

strong case that frontier LLMs can serve as additional expert raters in structured evaluation pipelines,

Our headline finding is that the best-performing model (GPT-5 Pro) matches or exceeds pairwise human inter-rater rank agreement on overall quality,

against expert evaluations for 60 economics and social-science working papers

multi-dimensional

These developments make the evidentiary gap salient: funders, editors, and policymakers need to know when AI evaluation outputs are trustworthy enough to use, and when they are unstable, biased, or manipulable. Recent work highlights all three concerns. First, reproducibility can be “jagged”: repeated runs of the same models on the same corpus over time can be highly consistent for some tasks and models, but much less so for others (Thomas, Romasanta, and Pujol Priego 2026); robustness may require separating scientific judgment from computational execution (Xu and Yang 2026); and even without overt adversarial intent, subtle reframings of the same task can induce systematic shifts in outputs—a form of LLM “specification search”—raising concerns about frame-sensitive biases when models serve as measurement instruments (Asher et al. 2026). Second, adversarial manipulation is not hypothetical: invisible-text “prompt injection” can substantially inflate LLM-assigned review scores and acceptance recommendations in simulated peer review (Choi et al. 2026), and prompt-injection vulnerabilities are also documented in other high-stakes advice settings (Lee et al. 2025). Third, even when outputs look fluent and plausible, it remains unclear whether AI models approximate expert judgment: AI-generated reviews tend to cover more surface-level sections while being less thematically diverse and less focused on interpretation, originality, and applicability than human reviews (Rajakumar et al. 2026); LLMs used as manuscript quality checkers identify only a small fraction of confirmed critical errors even with the strongest reasoning models (Zhang and Abernethy 2025); and LLM scoring exhibits systematic range restriction and halo effects that can distort agreement metrics (Wang et al. 2025).

Meanwhile, publishers are formalizing policies that treat manuscripts and reviews as confidential and prohibit reviewers from uploading them into general-purpose generative AI tools

under strain

how much do plant-based products actually substitute for animal products?

irr::kripp.alpha(M, method = "interval")$value }, error = function(e) NA_real_)

0.07

α_HL (GPT-5.2 Pro)

Table A.3: Krippendorff’s αHH

At 20 kTA reference scale

CAPEX scaling

Scalable GF technology 50% Switches to “cheap” GF prices Pivotal uncertaint

yments:

The slider complements this by letting users explore “what if progress is partial?” scenarios.

If any one of these succeeds at commercial scale, the “cheap” price regime applies

$0.20 – $1.20

Consider correlated scenarios via the maturity slider

Use for relative comparisons rather than absolute predictions

Equipment depreciation period

Weighted average cost of capital

Breakthrough technologies that could trigger the “cheap” scenario: - Autocrine cell lines (cells produce own FGF2) - Plant molecular farming ($1-10/g target) - Precision fermentation at scale - Polyphenol substitution (reduces GF requirements by 80%)

30-200 g/L Final biomass at harvest Cycle time 0.5-5 days Time per production batch Media turnover 1-10 ratio 1=batch, >1=perfusion

Distribution

Why correlate? In “good worlds” for cultured chicken: - Technologies are more likely adopted (higher P) - Custom reactors are more common (lower CAPEX) - Financing is cheaper (lower WACC) This prevents unrealistic scenarios where technology succeeds but financing remains prohibitively expensive.

The model uses a latent maturity factor (0–1) to correlate technology adoption, reactor costs, and financing: Padopted=bound(Pbase+k⋅(m−0.5),0,1) What does “bound” mean? bound(x, 0, 1) ensures the result stays between 0 and 1. Also c

The GF progress slider interpolates between current and target prices: PGF=Pcurrent×(0.01)progress At 0% progress: current prices ($5,000–500,000/g) At 100% progress: target prices ($1–100/g for cheap scenario)

Example calculation: - Cell density: 50 g/L → need 1000/50 = 20 L per kg - Media turnover: 3× (perfusion system) → 20 × 3 = 60 L/kg - Media price: $0.50/L (hydrolysates) → 60 × 0.50 = $30/kg

uld the resulting adjustments change the cost-effectiven

WELLBY is a reasonably useful measure in this context

Given the available collected data [...], how should [funders] measure the impact on wellbeing? [...] What measures of well-being should charities, NGOs, and RCTs collect for impact analysis?

How reliable is the WELLBY measure [...] relative to other available measures in the 'wellbeing space'? How much insight is lost by using WELLBY and when will it steer us wrong?

More detailed questions on WELLBY reliability

"Meaningful change" = at least one intervention currently in the top 5 moves out of the top 5, OR the #1 ranked intervention changes. This assumes future RCTs incorporate these methods and Founders Pledge updates their CEA accordingly.