Scores combine AI estimates with human ratings where available
Should they do that even if I select "AI ratings only" in the Rating source option above?
6 Matching Annotations
- Sep 2026
-
uj-prioritization-dashboard.netlify.app uj-prioritization-dashboard.netlify.app
-
-
quick votes map from Strong No through Strong Yes to 10, 30, 50, 70, and 90
My sense is that it might be clearer just to show the numbers for quick votes (rather than the signs that may be read differently by people who don't read instructions carefully) (?)
-
What is this score?
I think this is called "Score weights" now?
-
Calibration and stability answer different questions. Calibration asks whether AI ratings line up with human judgments. Stability asks whether the same papers keep similar ratings when the model is run again or a substantively equivalent instruction is reworded. A system can be stable but wrong, so stability is a reproducibility diagnostic rather than an accuracy claim. Our pilot covers 24 selected papers with 192 repeated ratings and remains exploratory; it is not a full-dashboard reliability estimate.
I might misunderstand this, but should there be an assessment of prompt and run stability (rather than just an explanation of what it means)?
-
Show planned workflow
Minor: Is there any reason why this isn't formatted in the toggle-style used for other expandable sections on the site?
-
add your judgment
Maybe specify that this is a judgement of relevance (rather than of evidence quality/reliability)?
-