Where is the 'pre-selection into evaluation' probability in this diagra (Po)

what does 'adverse record' mean below? I guess we are talking about the evaluation rating here?

interesting but you haven't really connected this to the discussion or model. Why does evaluator anonymity matter to the author's decision?

also, as with these signaling games, there's usually multl equilibria, including a sort of 'babbling' one iirc

this section is not written well. As far as I know only the things linked to Propositions 1 and 2 even engage the model ... you can mention other possible costs and benefits not considered by the model, which are largely empirical questions

I don't see how this is a 'result' -- it's just an obvious construction

simplfy this ... that's obviously an empirical issue.

link and tooltip "Proposition 2" -- it hasn't been introduced yet

'backs' or 'focuses on'

what 'third category'?

remove 'not formal publication' -- already noted, but the 'formal publication' is not a benefit per se

'downside risk'

The diagram below is underexplained. Also note we geve evaluators the opportunity to revise if the author points out clear omissions or errors

this needs clarification or a tooltip. I think you are talking about the 'conditional embargos'?

The probability of why should the probability of each of these be the same? That's weird and seems very limiting to me. The model should be made more general.

Use variable names for these also.

This was the first time that you mentioned the variable y - you need to define it better.

This could be fleshed out more carefully and slowly with more explanation.

encouraging authors to respond, and evaluators to update their evaluations if authors find clear mistakes or oversight.

Tooltip: We're also working to build and coach our evaluator pool, and hope to provide paid calibration workshops in future.

"The Unjournal works to limit the noise component through..."

is accuracy a 'false negative and positive rate' or what -- give a tooltip on what q means

Flesh out -- why would this be a particular concern for public evaluation? (Explain briefly, tooltip a longer explanatino)

wait -- 'concern with what', 'relucatance to do what'? Needs clarification, seems somewhat inconsistnet

link please, and give details in tooltips

this is a bit too stark. I don't know if we can be sure that UJ fully changes these situations, nor whether in the 'risky' cases the risk is substantial. State it more tentatively or diretionally

'publication tier bar'

sklip 'visa'

We discuss a series of reasonable considerations

and, evaluator updates, and evaluation manager synthesis.

quantified evaluation and a detailed report

'mistakenly being connected'

Should we also mention the 'submitting your work for public evaluation can also be a strong signal of your confidence in the credibility of your work?'

LaTeX is not really rendered in the diagram below for "Bar C", and also it's not clear what these variables are that need better labeling.

"I don't like 'read it as a working decision aid' so much" -- it's a bit of this, but also a bit of a 'here's how to think about this' discussion document. You can say 'this does not represent The Unjournal's official position or advice' and link our Author FAQ https://globalimpact.gitbook.io/the-unjournal-project-and-communication-space/faq-interaction/for-researchers-authors for the latter. But also avoid the 'not this but that' typical AI language. These are 2 separate things

why 'seminar-defensible' -- this could confuse things, people will say "why not present it at seminars"? Also, you mean for such a paper; this could be interpreted as public (UJ) evaluations will make it seminar defensible

Can we have a 'separating equilibrium' or other image here. Let's focus this slide on the "willing to recieve and respond to public criticism signals research strength' part ... and then the 'immediate underdog benefit thing' is the next slide (which you can tease)

use the space on this slide bette

1st and second box ... thought bubbles or callouts (0-100 percentile relative to pool); Both suggestor and assessor writes a motivating explanation/discussion.

"Whole team votes" -- 5 point approval scale (strong/weak/neutral)

the text on this 'all green slide' is a bit hard to read. Make it more readable and clear.

prioritization and evaluation outputs

and usefullness, providing multodimensional ratings and discussion -- not just "which venue published it".

space this slide out more

add a vertical space before this sentence

in the prioritized pool

I think I wanted more images, more faces, here or below. I had a comment about that

remove the disclaimer

center the image.

adjust these examples to have some with wider and others w narrower bands

leave this off; that's not quite what we ask, we ask about implications

--> with quantified uncertainty

This bit is cut off at the bottom of the page. Need to use the vertical space a little more conservatively here. Have smaller fonts or more use of the horizontal space.

still not embedding

Add image of our Google Scholar visibility here : https://scholar.google.com/scholar?hl=en&as_sdt=0%2C5&q=source%3Aunjournal&btnG= as one example

Obviously, journals never teach this. Also, our research education system doesn't tend to teach this. But it's also not clear that this alone is something that has a lot of career value. It's the methodology, theory, context, etc. that has the career value, as it allows you to do better work.

I don't know that I would emphasize this quite so much. First of all, this only (mainly) holds if you choose to sign your reviews, which not everyone wants to do, particularly early in their career . But also, it does show some value and helps you demonstrate your understanding, but I don't think that the profession rewards refereeing and reviewing quite so much.

A nice thing is you will get feedback on your evaluation from us and potentially from the authors, which will help you improve and learn.

I think this is missing some of the key benefits here. Our public evaluations help you understand what issues other economists care about, as well as, to some extent practitioners, funders and people interested in impact. It's a methodological discussion that will help your own work, as well as help you understand the ways to engage in the peer review process .

This also helps make you part of a conversation involving funders, grantmakers, and people that might be able to help your career and help you have more impact.

This is a bit vague. I'm not sure about this wording here.

I can see something like "sensitive career moments + Work that likely passes the bar" Being a situation where you wouldn't want to have this sort of public criticism or these additional signals. But it depends. If you're at a sensitive career moment but you think you're coming up just below the bar, or you are being systematically undervalued, then it might be helpful to have these additional signals. And if making a commitment to public evaluation itself becomes a positive signal in a screening, sorting equilibrium, you will definitely want to do it in such a situation.

Ask who? What am I supposed to say about this?

I don't see this as clearly argued - The Unjournal isn't going to give these sorts of critiques in a way that I think will be harmful, and I don't want people to keep thinking that public criticism is somehow a deeply harmful thing.

Not sure about mentioning "co-author downside" - rephrase this somehow.

I'm not sure "test" is the right word. It's more like a commitment and a signal.

This seems a bit small, not using the whole page. ?

And Pivotal Questions Advisor

The Metaculus thing is a bit separate now. Go back to the Pivotal questions knowledge base and update on this.

Hyperlink. https://info.unjournal.org/pivotal-questions.html here

Hyperlink the workshops here.

Those aren't the partners on this. The partners on this include Founders Pledge and Animal Charity Evaluators. People from Coefficient Giving and many other organizations have participated in our workshops on these pivotal questions as well.

I think the earlier version of this page was better. It's a little bit confusing because you're talking about evaluators, but you're showing the management team and advisory board here.

I don't like the "plus" here. The substantive reports are equally important, if not more important, than the ratings.

This benchmarking is necessary because we don't quote, accept, or reject, and we don't have a journal tier. So this is the path to Unjournal evaluations being something that has career value as well as value for research users.

This doesn't quite make sense. First of all, we don't publish the research; we publish evaluations. Second of all, where are you getting the 330 figure? Maybe leave this off.

That's not quite getting at the right thing because we actually do favor research that is more well known, as it's likely to be more influential. What we don't favor is research that is just simply seen as deeply intellectually interesting or clever

It shows up as too small on the slide. Can we rearrange it somehow? Maybe make it horizontal?

"lands"

can be

'positive' is more about their work and not about the evaluation ... look for better feedback, including from authors

the "180 evaluators" don't stand behind every evaluation ... this doesn't make sense. We have usually 2 evaluators per evaluation. Also mention the field specialist counts too

incorporate image/text of one of the award-winning evaluations here (or as a bonus vertical slide)

ilustration is OK but it's a bit too 'obvious' -- it should note that a signal could go in either direction ... perhaps should be nested in a graph considering both internal and external signals ... and illstrate cases where the expected value is positive

what about the case that 'willingness to make all work available for evaluation could be a strong signal of your confidence and credibility'?

More like, "How does AI evaluation compare to humans?"

And I'd frame this more as an open question, one we're exploring, but at the moment the general attitude seems to be that there needs to be a human in the loop, at the very least, making the final judgment calls, prioritization, and communication

I'd add things like: - How could this invigorate teaching and research training? - How could it help with building agendas, attracting funding, and demonstrating value for exercises like the REF?

This is perhaps the most important question here - maybe put this one first. ... What would make it reliable, meaningful, and valued?

Not quote in economics. That's asking too much. Leave the last bit out, but presumably they'll understand that we're asking about what would be useful to them.

This slide should have the header "Pivotal questions project" Not

We don't have such strong evidence on this to say it's not a line enough to substitute, to be honest. We only have one trial that we attempted. This slide overstates things, and it would be better to have it link and show some of our output, just so people know what was done in our trial.

This seems like a claim, but what backs it up?

"efficient and transparent evaluation" ... "nd connection to 'real stakeholders and impact'"

Generated by AI tools, which may or may not actually be correct or useful

Those are not the relevant partners. The relevant partners are: - Founders Pledge - Animal Charity Evaluators We've had participants in workshops from Coefficient Giving and many other organizations..

Metaculus is not really at the center of this at the moment. ... It's more on our own pages and platforms. https://uj-wellbeing-workshop.netlify.app/beliefs the Metaculus thing is sort of an extension.

shoulf hyperlink https://info.unjournal.org/pivotal-questions.html

Make this bit larger and more visible. Maybe include some of the visuals on the composition of the evaluator pool.

I don't think labor markets get at it here. We're talking about the impact of transformative technological change on labor markets, not labor markets on their own.

Accepted after substantial revisions ... I don't know why we would use the word "certified" here.

At best

Versus a traditional journal

Add the image of the fish banding together on this slide. You can find that on the previous PowerPoint/Google slide presentations.

Add "funding and grantmaker incentives will help".

And maybe replace this with "solving the coordination problem".

Also a bullet about how we're making ourselves prominent in the ecosystem so that the evaluations and ratings will be seen before the paper is reviewed by conventional journals.

Why is this quote here? It doesn't seem to relate to the slide.

Hard to see the diagram below. Can we make it more visible somehow? Larger or horizontal, or allow us to zoom in?

Feel free to hyperlink these.

Skip this slogan. It's not just about speed and timing.

And they want more feedback then simply "which journal published it" ?

Make it clear we're a non-profit and we focus on research with potential for global impact.

Publicly Hosted

What do we mean by "decoupled from gatekeeping"? I'm not sure about that one.

Use a colon here, not an em dash

You can publish in a journal too

Add some more vertical slides with the other costs of the existing system/benefits of separating evaluation from "publication", And making the evaluation public

Why is this in bold?

It doesn't really govern dissemination, as noted in the other bullet point... It governs careers and research credibility.

These green slides are not so visually compelling. The text is small, the numbering is not particularly helpful, and there's no image or anything that makes it seem interesting.

That's nice, but I also want to emphasize the value that we're providing in the medium term.

What was meant by "pieces" here?

Recruit an evaluation manager and ...

~2

Use the word 'research' here, not work.

As a team

And evaluators can adjust to this response.

And an evaluation manager summary/synthesis

I don't think this is the right diagram. I think we want the other one illustrating just what the process has been ... People on the team suggest it, give it a rating for prioritization/potential for impact, the whole team votes on it, we finalize it, and liaise with the authors, etc. This is just a diagram about a particular way that we do or do not consider certain things in doing this prioritization.

Drop this. This is obviously already a point being made.

I don't know which building I'm going to be in. Maybe say on this campus or something like that.

"here" Is vague.

Hyperlink the workshop here. Both of them

Research leadership: ...

colore on black text here is very hard to read

Would not attend live but offered to follow up asynchronously.

Fix this. It takes too long to generate.

how many use it now -- put this in context

check and annotate -- what does e.g., 5 wellbys per DALY mean in context, and how does it compare with what people currently do?

This seems like a valid objection, but I think we still phrase the question such that you would give a meaningful answer, or you could give a meaningful answer in this case in terms of the value generated if you were forced to use a single conversion.

The discussion might be more valuable, or I would say is likely to be more valuable than the response, particularly for this question.

let's do better to differentiate this from calibrated well-being. It's not fully clear to someone glancing at this briefly.

this is not a 'subquestion' ... label this page header differently

Add backlink to the workshop and to the elicitation form here.

Make sure you can access the literal question from this interface to know exactly what the respondents are answering. If these things get very long, you can use tooltips.

some visulaizations (bar or donut or mosaic graph) would be helpful here

I don't htink this is a stochastic model? Perhaps an extension of this should give these (correlated?) distributions. ... Squiggle-type modeling

There should also be discplay of the actual equation behind the model, and a folding box or linked page explaining it in more detail

Maybe put this model at the top, after a short preamble?

any prediction markets deal with the fundraising/donation/nonprofit askpets of this?

why? is this just a truism?

this advice seems on the overly generic side?

what is this -- things like this need definitions, in the text with a link and in tooltips

title font too big, taking upd too much screen saace

I'll try to respond / adapt to hypothes.is comments, especially if you flag @daaronr

Unjournal data

There's some text overlap in the diagram.

what 'costs of negative findings'? This is a bit of a vague sentence.

also state and interpret the equation in terms of 'benefits vs cost' -- easier to interpret I think

remove 'or policy relevance' perhaps -- The Unjournal prioritizes research with global impact potential (although that's not what we mainly rate the research on)

not sure I understand the logic behind the latter part

how would someone know this?

This is a possible set of considerations -- I don't want to state it as definitive

maybe the 'gatekeeper' or 'editor' instead of the 'reader'?

Is it a truly 'public' signal, or only one the evluator sees after reading the paper? Or should we have both?

notation needs improvement, and it should be explained more -- how derived, how to interpret it? Tooltips and expanding sections could help

I suspect another paper has dealt with this question ... 'when noisy signals help the seller' or some such

this needs clarification, I don't quite see why this is the case. Isn't it possible that the author's signal is positive so they submit, but the evaluator reading the paper gets a negative signal?

relevant to whom?

what is 'acts' and what is 'c' -- underexplained

is this 'binary' rhe relevant threshold? Where did it com from? is it sort of generalizable? Consider if it misses some important nuance

who is the 'reader'? I think we mean the later journal evaluator, or career gatekeeper

base

simple

why?

this should be a fold or footnote -- give a quick statistic and footnote yow it was captured

too strong. And takes the 'high dowside' idea for granted too much IMO

too much 'not this but that' AI speak. And 'publicity' is vague here

too much use of bold within paragraphs and prose

this needs further explanation and clarification -- 'usual channels' should already encompass clarification

this is the 'AI language of dichotomy' overused

The language of this is a bit unclear. Try to make it easier to understand.

Don't use bold here.

this is meta -- I don't want meta, or at least put that into an 'opt-in' list

We can swap in here the legal prioritization prototype -- https://uj-prioritization-prototype.netlify.app/legal/ -- please do this -- and note that we're looking for feedback and examples to help improve and train this. Note that we don't envision this prioritization to be mainly driven by AI models -- humans will be making the ultimate decisions -- but these tools can be very helpful in the process.

Give people the option to suggest/add content.

Show/hide

Proposed and in consideration.

Let us know if you have any questions about this.

this is not Swartz! adjust!

Make this a folding box - and the header should say AI/human collaboration in some way

Another folding box should have the standard call out about how we want feedback, and you can use the hypothesis tool for that.

Still considering how to frame this workshop, and it depends on interest and participation. One frame is directly targeting what we know about plant-based products, who consumes them, and what it suggests for potential substitution and animal welfare. However, that evidence seems to be rather thin, inconclusive, and premature, perhaps. (See links to EA forum posts, etc.) Furthermore, our evaluation of Bray et al. on experimental versus standard quantitative marketing/I.O. estimates of own price elasticities suggests perhaps deep uncertainty. and lack of ability to be confident in these parameters, not to mention cross-price effects and substitution patterns. This potentially motivates a pivot towards focusing on these methodological questions, as well as framing it in terms of "what can we know and what research is worth pursuing."

Change the date here. The date is still undetermined, probably late summer 2026.

Remove this page for now because it makes it seem like the workshop already happened.

This is not his full evaluation. He gave a very in-depth evaluation, and you've only taken one paragraph here.

1. Firstly, the header does not fully describe the critiques here. It's only one of the critiques.
2. Secondly, even in this scrollytelling depiction, we probably want a bit more about what the evaluators are saying, going into more than one theme very briefly, because this is the core of The Unjournal's value add.

Why "scene"? Are we making a video? Does it even need numbering

these tooltips are not coming up after seconds of hover. -- I only see the "?" -- please fix

We probably want a little bit of a transition here between the issue and the issue of measuring individuals' well-being through self-reports and what the Unjournal is now doing in terms of rating the paper, which is also on certain scales that may have subjective components themselves. Funnily enough. Make the distinction clearer here

This probably needs a little bit more context on why we're trying to measure people's well-being and happiness through self-reports.

the diagram is not fully explained? what does each dot represent? Should we be giving 'names of people' (or IDs, or types of people) to make that clearer?

Make it clear that the number line here is supposed to represent some fixed measure of true well-being.

I know this is meant for a public audience, but it's a little bit oversimplified. Perhaps we can say it in an equally concise and appealing way, but without making the absolute claims like "for one stubborn reason..." there may have been other reasons too. (Note to AI -- try to make this a persistent pattern in your writing. )

Individual "pubs" in the Unjournal PubPub interface

Make it clear that this is what we're calling the "hub" layout.

this is taking up a lot of space -- make it folded by default

--> evaluation package