The units are only "comparable across studies" if there weren't any mishaps (eg, clinical or methodological heterogeneity). If there's clinical heterogeneity, then we're probably comparing apples to oranges (ie, either participants, interventions, or outcomes are different among studies). If there's methodological heterogeneity, then that means there's a difference in study design

Nehal, K. R., Steendam, L. M., Campos Ponce, M., van der Hoeven, M., & Smit, G. S. A. (2021). Worldwide Vaccination Willingness for COVID-19: A Systematic Review and Meta-Analysis. Vaccines, 9(10), 1071. https://doi.org/10.3390/vaccines9101071

Brewer, N. T., DeFrank, J. T., & Gilkey, M. B. (2016). Anticipated Regret and Health Behavior: A Meta-Analysis. Health Psychology : Official Journal of the Division of Health Psychology, American Psychological Association, 35(11), 1264–1275. https://doi.org/10.1037/hea0000294

Breznau, N., Rinke, E. M., Wuttke, A., Adem, M., Adriaans, J., Alvarez-Benjumea, A., Andersen, H. K., Auer, D., Azevedo, F., Bahnsen, O., Balzer, D., Bauer, G., Bauer, P. C., Baumann, M., Baute, S., Benoit, V., Bernauer, J., Berning, C., Berthold, A., … Nguyen, H. H. V. (2021). Observing Many Researchers using the Same Data and Hypothesis Reveals a Hidden Universe of Data Analysis. MetaArXiv. https://doi.org/10.31222/osf.io/cd5j9

Robertson, O. M., & Pownall, M. (2020). The Expertise Paradox: Opportunities and Challenges of a Public Psychology Framework [Preprint]. PsyArXiv. https://doi.org/10.31234/osf.io/sfnb9

Hoogeveen, S., Sarafoglou, A., & Wagenmakers, E.-J. (2020). Laypeople Can Predict Which Social-Science Studies Will Be Replicated Successfully: Advances in Methods and Practices in Psychological Science. https://doi.org/10.1177/2515245920919667

Gligorić, Vukašin, Allard Feddes, and Bertjan Doosje. ‘Political Bullshit Receptivity and Its Correlates: A Cross-Cultural Validation of the Concept’. PsyArXiv, 27 October 2020. https://doi.org/10.31234/osf.io/u9pe3.

Aczel, Balazs, Marton Kovacs, and Rink Hoekstra. ‘The Role of Human Fallibility in Psychological Research: A Survey of Mistakes in Data Management’. PsyArXiv, 5 November 2020. https://doi.org/10.31234/osf.io/xcykz.

Tomfohr-Madsen, L., Giesbrecht, G., Lebel, C., Racine, N., & Madigan, S. (2021). Depression and Anxiety in Pregnancy during COVID-19: A Rapid Review and Meta-Analysis. PsyArXiv. https://doi.org/10.31234/osf.io/n8b7x

Hohn, R. E., Slaney, K. L., & Tafreshi, D. (2020). An Empirical Review of Research and Reporting Practices in Psychological Meta-Analyses. Review of General Psychology, 1089268020918844. https://doi.org/10.1177/1089268020918844

Moreau, D., & Gamble, B. (2020). Conducting a Meta-Analysis in the Age of Open Science: Tools, Tips, and Practical Recommendations [Preprint]. PsyArXiv. https://doi.org/10.31234/osf.io/t5dwg

Lees, J. M., & Cikara, M. (2020, July 29). Understanding and Combating False Polarization. https://doi.org/10.31234/osf.io/ncwez

Webster, G. D., Mahar, E., & Wongsomboon, V. (2020). American Psychology Is Becoming More International, But Too Slowly: Comment on Thalmayer et al. (2020). https://doi.org/10.31234/osf.io/wqmer Ame

Olsson-Collentine, A., van Assen, M. A. L. M., & Wicherts, J. M. (2020). Postprint—Heterogeneity in direct replications in psychology and its association with effect size [Preprint]. PsyArXiv. https://doi.org/10.31234/osf.io/m23v4

Edgcumbe, D. (2020). PrePrint Version (Edgcumbe, 2020): The developmental trajectory of open-mindedness: from 18 to 87-years of age. [Preprint]. PsyArXiv. https://doi.org/10.31234/osf.io/fnrmv

Puthillam, Arathy. ‘Too WEIRD, Too Fast: Preprints about COVID-19 in the Psychological Sciences’. Preprint. PsyArXiv, 10 June 2020. https://doi.org/10.31234/osf.io/5w7du.

Forscher, P. S., Wagenmakers, E.-J., DeBruine, L. M., Coles, N. A., Silan, M. A., & IJzerman, H. (2020). A Manifesto for Team Science [Preprint]. PsyArXiv. https://doi.org/10.31234/osf.io/2mdxh

Grossmann, I., Weststrate, N. M., Ardelt, M., Brienza, J. P., Dong, M., Ferrari, M., Fournier, M. A., Hu, C. S., Nusbaum, H., & Vervaeke, J. (2020). The Science of Wisdom in a Polarized World: Knowns and Unknowns [Preprint]. PsyArXiv. https://doi.org/10.31234/osf.io/w9ygc

Under review at Meta-Psychology. Contribute with open community peer review comments directly on the preprint.

The fully transparent editorial process can be found here: https://osf.io/4tgq9/

Submitted to Meta-Psychology. Follow the fully transparent review process here: link

Contribute with open community peer review by commenting directly on this preprint.

This manuscript is under peer review at Meta-Psychology. Please contribute with open community peer review directly at the preprint.

The transparent editorial history can be found here:

https://osf.io/y369z/

Under review at Meta-Psychology. Follow the transparent editorial process here:

https://osf.io/q9gb3/

Under review at Meta-Psychology. Follow the history here:

https://osf.io/f9udb/

In his commentary, Alex Holcombe makes the argument that only ‘one or two exemplars of a color category’ are typically examined in color studies, and this is problematic because a color such as ‘red’ is a category, not a single hue.

Although in some fields it is very important to examine a range of stimuli, and in general examining the generalizability of findings has an important place in research lines, I do not think that currently this issue is a pressing concern in color psychology. Small variations in hue and brightness naturally occur in online studies, and these are assumed not to matter for the underlying mechanism. Schietecat, Lakens, IJsselsteijn, and De Kort (2018) write: “In addition, we conducted Experiments 1 and 3 in a laboratory environment, but Experiments 2, 4, and 5 were conducted in participants’ homes with an internet-based method. Therefore, we could not be completely sure that the presentation of the stimuli on their personal computers was identical for every participant in those experiments. However, we expected that the impact of these variations on our results is not substantial. The labels of the IAT (i.e., red vs blue) increased the salience of the relevant hue dimension, and we do not expect our results to hold for very specific hues, but for colors that are broadly categorized as red, blue, and green. The similar associative patterns across Experiments 2 and 3 seem to support this expectation.”

We wrote this because there is nothing specific about the hue that is expected to drive the effects in association based accounts of psychological effects of colors. If the color ‘red’ is associated with specific concepts (and the work by Schietecat at all supports the idea that red can activate associations related to either activity and evaluation, such as aggression or enthusiasm, depending on the context). This means that the crucial role of the stimulus is to activate the association with ‘red’, no the perceptual stimulation of the eye in any specific way. The critical manipulation check would thus be is people categorize a stimulus as ‘red’. As long as this is satisfied, we can assume the concept ‘red’ is activated, which can then activate related associations, depending on the context.

Obviously, the author is correct that there are benefits in testing multiple variations of the color ‘red’ to demonstrate the generalizability of observed effects. However, the authors is writing too much as a perception researcher I fear. If there is a strong theoretical reason to assume slightly different hues and chromas will not matter (because as long as a color is recognized as ‘red’ it will activate specific associations) the research priority of varying colors is much lower than in other fields (e.g., research on human faces) where it is more plausible that the specifics of the stimuli matter. A similar argument holds for the question whether “any link is specifically to red, rather than extending to green, yellow, purple, and brown”. This is too a-theoretical, and even though not all color research has been replicable, and many studies suffered from problems identified during the replication crisis, the theoretical models are still plausible, and specific to predictions about certain hues. We know quite a lot about color associations for prototypical colors in terms of their associations with valence and activity (e.g., Russell & Mehrabian, 1977) and this can be used to make more specific predictions than to a-theoretically test the entire color spectrum.

Indeed, across the literature many slightly different variations of red are used, or in online studies (Schietecat et al., 2018) studies have been performed online, where different computer screens will naturally lead to some variation in the exact colors presented. This doesn’t mean that more dedicated exploration of the boundaries of these effects can be worthwhile in the future. But currently, the literature is more focused on examining whether these effects are reliable to begin with, and explaining basic questions about their context dependency, than that they are concerned about testing the range of hues for which effects can be observed. So, although in principle it is often true that the generalizability of effects is understudies and deserved more attention, it is not color psychology’s most pressing concern, because we have theoretical predictions about specific colors, and because theoretically as long as a color activates the concept (e.g., ‘red’), the associated concepts that influence subsequent psychological responses are assumed to be activated, irrespective of minor differences in for example hue or brightness.

Daniel Lakens

References

Russell, J. A., & Mehrabian, A. (1977). Evidence for a three-factor theory of emotions. Journal of Research in Personality, 11(3), 273–294. DOI: https://doi.org/10.1016/0092-6566(77)90037-X Schietecat, A. C., Lakens, D., IJsselsteijn, W. A., & Kort, Y. A. W. de. (2018). Predicting Context-dependent Cross-modal Associations with Dimension-specific Polarity Attributions. Part 2: Red and Valence. Collabra: Psychology, 4(1). https://doi.org/10.1525/COLLABRA.126

Submitted to Meta-Psychology. Contribute with peer review directly on this preprint. The editorial process can be found here:

https://osf.io/vg5wa/

Under review at Meta-Psychology. Contribute with peer review here. The editorial process can be followed at:

https://osf.io/s6yr4/