Reviewer #1 (Public review):
The revision by Ruan et al clarifies several aspects of the original manuscript that were difficult to understand, and I think it presents some useful and interesting ideas. I understand that the authors are distinguishing their model from the standard Wright-Fisher model in that the population size is not imposed externally, but is instead a consequence of the stochastic reproduction scheme. Here, the authors chose a branching process but in principle any Markov chain can probably be used. Within this framework, the authors are particularly interested in cases where the variance in reproductive success changes through time, as explored by the DDH model, for example. They argue with some experimental results that there is a reason to believe that the variance in reproductive success does change over time.
One of the key aspects of the original manuscript that I want to engage with is the DDH model. As the authors point out, their equations 5 and 6 are assumptions, and not derived from any principles. In essence, the authors are positing that that the variance in reproductive success, given by 6, changes as a function of the current population size. There is nothing "inherent" to a negative binomial branching mechanism that results in this: in fact, the the variance in offspring number could in principle be the same for all time. As relates to models that exist in the literature, I believe that this is the key difference: unlike Cannings models, the authors allow for a changing variance in reproduction through time.
This is, of course, an interesting thing to consider, and I think that the situation the authors point out, in which drift is lower at small population sizes and larger at large population sizes, is not appreciated in the literature. However, I am not so sure that there is anything that needs to be resolved in Paradox 1. A very strong prediction of that model is that Ne and N could be inversely related, as shown by the blue line in Fig 3b. This suggests that you could see something very strange if you, for example, infer a population size history using a Wright-Fisher framework, because you would infer a population *decline* when there is in fact a population *expansion*. However, as far as I know there are very few "surprising population declines" found in empirical data. An obvious case where we know there is very rapid population growth is human populations; I don't think I've ever seen an inference of recent human demographic history from genetic data that suggests anything other than a massive population expansion. While I appreciate the authors empirical data supporting their claim of Paradox 1 (more on the empirical data later), it's not clear to me that there's a "paradox" in the literature that needs explaining so much as this is a "words of caution about interpreting inferred effective population sizes". To be clear, I think those words of caution are important, and I had never considered that you might be so fundamentally misled as to infer decline when there is growth, but calling it a "paradox" seems to suggest that this is an outstanding problem in the literature, when in fact I think the authors are raising a *new* and important problem. Perhaps an interesting thing for the authors to do to raise the salience of this point would be to perform simulations under this model and then infer effective population sizes using e.g. dadi or psmc and show that you could identify a situation in which the true history is one of growth, but the best fit would be one of decline
The authors also highlight that their approach reflects a case where the population size is determined by the population dynamics themselves, as opposed to being imposed externally as is typical in Cannings models. I agree with the authors that this aspect of population regulation is understudied. Nonetheless, several manuscripts have dealt with the case of population genetic dynamics in populations of stochastically fluctuating size. For example, Kaj and Krone (2003) show that under pretty general conditions you get something very much like a standard coalescent; for example, combining their theorem 1 with their arguments on page 36 and 37, they find that exchangeable populations with stochastic population dynamics where the variance does not change with time still converge to exactly the coalescent you would expect from Cannings models. This is strongly suggestive that the authors key result isn't about stochastic population dynamics per se, but instead related to arguing that variance in reproductive success could change through time. In fact, I believe that the result of Kaj and Krone (2003) is substantially more general than the models considered in this manuscript. That being said, I believe that the authors of this manuscript do a much better job of making the implications for evolutionary processes clear than Kaj and Krone, which is important---it's very difficult to understand from Kaj and Krone the conditions under which effective population sizes will be substantially impacted by stochastic population dynamics.
I also find the authors exposition on Paradox 3 to be somewhat strange. First of all, I'm not sure there's a paradox there at all? The authors claim that the lack of dependence of the fixation probability on Ne is a paradox, but this is ultimately not surprising---fixation of a positively selected allele depends mostly on escaping the boundary layer, which doesn't really depend on the population size (see Gillespie's book "The Causes of Molecular Evolution" for great exposition on boundary layer effects). Moreover, the authors *use a Cannings-style argument* to get gain a good approximation of how the fixation probability changes when there is non-Poisson reproduction. So it's not clear that the WFH model is really doing a lot of work here. I suppose they raise the interesting point that the particularly simple form of p(fix) = 2s is due to the assumption that variance in offspring is equal to 1.
In addition, I raised some concerns about the analysis of empirical results on reproductive variance in my original review, and I don't believe that the authors responded to it at all. I'm not super worried about that analysis, but I think that the authors should probably respond to me.
Overall, I feel like I now have a better understanding of this manuscript. However, I think it still presents its results too strongly: Paradox 1 contains important words of caution that reflect what I am confident is an under appreciated possibility, and Paradox 3 is, as far as I'm concerned, not a paradox at all. I have not addressed Paradox 2 very much because I think that another reviewer had solid and interesting comments on that front and I am leaving it to them. That being said, I do think Paradox 2 actually presents a deep problem in the literature and that the authors' argument may actually represent a path toward a solution.
This manuscript can be a useful contribution to the literature, but as it's presented at the moment, I think most of it is worded too strongly and it continues to not engage appropriately with the literature. Theoretical advances are undoubtedly important, and I think the manuscript presents some interesting things to think about but ultimately needs to be better situated and several of the claims strongly toned down.
References:<br /> Kaj, I., & Krone, S. M. (2003). The coalescent process in a population with stochastically varying size. Journal of Applied Probability, 40(1), 33-48.