Reviewer #1 (Public review):
Summary:
This theoretical paper addresses how to optimize reward-rate-maximizing decisions in certain foraging-style environments. It presents a series of equations and graphical illustrations for quantities such as reward rates and time-related costs that a decision maker could estimate as a basis for such decisions. One of the main takeaways is that if the hypothetical agent underweights the time spent outside a focal reward pursuit relative to the time spent within it, this can predict a broadly realistic pattern of impatience in two alternative intertemporal choices paired with well-calibrated take-it-or-leave-it decisions. Another takeaway is that if the optimally estimated subjective value of a reward pursuit is plotted as a function of a range of temporal durations, the result resembles a hyperbolic discounting function and is affected in empirically realistic ways by the magnitude and sign of the reward. Thus, the rate-maximization framework might lead to a hypothesis about the basis for the magnitude and sign effects in discounting.
Strengths:
The paper makes a useful contribution by broadening the application of reward-rate maximization to time-related decision scenarios. The paper's breadth of scope includes applying the same framework to accept/reject decisions and multi-alternative discounting decisions. The figures take a creative approach to illustrating the internal quantities in the model. It's particularly useful that the paper gives consideration to internal distortions that could give rise to documented anomalies in decision behavior.
Weaknesses:
(1) Although there are many citations acknowledging relevant previous work, there often isn't a very granular attribution of individual previous findings to their sources. In the results section, it's sometimes ambiguous when the paper is recapping established background and when it is breaking new ground. For example, around equation 8 in the results (sv = r - rho*t), it would be good to refer to previous places where versions of this equation have been presented. Offhand, McNamara 1982 (Theoretical Population Biology) is one early instance and Fawcett et al. 2012 (Behavioural Processes) is a later one. Line 922 of the discussion seems to imply this formulation is novel here.
(2) The choice environments that are considered in detail in the paper are very simple. The simplicity facilitates concrete examples and visualizations, but it would be worth further consideration of whether and how the conclusions generalize to more complex environments. The paper considers "forgo" scenario in which the agent can choose between sequences of pursuits like A-B-A-B (engaging with option B at all opportunities, which are interleaved with a default pursuit A) and A-A-A-A (forgoing option B). It considers "choice" scenarios where the agent can choose between sequences like A-B-A-B and A-C-A-C (where B and C are larger-later and smaller-sooner rewards, either of which can be interleaved with the default pursuit). Several forms of additional complexity would be valuable to consider. One would be a greater number of unique pursuits, not repeated identically in a predictable sequence, akin to a prey-selection paradigm. It seems to me this would cause t_out and r_out (the time and reward outside of the focal prospect) to be policy-dependent, making the 'apportionment cost' more challenging to ascertain. Another relevant form of complexity would be if there were variance or uncertainty in reward magnitudes or temporal durations or if the agent had the ability to discontinue a pursuit such as in patch-departure scenarios.
(3) I had a hard time arriving at a solid conceptual understanding of the 'apportionment cost' around Figure 5. I understand the arithmetic, but it would help if it were possible to formulate a more succinct verbal description of what makes the apportionment cost a useful and meaningful quality to focus on. I think Figure 6C relates to this, but I had difficulty relating the axis labels to the points, lines, and patterned regions in the plot. I also was a bit confused by how the mathematical formulation was presented. As I understood it, the apportionment cost essentially involves scaling the rest of the SV expression by t_out/(t_in + t_out). The way this scaling factor is written in Figure 5C, as 1/(1 + (1/t_out)t_in), seems less clear than it could be. Also, the apportionment cost is described in the text as being subtracted from SV rather than as a multiplicative scaling factor. It could be written as a subtraction, by subtracting a second copy of the rest of the SV expression scaled by t_in/(t_in + t_out). But that shows the apportionment cost to depend on the opportunity cost, which is odd because the original motivation on line 404 was to resolve the lack of independence between terms in the SV expression.
(4) In the analysis of discounting functions (line 664 and beyond), the paper doesn't say much about the fact that many discounting studies take specific measures to distinguish true time preferences from opportunity costs and reward-rate maximization. In many of the human studies, delay time doesn't preclude other activities. In animal studies, rate maximization can serve as a baseline against which to measure additional effects of temporal discounting. This is an important caveat to claims about discounting anomalies being rational under rate maximization (e.g., line 1024).
(5) The paper doesn't feature any very concrete engagement with empirical data sets. This is ok for a theoretical paper, but some of the characterizations of empirical results that the model aims to match seem oversimplified. An example is the contention that real decision-makers are optimal in accept/reject decisions (line 816 and elsewhere). This isn't always true; sometimes there is evidence of overharvesting, for example.
(6) Related to the point above, it would be helpful to discuss more concretely how some of this paper's theoretical proposals could be empirically evaluated in the future. Regarding the magnitude and sign effects of discounting, there is not a very thorough overview of the several other explanations that have been proposed in the literature. It would be helpful to engage more deeply with previous proposals and consider how the present hypothesis might make unique predictions and could be evaluated against them. A similar point applies to the 'malapportionment hypothesis' although in this case there is a very helpful section on comparisons to prior models (line 1163). The idea being proposed here seems to have a lot in common conceptually with Blanchard et al. 2013, so it would be worth saying more about how data could be used to test or reconcile these proposals.