These are definitely very very important. They are also very very tricky to do right; will need to put a lot of thought into how to handle these!

For both contact rate and chase rate - understand this, but again, let's please not explore incorporating fixed effects at any point during this exercise until we feel phenomenal about everything else.

I think doing something LIKE the Heckman selection model would be good, but actually think it should be a little different than that model. We can talk a little more about this offline, but doing something in-model to account for survivor bias would be fantastic.

Yes!

Assuming we're good to go with the sample size stuff - this is great!!

Great! One big question though.

Who are you including in this evaluation? Anyone who had at least __ PA in 2023 AND 2024, or just anyone who had at least 1 PA in each?

If we aren't doing it already, can we re-calculate comparing players who had at least 50 PA in both 2023 and 2024?

I think Figure 21 was sufficient here

Can we get something to explain this before we see the plots? I'm also still not 100% sure I understand this, so we can talk about this in person!

Agreed - however, I don't think that the model's predictions are "good" necessarily. I think the model is correctly identifying that there is a TON of uncertainty around these estimates. We want a model that (1) produces better point estimate predictions with (2) smaller posterior variances.

I agree with the fact that this is struggling because the blue lines and HDIs are too stiff. I agree that this is probably because the latent rate is not flexible enough, and because splines are struggling.

I actually disagree that this is because the model is lacking informative covariates. That would certainly improve predictive power of the model! However, I think this is the least important of the three things that you've diagnosed in this setting. We can (and should be able to) build a perfectly fine projection system WITHOUT fixed effects like chase rate.

Long term, when we're projecting different outputs, we will definitely want to include fixed effects (i.e. FB velo when projecting FB pitch grade). However, my strong prior is that these are the last improvements we should incorporate because they are (1) the trickiest to do well, and (2) move the needle primarily at the edges.

This doesn't feel like it's horrendous, but it doesn't really feel great either.

Would be interested in any methods you have in mind for calibrating Bayesian models, beyond something like the typical point estiamte calibrations applied to all posterior samples.

Some of the posterior predictive intervals are just crazy wide for some batters. Is this because their sample sizes are so low? Would probably make sense to only show posterior predictive samples for guys who have some reasonable number of PA (maybe at least 50).

I don't know if I agree with this; it does look like there are some calibration issues. I also imagine the plot would look worse if the x and y axes had the same scales.

One question - is this run for ALL batter/seasons in the training set? Would be interested in what this looks like if we restrict the population to player/seasons with some sample size threshold (maybe \(PA > 50\). Don't know if that's the right way to evaluate the model, but just something I'm curious about. My prior is that it would make the calibration look even worse, since the model will be more confident about their true talent and their sample size makes the expected noise drop.

Regardless - we need to think more about what this is telling us. In my mind, it's saying that the model is overconfident. It's estimating true talent too close to the observed values in some cases (too much coverage of low probabilities), and that's likely what's hurting the top end as well (not enough coverage of high probabilities).

This is an interesting interpretation. I usually think of it as "the batter's true underlying strikeout rate ability". Not sure if you think that's less appropriate phrasing - interested in your thoughts

Completely agree that we need to figure out how to adjust for survivor bias.

I don't agree with the statement:

"The behavior after ~30 years shouldn't be taken too seriously"

I get the sentiment - don't take the spline estimates too seriously - but this is where we need to be as accurate as possible. We want to be able to consider long-term contract valuation, and we can't do that without having good projections for batters into their late 30s/early 40s.

Really enjoyed this explanation! Thank you for diving into this! Very intuitive.

Could we also see some traceplots here?

I'm not super familiar with implementing rank plots, and the scales here seem hard to really visualize well. For example, Chain 2 for sigma_matchup actually has some spots that don't look uniform, but I don't have good intuition for what a bad plot looks like.

Can you discuss how/why you set Gamma(2,2) priors on the different sigma parameters?

Agreed that this is interesting - things would probably change a little bit if we could adjust for quality of pitcher.

Either way though, another interesting thing coming out of here is the difference in platoon effects by batter handedness. The difference between LL/LR is bigger than RR/RL.

I can't quite tell from the formula here. Are you essentially saying, in Wilkinson notation, the following:

SO ~ (1 | batter) + (1 | matchup)

or

SO ~ matchup + (1 | batter)

Additionally, is there a global intercept in the model? Don't think it's needed but just wanted to confirm.