I'm not convinced that this rejects "leakage" as an important driver of the innacuracy.
Need to think about this a bit more myself: Suppose that model A1 was fit on "training data A" and part of the 'predictors in training data A' actually reflected outcomes, aka, there were 'leaks'.
We can compare it with "model A2" -- a model fit on training data A that did not include the leaks/contaminated predictor.
Suppose that both model A1 and model A2 used appropriate penalization/normalization and cross-validation, so as to not overfit (other than this leaks issue).
Consider "out of time testing data B", presuming the structural relationships did not change between the earlier and later dates
A1 should predict worse on testing data B than it did on data A . However, A2 should predict roughly as well on testing data B as it did on A.
A2 should predict better on B than does A1. That's ~because A2 did not get contaminated by these leaky outcome variables, which would throw off the rest of the model.
When you say "dropping those predictors barely changes accuracy" is this doing something like "running model A2 and then comparing A1 and A2 in predicting B"?