While it may be slightly more difficult to find larger metal/wood cases for the 4x6 or 5x8 cards, it's a minor nuisance and anyone who wants them will eventually find the right thing for them. Beyond this, choose the card size that feels right to you.
If you don't have an idea of what you need or like, try things out for 10-20 cards and see how it works for you, your handwriting size, and general needs. People have been using 3x5, 4x6, and even larger for hundreds of years without complaining about any major issues. If Carl Linnaeus managed to be okay with 3x5, which he hand cut by the way, I suspect you'll manage too.
Of course I won't mention to the Americans the cleverness of the A6, A5, A4 paper standards which allows you to fold the larger sizes in half to get the exact next smaller size down. Then you might get the benefit of the smaller size as well as the larger which could be folded into your collection of smaller cards, you just have to watch out for accidentally wrapping ("taco-ing") a smaller card inside of a larger one and losing it. I suppose you could hand cut your own 5" x 6" larger cards to do this if you found that you occasionally needed them.
For the pocketbook conscious, 3x5 does have the benefit of lower cost as well as many more options and flexibility than larger sizes.
At least commercial card sizes are now largely standardized, so you don't have deal with changing sizes the way Roland Barthes did over his lifetime.
My personal experience and a long history of so many manuals on the topic saying "cards of the same size" indicates that you assuredly won't have fun mixing different sized slips together. I personally use 3x5" cards in a waste book sense, but my main/permanent collection is in 4x6" format. Sometimes I think I should have done 3 x 5, but it's more like jealousy than regret, particularly when it comes to the potential of a restored fine furniture card catalog. But then again...