The performance of the Gamba models in comparison to much larger baselines is pretty remarkable! I also thought that introducing more difficult, less random versions of the classification experiments you ran was a smart way of investigating the models’ discriminative capabilities. However, I’m a little curious why the MEM training task predicts evolutionary rates for all tokens, rather than just the masked ones. Did you compare this performance to predicting only the masked positions? Or the performance if there was no mask, and the model predicted at all positions?

Really interesting work! The SAE analysis is a great analogue of the distributional hypothesis in NLP. I’ve seen papers calling for architectures and datasets built around the concept for genes, but it’s cool to see compelling evidence that biologically meaningful feature neighborhoods can be learned from sequence prediction alone. It’d be especially interesting to see how closely the distances in these neighborhoods correlate with co-occurrence in the training data versus independent structural, biochemical, or functional similarity.