We find that optimization becomes more reliable when a small intermediate-state regularizer is added on top of token-level distillation.
这个发现提供了一个有价值的见解：在模型级别的蒸馏过程中添加中间状态的正则化项，可以提高优化的可靠性。这表明，除了关注输出分布的匹配外，保持内部表示轨迹的几何一致性对于模型转换也很重要。这种见解可能对其他模型转换和蒸馏方法有启发意义。
Title: Delays, Detours, and Forks in the Road: Latent State Models of Training Dynamics Authors: Michael Y. Hu1 Angelica Chen1 Naomi Saphra1 Kyunghyun Cho Note: This paper seems cool, using older interpretable machine learning models, graphical models to understand what is going on inside a deep neural network
LocalNorm: Robust Image Classification through Dynamically Regularized Normalization
提出了新的 LocalNorm。既然Norm都玩得这么嗨了，看来接下来就可以研究小 GeneralizedNorm 或者 AnyRandomNorm 啥的了。。。[doge]
λR(W)
In assignment 1, function svm_loss_naive, there is a 1/2 term before the regularization loss. Where does this come from?
loss += 0.5 reg np.sum(W * W)