a lot of the improvements does not come from new algorithms. It comes from finding small bugs here and there in the data pipeline, in the model training pipeline.
大多数人认为模型性能的提升主要来自于算法创新和架构改进，但作者认为最大的提升往往来自于数据管道和训练管道中的小错误修复。这挑战了人们对AI模型开发过程的主流认知，暗示了工程优化可能比算法创新更重要。
As of March 2026, AI systems are able to post-train models to get about half as much of the uplift as ones trained by humans. The specific eval scores are derived by a 'weighted average is taken across all post-trained LLMs... The top-scoring systems as of April get 25%-28% (Opus 4.6, and GPT 5.4), compared to a human score of 51%.'
在模型微调任务上，AI系统已能达到人类研究员51%性能的一半，显示出AI在科研任务上的显著进步。
E2B LoRA：8-10GB 显存即可训练
令人惊讶的是：即使是大型语言模型，现在只需要8-10GB的显存就能进行微调，这大大降低了AI模型训练的硬件门槛，使更多研究者和开发者能够参与模型定制。
On a single H200 GPU with 1.5TB host memory, MegaTrain reliably trains models up to 120B parameters.
令人惊讶的是：仅使用一块配备1.5TB主机内存的H200 GPU就能训练1200亿参数的模型，这打破了人们对大规模模型必须依赖多GPU集群的固有印象。这一技术突破可能使超大规模模型训练变得更加普及和经济。
SOTA models of different architectures and parameter scales exhibit highly consistent failure patterns on the same set of hard samples, suggesting that the performance bottleneck stems from shared deficiencies in training data rather than architecture itself.
大多数人认为不同架构的模型会有不同的失败模式和弱点，但作者发现无论架构和参数规模如何，SOTA模型在相同困难样本上表现出高度一致的失败模式，这表明性能瓶颈源于训练数据的共同缺陷，而非架构差异，这一发现挑战了模型多样化的传统观点。
this other sort of development also happened in the last couple years just clip models um and this enables us to do predictive 00:09:47 modeling across domains um what do I mean by that it means that you can understand and provide the model information in one modality and it can essentially translate it into another
for: definition, definition - CLIP models
definition: CLIP model
ORWG Virtual Meeting 08/09/2020 https://www.youtube.com/playlist?list=PLOA0aRJ90NxvXtMt5Si5ukmR9LYfvDueB (n.d.)
Britwum, K., Catrone, R., Smith, G. D., & Koch, D. S. (2020, May 5). A University Based Social Services Parent Training Model: A Telehealth Adaptation During the COVID-19 Pandemic. https://doi.org/10.31234/osf.io/gw3cd