6 Matching Annotations
  1. May 2023
    1. A new import is the _LRScheduler which we will use to implement our learning rate finder.

    2. We will also show how to initialize the weights of our neural network and how to find a suitable learning rate using a modified version of the learning rate finder.

  2. Jul 2021
  3. Nov 2018
    1. Learning with Random Learning Rates

      作者提出了一种新的Alrao优化算法,让网络中每个 unit 或 feature 都各自从不同级别的随机分布中采样获得其自己的学习率。该算法没有额外计算损耗,可以更快速达到理想 lr 下的SGD性能,用来测试 DL 模型很棒!

  4. Oct 2018
    1. Approximate Fisher Information Matrix to Characterise the Training of Deep Neural Networks

      深度神经网络训练(收敛/泛化性能)的近似Fisher信息矩阵表征,可自动优化mini-batch size/learning rate


      挺有趣的 paper,提出了从 Fisher 矩阵抽象出新的量用来衡量训练过程中的模型表现,来优化mini-batch sizes and learning rates | 另外 paper 中的figure画的很好看 | 作者认为逐步增加batch sizes的传统理解只是partially true,存在逐步递减该 size 来提高 model 收敛和泛化能力的可能。