Contents

biotransformers.lightning_utils.optimizer

biotransformers.lightning_utils.optimizer

Module Contents

Functions

lr_update(num_updates: int, warmup_updates: int, warmup_init_lr: float, lr_step: float, decay_factor: float) → float

InverseSquareRootSchedule.
biotransformers.lightning_utils.optimizer.lr_update(num_updates: int, warmup_updates: int, warmup_init_lr: float, lr_step: float, decay_factor: float) → float

InverseSquareRootSchedule.

https://github.com/pytorch/fairseq/blob/master/fairseq/optim/lr_scheduler/inverse_square_root_schedule.py#L32

Parameters

  • num_updates – number of batches already used.

  • warmup_updates – number of batch steps for warm up.

  • warmup_init_lr – initial learning rate.

  • lr_step – step for increasing learning rate during warm up.

  • decay_factor – factor for decreasing learning rate after warm up.

Returns

learning rate multiplicate factor