biotransformers.lightning_utils.optimizer#

Module Contents#

Functions#

lr_update(num_updates: int, warmup_updates: int, warmup_init_lr: float, lr_step: float, decay_factor: float) → float

InverseSquareRootSchedule.
biotransformers.lightning_utils.optimizer.lr_update(num_updates: int, warmup_updates: int, warmup_init_lr: float, lr_step: float, decay_factor: float) → float#

InverseSquareRootSchedule.

pytorch/fairseq

Parameters

  • num_updates – number of batches already used.

  • warmup_updates – number of batch steps for warm up.

  • warmup_init_lr – initial learning rate.

  • lr_step – step for increasing learning rate during warm up.

  • decay_factor – factor for decreasing learning rate after warm up.

Returns

learning rate multiplicate factor