model training

Hyperparameters should be set explicitly.

Free memory in time.

Set deterministic algorithm option to True during the development process, and use the option that provides better performance in the production.

Set random seed explicitly during the development process whenever a possible random procedure is involved in the application.

Add a mask for possible invalid values. For example, developers should add a mask for the input for tf.log() API.

Use the broadcasting feature in TensorFlow 2 to be more memory efficient.

Use tf.TensorArray() in TensorFlow 2 if the value of the array will change in the loop.

Call the training mode in the appropriate place in PyTorch code to avoid forgetting to toggle back the training mode after the inference step.

Use self.net() in PyTorch to forward the input to the network instead of self.net.forward().

Use optimizer.zero_grad(), loss_fn.backward(), optimizer.step() together in order in PyTorch. Do not forget to use optimizer.zero_grad() before loss_fn.backward() to clear gradients.