model training

Hyperparameters should be set explicitly.

Free memory in time.

Set deterministic algorithm option to True during the development process, and use the option that provides better performance in the production.

Add a mask for possible invalid values. For example, developers should add a mask for the input for tf.log() API.

Set random seed explicitly during the development process whenever a possible random procedure is involved in the application.

Use the broadcasting feature in TensorFlow 2 to be more memory efficient.

Use tf.TensorArray() in TensorFlow 2 if the value of the array will change in the loop.

Use self.net() in PyTorch to forward the input to the network instead of self.net.forward().

Call the training mode in the appropriate place in PyTorch code to avoid forgetting to toggle back the training mode after the inference step.

Use optimizer.zero_grad(), loss_fn.backward(), optimizer.step() together in order in PyTorch. Do not forget to use optimizer.zero_grad() before loss_fn.backward() to clear gradients.