Description
Context
Hyperparameters are usually set before the actual learning process begins and control the learning process. These parameters directly influence the behavior of the training algorithm and therefore have a significant impact on the model’s performance.
Problem
The default parameters of learning algorithm APIs may not be optimal for a given data or problem, and may lead to local optima. In addition, while the default parameters of a machine learning library may be adequate for some time, these default parameters may change in new versions of the library. Furthermore, not setting the hyperparameters explicitly is inconvenient for replicating the model in a different programming language.
Solution
Hyperparameters should be set explicitly and tuned for improving the result’s quality and reproducibility.
Type
Generic
Existing Stage
Model Training
Effect
Error-prone & Reproducibility
Example
### Scikit-Learn
from sklearn.cluster import KMeans
- kmeans = KMeans()
+ kmeans = KMeans(n_clusters=8, random_state=0)
### PyTorch
from torch.optim import SGD
- optimizer = SGD()
+ optimizer = SGD(lr=0.01, momentum=0.9, weight_decay=0)
Source:
Paper
- MPA Haakman. 2020. Studying the Machine Learning Lifecycle and ImprovingCode Quality of Machine Learning Applications. (2020).
- Eric Breck, Shanqing Cai, Eric Nielsen, Michael Salib, and D Sculley. 2017. TheML test score: A rubric for ML production readiness and technical debt reduction.In2017 IEEE International Conference on Big Data (Big Data). IEEE, 1123–1132.
- Gopi Krishnan Rajbahadur, Gustavo Ansaldi Oliva, Ahmed E Hassan, and Juer-gen Dingel. 2019. Pitfalls Analyzer: Quality Control for Model-Driven DataScience Pipelines. In2019 ACM/IEEE 22nd International Conference on ModelDriven Engineering Languages and Systems (MODELS). IEEE, 12–22.
- Nargiz Humbatova, Gunel Jahangirova, Gabriele Bavota, Vincenzo Riccio, AndreaStocco, and Paolo Tonella. 2020. Taxonomy of real faults in deep learning sys-tems. InProceedings of the ACM/IEEE 42nd International Conference on SoftwareEngineering. 1110–1121.