Skip to main content

Diving Into Data Science

This document aims to help new machine learning (ML) practitioners as they embark on their journey. The following information is not specific to the Chariot platform, though we will (likely) reference it numerous times. Concepts will be explained, as much as possible, without detailed mathematics or statistics to be accessible to the broadest possible audience. However, we will provide references for those interested in digging deeper.

The remainder of this page is organized to walk you through a workflow, culminating in a trained model ready to be deployed into a workflow. Consider this guide a map and compass.

Getting Started​

All models will require some input (data) and produce some corresponding output (prediction). Therefore, the essential requirements for this process are data and a model.

Choosing a Model​

The type of learning task you require will decide which specific model to use. You may need to perform regression, classification, object detection, or segmentation. If none of those tasks sounds right, or if you’re unsure how to tell the model what it should produce as output, you might want to examine the distinction between supervised and unsupervised learning. Remember that Chariot is designed to support supervised learning tasks—though in some cases, others may be possible.

Preparing Data​

To train the model, you’ll need data similar to the data against which you will eventually deploy the trained model at the end of this process. This data may already exist (e.g., as a dataset associated with a project in Chariot), or you may need to curate it.

The following are essential steps in data preparation:

Those links include the tradeoff between recycling old data and collecting new data, how much data you need, what to tell your model about the data, and the what, how, and why of creating data splits.

Training a Model​

After your data has been collected and sufficiently annotated for the model’s intended task, you can train the model. The model training section includes a description of what it means to train a model, how the process works (behind the scenes), and a dictionary of standard terms associated with training and terms you'll see in Chariot's training run wizard. Notably, the dictionary provides several suggestions for how to start training, as many options can become overwhelming.

Evaluating Your Model​

The metrics section contains extensive information about the various metrics that may be useful in evaluating your model. It also provides several examples to help you understand which metrics are appropriate and when.

Part of the evaluation process is monitoring for overfitting. Ensure your model does not overfit; otherwise, it’ll be useless.

What Can I Do If the Model Isn't Good Enough?​

The most important thing to understand is that model training is a stochastic process; many random aspects of the training can cause the results of two otherwise identical runs to be pretty different. Many parameters can also be adjusted that may impact the quality of the trained model. This section briefly suggests things to try if your model either won’t train or trains, but the accuracy is never good enough for your application.

Your model was improving when the training finished…​

  • Resume training from the last checkpoint and let it train longer; you're on the right track!

Your model initially improved but stopped improving before training finished…​

  • Resume training from the best checkpoint and reduce the learning rate by half or more.
    Reducing the learning rate of a stalled training run can boost the model. Neural network literature is filled with examples of learning rate schedules or strategies. Learning rate schedules can automatically reduce learning rates at fixed points during a training run (e.g., every 10,000 training steps). Learning rate strategies may also automatically reduce learning rates dynamically when criteria are met (e.g., training fails to improve over 1,000 batches).

Your model never improved, with metrics that are flat or noisy…​

This case can be the trickiest. Many things could influence this. Below are a few suggestions in descending order that an experienced data scientist would try. In each case, experiment by making short training runs (about 3000–4000 training steps) and see if significant differences arise. If you see positive differences, restart the training run for a standard length of time and evaluate.

  • Tweak the Learning Rate Try doubling or halving the learning rate.

  • Try Other Optimizer Tweaks Make sure you are using an optimizer with momentum. If the optimizer uses momentum, that value will be a parameter you can change in Chariot under the Advanced Settings tab. Make adjustments to momentum (if it is high, reduce it; if it is low, increase it), weight decay (if using weight decay, halve it or set it to zero), or try a different optimizer (Adam, RMSProp, NAdam, etc.).

  • Try a Different Model Finding a suitable model architecture can be daunting. This guide provides a guide to help find a good architecture.

  • Start From a Model Pre-Trained on Some Related Dataset Often, model training starts from models pre-trained on ImageNet, and that's frequently useful. However, if you have a dataset that is more similar to the dataset of interest than ImageNet, you should start training from a model that performs well on that dataset. This is not always possible.

  • Your Dataset May Have Issues This could include many incorrect labels, annotations, or imbalances. It may also be that your dataset is too small, lacks diversity, or has other problems that cause issues.

If none of these things work, don’t panic. Seek help from trusted colleagues or professionals, and keep trying. Keeping detailed notes of what has been tried and its impact will be worth the effort when you get down this process. Feel free to try the same thing twice or thrice—training is a stochastic process. Use your notes to make sure that any repeating of experiments is intentional and recorded, not accidental and a waste of time and effort.