We implement a simple automated machine learning (AutoML) system which includes preprocessing, a switch between multiple learners and hyperparameter tuning.

This is the third part of the practical tuning series. The other parts can be found here:

- Part I - Tune a Support Vector Machine
- Part II - Tune a Preprocessing Pipeline
- Part IV - Tuning and Parallel Processing

In this post, we implement a simple automated machine learning (AutoML) system which includes preprocessing, a switch between multiple learners and hyperparameter tuning. For this, we build a pipeline with the mlr3pipelines extension package. Additionally, we use nested resampling to get an unbiased performance estimate of our AutoML system.

We load the mlr3verse package which pulls in the most important packages for this example.

We initialize the random number generator with a fixed seed for reproducibility, and decrease the verbosity of the logger to keep the output clearly represented. The `lgr`

package is used for logging in all mlr3 packages. The mlr3 logger prints the logging messages from the base package, whereas the bbotk logger is responsible for logging messages from the optimization packages (e.g. mlr3tuning ).

```
set.seed(7832)
lgr::get_logger("mlr3")$set_threshold("warn")
lgr::get_logger("bbotk")$set_threshold("warn")
```

In this example, we use the Pima Indians Diabetes data set which is used to to predict whether or not a patient has diabetes. The patients are characterized by 8 numeric features and some have missing values.

```
task = tsk("pima")
```

We use three popular machine learning algorithms: k-nearest-neighbors, support vector machines and random forests.

The `PipeOpBranch`

allows us to specify multiple alternatives paths. In this graph, the paths lead to the different learner models. The `selection`

hyperparameter controls which path is executed i.e., which learner is used to fit a model. It is important to use the `PipeOpBranch`

after the branching so that the outputs are merged into one result object. We visualize the graph with branching below.

Alternatively, we can use the `ppl()`

-shortcut to load a predefined graph from the `mlr_graphs`

dictionary. For this, the learner list must be named.

The task has missing data in five columns.

```
round(task$missings() / task$nrow, 2)
```

```
diabetes age glucose insulin mass pedigree pregnant pressure triceps
0.00 0.00 0.01 0.49 0.01 0.00 0.00 0.05 0.30
```

The pipeline `"robustify"`

function creates a preprocessing pipeline based on our task. The resulting pipeline imputes missing values with `PipeOpImputeHist`

and creates a dummy column (`PipeOpMissInd`

) which indicates the imputed missing values. Internally, this creates two paths and the results are combined with `PipeOpFeatureUnion`

. In contrast to `PipeOpBranch`

, both paths are executed. Additionally, `"robustify"`

adds `PipeOpEncode`

to encode factor columns and `PipeOpRemoveConstants`

to remove features with a constant value.

We could also create the preprocessing pipeline manually.

```
gunion(list(po("imputehist"),
po("missind", affect_columns = selector_type(c("numeric", "integer"))))) %>>%
po("featureunion") %>>%
po("encode") %>>%
po("removeconstants")
```

```
Graph with 5 PipeOps:
ID State sccssors prdcssors
imputehist <<UNTRAINED>> featureunion
missind <<UNTRAINED>> featureunion
featureunion <<UNTRAINED>> encode imputehist,missind
encode <<UNTRAINED>> removeconstants featureunion
removeconstants <<UNTRAINED>> encode
```

We use `as_learner()`

to create a `GraphLearner`

which encapsulates the pipeline and can be used like a learner.

```
graph_learner = as_learner(graph)
```

The parameter set of the graph learner includes all hyperparameters from all contained learners. The hyperparameter ids are prefixed with the corresponding learner ids. The hyperparameter `branch.selection`

controls which learner is used.

```
as.data.table(graph_learner$param_set)
```

id | class | lower | upper | nlevels |
---|---|---|---|---|

imputehist.affect_columns | ParamUty | NA | NA | Inf |

missind.which | ParamFct | NA | NA | 2 |

missind.type | ParamFct | NA | NA | 4 |

missind.affect_columns | ParamUty | NA | NA | Inf |

encode.method | ParamFct | NA | NA | 5 |

encode.affect_columns | ParamUty | NA | NA | Inf |

removeconstants.ratio | ParamDbl | 0 | 1 | Inf |

removeconstants.rel_tol | ParamDbl | 0 | Inf | Inf |

removeconstants.abs_tol | ParamDbl | 0 | Inf | Inf |

removeconstants.na_ignore | ParamLgl | NA | NA | 2 |

removeconstants.affect_columns | ParamUty | NA | NA | Inf |

kknn.k | ParamInt | 1 | Inf | Inf |

kknn.distance | ParamDbl | 0 | Inf | Inf |

kknn.kernel | ParamFct | NA | NA | 10 |

kknn.scale | ParamLgl | NA | NA | 2 |

kknn.ykernel | ParamUty | NA | NA | Inf |

svm.cachesize | ParamDbl | -Inf | Inf | Inf |

svm.coef0 | ParamDbl | -Inf | Inf | Inf |

svm.cost | ParamDbl | 0 | Inf | Inf |

svm.cross | ParamInt | 0 | Inf | Inf |

svm.degree | ParamInt | 1 | Inf | Inf |

svm.gamma | ParamDbl | 0 | Inf | Inf |

svm.kernel | ParamFct | NA | NA | 4 |

svm.nu | ParamDbl | -Inf | Inf | Inf |

svm.shrinking | ParamLgl | NA | NA | 2 |

svm.tolerance | ParamDbl | 0 | Inf | Inf |

svm.type | ParamFct | NA | NA | 2 |

svm.fitted | ParamLgl | NA | NA | 2 |

svm.scale | ParamUty | NA | NA | Inf |

svm.class.weights | ParamUty | NA | NA | Inf |

svm.decision.values | ParamLgl | NA | NA | 2 |

ranger.alpha | ParamDbl | -Inf | Inf | Inf |

ranger.always.split.variables | ParamUty | NA | NA | Inf |

ranger.class.weights | ParamDbl | -Inf | Inf | Inf |

ranger.holdout | ParamLgl | NA | NA | 2 |

ranger.importance | ParamFct | NA | NA | 4 |

ranger.keep.inbag | ParamLgl | NA | NA | 2 |

ranger.max.depth | ParamInt | 0 | Inf | Inf |

ranger.min.node.size | ParamInt | 1 | Inf | Inf |

ranger.min.prop | ParamDbl | -Inf | Inf | Inf |

ranger.minprop | ParamDbl | -Inf | Inf | Inf |

ranger.mtry | ParamInt | 1 | Inf | Inf |

ranger.mtry.ratio | ParamDbl | 0 | 1 | Inf |

ranger.num.random.splits | ParamInt | 1 | Inf | Inf |

ranger.num.threads | ParamInt | 1 | Inf | Inf |

ranger.num.trees | ParamInt | 1 | Inf | Inf |

ranger.oob.error | ParamLgl | NA | NA | 2 |

ranger.regularization.factor | ParamUty | NA | NA | Inf |

ranger.regularization.usedepth | ParamLgl | NA | NA | 2 |

ranger.replace | ParamLgl | NA | NA | 2 |

ranger.respect.unordered.factors | ParamFct | NA | NA | 3 |

ranger.sample.fraction | ParamDbl | 0 | 1 | Inf |

ranger.save.memory | ParamLgl | NA | NA | 2 |

ranger.scale.permutation.importance | ParamLgl | NA | NA | 2 |

ranger.se.method | ParamFct | NA | NA | 2 |

ranger.seed | ParamInt | -Inf | Inf | Inf |

ranger.split.select.weights | ParamDbl | 0 | 1 | Inf |

ranger.splitrule | ParamFct | NA | NA | 2 |

ranger.verbose | ParamLgl | NA | NA | 2 |

ranger.write.forest | ParamLgl | NA | NA | 2 |

branch.selection | ParamFct | NA | NA | 3 |

We will only tune one hyperparameter for each learner in this example. Additionally, we tune the branching parameter which selects one of the three learners. We have to specify that a hyperparameter is only valid for a certain learner by using `depends = branch.selection == <learner_id>`

.

```
# branch
graph_learner$param_set$values$branch.selection =
to_tune(c("kknn", "svm", "ranger"))
# kknn
graph_learner$param_set$values$kknn.k =
to_tune(p_int(3, 50, logscale = TRUE, depends = branch.selection == "kknn"))
# svm
graph_learner$param_set$values$svm.cost =
to_tune(p_dbl(-1, 1, trafo = function(x) 10^x, depends = branch.selection == "svm"))
# ranger
graph_learner$param_set$values$ranger.mtry =
to_tune(p_int(1, 8, depends = branch.selection == "ranger"))
# short learner id for printing
graph_learner$id = "graph_learner"
```

We define a tuning instance and select a random search which is stopped after 20 evaluated configurations.

The following shows a quick way to visualize the tuning results.

We add the optimized hyperparameters to the graph learner and train the learner on the full dataset.

```
learner = as_learner(graph)
learner$param_set$values = instance$result_learner_param_vals
learner$train(task)
```

The trained model can now be used to make predictions on new data. A common mistake is to report the performance estimated on the resampling sets on which the tuning was performed (`instance$result_y`

) as the model’s performance. Instead we have to use nested resampling to get an unbiased performance estimate.

We use nested resampling to get an unbiased estimate of the predictive performance of our graph learner.

```
graph_learner = as_learner(graph)
graph_learner$param_set$values$branch.selection =
to_tune(c("kknn", "svm", "ranger"))
graph_learner$param_set$values$kknn.k =
to_tune(p_int(3, 50, logscale = TRUE, depends = branch.selection == "kknn"))
graph_learner$param_set$values$svm.cost =
to_tune(p_dbl(-1, 1, trafo = function(x) 10^x, depends = branch.selection == "svm"))
graph_learner$param_set$values$ranger.mtry =
to_tune(p_int(1, 8, depends = branch.selection == "ranger"))
graph_learner$id = "graph_learner"
inner_resampling = rsmp("cv", folds = 3)
at = AutoTuner$new(
learner = graph_learner,
resampling = inner_resampling,
measure = msr("classif.ce"),
terminator = trm("evals", n_evals = 10),
tuner = tnr("random_search")
)
outer_resampling = rsmp("cv", folds = 3)
rr = resample(task, at, outer_resampling, store_models = TRUE)
```

We check the inner tuning results for stable hyperparameters. This means that the selected hyperparameters should not vary too much. We might observe unstable models in this example because the small data set and the low number of resampling iterations might introduce too much randomness. Usually, we aim for the selection of stable hyperparameters for all outer training sets.

iteration | kknn.k | svm.cost | ranger.mtry | branch.selection | classif.ce | task_id | learner_id | resampling_id |
---|---|---|---|---|---|---|---|---|

1 | NA | NA | 6 | ranger | 0.2344341 | pima | graph_learner.tuned | cv |

2 | NA | 0.1235806 | NA | svm | 0.2284371 | pima | graph_learner.tuned | cv |

3 | NA | NA | 5 | ranger | 0.2596950 | pima | graph_learner.tuned | cv |

Next, we want to compare the predictive performances estimated on the outer resampling to the inner resampling. Significantly lower predictive performances on the outer resampling indicate that the models with the optimized hyperparameters overfit the data.

```
rr$score()
```

iteration | task_id | learner_id | resampling_id | classif.ce |
---|---|---|---|---|

1 | pima | graph_learner.tuned | cv | 0.2539062 |

2 | pima | graph_learner.tuned | cv | 0.2578125 |

3 | pima | graph_learner.tuned | cv | 0.2148438 |

The aggregated performance of all outer resampling iterations is essentially the unbiased performance of the graph learner with optimal hyperparameter found by random search.

```
rr$aggregate()
```

```
classif.ce
0.2421875
```

Applying nested resampling can be shortened by using the `tune_nested()`

-shortcut.

```
graph_learner = as_learner(graph)
graph_learner$param_set$values$branch.selection =
to_tune(c("kknn", "svm", "ranger"))
graph_learner$param_set$values$kknn.k =
to_tune(p_int(3, 50, logscale = TRUE, depends = branch.selection == "kknn"))
graph_learner$param_set$values$svm.cost =
to_tune(p_dbl(-1, 1, trafo = function(x) 10^x, depends = branch.selection == "svm"))
graph_learner$param_set$values$ranger.mtry =
to_tune(p_int(1, 8, depends = branch.selection == "ranger"))
graph_learner$id = "graph_learner"
rr = tune_nested(
method = "random_search",
task = task,
learner = graph_learner,
inner_resampling = rsmp ("cv", folds = 3),
outer_resampling = rsmp("cv", folds = 3),
measure = msr("classif.ce"),
term_evals = 10,
)
```

The mlr3book includes chapters on pipelines and hyperparameter tuning. The mlr3cheatsheets contain frequently used commands and workflows of mlr3.

For attribution, please cite this work as

Becker, et al. (2021, March 11). mlr3gallery: Practical Tuning Series - Build an Automated Machine Learning System. Retrieved from https://mlr3gallery.mlr-org.com/posts/2021-03-11-practical-tuning-series-build-an-automated-machine-learning-system/

BibTeX citation

@misc{becker2021practical, author = {Becker, Marc and Ullmann, Theresa and Lang, Michel and Bischl, Bernd and Richter, Jakob and Binder, Martin}, title = {mlr3gallery: Practical Tuning Series - Build an Automated Machine Learning System}, url = {https://mlr3gallery.mlr-org.com/posts/2021-03-11-practical-tuning-series-build-an-automated-machine-learning-system/}, year = {2021} }