How to tune integer hyperparameters with tuners that can only propose real numbers.

`Tuner`

for real-valued search spaces are not able to tune on integer hyperparameters. However, it is possible to round the real values proposed by a `Tuner`

to integers before passing them to the learner in the evaluation. We show how to apply a parameter transformation to a `ParamSet`

and use this set in the tuning process.

We load the mlr3verse package which pulls in the most important packages for this example and we decrease the verbosity of the logger to keep the output clearly represented.

```
library(mlr3verse)
lgr::get_logger("mlr3")$set_threshold("warn")
lgr::get_logger("bbotk")$set_threshold("warn")
```

In this example, we use the k-Nearest-Neighbor classification learner. We want to tune the integer-valued hyperparameter `k`

which defines the numbers of neighbors.

```
id class lower upper levels default
1: k ParamInt 1 Inf 7
```

We choose the `iris`

dataset to demonstrate the tuning.

```
<TaskClassif:iris> (150 x 5)
* Target: Species
* Properties: multiclass
* Features (4):
- dbl (4): Petal.Length, Petal.Width, Sepal.Length, Sepal.Width
```

We choose generalized simulated annealing as tuning strategy. The `param_classes`

field of `TunerGenSA`

states that the tuner only supports real-valued (`ParamDbl`

) hyperparameter tuning.

```
<TunerGenSA>
* Parameters: list()
* Parameter classes: ParamDbl
* Properties: single-crit
* Packages: GenSA
```

To get integer-valued hyperparameter values for `k`

, we construct a search space with a transformation function. The `as.integer()`

function converts any real valued number to an integer by removing the decimal places.

We construct the other objects needed for tuning.

```
instance = TuningInstanceSingleCrit$new(
task = task,
learner = learner,
resampling = rsmp("holdout"),
measure = msr("classif.ce"),
terminator = trm("evals", n_evals = 20),
search_space = search_space)
```

We start the tuning and compare the results of the search space to the results in the space of the learners hyperparameter set.

```
tuner$optimize(instance)
```

```
k learner_param_vals x_domain classif.ce
1: 7.532552 <list[1]> <list[1]> 0.04
```

The optimal `k`

is still a real number in the search space.

```
instance$result_x_search_space
```

```
k
1: 7.532552
```

However, in the learners hyperparameters space, `k`

is an integer value.

```
instance$result_x_domain
```

```
$k
[1] 7
```

The archive shows us that for all real-valued `k`

proposed by GenSA, an integer-valued `k`

in the learner hyperparameter space (`x_domain_k`

) was created.

```
as.data.table(instance$archive)[, c("k", "classif.ce", "x_domain_k")]
```

```
k classif.ce x_domain_k
1: 3.487624 0.06 3
2: 5.657087 0.06 5
3: 5.602096 0.06 5
4: 3.487624 0.06 3
5: 3.487625 0.06 3
6: 3.487623 0.06 3
7: 6.226820 0.06 6
8: 3.177346 0.06 3
9: 7.532552 0.04 7
10: 3.989377 0.06 3
11: 7.532552 0.04 7
12: 7.532553 0.04 7
13: 7.532551 0.04 7
14: 5.197850 0.06 5
15: 7.425378 0.04 7
16: 6.434803 0.06 6
17: 3.079410 0.06 3
18: 5.725471 0.06 5
19: 7.049180 0.04 7
20: 7.931102 0.04 7
```

Internally, `TunerGenSA`

was given the parameter types of the search space and therefore suggested real numbers for `k`

. Before the performance of the different `k`

values was evaluated, the transformation function of the `search_space`

parameter set was called and `k`

was transformed to an integer value.

Note that the tuner is not aware of the transformation. This has two problematic consequences: First, the tuner might propose different real valued configurations that after rounding end up to be already evaluated configurations and we end up with re-evaluating the same hyperparameter configuration. This is only problematic, if we only optimze integer parameters. Second, the rounding introduces discontinuities which can be problematic for some tuners.

We successfully tuned a integer-valued hyperparameter with `TunerGenSA`

which is only suitable for an real-valued search space. This technique is not limited to tuning problems. `Optimizer`

in bbotk can be also used in the same way to produce points with integer parameters.

For attribution, please cite this work as

Becker (2021, Jan. 19). mlr3gallery: Integer Hyperparameters in Tuners for Real-valued Search Spaces. Retrieved from https://mlr3gallery.mlr-org.com/posts/2021-01-19-integer-hyperparameters-in-tuners-for-real-valued-search-spaces/

BibTeX citation

@misc{becker2021integer, author = {Becker, Marc}, title = {mlr3gallery: Integer Hyperparameters in Tuners for Real-valued Search Spaces}, url = {https://mlr3gallery.mlr-org.com/posts/2021-01-19-integer-hyperparameters-in-tuners-for-real-valued-search-spaces/}, year = {2021} }