mlr3tuning Tutorial - German Credit

mlr3tuning tuning optimization nested resampling german credit data set classification

In this use case, we continue working with the German credit dataset. We work on hyperparameter tuning and apply nested resampling.

Martin Binder , Florian Pfisterer
03-11-2020

Intro

This is the second part of a serial of tutorials. The other parts of this series can be found here:

We will continue working with the German credit dataset. In Part I, we peeked into the dataset by using and comparing some learners with their default parameters. We will now see how to:

Prerequisites

First, load the packages we are going to use:

We use the same Task as in Part I:

task = tsk("german_credit")

We also might want to use multiple cores to reduce long run times of tuning runs.

# future::plan("multiprocess") # uncomment for parallelization

Evaluation

We will evaluate all hyperparameter configurations using 10-fold CV. We use a fixed train-test split, i.e. the same splits for each evaluation. Otherwise, some evaluation could get unusually “hard” splits, which would make comparisons unfair.

set.seed(8008135)
cv10_instance = rsmp("cv", folds = 10)

# fix the train-test splits using the $instantiate() method
cv10_instance$instantiate(task)

# have a look at the test set instances per fold
cv10_instance$instance
      row_id fold
   1:      5    1
   2:     20    1
   3:     28    1
   4:     35    1
   5:     37    1
  ---            
 996:    936   10
 997:    950   10
 998:    963   10
 999:    985   10
1000:    994   10

Simple Parameter Tuning

Parameter tuning in mlr3 needs two packages:

  1. The paradox package is used for the search space definition of the hyperparameters
  2. The mlr3tuning package is used for tuning the hyperparameters

Search Space and Problem Definition

First, we need to decide what Learner we want to optimize. We will use LearnerClassifKKNN, the “kernelized” k-nearest neighbor classifier. We will use kknn as a normal kNN without weighting first (i.e., using the rectangular kernel):

knn = lrn("classif.kknn", predict_type = "prob")
knn$param_set$values$kernel = "rectangular"

As a next step, we decide what parameters we optimize over. Before that, though, we are interested in the parameter set on which we could tune:

knn$param_set
<ParamSet>
         id    class lower upper nlevels default       value
1:        k ParamInt     1   Inf     Inf       7            
2: distance ParamDbl     0   Inf     Inf       2            
3:   kernel ParamFct    NA    NA      10 optimal rectangular
4:    scale ParamLgl    NA    NA       2    TRUE            
5:  ykernel ParamUty    NA    NA     Inf                    

We first tune the k parameter (i.e. the number of nearest neighbors), between 3 to 20. Second, we tune the distance function, allowing L1 and L2 distances. To do so, we use the paradox package to define a search space (see the online vignette for a more complete introduction.

search_space = ParamSet$new(list(
  ParamInt$new("k", lower = 3, upper = 20),
  ParamInt$new("distance", lower = 1, upper = 2)
))

As a next step, we define a TuningInstanceSingleCrit that represents the problem we are trying to optimize.

instance_grid = TuningInstanceSingleCrit$new(
  task = task,
  learner = knn,
  resampling = cv10_instance,
  measure = msr("classif.ce"),
  terminator = trm("none"),
  search_space = search_space
)

After having set up a tuning instance, we can start tuning. Before that, we need a tuning strategy, though. A simple tuning method is to try all possible combinations of parameters: Grid Search. While it is very intuitive and simple, it is inefficient if the search space is large. For this simple use case, it suffices, though. We get the grid_search tuner via:

set.seed(1)
tuner_grid = tnr("grid_search", resolution = 18, batch_size = 36)

Tuning works by calling $optimize(). Note that the tuning procedure modifies our tuning instance (as usual for R6 class objects). The result can be found in the instance object. Before tuning it is empty:

instance_grid$result
NULL

Now, we tune:

tuner_grid$optimize(instance_grid)

The result is returned by $optimize() together with its performance. It can be also accessed with the $result slot:

instance_grid$result
   k distance learner_param_vals  x_domain classif.ce
1: 9        2          <list[3]> <list[2]>       0.25

We can also look at the Archive of evaluated configurations:

as.data.table(instance_grid$archive)
     k distance classif.ce                                uhash           timestamp batch_nr x_domain_k
 1:  3        1      0.271 330eee18-e3ce-4ed1-be4c-9691a420f5a1 2021-06-13 10:09:16        1          3
 2:  3        2      0.273 05f23cd6-e9c5-49f7-af20-b1ce39a974dd 2021-06-13 10:09:16        1          3
 3:  4        1      0.292 13cfb43f-b88e-46dc-b13b-71ae05eac38e 2021-06-13 10:09:16        1          4
 4:  4        2      0.279 73076c78-077a-40ab-9ac5-70fc13646c09 2021-06-13 10:09:16        1          4
 5:  5        1      0.271 56684617-de2a-4227-b88e-2d7daed2f682 2021-06-13 10:09:16        1          5
 6:  5        2      0.274 816848ac-0df0-4307-9b18-4ba62ca869d9 2021-06-13 10:09:16        1          5
 7:  6        1      0.278 db3bcd57-9419-49e8-a8c7-3e5c6e2edf87 2021-06-13 10:09:16        1          6
 8:  6        2      0.273 81f4daef-2c5e-4823-8156-c5888b42a547 2021-06-13 10:09:16        1          6
 9:  7        1      0.257 b9c613e7-2b54-4555-92d3-773a23013e48 2021-06-13 10:09:16        1          7
10:  7        2      0.258 c0d26ac7-6a73-454a-b4fe-3420e40a4864 2021-06-13 10:09:16        1          7
11:  8        1      0.264 b00b0695-2543-45e1-9249-e6de2c2399ce 2021-06-13 10:09:16        1          8
12:  8        2      0.256 48bfeabd-2757-4375-8468-fd6219cc1e0b 2021-06-13 10:09:16        1          8
13:  9        1      0.251 803f5860-b991-47ca-b12c-b3b5a1000fb4 2021-06-13 10:09:16        1          9
14:  9        2      0.250 df3e5618-f84b-4697-9512-034753d7dfac 2021-06-13 10:09:16        1          9
15: 10        1      0.261 9f8b291c-bd74-43d4-aaa1-c64833b8df10 2021-06-13 10:09:16        1         10
16: 10        2      0.250 e97a8c4f-3677-4722-8652-5f1f4b999d61 2021-06-13 10:09:16        1         10
17: 11        1      0.256 cd130866-1a7c-4d0a-90f7-acc631b88cf8 2021-06-13 10:09:16        1         11
18: 11        2      0.254 3c6e4b39-e58c-46d0-8d71-0f9512e85eec 2021-06-13 10:09:16        1         11
19: 12        1      0.260 e22d1f24-e609-4d4f-b60a-d320dae532d1 2021-06-13 10:09:16        1         12
20: 12        2      0.259 4178a195-35c9-46b0-a6ec-62b9a3a895f0 2021-06-13 10:09:16        1         12
21: 13        1      0.268 9c424d21-7829-4e01-9220-b425a48570c4 2021-06-13 10:09:16        1         13
22: 13        2      0.258 b70302e0-8e19-4dbc-afa2-68f3bbc01f10 2021-06-13 10:09:16        1         13
23: 14        1      0.265 2702f418-420f-478e-9f26-34c95072b201 2021-06-13 10:09:16        1         14
24: 14        2      0.263 ef1aa831-60cf-4ec3-84b0-9553f04b73b2 2021-06-13 10:09:16        1         14
25: 15        1      0.268 151ad6da-2765-4a23-8aad-ba8392e5bd47 2021-06-13 10:09:16        1         15
26: 15        2      0.264 7d04d9fd-2cd3-4324-950a-99f51853947f 2021-06-13 10:09:16        1         15
27: 16        1      0.267 1753cc38-048c-467e-9a28-3508d041aa55 2021-06-13 10:09:16        1         16
28: 16        2      0.262 cdc58017-a51e-4bce-b75e-7e86e741897d 2021-06-13 10:09:16        1         16
29: 17        1      0.264 5875e1f9-3370-4925-9332-a32e4994cb40 2021-06-13 10:09:16        1         17
30: 17        2      0.267 3051ff93-69f0-4ca4-94ae-4d53983e0c86 2021-06-13 10:09:16        1         17
31: 18        1      0.273 fd25b4de-924f-45ef-a419-ae69e7c8c00b 2021-06-13 10:09:16        1         18
32: 18        2      0.271 236b6031-d0c6-4272-b5f1-ea2f629417c2 2021-06-13 10:09:16        1         18
33: 19        1      0.269 98dbde75-25c7-470d-86a3-65438ecd0c27 2021-06-13 10:09:16        1         19
34: 19        2      0.269 c7ff1076-088d-4b13-8374-881fed14942a 2021-06-13 10:09:16        1         19
35: 20        1      0.268 61a7ac6c-7b98-4f6d-ab21-f1f1eaa0676a 2021-06-13 10:09:16        1         20
36: 20        2      0.269 db7894cb-95b8-4d38-8ff4-3589d8f07d25 2021-06-13 10:09:16        1         20
     k distance classif.ce                                uhash           timestamp batch_nr x_domain_k
    x_domain_distance
 1:                 1
 2:                 2
 3:                 1
 4:                 2
 5:                 1
 6:                 2
 7:                 1
 8:                 2
 9:                 1
10:                 2
11:                 1
12:                 2
13:                 1
14:                 2
15:                 1
16:                 2
17:                 1
18:                 2
19:                 1
20:                 2
21:                 1
22:                 2
23:                 1
24:                 2
25:                 1
26:                 2
27:                 1
28:                 2
29:                 1
30:                 2
31:                 1
32:                 2
33:                 1
34:                 2
35:                 1
36:                 2
    x_domain_distance

We plot the performances depending on the sampled k and distance:

ggplot(as.data.table(instance_grid$archive), aes(x = k, y = classif.ce, color = as.factor(distance))) +
  geom_line() + geom_point(size = 3)

On average, the Euclidean distance (distance = 2) seems to work better. However, there is much randomness introduced by the resampling instance. So you, the reader, may see a different result, when you run the experiment yourself and set a different random seed. For k, we find that values between 7 and 13 perform well.

Random Search and Transformation

Let’s have a look at a larger search space. For example, we could tune all available parameters and limit k to large values (50). We also now tune the distance param continuously from 1 to 3 as a double and tune distance kernel and whether we scale the features.

We may find two problems when doing so:

First, the resulting difference in performance between k = 3 and k = 4 is probably larger than the difference between k = 49 and k = 50. While 4 is 33% larger than 3, 50 is only 2 percent larger than 49. To account for this we will use a transformation function for k and optimize in log-space. We define the range for k from log(3) to log(50) and exponentiate in the transformation. Now, as k has become a double instead of an int (in the search space, before transformation), we round it in the trafo.

large_searchspace = ParamSet$new(list(
  ParamDbl$new("k", lower = log(3), upper = log(50)),
  ParamDbl$new("distance", lower = 1, upper = 3),
  ParamFct$new("kernel", c("rectangular", "gaussian", "rank", "optimal")),
  ParamLgl$new("scale")
))

large_searchspace$trafo = function(x, param_set) {
  x$k = round(exp(x$k))
  x
}

The second problem is that grid search may (and often will) take a long time. For instance, trying out three different values for k, distance, kernel, and the two values for scale will take 54 evaluations. Because of this, we use a different search algorithm, namely the Random Search. We need to specify in the tuning instance a termination criterion. The criterion tells the search algorithm when to stop. Here, we will terminate after 36 evaluations:

tuner_random = tnr("random_search", batch_size = 36)

instance_random = TuningInstanceSingleCrit$new(
  task = task,
  learner = knn,
  resampling = cv10_instance,
  measure = msr("classif.ce"),
  terminator = trm("evals", n_evals = 36),
  search_space = large_searchspace,
)
tuner_random$optimize(instance_random)

Like before, we can review the Archive. It includes the points before and after the transformation. The archive includes a column for each parameter the Tuner sampled on the search space (values before the transformation) and additional columns with prefix x_domain_* that refer to the parameters used by the learner (values after the transformation):

as.data.table(instance_random$archive)
           k distance      kernel scale classif.ce                                uhash           timestamp batch_nr
 1: 2.654531 2.921236 rectangular FALSE      0.314 18bb6012-634b-4857-95ed-df31b1a692e9 2021-06-13 10:09:24        1
 2: 2.588931 1.869319        rank  TRUE      0.254 0cc2db5c-96ad-40ec-b5a4-478ba1c28d16 2021-06-13 10:09:24        1
 3: 3.319396 2.425029 rectangular  TRUE      0.272 b2504e3c-0706-49ed-bf2e-11408d1c9bed 2021-06-13 10:09:24        1
 4: 1.164253 1.799989    gaussian FALSE      0.364 3a42895c-265b-4894-aef3-4de9a3121b98 2021-06-13 10:09:24        1
 5: 2.441256 1.650704        rank  TRUE      0.253 1889b349-d879-4875-bbd1-df9d8f8a692f 2021-06-13 10:09:24        1
 6: 3.158912 2.514174     optimal FALSE      0.305 e6ee6cc9-42e5-4dea-92dd-ebd911b9633f 2021-06-13 10:09:24        1
 7: 3.047551 1.405385    gaussian  TRUE      0.257 659a3737-6bd8-4cff-a129-6683f2a9fd33 2021-06-13 10:09:24        1
 8: 2.442352 2.422242    gaussian  TRUE      0.270 dbaa0001-841a-4ef1-89d8-e4b1a5bdc993 2021-06-13 10:09:24        1
 9: 3.521548 1.243384 rectangular  TRUE      0.271 07e4ce0b-223d-4a13-b339-4eb17336691c 2021-06-13 10:09:24        1
10: 2.331159 1.490977     optimal  TRUE      0.252 a086cf24-4978-402a-a867-970c267b5814 2021-06-13 10:09:24        1
11: 1.787328 1.286609    gaussian FALSE      0.345 e9d289a7-3cb5-49df-bc82-e6d434cd6c94 2021-06-13 10:09:24        1
12: 1.297461 1.479259        rank  TRUE      0.274 2152c7bd-0b08-452d-ace7-bd2cbee414d2 2021-06-13 10:09:24        1
13: 1.378451 1.117869 rectangular FALSE      0.357 2cb224f2-be42-458f-802b-9be2dd8a0d34 2021-06-13 10:09:24        1
14: 1.988414 2.284577 rectangular FALSE      0.313 55b2facd-194b-426e-a15a-9de0eeb84f6c 2021-06-13 10:09:24        1
15: 2.557743 2.752538        rank  TRUE      0.267 00727883-2b74-4395-a231-9f4b664ac3ca 2021-06-13 10:09:24        1
16: 2.961104 2.557829        rank  TRUE      0.264 9e9af8a8-95f5-4e71-abab-af31a9059550 2021-06-13 10:09:24        1
17: 2.243193 2.594618 rectangular  TRUE      0.256 7a4ce32d-4545-4678-9651-c4decc8fd9da 2021-06-13 10:09:24        1
18: 3.666907 1.910549 rectangular FALSE      0.292 a165b081-c8e0-4e4f-89d7-d1952f0ef4e1 2021-06-13 10:09:24        1
19: 1.924639 1.820168        rank  TRUE      0.256 d6e10491-a947-4365-81a9-5d625f0a1096 2021-06-13 10:09:24        1
20: 2.390153 2.621740     optimal FALSE      0.339 dc97f46e-df14-496f-ab19-fead6e477d55 2021-06-13 10:09:24        1
21: 2.033775 2.209867        rank FALSE      0.332 3f341002-79d4-406f-ba2d-4c581eec8037 2021-06-13 10:09:24        1
22: 2.929778 2.309448        rank FALSE      0.302 c4003832-ff99-42d6-91c3-d59ccd3ad0b2 2021-06-13 10:09:24        1
23: 1.824519 1.706395        rank  TRUE      0.261 15825c1f-debf-4f99-9e48-fa2c19c4feb7 2021-06-13 10:09:24        1
24: 2.444957 1.540520     optimal  TRUE      0.249 0f7efb99-c906-4066-95d6-2475139744db 2021-06-13 10:09:24        1
25: 3.254559 2.985368        rank FALSE      0.302 237af02c-37c8-4505-a714-11c94f81c405 2021-06-13 10:09:24        1
26: 1.335633 2.266987        rank FALSE      0.361 e503439b-1756-4ffa-b649-f2d2c6643857 2021-06-13 10:09:24        1
27: 3.561251 1.426416        rank FALSE      0.296 e17db6a0-a786-4bfd-bb80-181a67f5b557 2021-06-13 10:09:24        1
28: 2.052564 1.258745 rectangular FALSE      0.324 62cff254-fb37-418f-9658-e7f6e7570156 2021-06-13 10:09:24        1
29: 3.460303 1.956236    gaussian FALSE      0.296 f6c5f7ec-d99c-4662-af8f-9b4560ccaba9 2021-06-13 10:09:24        1
30: 2.073975 2.848149        rank  TRUE      0.269 46d72616-df4e-43dd-af5f-b128889a23fc 2021-06-13 10:09:24        1
31: 2.037658 2.197522    gaussian  TRUE      0.267 d61a40f5-8f99-41a8-b938-e1a263f3adf5 2021-06-13 10:09:24        1
32: 2.438784 2.952341 rectangular FALSE      0.308 6af6c497-3a1e-49ea-b0c7-323590e64794 2021-06-13 10:09:24        1
33: 3.608733 2.463585        rank FALSE      0.294 18c84141-2057-4703-8087-e45ae8b104b7 2021-06-13 10:09:24        1
34: 3.530354 1.713454 rectangular FALSE      0.297 ebc26d06-4096-4558-8fb0-914af30ce437 2021-06-13 10:09:24        1
35: 2.195813 1.862947     optimal  TRUE      0.257 5dcf423d-f7f2-4ec1-807e-364c18213fb9 2021-06-13 10:09:24        1
36: 3.285535 1.296423        rank FALSE      0.300 bac5716d-9ed7-4d01-bf55-7233d846e5ea 2021-06-13 10:09:24        1
           k distance      kernel scale classif.ce                                uhash           timestamp batch_nr
    x_domain_k x_domain_distance x_domain_kernel x_domain_scale
 1:         14          2.921236     rectangular          FALSE
 2:         13          1.869319            rank           TRUE
 3:         28          2.425029     rectangular           TRUE
 4:          3          1.799989        gaussian          FALSE
 5:         11          1.650704            rank           TRUE
 6:         24          2.514174         optimal          FALSE
 7:         21          1.405385        gaussian           TRUE
 8:         12          2.422242        gaussian           TRUE
 9:         34          1.243384     rectangular           TRUE
10:         10          1.490977         optimal           TRUE
11:          6          1.286609        gaussian          FALSE
12:          4          1.479259            rank           TRUE
13:          4          1.117869     rectangular          FALSE
14:          7          2.284577     rectangular          FALSE
15:         13          2.752538            rank           TRUE
16:         19          2.557829            rank           TRUE
17:          9          2.594618     rectangular           TRUE
18:         39          1.910549     rectangular          FALSE
19:          7          1.820168            rank           TRUE
20:         11          2.621740         optimal          FALSE
21:          8          2.209867            rank          FALSE
22:         19          2.309448            rank          FALSE
23:          6          1.706395            rank           TRUE
24:         12          1.540520         optimal           TRUE
25:         26          2.985368            rank          FALSE
26:          4          2.266987            rank          FALSE
27:         35          1.426416            rank          FALSE
28:          8          1.258745     rectangular          FALSE
29:         32          1.956236        gaussian          FALSE
30:          8          2.848149            rank           TRUE
31:          8          2.197522        gaussian           TRUE
32:         11          2.952341     rectangular          FALSE
33:         37          2.463585            rank          FALSE
34:         34          1.713454     rectangular          FALSE
35:          9          1.862947         optimal           TRUE
36:         27          1.296423            rank          FALSE
    x_domain_k x_domain_distance x_domain_kernel x_domain_scale

Let’s now investigate the performance by parameters. This is especially easy using visualization:

ggplot(as.data.table(instance_random$archive),
  aes(x = x_domain_k, y = classif.ce, color = x_domain_scale)) +
  geom_point(size = 3) + geom_line()

The previous plot suggests that scale has a strong influence on performance. For the kernel, there does not seem to be a strong influence:

ggplot(as.data.table(instance_random$archive),
  aes(x = x_domain_k, y = classif.ce, color = x_domain_kernel)) +
  geom_point(size = 3) + geom_line()

Nested Resampling

Having determined tuned configurations that seem to work well, we want to find out which performance we can expect from them. However, this may require more than this naive approach:

instance_random$result_y
classif.ce 
     0.249 
instance_grid$result_y
classif.ce 
      0.25 

The problem associated with evaluating tuned models is overtuning. The more we search, the more optimistically biased the associated performance metrics from tuning become.

There is a solution to this problem, namely Nested Resampling.

The mlr3tuning package provides an AutoTuner that acts like our tuning method but is actually a Learner. The $train() method facilitates tuning of hyperparameters on the training data, using a resampling strategy (below we use 5-fold cross-validation). Then, we actually train a model with optimal hyperparameters on the whole training data.

The AutoTuner finds the best parameters and uses them for training:

grid_auto = AutoTuner$new(
  learner = knn,
  resampling = rsmp("cv", folds = 5), # we can NOT use fixed resampling here
  measure = msr("classif.ce"),
  terminator = trm("none"),
  tuner = tnr("grid_search", resolution = 18),
  search_space = search_space
)

The AutoTuner behaves just like a regular Learner. It can be used to combine the steps of hyperparameter tuning and model fitting but is especially useful for resampling and fair comparison of performance through benchmarking:

rr = resample(task, grid_auto, cv10_instance, store_models = TRUE)

We aggregate the performances of all resampling iterations:

rr$aggregate()
classif.ce 
     0.256 

Essentially, this is the performance of a “knn with optimal hyperparameters found by grid search”. Note that grid_auto is not changed since resample() creates a clone for each resampling iteration. The trained AutoTuner objects can be accessed by using

rr$learners[[1]]
<AutoTuner:classif.kknn.tuned>
* Model: list
* Parameters: list()
* Packages: kknn
* Predict Type: prob
* Feature types: logical, integer, numeric, factor, ordered
* Properties: multiclass, twoclass
rr$learners[[1]]$tuning_result
   k distance learner_param_vals  x_domain classif.ce
1: 9        2          <list[3]> <list[2]>       0.26

Appendix

Example: Tuning With A Larger Budget

It is always interesting to look at what could have been. The following dataset contains an optimization run result with 3600 evaluations – more than above by a factor of 100:

             k distance   kernel scale classif.ce                                uhash           timestamp batch_nr
   1: 2.191216 2.232217 gaussian FALSE      0.312 0bac4d72-2dad-4d0f-8502-22cc27284bb2 2020-12-09 15:08:29        1
   2: 3.549142 1.058476     rank FALSE      0.296 fe8de768-e91a-4a89-9fe2-521083a24466 2020-12-09 15:08:29        1
   3: 2.835727 2.121690  optimal  TRUE      0.251 bbeab2d5-19aa-4382-a351-d5ece72a58a2 2020-12-09 15:08:29        1
   4: 1.118085 1.275450     rank FALSE      0.368 93b5f285-66b9-4ea6-a8fc-700a11ed08a2 2020-12-09 15:08:29        1
   5: 2.790168 2.126899  optimal FALSE      0.320 1ab6ca63-96db-414e-9932-67f23592aab9 2020-12-09 15:08:29        1
  ---                                                                                                              
3596: 3.023075 1.413180  optimal FALSE      0.306 ae870063-e324-4f69-bfea-7c973ccabb02 2020-12-09 16:05:45      100
3597: 3.243131 1.827885 gaussian  TRUE      0.255 8f2d97f6-9ead-482d-96df-dade4277054c 2020-12-09 16:05:45      100
3598: 1.628957 2.254808     rank  TRUE      0.271 5b1d82a7-d330-48a6-b4be-af06d6a73a4c 2020-12-09 16:05:45      100
3599: 3.298112 2.984946  optimal FALSE      0.301 9e77de4a-f3ab-43fc-a89b-987e53be04a4 2020-12-09 16:05:45      100
3600: 3.855455 2.613641 gaussian FALSE      0.294 1ce1c23f-a756-482a-8b46-65abb1270fd2 2020-12-09 16:05:45      100
      x_domain_k x_domain_distance x_domain_kernel x_domain_scale
   1:          9          2.232217        gaussian          FALSE
   2:         35          1.058476            rank          FALSE
   3:         17          2.121690         optimal           TRUE
   4:          3          1.275450            rank          FALSE
   5:         16          2.126899         optimal          FALSE
  ---                                                            
3596:         21          1.413180         optimal          FALSE
3597:         26          1.827885        gaussian           TRUE
3598:          5          2.254808            rank           TRUE
3599:         27          2.984946         optimal          FALSE
3600:         47          2.613641        gaussian          FALSE

The scale effect is just as visible as before with fewer data:

ggplot(perfdata, aes(x = x_domain_k, y = classif.ce, color = scale)) +
  geom_point(size = 2, alpha = 0.3)

Now, there seems to be a visible pattern by kernel as well:

ggplot(perfdata, aes(x = x_domain_k, y = classif.ce, color = kernel)) +
  geom_point(size = 2, alpha = 0.3)

In fact, if we zoom in to (5, 40) \(\times\) (0.23, 0.28) and do decrease smoothing we see that different kernels have their optimum at different values of k:

ggplot(perfdata, aes(x = x_domain_k, y = classif.ce, color = kernel,
  group = interaction(kernel, scale))) +
  geom_point(size = 2, alpha = 0.3) + geom_smooth() +
  xlim(5, 40) + ylim(0.23, 0.28)

What about the distance parameter? If we select all results with k between 10 and 20 and plot distance and kernel we see an approximate relationship:

ggplot(perfdata[x_domain_k > 10 & x_domain_k < 20 & scale == TRUE],
  aes(x = distance, y = classif.ce, color = kernel)) +
  geom_point(size = 2) + geom_smooth()

In sum our observations are: The scale parameter is very influential, and scaling is beneficial. The distance type seems to be the least influential. Their seems to be an interaction between ‘k’ and ‘kernel’.

Citation

For attribution, please cite this work as

Binder & Pfisterer (2020, March 11). mlr3gallery: mlr3tuning Tutorial - German Credit. Retrieved from https://mlr3gallery.mlr-org.com/posts/2020-03-11-mlr3tuning-tutorial-german-credit/

BibTeX citation

@misc{binder2020mlr3tuning,
  author = {Binder, Martin and Pfisterer, Florian},
  title = {mlr3gallery: mlr3tuning Tutorial - German Credit},
  url = {https://mlr3gallery.mlr-org.com/posts/2020-03-11-mlr3tuning-tutorial-german-credit/},
  year = {2020}
}