Feature Selection on the Titanic Data Set

mlr3fselect optimization feature selection nested resampling titanic data set classification

Short introduction to feature selection with mlr3fselect.

Marc Becker
01-08-2021

Introduction

In this tutorial, we introduce the mlr3fselect package by comparing feature selection methods on the Titanic disaster data set. The objective of feature selection is to enhance the interpretability of models, speed up the learning process and increase the predictive performance.

We load the mlr3verse package which pulls in the most important packages for this example.

We initialize the random number generator with a fixed seed for reproducibility, and decrease the verbosity of the logger to keep the output clearly represented.

set.seed(7832)
lgr::get_logger("mlr3")$set_threshold("warn")
lgr::get_logger("bbotk")$set_threshold("warn")

Titanic Data Set

The Titanic data set contains data for 887 Titanic passengers, including whether they survived when the Titanic sank. Our goal will be to predict the survival of the Titanic passengers.

After loading the data set from the mlr3data package, we impute the missing age values with the median age of the passengers, set missing embarked values to "s" and remove character features. We could use feature engineering to create new features from the character features, however we want to focus on feature selection in this tutorial.

In addition to the survived column, the reduced data set contains the following attributes for each passenger:

Feature Description
age Age
sex Sex
sib_sp Number of siblings / spouses aboard
parch Number of parents / children aboard
fare Amount paid for the ticket
pc_class Passenger class
embarked Port of embarkation
library(mlr3data)

data("titanic", package = "mlr3data")
titanic$age[is.na(titanic$age)] = median(titanic$age, na.rm = TRUE)
titanic$embarked[is.na(titanic$embarked)] = "S"
titanic$ticket = NULL
titanic$name = NULL
titanic$cabin = NULL
titanic = titanic[!is.na(titanic$survived),]

We construct a binary classification task.

task = as_task_classif(titanic, target = "survived", positive = "yes")

Model

We use the logistic regression learner provided by the mlr3learners package.

library(mlr3learners)

learner = lrn("classif.log_reg")

To evaluate the predictive performance, we choose a 3-fold cross-validation and the classification error as the measure.

resampling = rsmp("cv", folds = 3)
measure = msr("classif.ce")

resampling$instantiate(task)

Classes

The FSelectInstanceSingleCrit class specifies a general feature selection scenario. It includes the ObjectiveFSelect object that encodes the black box objective function which is optimized by a feature selection algorithm. The evaluated feature sets are stored in an ArchiveFSelect object. The archive provides a method for querying the best performing feature set.

The Terminator classes determine when to stop the feature selection. In this example we choose a terminator that stops the feature selection after 10 seconds. The sugar functions trm() and trms() can be used to retrieve terminators from the mlr_terminators dictionary.

terminator = trm("run_time", secs = 10)
FSelectInstanceSingleCrit$new(
  task = task,
  learner = learner,
  resampling = resampling,
  measure = measure,
  terminator = terminator)
<FSelectInstanceSingleCrit>
* State:  Not optimized
* Objective: <ObjectiveFSelect:classif.log_reg_on_titanic>
* Search Space:
<ParamSet>
         id    class lower upper nlevels        default value
1:      age ParamLgl    NA    NA       2 <NoDefault[3]>      
2: embarked ParamLgl    NA    NA       2 <NoDefault[3]>      
3:     fare ParamLgl    NA    NA       2 <NoDefault[3]>      
4:    parch ParamLgl    NA    NA       2 <NoDefault[3]>      
5:   pclass ParamLgl    NA    NA       2 <NoDefault[3]>      
6:      sex ParamLgl    NA    NA       2 <NoDefault[3]>      
7:   sib_sp ParamLgl    NA    NA       2 <NoDefault[3]>      
* Terminator: <TerminatorRunTime>
* Terminated: FALSE
* Archive:
<ArchiveFSelect>
Null data.table (0 rows and 0 cols)

The FSelector subclasses describe the feature selection strategy. The sugar function fs() can be used to retrieve feature selection algorithms from the mlr_fselectors dictionary.

mlr_fselectors
<DictionaryFSelect> with 7 stored values
Keys: design_points, exhaustive_search, genetic_search, random_search, rfe, sequential,
  shadow_variable_search

Random search randomly draws feature sets and evaluates them in batches. We retrieve the FSelectorRandomSearch class with the fs() sugar function and choose TerminatorEvals. We set the n_evals parameter to 10 which means that 10 feature sets are evaluated.

terminator = trm("evals", n_evals = 10)
instance = FSelectInstanceSingleCrit$new(
  task = task,
  learner = learner,
  resampling = resampling,
  measure = measure,
  terminator = terminator)
fselector = fs("random_search", batch_size = 5)

The feature selection is started by passing the FSelectInstanceSingleCrit object to the $optimize() method of FSelectorRandomSearch which generates the feature sets. These features set are internally passed to the $eval_batch() method of FSelectInstanceSingleCrit which evaluates them with the objective function and stores the results in the archive. This general interaction between the objects of mlr3fselect stays the same for the different feature selection methods. However, the way how new feature sets are generated differs depending on the chosen FSelector subclass.

fselector$optimize(instance)
    age embarked fare parch pclass  sex sib_sp                         features classif.ce
1: TRUE    FALSE TRUE  TRUE   TRUE TRUE   TRUE age,fare,parch,pclass,sex,sib_sp  0.2020202

The ArchiveFSelect stores a data.table::data.table() which consists of the evaluated feature sets and the corresponding estimated predictive performances.

as.data.table(instance$archive)
age embarked fare parch pclass sex sib_sp classif.ce timestamp batch_nr
TRUE TRUE TRUE TRUE TRUE TRUE TRUE 0.2031425 2021-12-03 04:53:49 1
TRUE FALSE FALSE FALSE FALSE FALSE TRUE 0.3838384 2021-12-03 04:53:49 1
FALSE FALSE FALSE TRUE FALSE FALSE TRUE 0.3804714 2021-12-03 04:53:49 1
FALSE FALSE TRUE FALSE FALSE FALSE FALSE 0.3288440 2021-12-03 04:53:49 1
FALSE FALSE TRUE FALSE FALSE TRUE FALSE 0.2188552 2021-12-03 04:53:49 1
FALSE FALSE FALSE FALSE TRUE FALSE FALSE 0.3209877 2021-12-03 04:53:52 2
TRUE FALSE FALSE FALSE FALSE FALSE TRUE 0.3838384 2021-12-03 04:53:52 2
TRUE FALSE TRUE TRUE TRUE TRUE TRUE 0.2020202 2021-12-03 04:53:52 2
TRUE TRUE TRUE TRUE TRUE TRUE TRUE 0.2031425 2021-12-03 04:53:52 2
TRUE FALSE TRUE TRUE FALSE FALSE FALSE 0.3389450 2021-12-03 04:53:52 2

The associated resampling iterations can be accessed in the BenchmarkResult by calling

instance$archive$benchmark_result
<BenchmarkResult> of 30 rows with 10 resampling runs
 nr task_id             learner_id resampling_id iters warnings errors
  1 titanic select.classif.log_reg            cv     3        0      0
  2 titanic select.classif.log_reg            cv     3        0      0
  3 titanic select.classif.log_reg            cv     3        0      0
  4 titanic select.classif.log_reg            cv     3        0      0
  5 titanic select.classif.log_reg            cv     3        0      0
  6 titanic select.classif.log_reg            cv     3        0      0
  7 titanic select.classif.log_reg            cv     3        0      0
  8 titanic select.classif.log_reg            cv     3        0      0
  9 titanic select.classif.log_reg            cv     3        0      0
 10 titanic select.classif.log_reg            cv     3        0      0

We retrieve the best performing feature set with

instance$result
    age embarked fare parch pclass  sex sib_sp                         features classif.ce
1: TRUE    FALSE TRUE  TRUE   TRUE TRUE   TRUE age,fare,parch,pclass,sex,sib_sp  0.2020202

Sequential forward selection

We try sequential forward selection. We chose TerminatorStagnation that stops the feature selection if the predictive performance does not increase anymore.

terminator = trm("stagnation", iters = 5)
instance = FSelectInstanceSingleCrit$new(
  task = task,
  learner = learner,
  resampling = resampling,
  measure = measure,
  terminator = terminator)

fselector = fs("sequential")
fselector$optimize(instance)
     age embarked  fare parch pclass  sex sib_sp                features classif.ce
1: FALSE    FALSE FALSE  TRUE   TRUE TRUE   TRUE parch,pclass,sex,sib_sp  0.1964085

The FSelectorSequential object has a special method for displaying the optimization path of the sequential feature selection.

fselector$optimization_path(instance)
    age embarked  fare parch pclass   sex sib_sp classif.ce batch_nr
1: TRUE    FALSE FALSE FALSE  FALSE FALSE  FALSE  0.3838384        1
2: TRUE    FALSE FALSE FALSE  FALSE  TRUE  FALSE  0.2132435        2
3: TRUE    FALSE FALSE FALSE  FALSE  TRUE   TRUE  0.2087542        3
4: TRUE    FALSE FALSE FALSE   TRUE  TRUE   TRUE  0.2143659        4
5: TRUE    FALSE FALSE  TRUE   TRUE  TRUE   TRUE  0.2065095        5
6: TRUE    FALSE  TRUE  TRUE   TRUE  TRUE   TRUE  0.2020202        6

Recursive feature elimination

Recursive feature elimination utilizes the $importance() method of learners. In each iteration the feature(s) with the lowest importance score is dropped. We choose the non-recursive algorithm (recursive = FALSE) which calculates the feature importance once on the complete feature set. The recursive version (recursive = TRUE) recomputes the feature importance on the reduced feature set in every iteration.

learner = lrn("classif.ranger", importance = "impurity")
terminator = trm("none")
instance = FSelectInstanceSingleCrit$new(
  task = task,
  learner = learner,
  resampling = resampling,
  measure = measure,
  terminator = terminator,
  store_models = TRUE)

fselector = fs("rfe", recursive = FALSE)
fselector$optimize(instance)
    age embarked fare parch pclass  sex sib_sp                               features classif.ce
1: TRUE     TRUE TRUE  TRUE   TRUE TRUE   TRUE age,embarked,fare,parch,pclass,sex,...  0.1694725

We access the results.

as.data.table(instance$archive, exclude_columns = c("runtime_learners", "timestamp", "batch_nr", "resample_result", "uhash"))
age embarked fare parch pclass sex sib_sp classif.ce importance
TRUE TRUE TRUE TRUE TRUE TRUE TRUE 0.1694725 68.711046, 45.345443, 37.314977, 22.890378, 11.733529, 9.171937, 8.422415
TRUE FALSE TRUE FALSE FALSE TRUE FALSE 0.2143659 68.71105, 45.34544, 37.31498
FALSE FALSE FALSE FALSE FALSE TRUE FALSE 0.2132435 68.71105

Nested resampling

It is a common mistake to report the predictive performance estimated on resampling sets during the feature selection as the performance that can be expected from the combined feature selection and model training. The repeated evaluation of the model might leak information about the test sets into the model and thus leads to over-fitting and over-optimistic performance results. Nested resampling uses an outer and inner resampling to separate the feature selection from the performance estimation of the model. We can use the AutoFSelector class for running nested resampling. The AutoFSelector essentially combines a given Learner and feature selection method into a Learner with internal automatic feature selection. The inner resampling loop that is used to determine the best feature set is conducted internally each time the AutoFSelector Learner object is trained.

resampling_inner = rsmp("cv", folds = 5)
measure = msr("classif.ce")

at = AutoFSelector$new(
  learner = learner,
  resampling = resampling_inner,
  measure = measure,
  terminator = terminator,
  fselect = fs("sequential"),
  store_models = TRUE)

We put the AutoFSelector into a resample() call to get the outer resampling loop.

resampling_outer = rsmp("cv", folds = 3)

rr = resample(task, at, resampling_outer, store_models = TRUE)

The aggregated performance of all outer resampling iterations is the unbiased predictive performance we can expected from the logistic regression model with an optimized feature set found by sequential selection.

rr$aggregate()
classif.ce 
 0.1840629 

We check whether the feature sets that were selected in the inner resampling are stable. The selected feature sets should not differ too much. We might observe unstable models in this example because the small data set and the low number of resampling iterations might introduces too much randomness. Usually, we aim for the selection of similar feature sets for all outer training sets.

iteration age embarked fare parch pclass sex sib_sp classif.ce features task_id learner_id resampling_id
1 TRUE TRUE TRUE FALSE TRUE TRUE TRUE 0.1599202 age , embarked, fare , pclass , sex , sib_sp titanic classif.ranger.fselector cv
2 TRUE TRUE FALSE FALSE TRUE TRUE TRUE 0.1497650 age , embarked, pclass , sex , sib_sp titanic classif.ranger.fselector cv
3 TRUE TRUE FALSE TRUE FALSE TRUE TRUE 0.1902293 age , embarked, parch , sex , sib_sp titanic classif.ranger.fselector cv

Next, we want to compare the predictive performances estimated on the outer resampling to the inner resampling. Significantly lower predictive performances on the outer resampling indicate that the models with the optimized feature sets overfit the data.

rr$score()
iteration task_id learner_id resampling_id classif.ce
1 titanic classif.ranger.fselector cv 0.1649832
2 titanic classif.ranger.fselector cv 0.2289562
3 titanic classif.ranger.fselector cv 0.1582492

The archives of the AutoFSelectors gives us all evaluated feature sets with the associated predictive performances.

iteration age embarked fare parch pclass sex sib_sp classif.ce runtime_learners timestamp batch_nr resample_result task_id learner_id resampling_id
1 TRUE FALSE FALSE FALSE FALSE FALSE FALSE 0.4090585 1.005 2021-06-14 09:50:10 1 <environment: 0x56194a323ed0> titanic classif.ranger.fselector cv
1 FALSE TRUE FALSE FALSE FALSE FALSE FALSE 0.3635949 1.140 2021-06-14 09:50:10 1 <environment: 0x56194a2587d0> titanic classif.ranger.fselector cv
1 FALSE FALSE TRUE FALSE FALSE FALSE FALSE 0.3316906 1.268 2021-06-14 09:50:10 1 <environment: 0x56194a217950> titanic classif.ranger.fselector cv
1 FALSE FALSE FALSE TRUE FALSE FALSE FALSE 0.3754024 0.990 2021-06-14 09:50:10 1 <environment: 0x56194a1eda68> titanic classif.ranger.fselector cv
1 FALSE FALSE FALSE FALSE TRUE FALSE FALSE 0.3114798 0.657 2021-06-14 09:50:10 1 <environment: 0x56194a190c10> titanic classif.ranger.fselector cv
1 FALSE FALSE FALSE FALSE FALSE TRUE FALSE 0.2254949 0.752 2021-06-14 09:50:10 1 <environment: 0x56194a1734d0> titanic classif.ranger.fselector cv
1 FALSE FALSE FALSE FALSE FALSE FALSE TRUE 0.3720553 0.663 2021-06-14 09:50:10 1 <environment: 0x56194a145fe8> titanic classif.ranger.fselector cv
1 TRUE FALSE FALSE FALSE FALSE TRUE FALSE 0.2170916 0.716 2021-06-14 09:50:16 2 <environment: 0x56194a114b80> titanic classif.ranger.fselector cv
1 FALSE TRUE FALSE FALSE FALSE TRUE FALSE 0.2254949 0.700 2021-06-14 09:50:16 2 <environment: 0x56194a0b8038> titanic classif.ranger.fselector cv
1 FALSE FALSE TRUE FALSE FALSE TRUE FALSE 0.2254949 0.795 2021-06-14 09:50:16 2 <environment: 0x56194a08e790> titanic classif.ranger.fselector cv
1 FALSE FALSE FALSE TRUE FALSE TRUE FALSE 0.2238143 0.710 2021-06-14 09:50:16 2 <environment: 0x56194a068e10> titanic classif.ranger.fselector cv
1 FALSE FALSE FALSE FALSE TRUE TRUE FALSE 0.2238428 1.062 2021-06-14 09:50:16 2 <environment: 0x56194a03b6d0> titanic classif.ranger.fselector cv
1 FALSE FALSE FALSE FALSE FALSE TRUE TRUE 0.2221478 0.695 2021-06-14 09:50:16 2 <environment: 0x561949ffcb98> titanic classif.ranger.fselector cv
1 TRUE TRUE FALSE FALSE FALSE TRUE FALSE 0.2221336 0.817 2021-06-14 09:50:22 3 <environment: 0x561949fbee98> titanic classif.ranger.fselector cv
1 TRUE FALSE TRUE FALSE FALSE TRUE FALSE 0.2221336 0.897 2021-06-14 09:50:22 3 <environment: 0x561949fa5218> titanic classif.ranger.fselector cv
1 TRUE FALSE FALSE TRUE FALSE TRUE FALSE 0.2137445 0.791 2021-06-14 09:50:22 3 <environment: 0x561949f7e980> titanic classif.ranger.fselector cv
1 TRUE FALSE FALSE FALSE TRUE TRUE FALSE 0.2053269 0.787 2021-06-14 09:50:22 3 <environment: 0x561949f4d150> titanic classif.ranger.fselector cv
1 TRUE FALSE FALSE FALSE FALSE TRUE TRUE 0.2036320 0.799 2021-06-14 09:50:22 3 <environment: 0x561949ede6d0> titanic classif.ranger.fselector cv
1 TRUE TRUE FALSE FALSE FALSE TRUE TRUE 0.2087025 1.053 2021-06-14 09:50:28 4 <environment: 0x561949ea18c0> titanic classif.ranger.fselector cv
1 TRUE FALSE TRUE FALSE FALSE TRUE TRUE 0.1986469 1.199 2021-06-14 09:50:28 4 <environment: 0x561949e50a68> titanic classif.ranger.fselector cv
1 TRUE FALSE FALSE TRUE FALSE TRUE TRUE 0.2019798 0.963 2021-06-14 09:50:28 4 <environment: 0x561949e1b210> titanic classif.ranger.fselector cv
1 TRUE FALSE FALSE FALSE TRUE TRUE TRUE 0.2036604 0.970 2021-06-14 09:50:28 4 <environment: 0x561949dfdac0> titanic classif.ranger.fselector cv
1 TRUE TRUE TRUE FALSE FALSE TRUE TRUE 0.1952713 1.631 2021-06-14 09:50:33 5 <environment: 0x561949dbbf58> titanic classif.ranger.fselector cv
1 TRUE FALSE TRUE TRUE FALSE TRUE TRUE 0.1919100 1.100 2021-06-14 09:50:33 5 <environment: 0x561949d7ad48> titanic classif.ranger.fselector cv
1 TRUE FALSE TRUE FALSE TRUE TRUE TRUE 0.1817690 1.086 2021-06-14 09:50:33 5 <environment: 0x561949d4f720> titanic classif.ranger.fselector cv
1 TRUE TRUE TRUE FALSE TRUE TRUE TRUE 0.1599202 1.078 2021-06-14 09:50:36 6 <environment: 0x561948cd1988> titanic classif.ranger.fselector cv
1 TRUE FALSE TRUE TRUE TRUE TRUE TRUE 0.1716565 1.104 2021-06-14 09:50:36 6 <environment: 0x561947484488> titanic classif.ranger.fselector cv
1 TRUE TRUE TRUE TRUE TRUE TRUE TRUE 0.1666287 1.488 2021-06-14 09:50:38 7 <environment: 0x561947466980> titanic classif.ranger.fselector cv
2 TRUE FALSE FALSE FALSE FALSE FALSE FALSE 0.3973508 1.700 2021-06-14 09:50:10 1 <environment: 0x5619472106a0> titanic classif.ranger.fselector cv
2 FALSE TRUE FALSE FALSE FALSE FALSE FALSE 0.3366614 0.662 2021-06-14 09:50:10 1 <environment: 0x5619471e8420> titanic classif.ranger.fselector cv
2 FALSE FALSE TRUE FALSE FALSE FALSE FALSE 0.3147557 1.195 2021-06-14 09:50:10 1 <environment: 0x561947154b50> titanic classif.ranger.fselector cv
2 FALSE FALSE FALSE TRUE FALSE FALSE FALSE 0.3517875 1.073 2021-06-14 09:50:10 1 <environment: 0x56194710db40> titanic classif.ranger.fselector cv
2 FALSE FALSE FALSE FALSE TRUE FALSE FALSE 0.2963111 0.685 2021-06-14 09:50:10 1 <environment: 0x5619470c3f30> titanic classif.ranger.fselector cv
2 FALSE FALSE FALSE FALSE FALSE TRUE FALSE 0.1834354 0.651 2021-06-14 09:50:10 1 <environment: 0x56194707f698> titanic classif.ranger.fselector cv
2 FALSE FALSE FALSE FALSE FALSE FALSE TRUE 0.3383848 0.663 2021-06-14 09:50:10 1 <environment: 0x561947037d80> titanic classif.ranger.fselector cv
2 TRUE FALSE FALSE FALSE FALSE TRUE FALSE 0.1867968 0.748 2021-06-14 09:50:17 2 <environment: 0x561946fda978> titanic classif.ranger.fselector cv
2 FALSE TRUE FALSE FALSE FALSE TRUE FALSE 0.1834354 1.053 2021-06-14 09:50:17 2 <environment: 0x561946f8ef18> titanic classif.ranger.fselector cv
2 FALSE FALSE TRUE FALSE FALSE TRUE FALSE 0.1851161 0.781 2021-06-14 09:50:17 2 <environment: 0x561946f42d00> titanic classif.ranger.fselector cv
2 FALSE FALSE FALSE TRUE FALSE TRUE FALSE 0.1834354 0.687 2021-06-14 09:50:17 2 <environment: 0x561946f05338> titanic classif.ranger.fselector cv
2 FALSE FALSE FALSE FALSE TRUE TRUE FALSE 0.1834354 0.718 2021-06-14 09:50:17 2 <environment: 0x561946ecb2e0> titanic classif.ranger.fselector cv
2 FALSE FALSE FALSE FALSE FALSE TRUE TRUE 0.1817690 0.697 2021-06-14 09:50:17 2 <environment: 0x561946e97c80> titanic classif.ranger.fselector cv
2 TRUE FALSE FALSE FALSE FALSE TRUE TRUE 0.1716565 0.750 2021-06-14 09:50:22 3 <environment: 0x561946e58cd8> titanic classif.ranger.fselector cv
2 FALSE TRUE FALSE FALSE FALSE TRUE TRUE 0.1851303 0.752 2021-06-14 09:50:22 3 <environment: 0x561946e2f0b8> titanic classif.ranger.fselector cv
2 FALSE FALSE TRUE FALSE FALSE TRUE TRUE 0.1851303 0.807 2021-06-14 09:50:22 3 <environment: 0x561946dfab88> titanic classif.ranger.fselector cv
2 FALSE FALSE FALSE TRUE FALSE TRUE TRUE 0.1868110 0.712 2021-06-14 09:50:22 3 <environment: 0x561946dba468> titanic classif.ranger.fselector cv
2 FALSE FALSE FALSE FALSE TRUE TRUE TRUE 0.1817690 0.775 2021-06-14 09:50:22 3 <environment: 0x561946d647e0> titanic classif.ranger.fselector cv
2 TRUE TRUE FALSE FALSE FALSE TRUE TRUE 0.1733371 1.066 2021-06-14 09:50:28 4 <environment: 0x561946d1f438> titanic classif.ranger.fselector cv
2 TRUE FALSE TRUE FALSE FALSE TRUE TRUE 0.1700328 1.145 2021-06-14 09:50:28 4 <environment: 0x561946cdf310> titanic classif.ranger.fselector cv
2 TRUE FALSE FALSE TRUE FALSE TRUE TRUE 0.1733514 1.361 2021-06-14 09:50:28 4 <environment: 0x561946c951b0> titanic classif.ranger.fselector cv
2 TRUE FALSE FALSE FALSE TRUE TRUE TRUE 0.1548497 0.932 2021-06-14 09:50:28 4 <environment: 0x561946c4a210> titanic classif.ranger.fselector cv
2 TRUE TRUE FALSE FALSE TRUE TRUE TRUE 0.1497650 0.948 2021-06-14 09:50:32 5 <environment: 0x561946c31e48> titanic classif.ranger.fselector cv
2 TRUE FALSE TRUE FALSE TRUE TRUE TRUE 0.1616152 1.094 2021-06-14 09:50:32 5 <environment: 0x561946c06858> titanic classif.ranger.fselector cv
2 TRUE FALSE FALSE TRUE TRUE TRUE TRUE 0.1649765 0.950 2021-06-14 09:50:32 5 <environment: 0x5619456a4740> titanic classif.ranger.fselector cv
2 TRUE TRUE TRUE FALSE TRUE TRUE TRUE 0.1531833 1.054 2021-06-14 09:50:35 6 <environment: 0x561945684db0> titanic classif.ranger.fselector cv
2 TRUE TRUE FALSE TRUE TRUE TRUE TRUE 0.1582111 0.981 2021-06-14 09:50:35 6 <environment: 0x561945654c18> titanic classif.ranger.fselector cv
2 TRUE TRUE TRUE TRUE TRUE TRUE TRUE 0.1497792 1.114 2021-06-14 09:50:37 7 <environment: 0x5619456304b8> titanic classif.ranger.fselector cv
3 TRUE FALSE FALSE FALSE FALSE FALSE FALSE 0.4158809 1.486 2021-06-14 09:50:10 1 <environment: 0x5619454d7460> titanic classif.ranger.fselector cv
3 FALSE TRUE FALSE FALSE FALSE FALSE FALSE 0.3905569 0.692 2021-06-14 09:50:10 1 <environment: 0x5619454bac00> titanic classif.ranger.fselector cv
3 FALSE FALSE TRUE FALSE FALSE FALSE FALSE 0.3349665 1.271 2021-06-14 09:50:10 1 <environment: 0x561945497808> titanic classif.ranger.fselector cv
3 FALSE FALSE FALSE TRUE FALSE FALSE FALSE 0.3855291 0.677 2021-06-14 09:50:10 1 <environment: 0x561945471f08> titanic classif.ranger.fselector cv
3 FALSE FALSE FALSE FALSE TRUE FALSE FALSE 0.3517875 0.672 2021-06-14 09:50:10 1 <environment: 0x561945449380> titanic classif.ranger.fselector cv
3 FALSE FALSE FALSE FALSE FALSE TRUE FALSE 0.2306082 1.207 2021-06-14 09:50:10 1 <environment: 0x56194542ce68> titanic classif.ranger.fselector cv
3 FALSE FALSE FALSE FALSE FALSE FALSE TRUE 0.4241988 0.823 2021-06-14 09:50:10 1 <environment: 0x5619454071b0> titanic classif.ranger.fselector cv
3 TRUE FALSE FALSE FALSE FALSE TRUE FALSE 0.2221906 0.775 2021-06-14 09:50:17 2 <environment: 0x5619453dde60> titanic classif.ranger.fselector cv
3 FALSE TRUE FALSE FALSE FALSE TRUE FALSE 0.2306082 0.705 2021-06-14 09:50:17 2 <environment: 0x561946ba1628> titanic classif.ranger.fselector cv
3 FALSE FALSE TRUE FALSE FALSE TRUE FALSE 0.2306082 1.199 2021-06-14 09:50:17 2 <environment: 0x56194697c630> titanic classif.ranger.fselector cv
3 FALSE FALSE FALSE TRUE FALSE TRUE FALSE 0.2322888 0.733 2021-06-14 09:50:17 2 <environment: 0x56194695ca98> titanic classif.ranger.fselector cv
3 FALSE FALSE FALSE FALSE TRUE TRUE FALSE 0.2356929 0.740 2021-06-14 09:50:17 2 <environment: 0x56194693cb98> titanic classif.ranger.fselector cv
3 FALSE FALSE FALSE FALSE FALSE TRUE TRUE 0.2188292 0.721 2021-06-14 09:50:17 2 <environment: 0x56194691df60> titanic classif.ranger.fselector cv
3 TRUE FALSE FALSE FALSE FALSE TRUE TRUE 0.2087452 0.777 2021-06-14 09:50:22 3 <environment: 0x5619468f6d38> titanic classif.ranger.fselector cv
3 FALSE TRUE FALSE FALSE FALSE TRUE TRUE 0.2154679 0.763 2021-06-14 09:50:22 3 <environment: 0x5619468db8b0> titanic classif.ranger.fselector cv
3 FALSE FALSE TRUE FALSE FALSE TRUE TRUE 0.2171628 0.806 2021-06-14 09:50:22 3 <environment: 0x5619468bcd58> titanic classif.ranger.fselector cv
3 FALSE FALSE FALSE TRUE FALSE TRUE TRUE 0.2171486 0.774 2021-06-14 09:50:22 3 <environment: 0x5619468a72a0> titanic classif.ranger.fselector cv
3 FALSE FALSE FALSE FALSE TRUE TRUE TRUE 0.2154821 0.784 2021-06-14 09:50:22 3 <environment: 0x561946887de0> titanic classif.ranger.fselector cv
3 TRUE TRUE FALSE FALSE FALSE TRUE TRUE 0.2020225 1.118 2021-06-14 09:50:28 4 <environment: 0x56194686b020> titanic classif.ranger.fselector cv
3 TRUE FALSE TRUE FALSE FALSE TRUE TRUE 0.2221906 1.198 2021-06-14 09:50:28 4 <environment: 0x5619468475e0> titanic classif.ranger.fselector cv
3 TRUE FALSE FALSE TRUE FALSE TRUE TRUE 0.1919100 0.966 2021-06-14 09:50:28 4 <environment: 0x56194682c190> titanic classif.ranger.fselector cv
3 TRUE FALSE FALSE FALSE TRUE TRUE TRUE 0.1986469 0.993 2021-06-14 09:50:28 4 <environment: 0x56194680f868> titanic classif.ranger.fselector cv
3 TRUE TRUE FALSE TRUE FALSE TRUE TRUE 0.1902293 0.984 2021-06-14 09:50:33 5 <environment: 0x5619467f4530> titanic classif.ranger.fselector cv
3 TRUE FALSE TRUE TRUE FALSE TRUE TRUE 0.2188720 1.132 2021-06-14 09:50:33 5 <environment: 0x5619467c97b0> titanic classif.ranger.fselector cv
3 TRUE FALSE FALSE TRUE TRUE TRUE TRUE 0.1936334 1.002 2021-06-14 09:50:33 5 <environment: 0x56194679e158> titanic classif.ranger.fselector cv
3 TRUE TRUE TRUE TRUE FALSE TRUE TRUE 0.2239140 1.107 2021-06-14 09:50:36 6 <environment: 0x561946776cd8> titanic classif.ranger.fselector cv
3 TRUE TRUE FALSE TRUE TRUE TRUE TRUE 0.1986754 1.037 2021-06-14 09:50:36 6 <environment: 0x56194675be70> titanic classif.ranger.fselector cv
3 TRUE TRUE TRUE TRUE TRUE TRUE TRUE 0.2053981 1.113 2021-06-14 09:50:37 7 <environment: 0x561946734e88> titanic classif.ranger.fselector cv

Shortcuts

Selecting a feature subset can be shortened by using the fselect()-shortcut.

instance = fselect(
  method = "random_search",
  task = tsk("iris"),
  learner = lrn("classif.log_reg"),
  resampling = rsmp("cv", folds = 3),
  measure = msr("classif.ce"),
  term_evals = 10
)

Applying nested resampling can be shortened by using the fselect_nested()-shortcut.

rr = fselect_nested(
  method = "random_search",
  task = tsk("iris"),
  learner = lrn("classif.log_reg"),
  inner_resampling = rsmp ("cv", folds = 3),
  outer_resampling = rsmp("cv", folds = 3),
  measure = msr("classif.ce"),
  term_evals = 10
)

Citation

For attribution, please cite this work as

Becker (2021, Jan. 8). mlr3gallery: Feature Selection on the Titanic Data Set. Retrieved from https://mlr3gallery.mlr-org.com/posts/2020-09-14-mlr3fselect-basic/

BibTeX citation

@misc{becker2021feature,
  author = {Becker, Marc},
  title = {mlr3gallery: Feature Selection on the Titanic Data Set},
  url = {https://mlr3gallery.mlr-org.com/posts/2020-09-14-mlr3fselect-basic/},
  year = {2021}
}