This function provides the functionality to leave the interface of mlr3 for the computation of benchmark experiments and switch over to batchtools for a more fine grained control over the execution.

batchmark() populates a batchtools::ExperimentRegistry with jobs in a mlr3::benchmark() fashion. Each combination of mlr3::Task and mlr3::Resampling defines a batchtools::Problem, each mlr3::Learner is an batchtools::Algorithm.

After the jobs have been submitted and are terminated, results can be collected with reduceResultsBatchmark() which returns a mlr3::BenchmarkResult and thus to return to the interface of mlr3.

batchmark(design, store_models = FALSE, reg = batchtools::getDefaultRegistry())

Arguments

design

(data.frame())
Data frame (or data.table::data.table()) with three columns: "task", "learner", and "resampling". Each row defines a resampling by providing a Task, Learner and an instantiated Resampling strategy. The helper function benchmark_grid() can assist in generating an exhaustive design (see examples) and instantiate the Resamplings per Task. Additionally, you can set the additional column 'param_values', see benchmark_grid().

store_models

(logical(1))
Store the fitted model in the resulting object= Set to TRUE if you want to further analyse the models or want to extract information like variable importance.

reg

batchtools::ExperimentRegistry.

Value

data.table() with ids of created jobs (invisibly).

Examples

tasks = list(mlr3::tsk("iris"), mlr3::tsk("sonar"))
learners = list(mlr3::lrn("classif.featureless"), mlr3::lrn("classif.rpart"))
resamplings = list(mlr3::rsmp("cv", folds = 3), mlr3::rsmp("holdout"))

design = mlr3::benchmark_grid(
  tasks = tasks,
  learners = learners,
  resamplings = resamplings
)

reg = batchtools::makeExperimentRegistry(NA)
#> No readable configuration file found
#> Created registry in '/tmp/RtmpiZub69/registry57242d3d4705' using cluster functions 'Interactive'
batchmark(design, reg = reg)
#> Adding algorithm 'run_learner'
#> Adding problem '34a6beb27c3181c3'
#> Exporting new objects: 'a3b4bd84a3ce5fce' ...
#> Exporting new objects: '3431cc630a37510b' ...
#> Exporting new objects: '838c80b890e1544a' ...
#> Exporting new objects: 'ecf8ee265ec56766' ...
#> Overwriting previously exported object: 'ecf8ee265ec56766'
#> Adding 6 experiments ('34a6beb27c3181c3'[1] x 'run_learner'[2] x repls[3]) ...
#> Exporting new objects: '681f547de0d0db2f' ...
#> Adding 2 experiments ('34a6beb27c3181c3'[1] x 'run_learner'[2] x repls[1]) ...
#> Adding problem 'c0fcb04583948144'
#> Exporting new objects: 'cc50b6407545c334' ...
#> Adding 6 experiments ('c0fcb04583948144'[1] x 'run_learner'[2] x repls[3]) ...
#> Exporting new objects: 'e73b91a46604b440' ...
#> Adding 2 experiments ('c0fcb04583948144'[1] x 'run_learner'[2] x repls[1]) ...
batchtools::submitJobs(reg = reg)
#> Submitting 16 jobs in 16 chunks using cluster functions 'Interactive' ...

reduceResultsBatchmark(reg = reg)
#> <BenchmarkResult> of 16 rows with 8 resampling runs
#>  nr task_id          learner_id resampling_id iters warnings errors
#>   1    iris classif.featureless            cv     3        0      0
#>   2    iris       classif.rpart            cv     3        0      0
#>   3    iris classif.featureless       holdout     1        0      0
#>   4    iris       classif.rpart       holdout     1        0      0
#>   5   sonar classif.featureless            cv     3        0      0
#>   6   sonar       classif.rpart            cv     3        0      0
#>   7   sonar classif.featureless       holdout     1        0      0
#>   8   sonar       classif.rpart       holdout     1        0      0