This function provides the functionality to leave the interface of mlr3 for the computation of benchmark experiments and switch over to batchtools for a more fine grained control over the execution.
batchmark() populates a batchtools::ExperimentRegistry with jobs in a mlr3::benchmark() fashion.
Each combination of mlr3::Task and mlr3::Resampling defines a batchtools::Problem,
each mlr3::Learner is an batchtools::Algorithm.
After the jobs have been submitted and are terminated, results can be collected with reduceResultsBatchmark()
which returns a mlr3::BenchmarkResult and thus to return to the interface of mlr3.
batchmark(design, store_models = FALSE, reg = batchtools::getDefaultRegistry())(data.frame())
Data frame (or data.table::data.table()) with three columns: "task", "learner", and "resampling".
Each row defines a resampling by providing a Task, Learner and an instantiated Resampling strategy.
The helper function benchmark_grid() can assist in generating an exhaustive design (see examples) and
instantiate the Resamplings per Task.
Additionally, you can set the additional column 'param_values', see benchmark_grid().
(logical(1))
Store the fitted model in the resulting object=
Set to TRUE if you want to further analyse the models or want to
extract information like variable importance.
data.table() with ids of created jobs (invisibly).
tasks = list(mlr3::tsk("iris"), mlr3::tsk("sonar"))
learners = list(mlr3::lrn("classif.featureless"), mlr3::lrn("classif.rpart"))
resamplings = list(mlr3::rsmp("cv", folds = 3), mlr3::rsmp("holdout"))
design = mlr3::benchmark_grid(
tasks = tasks,
learners = learners,
resamplings = resamplings
)
reg = batchtools::makeExperimentRegistry(NA)
#> No readable configuration file found
#> Created registry in '/tmp/RtmpiZub69/registry57242d3d4705' using cluster functions 'Interactive'
batchmark(design, reg = reg)
#> Adding algorithm 'run_learner'
#> Adding problem '34a6beb27c3181c3'
#> Exporting new objects: 'a3b4bd84a3ce5fce' ...
#> Exporting new objects: '3431cc630a37510b' ...
#> Exporting new objects: '838c80b890e1544a' ...
#> Exporting new objects: 'ecf8ee265ec56766' ...
#> Overwriting previously exported object: 'ecf8ee265ec56766'
#> Adding 6 experiments ('34a6beb27c3181c3'[1] x 'run_learner'[2] x repls[3]) ...
#> Exporting new objects: '681f547de0d0db2f' ...
#> Adding 2 experiments ('34a6beb27c3181c3'[1] x 'run_learner'[2] x repls[1]) ...
#> Adding problem 'c0fcb04583948144'
#> Exporting new objects: 'cc50b6407545c334' ...
#> Adding 6 experiments ('c0fcb04583948144'[1] x 'run_learner'[2] x repls[3]) ...
#> Exporting new objects: 'e73b91a46604b440' ...
#> Adding 2 experiments ('c0fcb04583948144'[1] x 'run_learner'[2] x repls[1]) ...
batchtools::submitJobs(reg = reg)
#> Submitting 16 jobs in 16 chunks using cluster functions 'Interactive' ...
reduceResultsBatchmark(reg = reg)
#> <BenchmarkResult> of 16 rows with 8 resampling runs
#> nr task_id learner_id resampling_id iters warnings errors
#> 1 iris classif.featureless cv 3 0 0
#> 2 iris classif.rpart cv 3 0 0
#> 3 iris classif.featureless holdout 1 0 0
#> 4 iris classif.rpart holdout 1 0 0
#> 5 sonar classif.featureless cv 3 0 0
#> 6 sonar classif.rpart cv 3 0 0
#> 7 sonar classif.featureless holdout 1 0 0
#> 8 sonar classif.rpart holdout 1 0 0