When you tune the model hyperparameters you iteratively adjust the
hyperparameters while monitoring the changes in the evaluation metric
computed using the testing dataset. In this process, the information
contained in the testing dataset leaks in the model and therefore, at
the end of the process, the testing dataset doesn’t represent anymore an
independent set to evaluate the model Chollet and
Allaire (2018). A better strategy, than splitting the observation
locations in training and testing, would be to split them into training,
validation and testing datasets. The training dataset is then used to
train the model, the validation datasets to drive the hyperparameter
tuning and the testing dataset to evaluate the tuned model. The function
`trainValTest`

allows to split the data in three folds
containing the provided percentage of data. For illustration purpose
let’s split the presence locations in training (60%), validation (20%)
and testing (20%) datasets. The following steps are described in the
**basic-use** vignette, refer to it if the following code
is not clear:

```
library(SDMtune)
library(zeallot)
# Prepare data
<- list.files(path = file.path(system.file(package = "dismo"), "ex"),
files pattern = "grd", full.names = TRUE)
<- raster::stack(files)
predictors <- prepareSWD(species = "Virtual species", p = virtualSp$presence,
data a = virtualSp$background, env = predictors,
categorical = "biome")
# Split data in training, validation and testing datasets
c(train, val, test) %<-% trainValTest(data, val = 0.2, test = 0.2,
only_presence = TRUE, seed = 61516)
cat("# Training : ", nrow(train@data))
cat("\n# Validation: ", nrow(val@data))
cat("\n# Testing : ", nrow(test@data))
# Train Maxnet model with default settings
<- train("Maxnet", data = train) model
```

To see the effect of varying one hyperparameter on the model
performance we can use the function `gridSearch`

. The
function iterates through a set of predefined hyperparameter values,
train the model and displays in real-time the evaluation metric in the
RStudio viewer pane (hover over the points to get a tooltip with extra
information). Let’s see how the AUC changes varying the regularization
multiplier. First we have to define the values for the hyperparameter
that we want to test. For that we create a named list that we will use
as an argument for the function `gridSearch`

:

```
# Define the values for bg
<- list(reg = seq(0.2, 1, 0.1))
h # Call the gridSearch function
<- gridSearch(model, hypers = h, metric = "auc", test = val) exp_1
```

As you noticed we used the validation dataset as test argument. The
output of the function is an object of class `SDMtune`

. Let’s
print it:

` exp_1`

When you print the output, the text contains the models configuration
that have been used during the execution of the function. In our case,
only the regularization multiplier `reg`

has multiple values.
You can plot the `SDMtune`

object:

`plot(exp_1, title = "Experiment 1")`

and you can also recreate the interactive chart using:

`plot(exp_1, title = "Experiment 1", interactive = TRUE)`

The `SDMtune`

object stores the results in the slot
`@results`

:

`@results exp_1`

You can order them with:

`@results[order(-exp_1@results$test_AUC), ] exp_1`

In the next example we check how the TSS changes varying the regularization multiplier from 1 to 4:

```
# Define the values for reg
<- list(reg = 1:4)
h # Call the gridSearch function
<- gridSearch(model, hypers = h, metric = "tss", test = val) exp_2
```

and how AUC changes varying the feature combinations using the following values: l, lq, lh, lqp, lqph and lqpht:

```
# Define the values for fc
<- list(fc = c("l", "lq", "lh", "lqp", "lqph", "lqpht"))
h # Call the gridSearch function
<- gridSearch(model, hypers = h, metric = "auc", test = val) exp_3
```

Train a **Maxent** model and see how the AUC changes
varying the number of iterations from 300 to 1100 with increments of 200
(highlight to see the solution):

```
<- train("Maxent", data = data)
maxent_model # Define the values for fc
<- list("iter" = seq(300, 1100, 200))
h # Call the gridSearch function
<- gridSearch(maxent_model, hypers = h, metric = "auc", test = val) exp_4
```

To see which hyperparameters can be tuned in a given model use the
function `getTunableArgs`

. For example:

`getTunableArgs(model)`

To tune the model hyperparameters you should run all the possible combinations of hyperparameters. Here is an example using combinations of regularization multiplier and feature classes:

```
<- list(reg = seq(0.2, 2, 0.2),
h fc = c("l", "lq", "lh", "lqp", "lqph", "lqpht"))
<- gridSearch(model, hypers = h, metric = "auc", test = val) exp_5
```

This code takes already quite long as it has to train 60 models.
Imagine if you want to check more values for the regularization
multiplier and maybe add the number of iterations (in the case of a
**Maxent** model). The number of models to be trained
increases exponentially and consequently the execution time. In the next
two paragraphs we will present two possible alternative to the
`gridSearch`

function.

The function `randomSearch`

trains models taking a random
sample of the predefined configurations. In the next example we select
10 random configurations:

```
<- list(reg = seq(0.2, 5, 0.2), fc = c("l", "lq", "lh", "lp", "lqp", "lqph"))
h <- randomSearch(model, hypers = h, metric = "auc", test = val, pop = 10,
exp_6 seed = 65466)
```

The real-time chart plots two different graphs, one with the chosen
metric for each trained model and one with the evaluation metric for the
starting and the best found model. As you can see, the function is able
to find a better combination of the model hyperparameters compared to
the starting model; and this training only 10 instead of 150 models. The
results includes the 10 trained model. If you are not happy with the
solution, you can check the best hyperparameter combinations and this
gives you an intuition of which ones are the hyperparameters to “refine”
using the function `gridSearch`

. The `SDMtune`

object stores the results in a `data.frame`

than can be
accessed with the following command:

`@results exp_6`

The previous function doesn’t learn anything from the trained models,
it just selects n random combinations of hyperparameters. The function
`optimizeModel`

instead uses a *genetic algorithm* to
find an optimum or near optimum solution. Check the function
documentation to understand how it works, here we provide the code to
execute it:

```
<- optimizeModel(model, hypers = h, metric = "auc", test = val, pop = 15,
exp_7 gen = 2, seed = 798)
```

Let’s say we want to use the best tuned model found by the
`randomSearch`

function. Before evaluating the model using
the testing dataset, we can merge the training and the validation
datasets together to increase the number of locations and train a new
model with the merged observations and the tuned configuration. At this
point we may have removed variables using the `varSel`

or
`reduceVar`

function. If this is the case, we cannot merge
directly the initial datasets which contain all the environmental
variables. We can extract the train dataset with the selected variables
from the output of the experiment and merge it with the validation
dataset using the function `mergeSWD`

:

```
<- which.max(exp_6@results$test_AUC) # Index of the best model in the experiment
index <- exp_6@models[[index]]@data # New train dataset containing only the selected variables
new_train <- mergeSWD(new_train, val, only_presence = TRUE) # Merge only presence data merged_data
```

The `val`

dataset contains all the initial environmental
variables but the `mergeSWD`

function will merge only those
that are present in both datasets (in case you have performed variable
selection).

Then we get the model configuration from the experiment 6:

```
<- train("Maxnet", data = merged_data, fc = exp_6@results[index, 1],
final_model reg = exp_6@results[index, 2])
```

Now we can evaluate the final model using the held apart testing dataset:

`auc(final_model, test = test)`

Another approach would be to split the data in two folds: training and testing, use the cross validation strategy with the training dataset to tune the model hyperparameters, and evaluate the tuned model with the unseen held apart testing dataset.

```
# Create the folds from the training dataset
<- randomFolds(train, k = 4, only_presence = TRUE, seed = 25)
folds # Train the model
<- train("Maxent", data = train, folds = folds) cv_model
```

All the previous examples can be applied to the cross validation,
here an example with `randomSearch`

(note that in this case
the testing dataset is not provided as is taken from the folds stored in
the `SDMmodelCV`

):

```
<- list(reg = seq(0.2, 5, 0.2), fc = c("l", "lq", "lh", "lp", "lqp", "lqph"))
h <- randomSearch(cv_model, hypers = h, metric = "auc", pop = 10,
exp_8 seed = 65466)
```

The function `randomSearch`

orders the models according to
the best value of the metric for the testing dataset. In this case we
can train a model using the best hyperparameters configuration and
without cross validation with:

```
<- train("Maxent", data = exp_8@models[[1]]@data,
final_model fc = exp_8@results[1, 1], reg = exp_8@results[1, 2])
auc(final_model, test = test)
```

Chollet, François, and J. J. Allaire. 2018. *Deep learning with R*. 1st ed. Manning
Publications Co.

Müller, Andreas C., and Sarah Guido. 2016. *Introduction to machine learning with Python : a guide
for data scientists*.