1 Introduction

One benefit of working with R Markdown is that you can reproduce an analysis at the click of a button. Parameterized reports extend this concept one step further and allow users to specify one or more parameters to customize their analysis. This is useful when creating diagnostic plots because it lets you create your diagnostic plots once in an R Markdown template and then create those plots across multiple similar models.

We demonstrate how to generate typical diagnostic plots using pmplots and yspec on the Model Diagnostics page. Here we show you how to generate those plots from R Markdown templates and how, by changing the key inputs, you can render each template from an R script for multiple models with just a few lines of R code. (Figure 1).

Figure 1: Schematic showing where to find code for this projects diagnostic templates, the R scripts that render them and the format of the plots created

The output of these templates is flexible, here we make individual pdf and png versions of the report-ready plots for use in our Latex reports. Each template also generates a single html summary file too. We chose to render the summary file to an html (rather than a pdf) so that the rendered document includes a floating table of contents allowing easy navigation through all the summaries and diagnostics generated in the template. We include an example rendered diagnostics report to show what this output might look like.

Additional information about parameterized reports can be found on the R Markdown: The Definitive Guide website.

2 Tools used

2.1 MetrumRG packages

yspec Data specification, wrangling, and documentation for pharmacometrics.

pmplots Create exploratory and diagnostic plots for pharmacometrics.

mrggsave Label, arrange, and save annotated plots and figures.

bbr Manage, track, and report modeling activities, through simple R objects, with a user-friendly interface between R and NONMEM®.

2.2 CRAN packages

dplyr A grammar of data manipulation.

mrgmisc Format, manipulate and summarize data in the field of pharmacometrics.

3 Parameterized R Markdown templates

A parameterized report is particularly useful when you want to re-render the same Rmd code with distinct values for various key inputs. Parameters are declared once using the params field of the YAML section at the top of the Rmd document (see below) and then you can change these values for a given rendering by using the params argument in the rmarkdown::render function.

3.1 YAML header

The YAML section at the top of the diagnostics-report.Rmd looks like this:

---
title: "Report diagnostics"
output: 
  html_document:
    toc: true
    toc_float: true
    depth: 2
params:
  run: 102
  modelDir: "model/pk"
  script: "diagnostics-report.Rmd"
  yspec: "pk.yml"
  contCov: !r c("AGE","WT","ALB","EGFR")
  catCov: !r c("STUDY", "RF", "CP", "DOSE")
  etas: !r c("ETA1//ETA-KA", "ETA2//ETA-V/F", "ETA3//ETA-CL/F")
  drugNameUnits: "concentration (mg/L)"
  include_code: FALSE
  include_plots: TRUE
  run_mrggsave: TRUE
---

You first define the title and type of output (as you would for a typical R Markdown file): we chose an html file with a floating table of contents. Parameters are then specified using the params field.

3.2 The params field

The parameters defined in the params params field available for use in the R Markdown environment and the R console in RStudio. The params are essentially a list and can be accessed using params$run or params$modelDir. In this example we use params to define some model specific features and to control the behaviour of knitr.

3.2.1 Model-specific parameters:

run: model run number.
modelDir: model directory.
script: name of the script that creates the figures/tables (for mrggsave annotation).
yspec: name of your define spec file, e.g., pk.yml.
contCov: names of continuous covariates of interest (as they appear in your spec). Note that this parameter value is prefaced with !r to indicate that the value is an R expression rather than a literal value, e.g., !r c("AGE", "WT", "ALB", "EGFR").
catCov: names of continuous covariates of interest (as they appear in the spec and prefaced with !r).
etas: vector of the ETAs to be used by pmplots. If “ETA1”, “ETA2” etc. are used then this is how they appear on the plots. If NONMEMName//DisplayName is used then the plots use the display name, e.g., !r c("ETA1//ETA-KA", "ETA2//ETA-V/F", "ETA3//ETA-CL/F").
drugNameUnits: the drug label that should appear in plots.

3.2.2 Behaviour specific parameters:

include_code: define whether code should appear in the knitted html using true/false.
include_plots: define whether plots should appear in the knitted html using true/false.
run_mrggsave: define whether pdf versions of the plots should be saved out with mrggsave when rendering, using true/false.

3.2.3 Template specifics

When knitting your R Markdown template (see below), you can choose to either use the default values defined in the params section of the R Markdown, or you can update any params directly in the rmarkdown::render function call. Obviously, you can (and should) change the list of params included in the YAML section to suit your project. We walked through the code in the diagnostics-report.Rmd template, showing how to we create typical diagnostic plots using pmplots and yspec, on the Model Diagnostics page.

While this section focuses on the template we use to generate report-ready figures (in diagnostics-report.Rmd), the other templates have a similar set-up. They all include a YAML header with a params section and the code to make, and save out, plots. The id-dv-pred-plots.Rmd template creates individual DV-PRED-IPRED plots, and it saves these plots as pdfs but does not automatically display them in the html file to reduce the knitting time (this can easily be changed). The diagnostics-basic.Rmd creates a limited number of basic diagnostic plots and displays them in an html file for a quick look at your model fit. Given it is only meant to provide a quick look at your model fit, this template does not save any plots to pdf/png format.

4 Using R to knit the parameterized templates

Now you have your diagnostic templates setup, you can use the rmarkdown::render function to knit them. We use the model-diagnostics.R script to:

Create report-ready diagnostic plots described in diagnostics-report.Rmd for all six models.
Walk through the different ways you can render these templates with a few lines of code.
Showcase our helper function that will render a template for a given model and, if piped to “browseURL”, will pop open the html of model diagnostics in a new browser window.
Create individual fit plots using the id-dv-pred-plots.Rmd for the final model.

4.1 Required packages

library(tidyverse)
library(here)
library(rmarkdown)

4.2 Other set up

Source helper functions

source(here("script/functions-diagnostics.R"))

Define your model directory and the template you want to use

modelDir <- here("model", "pk")
rmd_template <- here("script", "diagnostic-templates", "diagnostics-report.Rmd")

4.3 Creating diagnostics

4.3.1 Customizing your params before knitting your template

This first example includes shows several params defined in the template that you might want to customize remotely but typically you’d only do this if you need to the change the default params you defined in your template (see later examples below).

modelSpecifics <- list(
  yspec = "pk.yml",
  contCov = c("AGE","WT","ALB","EGFR"), 
  catCov = c("STUDY", "RF", "CP", "DOSE"),
  etas = c("ETA1//ETA-KA", "ETA2//ETA-V/F", "ETA3//ETA-CL/F"),
  include_code = TRUE,
  include_plots = TRUE,
  run_mrggsave = TRUE)

You can then knit the R Markdown template using the rmarkdown::render function

rmarkdown::render(
  here("script", "diagnostic-templates", "diagnostics-report.Rmd"),  
  params = modelSpecifics,
  output_dir = here(modelDir, "100"),
  output_file = "diagnostic-report-100.html",
  envir = new.env() # good practice to render in new environment for reliability
)

Alternatively, you could use our model_diagnostics() helper function.

Code for model_diagnostics helper function

### This code can be found in functions-diagnostics.R ----------------------------

#' @details The only requirement for an `.Rmd` template to work with this
#' function is that it is parameterized and expects a parameter called `run`
#' (for the run number) and a parameter called `modelDir` (for the path to the
#' directory containing the model files).
#'
#' To be clear, **users do _not_ need to pass through these `run` and
#' `modelDir`** because they will be parsed internally.
#' 
#' @return Invisibly returns the path to the rendered HTML file. This is 
#' intended to make it easy to pipe into something like `browseURL()` for
#' opening in your browser.
#' 
#' @param .mod Either a path to a model output directory, or a `bbi_model`
#'   object (an object returned from [bbr::read_model()]/[bbr::new_model()] or
#'   [bbr::model_summary()])
#' @param .p A named list of parameters to pass through to the `.Rmd`
#'   template.
#' @param template Path to the `.Rmd` template to render. Defaults to
#'   `script/diagnostic-templates/diagnostics-basic.Rmd`.
#' @param dest_dir Destination directory to write the rendered HTML into. If
#'   `NULL`, the default, will render to `bbr::get_output_dir(.mod)`.
model_diagnostics <- function(
  .mod, 
  .p, 
  template = here::here("script", "diagnostic-templates", "diagnostics-basic.Rmd"),
  dest_dir = NULL
) {
  if (inherits(.mod, "bbi_model")) {
    .mod <- .mod$absolute_model_path
  }
  checkmate::assert_string(.mod)
  checkmate::assert_directory_exists(.mod)
  
  checkmate::assert_string(template)
  checkmate::assert_file_exists(template)
  
  checkmate::assert_list(.p, names = "named")
  checkmate::assert_string(dest_dir, null.ok = TRUE)
  
  # extract model id and build paths
  if (any(c("run", "modelDir", "script") %in% names(.p))) {
    warning(paste(
      "No need to pass `run`, `modelDir`, or `script` into model_diagnostics() because they are inferred from function inputs.",
      "Passed values will be ignored."
    ))
  }
  run <- basename(.mod)
  .p$run <- run
  .p$modelDir <- dirname(.mod)
  .p$script <- basename(template)
  
  stem <- tools::file_path_sans_ext(basename(template))
  html_file <- glue::glue("{stem}-{run}.html")
  
  # if no destination dir passed, put in model output dir
  if (is.null(dest_dir)) {
    dest_dir <- .mod
  }
  
  # pass through to render
  rmarkdown::render(
    template,
    params = .p,
    output_dir = dest_dir,
    output_file = html_file,
    envir = new.env() # good practice to render in new environment for reliability
  )
  
  invisible(return(file.path(dest_dir, html_file)))
}

The model_diagnostics() helper function knits the R Markdown template to an HTML file located in your model output directory and creates pdf/png versions of each figure in deliv directory. You can either pass this function a bbr model object, which is particularly useful if you’re doing this somewhere like a model management or submission script, or you can pass the function a path to your model directory.

mod <- bbr::read_model(file.path(modelDir, 100))

mod %>%
  model_diagnostics(
    modelSpecifics,
    template = rmd_template
  )

4.3.2 Different options for defining parameters or knitting the template

The params defined in the YAML header of your template will be used by default and so you often only need to update the parameters that you need to change between models.

modelSpecifics <- list(
  contCov = c("AGE", "WT","ALB","EGFR"),
  catCov = c("STUDY", "RF", "CP", "DOSE"),
  etas = c("ETA1//ETA-KA", "ETA2//ETA-V/F", "ETA3//ETA-CL/F")
)

You can pass a path to a model output directory (instead of a bbr object) to the model_diagnostics function. You can automatically pop open the compiled html in a new web browser window by piping the output of model_diagnostics to browseURL. This is possible because the model_diagnostics function invisibly returns the path to the html file.

model_diagnostics(
  file.path(modelDir, 102),
  modelSpecifics,
  template = rmd_template
) %>%
  browseURL()

You can also choose to define your params directly inside the model_diagnostics function

model_diagnostics(
  .mod = file.path(modelDir, 103),
  .p = list(
    contCov = c("AGE", "WT", "ALB", "EGFR"),
    catCov = c("STUDY", "RF", "CP", "DOSE"),
    etas = c("ETA1//ETA-KA", "ETA2//ETA-V/F", "ETA3//ETA-CL/F")
  ),
  template = rmd_template
) %>%
  browseURL()

If nothing has changed compared to the previous modelSpecifics list, you can simply pass through the previously defined list.

model_diagnostics(
  file.path(modelDir, 105),
  modelSpecifics,
  template = rmd_template
) %>%
  browseURL()

4.4 Creating individual diagnostic plots

As with the GOF diagnostic template above, the id-dv-pred-plots.Rmd template creates a series of individual plots of DV, PRED and IPRED versus time or TAD (linear and log scale) and saves them out as pdfs, pngs and and html. Depending on the size of your dataset this can be a little slow to run and so we typically only run this as needed.

modelSpecifics = list(
  yspec = "pk.yml",
  drugNameUnits = "Mockdralazine concentration (mg/L)",
  include_plots = FALSE )

model_diagnostics(
  file.path(modelDir, 106),
  modelSpecifics,
  template = here("script", "diagnostic-templates", "id-dv-pred-plots.Rmd")
)

4.5 Using the basic diagnostic template

The templates discussed so far focussed on creating report-ready figures but sometimes, particularly during early model development, you may want to take a quick look at some basic diagnostic plots. To address this we created a diagnostics-basic template that knits to a html only. The idea is that you might create these plots from directly within your model management script during model development. To showcase some of the features of this basic template and provide some useful code examples we put together the model-diagnostics-basic.R script.

Code for general set up

### Libraries ----------------------------
library(tidyverse)
library(rmarkdown)
library(here)
library(bbr)

### source helper functions ----------------------------
source(here("script/functions-diagnostics.R"))

4.5.1 A simple example

Define model specifics once, this is particularly useful if you’re going to reuse the same modelSpecifics for multiple models.

modelSpecifics <- list(
  yspec = "data/derived/pk.yml",
  drugNameUnits = "concentration (mg/L)"
)

Pass the model output directory path to render diagnostics and open the html of diagnostics in a new window

here("model", "pk", 102) %>%
  model_diagnostics(modelSpecifics) %>%
  browseURL()

4.5.2 Specifying specific covariates

While this is not something we do in our example, you can define the specific covariates of interest for a given model and this will overwrite any covariates defined in the template or your data spec file

modelSpecifics <- list(
  diagContCov = c("AGE"),
  diagCatCov = c("RF", "CP")
)

4.5.3 Pass in model objects

As mentioned above, you can also pass a bbr model object (anything with class bbi_model) to the model_diagnostics function. This is particularly helpful given this template will often be knitted from within your model management file where, presumably, you’ll already be working with the model objects.

mod <- read_model(here("model", "pk", 102))

mod %>% 
  model_diagnostics(modelSpecifics) %>%
  browseURL()

4.5.4 Excluding ETAs from ETA plots

Again, this is not something we do in our example, or something we expect you’ll need to do often but it maybe useful to exclude specific ETAs from ETA plots. For example, if your model has fixed ETAs that don’t need plots or multiple IOV etas, etc.

here("model", "pk", 106) %>% 
  model_diagnostics(list(excludeETAs = c("ETA1"))) %>%
  browseURL()

5 Other resources

The following scripts from the Github repository are discussed on this page. If you’re interested running this code, visit the About the Github Repo page first.

Script to render diagnostics templates: model-diagnostics.R
Diagnostics Rmd templates: