Model diagnostics
Run number 1100
To produce a set of diagnostic plots that will be included in a report. Please note that these plots are just meant to provide an example of what could be created and how. They are not an exhaustive list of every possible plot and were chosen with the project aims in mind.
Set up
Model location
Define modelName and path to the model directory
(MODEL_DIR).
Figure location
If saving figures out to pdf, define where those pdfs should be saved
to. Here the figures are saved to
deliv > figure > model_run_number
Model details
A summary of high-level model details.
Read in the model details using read_model. Details
stored in the mod object can be used to identify the
location of the source data (used in $DATA) - to see how this is done
look at the bbr::get_data_path() and
bbr::build_path_from_model() helper functions.
── Status ─────────────────────────────────
• Finished Running
── Absolute Model Path ───────────────────────────
• /data/bbr-nonmem-poppk-bayes/model/pk/1100
── YAML & Model Files ───────────────────────────
• /data/bbr-nonmem-poppk-bayes/model/pk/1100.yaml
• /data/bbr-nonmem-poppk-bayes/model/pk/1100.ctl
── Description ───────────────────────────────
• Final Model
── Tags ──────────────────────────────────
• two-compartment + absorption
• ETA-CL
• ETA-KA
• ETA-V2
• CLWT-allo
• V2WT-allo
• QWT-allo
• V3WT-allo
• proportional RUV
• CLEGFR
• CLAGE
• CLALB
── Notes ──────────────────────────────────
• 1: All diagnostics look reasonable
Read output
Parameter estimates
Compute summaries of posterior distributions as well as some diagnostics.
Bulk effective sample size (ESS) is a measure of sampling efficiency for the location of the distribution, while Tail ESS is a measure of sampling efficiency for the tails (5% and 95% quantiles) of the distribution. Higher values indicate greater sampling efficiency. A very rough rule of thumb is to aim for at least 400 for each parameter.
R-hat is a convergence diagnostic that compares the between- and within-chain variances of model parameters. Values close to 1 indicate that the chains have converged to similar distributions. Aim for less than about 1.05 for all parameters.
This is a repeat of the summary table produces in the MCMC diagnostics template, but can be usedful to include here to refer to when considering model diagnostics.
| parameter | mean | median | sd | mad | 90% CI | ess_bulk | ess_tail | rhat |
|---|---|---|---|---|---|---|---|---|
| THETA[1] | 0.478 | 0.478 | 0.0628 | 0.0577 | (0.373, 0.579) | 1.08e+03 | 1.3e+03 | 1 |
| THETA[2] | 4.1 | 4.1 | 0.0278 | 0.0279 | (4.06, 4.15) | 803 | 1.08e+03 | 1 |
| THETA[3] | 1.16 | 1.16 | 0.0291 | 0.029 | (1.11, 1.21) | 948 | 1.24e+03 | 1 |
| THETA[4] | 4.23 | 4.23 | 0.0248 | 0.0261 | (4.19, 4.27) | 1.83e+03 | 1.6e+03 | 1 |
| THETA[5] | 1.29 | 1.29 | 0.0379 | 0.0373 | (1.23, 1.35) | 2.12e+03 | 1.68e+03 | 1 |
| THETA[6] | 0.488 | 0.487 | 0.0485 | 0.0491 | (0.409, 0.568) | 1.38e+03 | 1.37e+03 | 1 |
| THETA[7] | -0.0413 | -0.0399 | 0.0736 | 0.0758 | (-0.163, 0.0766) | 1.1e+03 | 1.37e+03 | 1 |
| THETA[8] | 0.423 | 0.423 | 0.0853 | 0.086 | (0.282, 0.566) | 1.81e+03 | 1.6e+03 | 1 |
| OMEGA[1,1] | 0.237 | 0.231 | 0.0565 | 0.0536 | (0.155, 0.34) | 864 | 1.36e+03 | 1 |
| OMEGA[2,1] | 0.0658 | 0.0644 | 0.0199 | 0.0195 | (0.0359, 0.101) | 397 | 895 | 1.01 |
| OMEGA[2,2] | 0.0837 | 0.0831 | 0.0115 | 0.0114 | (0.0666, 0.104) | 594 | 943 | 1.01 |
| OMEGA[3,1] | 0.118 | 0.116 | 0.0225 | 0.0216 | (0.0852, 0.156) | 492 | 997 | 1 |
| OMEGA[3,2] | 0.0686 | 0.0679 | 0.0102 | 0.00969 | (0.0528, 0.0865) | 664 | 1.21e+03 | 1 |
| OMEGA[3,3] | 0.116 | 0.114 | 0.0136 | 0.0129 | (0.0949, 0.14) | 645 | 1.29e+03 | 1 |
| OMEGA[4,1] | 0 | 0 | 0 | 0 | ( 0, 0) | NA | NA | NA |
| OMEGA[4,2] | 0 | 0 | 0 | 0 | ( 0, 0) | NA | NA | NA |
| OMEGA[4,3] | 0 | 0 | 0 | 0 | ( 0, 0) | NA | NA | NA |
| OMEGA[4,4] | 0.025 | 0.025 | 0 | 0 | (0.025, 0.025) | NA | NA | NA |
| OMEGA[5,1] | 0 | 0 | 0 | 0 | ( 0, 0) | NA | NA | NA |
| OMEGA[5,2] | 0 | 0 | 0 | 0 | ( 0, 0) | NA | NA | NA |
| OMEGA[5,3] | 0 | 0 | 0 | 0 | ( 0, 0) | NA | NA | NA |
| OMEGA[5,4] | 0 | 0 | 0 | 0 | ( 0, 0) | NA | NA | NA |
| OMEGA[5,5] | 0.025 | 0.025 | 0 | 0 | (0.025, 0.025) | NA | NA | NA |
| SIGMA[1,1] | 0.0394 | 0.0394 | 0.00113 | 0.00116 | (0.0376, 0.0412) | 2.39e+03 | 1.5e+03 | 1 |
Read in data
Model output (EPRED, IPRED, NPDE, EWRES, ETAs) is read in from either
the output of simulations (see ?bbr.bayes::nm_join_bayes)
or from NONMEM output only. When only NONMEM output is used, medians of
these values across all chains are calculated.
After reading in the NONMEM dataset and the output dataset they’re
joined by a NUM column. This assumes that a row
number column (called NUM) was included during data
assembly. The idea here is that in NONMEM, you table just
NUM and none of the other input data items. They all will
get joined back to the NONMEM output … even character columns.
The data used in the diagnostic plots has been filtered
to only include the observations (i.e. EVID==0). Note that
further modifications may be needed, for example, if BLQ data was
included in the model or if the DV was log-transformed. The
dataset also converts the categorical covariates of interest to factors
using the yspec_add_factors function and details described
in the spec file.
The id subset gets the first record per ID. This would
usually be the baseline value but consider filtering on a baseline flag
if available. Also, if the model includes inter-occasion variability
(IOV), the occasion variable should be included within the
distinct function.
General diagnostic plots
The following plots assume that the preferred x-axis labels are defined here.
DV vs population and individual predictions
Create plots of DV vs population and individual predictions for the full dataset and stratified by renal function and hepatic function.
Population predictions are medians of 1000 simulated values incorporating between- and within-subject variability, as well as uncertainty in population parameter estimates via sampling from the posterior distribution. Individual predictions are medians of 1000 simulated values incorporating conditional estimates of individual parameters and include within-subject variability, as well as uncertainty in population parameter estimates via sampling from the posterior distribution.
DV vs population and individual predictions by renal function
DV vs population and individual predictions by hepatic function
NPDE plots
Normalized prediction distribution errors (NPDE) are Monte-Carlo generated diagnostics, using 1000 simulations incorporating between- and within-subject variability, as well as uncertainty in population parameter estimates via sampling from the posterior distribution.
NPDE vs population predictions, time and time after dose.
Population predictions are medians of 1000 simulated values incorporating between- and within-subject variability, as well as uncertainty in population parameter estimates via sampling from the posterior distribution.
NPDE vs continuous covariates
NPDE vs categorical covariates.
NPDE density histogram
EWRES vs population predictions, time and time after dose
Expected weighted residuals (EWRES) are Monte-Carlo generated residuals, using 1000 simulations incorporating between- and within-subject variability, as well as uncertainty in population parameter estimates via sampling from the posterior distribution.
Population predictions are medians of 1000 simulated values incorporating between- and within-subject variability, as well as uncertainty in population parameter estimates via sampling from the posterior distribution.
EWRES qq and density plot
EBEs-based diagnostics
ETAs are medians of 1000 posterior ETAs across the 4 chains.
ETA pairs plot
Continuous covariate plots
These plots uses yspec to automatically rename the axis labels.
ETA vs continuous covariates
Note that here we use a function that maps over the ETAs (not the covariates) because the purpose of these plots was to determine whether there were any trends in the covariates for a given ETA. This may need to be edited to address different study specific questions
## [[1]]
##
## [[2]]
##
## [[3]]
Continuous covariate pairs plot
Categorical covariate plots
These plots uses the yspec to automatically rename the axis labels.
ETA vs categorical covariates
Note that here we use a function that maps over the covariates (not the ETAs) because the purpose of these plots was to determine whether there is any difference in the distribution of ETAs across studies, dosing groups and disease states. This should be updated to reflect the questions you’re trying to address.
## $STUDY
##
## $RF
##
## $CP
##
## $DOSE
Session details
It is considered good practice to include these details at the end of all rmd scripts
Sys.getenv("AMI_NAME")## [1] ""sessioninfo::session_info()## ─ Session info ───────────────────────────────────────────────────────────────────────────────────────────
## setting value
## version R version 4.1.3 (2022-03-10)
## os Ubuntu 18.04.6 LTS
## system x86_64, linux-gnu
## ui RStudio
## language (EN)
## collate en_US.UTF-8
## ctype en_US.UTF-8
## tz America/New_York
## date 2024-05-12
## rstudio 2022.02.4+500.pro1 Prairie Trillium (server)
## pandoc 2.17.1.1 @ /usr/lib/rstudio-server/bin/quarto/bin/ (via rmarkdown)
##
## ─ Packages ───────────────────────────────────────────────────────────────────────────────────────────────
## package * version date (UTC) lib source
## abind 1.4-5 2016-07-21 [1] CRAN (R 4.1.3)
## assertthat 0.2.1 2019-03-21 [1] CRAN (R 4.1.3)
## backports 1.4.1 2021-12-13 [1] CRAN (R 4.1.3)
## bayesplot * 1.11.1 2024-02-15 [1] CRAN (R 4.1.3)
## bbr * 1.10.0 2024-03-07 [1] MPNDEV (R 4.1.3)
## bbr.bayes * 0.2.0 2024-03-07 [1] MPNDEV (R 4.1.3)
## bit 4.0.5 2022-11-15 [1] CRAN (R 4.1.3)
## bit64 4.0.5 2020-08-30 [1] CRAN (R 4.1.3)
## bslib 0.6.1 2023-11-28 [1] CRAN (R 4.1.3)
## cachem 1.0.8 2023-05-01 [1] CRAN (R 4.1.3)
## checkmate 2.3.1 2023-12-04 [1] CRAN (R 4.1.3)
## cli 3.6.2 2023-12-11 [1] CRAN (R 4.1.3)
## codetools 0.2-19 2023-02-01 [1] CRAN (R 4.1.3)
## colorspace 2.1-0 2023-01-23 [1] CRAN (R 4.1.3)
## cowplot * 1.1.3 2024-01-22 [1] CRAN (R 4.1.3)
## crayon 1.5.2 2022-09-29 [1] CRAN (R 4.1.3)
## data.table 1.15.2 2024-02-29 [1] CRAN (R 4.1.3)
## diffobj 0.3.5 2021-10-05 [1] CRAN (R 4.1.3)
## digest 0.6.34 2024-01-11 [1] CRAN (R 4.1.3)
## distributional 0.4.0 2024-02-07 [1] CRAN (R 4.1.3)
## dplyr * 1.1.4 2023-11-17 [1] CRAN (R 4.1.3)
## evaluate 0.23 2023-11-01 [1] CRAN (R 4.1.3)
## fansi 1.0.6 2023-12-08 [1] CRAN (R 4.1.3)
## farver 2.1.1 2022-07-06 [1] CRAN (R 4.1.3)
## fastmap 1.1.1 2023-02-24 [1] CRAN (R 4.1.3)
## forcats * 1.0.0 2023-01-29 [1] CRAN (R 4.1.3)
## fs 1.6.3 2023-07-20 [1] CRAN (R 4.1.3)
## furrr * 0.3.1 2022-08-15 [1] CRAN (R 4.1.3)
## future * 1.33.1 2023-12-22 [1] CRAN (R 4.1.3)
## future.apply 1.11.1 2023-12-21 [1] CRAN (R 4.1.3)
## generics 0.1.3 2022-07-05 [1] CRAN (R 4.1.3)
## GGally 2.2.1 2024-02-14 [1] CRAN (R 4.1.3)
## ggplot2 * 3.4.4 2023-10-12 [1] CRAN (R 4.1.3)
## ggstats 0.5.1 2023-11-21 [1] CRAN (R 4.1.3)
## globals 0.16.2 2022-11-21 [1] CRAN (R 4.1.3)
## glue * 1.7.0 2024-01-09 [1] CRAN (R 4.1.3)
## gridExtra 2.3 2017-09-09 [1] CRAN (R 4.1.3)
## gridGraphics 0.5-1 2020-12-13 [1] CRAN (R 4.1.3)
## gtable 0.3.4 2023-08-21 [1] CRAN (R 4.1.3)
## here * 1.0.1 2020-12-13 [1] CRAN (R 4.1.3)
## highr 0.10 2022-12-22 [1] CRAN (R 4.1.3)
## hms 1.1.3 2023-03-21 [1] CRAN (R 4.1.3)
## htmltools 0.5.7 2023-11-03 [1] CRAN (R 4.1.3)
## inline 0.3.19 2021-05-31 [1] CRAN (R 4.1.3)
## jquerylib 0.1.4 2021-04-26 [1] CRAN (R 4.1.3)
## jsonlite 1.8.8 2023-12-04 [1] CRAN (R 4.1.3)
## knitr 1.45 2023-10-30 [1] CRAN (R 4.1.3)
## labeling 0.4.3 2023-08-29 [1] CRAN (R 4.1.3)
## lattice 0.22-5 2023-10-24 [1] CRAN (R 4.1.3)
## lifecycle 1.0.4 2023-11-07 [1] CRAN (R 4.1.3)
## listenv 0.9.1 2024-01-29 [1] CRAN (R 4.1.3)
## loo * 2.7.0 2024-02-24 [1] CRAN (R 4.1.3)
## lubridate * 1.9.3 2023-09-27 [1] CRAN (R 4.1.3)
## magrittr 2.0.3 2022-03-30 [1] CRAN (R 4.1.3)
## Matrix 1.6-5 2024-01-11 [1] CRAN (R 4.1.3)
## matrixStats 1.2.0 2023-12-11 [1] CRAN (R 4.1.3)
## mclust 6.1 2024-02-23 [1] CRAN (R 4.1.3)
## mgcv 1.9-1 2023-12-21 [1] CRAN (R 4.1.3)
## mrggsave * 0.4.5 2023-10-17 [1] MPNDEV (R 4.1.3)
## mrgsolve * 1.4.1 2024-03-07 [1] MPNDEV (R 4.1.3)
## munsell 0.5.0 2018-06-12 [1] CRAN (R 4.1.3)
## nlme 3.1-164 2023-11-27 [1] CRAN (R 4.1.3)
## nmrec 0.4.0 2024-03-07 [1] MPNDEV (R 4.1.3)
## npde 3.5 2024-02-07 [1] CRAN (R 4.1.3)
## parallelly 1.37.1 2024-02-29 [1] CRAN (R 4.1.3)
## patchwork * 1.2.0 2024-01-08 [1] CRAN (R 4.1.3)
## pillar 1.9.0 2023-03-22 [1] CRAN (R 4.1.3)
## pkgbuild 1.4.3 2023-12-10 [1] CRAN (R 4.1.3)
## pkgconfig 2.0.3 2019-09-22 [1] CRAN (R 4.1.3)
## plyr 1.8.9 2023-10-02 [1] CRAN (R 4.1.3)
## pmplots * 0.4.0 2024-03-07 [1] MPNDEV (R 4.1.3)
## pmtables * 0.6.0 2023-10-17 [1] MPNDEV (R 4.1.3)
## posterior * 1.5.0 2023-10-31 [1] CRAN (R 4.1.3)
## prettyunits 1.2.0 2023-09-24 [1] CRAN (R 4.1.3)
## processx 3.8.3 2023-12-10 [1] CRAN (R 4.1.3)
## progress 1.2.3 2023-12-06 [1] CRAN (R 4.1.3)
## progressr 0.14.0 2023-08-10 [1] CRAN (R 4.1.3)
## ps 1.7.6 2024-01-18 [1] CRAN (R 4.1.3)
## purrr * 1.0.2 2023-08-10 [1] CRAN (R 4.1.3)
## QuickJSR 1.1.3 2024-01-31 [1] CRAN (R 4.1.3)
## R6 2.5.1 2021-08-19 [1] CRAN (R 4.1.3)
## RColorBrewer 1.1-3 2022-04-03 [1] CRAN (R 4.1.3)
## Rcpp 1.0.12 2024-01-09 [1] CRAN (R 4.1.3)
## RcppParallel 5.1.7 2023-02-27 [1] CRAN (R 4.1.3)
## readr * 2.1.5 2024-01-10 [1] CRAN (R 4.1.3)
## renv 0.16.0 2022-09-29 [1] CRAN (R 4.1.3)
## reshape2 1.4.4 2020-04-09 [1] CRAN (R 4.1.3)
## rlang 1.1.3 2024-01-10 [1] CRAN (R 4.1.3)
## rmarkdown 2.25 2023-09-18 [1] CRAN (R 4.1.3)
## rprojroot 2.0.4 2023-11-05 [1] CRAN (R 4.1.3)
## rstan 2.32.5 2024-01-10 [1] CRAN (R 4.1.3)
## rstudioapi 0.15.0 2023-07-07 [1] CRAN (R 4.1.3)
## sass 0.4.8 2023-12-06 [1] CRAN (R 4.1.3)
## scales 1.3.0 2023-11-28 [1] CRAN (R 4.1.3)
## sessioninfo 1.2.2 2021-12-06 [1] CRAN (R 4.1.3)
## StanHeaders 2.32.6 2024-03-01 [1] CRAN (R 4.1.3)
## stringi 1.8.3 2023-12-11 [1] CRAN (R 4.1.3)
## stringr * 1.5.1 2023-11-14 [1] CRAN (R 4.1.3)
## tensorA 0.36.2.1 2023-12-13 [1] CRAN (R 4.1.3)
## tibble * 3.2.1 2023-03-20 [1] CRAN (R 4.1.3)
## tidyr * 1.3.1 2024-01-24 [1] CRAN (R 4.1.3)
## tidyselect 1.2.0 2022-10-10 [1] CRAN (R 4.1.3)
## tidyverse * 2.0.0 2023-02-22 [1] CRAN (R 4.1.3)
## timechange 0.3.0 2024-01-18 [1] CRAN (R 4.1.3)
## tzdb 0.4.0 2023-05-12 [1] CRAN (R 4.1.3)
## utf8 1.2.4 2023-10-22 [1] CRAN (R 4.1.3)
## vctrs 0.6.5 2023-12-01 [1] CRAN (R 4.1.3)
## vroom 1.6.5 2023-12-05 [1] CRAN (R 4.1.3)
## withr 3.0.0 2024-01-16 [1] CRAN (R 4.1.3)
## xfun 0.42 2024-02-08 [1] CRAN (R 4.1.3)
## xtable 1.8-4 2019-04-21 [1] CRAN (R 4.1.3)
## yaml 2.3.8 2023-12-11 [1] CRAN (R 4.1.3)
## yspec * 0.6.1 2023-10-17 [1] MPNDEV (R 4.1.3)
##
## [1] /data/bbr-nonmem-poppk-bayes/renv/library/R-4.1/x86_64-pc-linux-gnu
## [2] /data/home/timw/R/renvExt
## [3] /data/bbr-nonmem-poppk-bayes/renv/sandbox/R-4.1/x86_64-pc-linux-gnu/dbd573bd
##
## ──────────────────────────────────────────────────────────────────────────────────────────────────────────bbr::bbi_version()## [1] "3.3.0"