mice: Multivariate Imputation by Chained Equations

The mice package implements a method to deal with missing data. The package creates multiple imputations (replacement values) for multivariate missing data. The method is based on Fully Conditional Specification, where each incomplete variable is imputed by a separate model. The MICE algorithm can impute mixes of continuous, binary, unordered categorical and ordered categorical data. In addition, MICE can impute continuous two-level data, and maintain consistency between imputations by means of passive imputation. Many diagnostic plots are implemented to inspect the quality of the imputations.

Generates Multivariate Imputations by Chained Equations (MICE)

Usage

mice(
  data,
  m = 5,
  method = NULL,
  predictorMatrix,
  ignore = NULL,
  where = NULL,
  blocks,
  visitSequence = NULL,
  formulas,
  blots = NULL,
  post = NULL,
  defaultMethod = c("pmm", "logreg", "polyreg", "polr"),
  maxit = 5,
  printFlag = TRUE,
  seed = NA,
  data.init = NULL,
  ...
)

Arguments

data: A data frame or a matrix containing the incomplete data. Missing values are coded as NA.
m: Number of multiple imputations. The default is m=5.
method: Can be either a single string, or a vector of strings with length length(blocks), specifying the imputation method to be used for each column in data. If specified as a single string, the same method will be used for all blocks. The default imputation method (when no argument is specified) depends on the measurement level of the target column, as regulated by the defaultMethod argument. Columns that need not be imputed have the empty method "". See details.
predictorMatrix: A numeric matrix of length(blocks) rows and ncol(data) columns, containing 0/1 data specifying the set of predictors to be used for each target column. Each row corresponds to a variable block, i.e., a set of variables to be imputed. A value of 1 means that the column variable is used as a predictor for the target block (in the rows). By default, the predictorMatrix is a square matrix of ncol(data) rows and columns with all 1's, except for the diagonal. Note: For two-level imputation models (which have "2l" in their names) other codes (e.g, 2 or -2) are also allowed.
ignore: A logical vector of nrow(data) elements indicating which rows are ignored when creating the imputation model. The default NULL includes all rows that have an observed value of the variable to imputed. Rows with ignore set to TRUE do not influence the parameters of the imputation model, but are still imputed. We may use the ignore argument to split data into a training set (on which the imputation model is built) and a test set (that does not influence the imputation model estimates). Note: Multivariate imputation methods, like mice.impute.jomoImpute() or mice.impute.panImpute(), do not honour the ignore argument.
where: A data frame or matrix with logicals of the same dimensions as data indicating where in the data the imputations should be created. The default, where = is.na(data), specifies that the missing data should be imputed. The where argument may be used to overimpute observed data, or to skip imputations for selected missing values. Note: Imputation methods that generate imptutations outside of mice, like mice.impute.panImpute() may depend on a complete predictor space. In that case, a custom where matrix can not be specified.
blocks: List of vectors with variable names per block. List elements may be named to identify blocks. Variables within a block are imputed by a multivariate imputation method (see method argument). By default each variable is placed into its own block, which is effectively fully conditional specification (FCS) by univariate models (variable-by-variable imputation). Only variables whose names appear in blocks are imputed. The relevant columns in the where matrix are set to FALSE of variables that are not block members. A variable may appear in multiple blocks. In that case, it is effectively re-imputed each time that it is visited.
visitSequence: A vector of block names of arbitrary length, specifying the sequence of blocks that are imputed during one iteration of the Gibbs sampler. A block is a collection of variables. All variables that are members of the same block are imputed when the block is visited. A variable that is a member of multiple blocks is re-imputed within the same iteration. The default visitSequence = "roman" visits the blocks (left to right) in the order in which they appear in blocks. One may also use one of the following keywords: "arabic" (right to left), "monotone" (ordered low to high proportion of missing data) and "revmonotone" (reverse of monotone). Special case: If you specify both visitSequence = "monotone" and maxit = 1, then the procedure will edit the predictorMatrix to conform to the monotone pattern. Realize that convergence in one iteration is only guaranteed if the missing data pattern is actually monotone. The procedure does not check this.
formulas: A named list of formula's, or expressions that can be converted into formula's by as.formula. List elements correspond to blocks. The block to which the list element applies is identified by its name, so list names must correspond to block names. The formulas argument is an alternative to the predictorMatrix argument that allows for more flexibility in specifying imputation models, e.g., for specifying interaction terms.
blots: A named list of alist's that can be used to pass down arguments to lower level imputation function. The entries of element blots[[blockname]] are passed down to the function called for block blockname.
post: A vector of strings with length ncol(data) specifying expressions as strings. Each string is parsed and executed within the sampler() function to post-process imputed values during the iterations. The default is a vector of empty strings, indicating no post-processing. Multivariate (block) imputation methods ignore the post parameter.
defaultMethod: A vector of length 4 containing the default imputation methods for 1) numeric data, 2) factor data with 2 levels, 3) factor data with > 2 unordered levels, and 4) factor data with > 2 ordered levels. By default, the method uses pmm, predictive mean matching (numeric data) logreg, logistic regression imputation (binary data, factor with 2 levels) polyreg, polytomous regression imputation for unordered categorical data (factor > 2 levels) polr, proportional odds model for (ordered, > 2 levels).
maxit: A scalar giving the number of iterations. The default is 5.
printFlag: If TRUE, mice will print history on console. Use print=FALSE for silent computation.
seed: An integer that is used as argument by the set.seed() for offsetting the random number generator. Default is to leave the random number generator alone.
data.init: A data frame of the same size and type as data, without missing data, used to initialize imputations before the start of the iterative process. The default NULL implies that starting imputation are created by a simple random draw from the data. Note that specification of data.init will start all m Gibbs sampling streams from the same imputation.
...: Named arguments that are passed down to the univariate imputation functions.

Value

Returns an S3 object of class mids (multiply imputed data set)

Details

The mice package contains functions to

Inspect the missing data pattern
Impute the missing data m times, resulting in m completed data sets
Diagnose the quality of the imputed values
Analyze each completed data set
Pool the results of the repeated analyses
Store and export the imputed data in various formats
Generate simulated incomplete data
Incorporate custom imputation methods

Generates multiple imputations for incomplete multivariate data by Gibbs sampling. Missing data can occur anywhere in the data. The algorithm imputes an incomplete column (the target column) by generating 'plausible' synthetic values given other columns in the data. Each incomplete column must act as a target column, and has its own specific set of predictors. The default set of predictors for a given target consists of all other columns in the data. For predictors that are incomplete themselves, the most recently generated imputations are used to complete the predictors prior to imputation of the target column.

A separate univariate imputation model can be specified for each column. The default imputation method depends on the measurement level of the target column. In addition to these, several other methods are provided. You can also write their own imputation functions, and call these from within the algorithm.

The data may contain categorical variables that are used in a regressions on other variables. The algorithm creates dummy variables for the categories of these variables, and imputes these from the corresponding categorical variable.

Built-in univariate imputation methods are:

`pmm`	any	Predictive mean matching
`midastouch`	any	Weighted predictive mean matching
`sample`	any	Random sample from observed values
`cart`	any	Classification and regression trees
`rf`	any	Random forest imputations
`mean`	numeric	Unconditional mean imputation
`norm`	numeric	Bayesian linear regression
`norm.nob`	numeric	Linear regression ignoring model error
`norm.boot`	numeric	Linear regression using bootstrap
`norm.predict`	numeric	Linear regression, predicted values
`lasso.norm`	numeric	Lasso linear regression
`lasso.select.norm`	numeric	Lasso select + linear regression
`quadratic`	numeric	Imputation of quadratic terms
`ri`	numeric	Random indicator for nonignorable data
`logreg`	binary	Logistic regression
`logreg.boot`	binary	Logistic regression with bootstrap
`lasso.logreg`	binary	Lasso logistic regression
`lasso.select.logreg`	binary	Lasso select + logistic regression
`polr`	ordered	Proportional odds model
`polyreg`	unordered	Polytomous logistic regression
`lda`	unordered	Linear discriminant analysis
`2l.norm`	numeric	Level-1 normal heteroscedastic
`2l.lmer`	numeric	Level-1 normal homoscedastic, lmer
`2l.pan`	numeric	Level-1 normal homoscedastic, pan
`2l.bin`	binary	Level-1 logistic, glmer
`2lonly.mean`	numeric	Level-2 class mean
`2lonly.norm`	numeric	Level-2 class normal
`2lonly.pmm`	any	Level-2 class predictive mean matching

These corresponding functions are coded in the mice library under names mice.impute.method, where method is a string with the name of the univariate imputation method name, for example norm. The method argument specifies the methods to be used. For the j'th column, mice() calls the first occurrence of paste('mice.impute.', method[j], sep = '') in the search path. The mechanism allows uses to write customized imputation function, mice.impute.myfunc. To call it for all columns specify method='myfunc'. To call it only for, say, column 2 specify method=c('norm','myfunc','logreg',...{}).

Skipping imputation: The user may skip imputation of a column by setting its entry to the empty method: "". For complete columns without missing data mice will automatically set the empty method. Setting t he empty method does not produce imputations for the column, so any missing cells remain NA. If column A contains NA's and is used as predictor in the imputation model for column B, then mice produces no imputations for the rows in B where A is missing. The imputed data for B may thus contain NA's. The remedy is to remove column A from the imputation model for the other columns in the data. This can be done by setting the entire column for variable A in the predictorMatrix equal to zero.

Passive imputation: mice() supports a special built-in method, called passive imputation. This method can be used to ensure that a data transform always depends on the most recently generated imputations. In some cases, an imputation model may need transformed data in addition to the original data (e.g. log, quadratic, recodes, interaction, sum scores, and so on).

Passive imputation maintains consistency among different transformations of the same data. Passive imputation is invoked if ~ is specified as the first character of the string that specifies the univariate method. mice() interprets the entire string, including the ~ character, as the formula argument in a call to model.frame(formula, data[!r[,j],]). This provides a simple mechanism for specifying deterministic dependencies among the columns. For example, suppose that the missing entries in variables data$height and data$weight are imputed. The body mass index (BMI) can be calculated within mice by specifying the string '~I(weight/height^2)' as the univariate imputation method for the target column data$bmi. Note that the ~ mechanism works only on those entries which have missing values in the target column. You should make sure that the combined observed and imputed parts of the target column make sense. An easy way to create consistency is by coding all entries in the target as NA, but for large data sets, this could be inefficient. Note that you may also need to adapt the default predictorMatrix to evade linear dependencies among the predictors that could cause errors like Error in solve.default() or Error: system is exactly singular. Though not strictly needed, it is often useful to specify visitSequence such that the column that is imputed by the ~ mechanism is visited each time after one of its predictors was visited. In that way, deterministic relation between columns will always be synchronized.

A new argument ls.meth can be parsed to the lower level .norm.draw to specify the method for generating the least squares estimates and any subsequently derived estimates. Argument ls.meth takes one of three inputs: "qr" for QR-decomposition, "svd" for singular value decomposition and "ridge" for ridge regression. ls.meth defaults to ls.meth = "qr".

Auxiliary predictors in formulas specification: For a given block, the formulas specification takes precedence over the corresponding row in the predictMatrix argument. This precedence is, however, restricted to the subset of variables specified in the terms of the block formula. Any variables not specified by formulas are imputed according to the predictMatrix specification. Variables with non-zero type values in the predictMatrix will be added as main effects to the formulas, which will act as supplementary covariates in the imputation model. It is possible to turn off this behavior by specifying the argument auxiliary = FALSE.

Functions

The main functions are:

`mice()`	Impute the missing data m times
`with()`	Analyze completed data sets
`pool()`	Combine parameter estimates
`complete()`	Export imputed data
`ampute()`	Generate missing data

Vignettes

There is a detailed series of six online vignettes that walk you through solving realistic inference problems with mice.

We suggest going through these vignettes in the following order

Van Buuren, S. (2018). Boca Raton, FL.: Chapman & Hall/CRC Press. The book Flexible Imputation of Missing Data. Second Edition. contains a lot of example code.

Methodology

The mice software was published in the Journal of Statistical Software (Van Buuren and Groothuis-Oudshoorn, 2011). doi:10.18637/jss.v045.i03 . The first application of the method concerned missing blood pressure data (Van Buuren et. al., 1999). The term Fully Conditional Specification was introduced in 2006 to describe a general class of methods that specify imputations model for multivariate data as a set of conditional distributions (Van Buuren et. al., 2006). Further details on mixes of variables and applications can be found in the book Flexible Imputation of Missing Data. Second Edition. Chapman & Hall/CRC. Boca Raton, FL.

Enhanced linear algebra

Updating the BLAS can improve speed of R, sometime considerably. The details depend on the operating system. See the discussion in the "R Installation and Administration" guide for further information.

References

van Buuren, S., Boshuizen, H.C., Knook, D.L. (1999) Multiple imputation of missing blood pressure covariates in survival analysis. Statistics in Medicine, 18, 681–694.

van Buuren, S., Brand, J.P.L., Groothuis-Oudshoorn C.G.M., Rubin, D.B. (2006) Fully conditional specification in multivariate imputation. Journal of Statistical Computation and Simulation, 76, 12, 1049–1064.

van Buuren, S., Groothuis-Oudshoorn, K. (2011). mice: Multivariate Imputation by Chained Equations in R. Journal of Statistical Software, 45(3), 1–67. doi:10.18637/jss.v045.i03

Van Buuren, S. (2018). Flexible Imputation of Missing Data. Second Edition. Chapman & Hall/CRC. Boca Raton, FL.

Van Buuren, S., Groothuis-Oudshoorn, K. (2011). mice: Multivariate Imputation by Chained Equations in R. Journal of Statistical Software, 45(3), 1-67. doi:10.18637/jss.v045.i03

Van Buuren, S. (2018). Flexible Imputation of Missing Data. Second Edition. Chapman & Hall/CRC. Boca Raton, FL.

Van Buuren, S., Brand, J.P.L., Groothuis-Oudshoorn C.G.M., Rubin, D.B. (2006) Fully conditional specification in multivariate imputation. Journal of Statistical Computation and Simulation, 76, 12, 1049–1064.

Van Buuren, S. (2007) Multiple imputation of discrete and continuous data by fully conditional specification. Statistical Methods in Medical Research, 16, 3, 219–242.

Van Buuren, S., Boshuizen, H.C., Knook, D.L. (1999) Multiple imputation of missing blood pressure covariates in survival analysis. Statistics in Medicine, 18, 681–694.

Brand, J.P.L. (1999) Development, implementation and evaluation of multiple imputation strategies for the statistical analysis of incomplete data sets. Dissertation. Rotterdam: Erasmus University.

Author

Maintainer: Stef van Buuren stef.vanbuuren@tno.nl

Authors:

Karin Groothuis-Oudshoorn c.g.m.oudshoorn@utwente.nl

Other contributors:

Gerko Vink g.vink@uu.nl [contributor]
Rianne Schouten R.M.Schouten@uu.nl [contributor]
Alexander Robitzsch robitzsch@ipn.uni-kiel.de [contributor]
Patrick Rockenschaub rockenschaub.patrick@gmail.com [contributor]
Lisa Doove lisa.doove@ppw.kuleuven.be [contributor]
Shahab Jolani s.jolani@maastrichtuniversity.nl [contributor]
Margarita Moreno-Betancur margarita.moreno@mcri.edu.au [contributor]
Ian White ian.white@ucl.ac.uk [contributor]
Philipp Gaffert philipp.gaffert@gfk.com [contributor]
Florian Meinfelder florian.meinfelder@uni-bamberg.de [contributor]
Bernie Gray bfgray3@gmail.com [contributor]
Vincent Arel-Bundock vincent.arel-bundock@umontreal.ca [contributor]
Mingyang Cai m.cai@uu.nl [contributor]
Thom Volker t.b.volker@uu.nl [contributor]
Edoardo Costantini e.costantini@tilburguniversity.edu [contributor]
Caspar van Lissa c.j.vanlissa@uu.nl [contributor]
Hanne Oberman h.i.oberman@uu.nl [contributor]
Stephen Wade stephematician@gmail.com [contributor]

Stef van Buuren stef.vanbuuren@tno.nl, Karin Groothuis-Oudshoorn c.g.m.oudshoorn@utwente.nl, 2000-2010, with contributions of Alexander Robitzsch, Gerko Vink, Shahab Jolani, Roel de Jong, Jason Turner, Lisa Doove, John Fox, Frank E. Harrell, and Peter Malewski.

Examples

# do default multiple imputation on a numeric matrix
imp <- mice(nhanes)
#> 
#>  iter imp variable
#>   1   1  bmi  hyp  chl
#>   1   2  bmi  hyp  chl
#>   1   3  bmi  hyp  chl
#>   1   4  bmi  hyp  chl
#>   1   5  bmi  hyp  chl
#>   2   1  bmi  hyp  chl
#>   2   2  bmi  hyp  chl
#>   2   3  bmi  hyp  chl
#>   2   4  bmi  hyp  chl
#>   2   5  bmi  hyp  chl
#>   3   1  bmi  hyp  chl
#>   3   2  bmi  hyp  chl
#>   3   3  bmi  hyp  chl
#>   3   4  bmi  hyp  chl
#>   3   5  bmi  hyp  chl
#>   4   1  bmi  hyp  chl
#>   4   2  bmi  hyp  chl
#>   4   3  bmi  hyp  chl
#>   4   4  bmi  hyp  chl
#>   4   5  bmi  hyp  chl
#>   5   1  bmi  hyp  chl
#>   5   2  bmi  hyp  chl
#>   5   3  bmi  hyp  chl
#>   5   4  bmi  hyp  chl
#>   5   5  bmi  hyp  chl
imp
#> Class: mids
#> Number of multiple imputations:  5 
#> Imputation methods:
#>   age   bmi   hyp   chl 
#>    "" "pmm" "pmm" "pmm" 
#> PredictorMatrix:
#>     age bmi hyp chl
#> age   0   1   1   1
#> bmi   1   0   1   1
#> hyp   1   1   0   1
#> chl   1   1   1   0

# list the actual imputations for BMI
imp$imp$bmi
#>       1    2    3    4    5
#> 1  29.6 25.5 22.0 30.1 25.5
#> 3  28.7 27.2 29.6 26.3 28.7
#> 4  22.5 21.7 22.5 20.4 25.5
#> 6  25.5 24.9 25.5 25.5 22.5
#> 10 20.4 22.5 28.7 21.7 30.1
#> 11 27.5 27.2 35.3 30.1 27.4
#> 12 27.5 27.2 27.5 22.5 27.4
#> 16 29.6 33.2 28.7 22.0 28.7
#> 21 20.4 22.7 30.1 30.1 33.2

# first completed data matrix
complete(imp)
#>    age  bmi hyp chl
#> 1    1 29.6   1 238
#> 2    2 22.7   1 187
#> 3    1 28.7   1 187
#> 4    3 22.5   2 186
#> 5    1 20.4   1 113
#> 6    3 25.5   2 184
#> 7    1 22.5   1 118
#> 8    1 30.1   1 187
#> 9    2 22.0   1 238
#> 10   2 20.4   1 187
#> 11   1 27.5   1 187
#> 12   2 27.5   1 218
#> 13   3 21.7   1 206
#> 14   2 28.7   2 204
#> 15   1 29.6   1 238
#> 16   1 29.6   1 238
#> 17   3 27.2   2 284
#> 18   2 26.3   2 199
#> 19   1 35.3   1 218
#> 20   3 25.5   2 206
#> 21   1 20.4   1 187
#> 22   1 33.2   1 229
#> 23   1 27.5   1 131
#> 24   3 24.9   1 186
#> 25   2 27.4   1 186

# imputation on mixed data with a different method per column
mice(nhanes2, meth = c("sample", "pmm", "logreg", "norm"))
#> 
#>  iter imp variable
#>   1   1  bmi  hyp  chl
#>   1   2  bmi  hyp  chl
#>   1   3  bmi  hyp  chl
#>   1   4  bmi  hyp  chl
#>   1   5  bmi  hyp  chl
#>   2   1  bmi  hyp  chl
#>   2   2  bmi  hyp  chl
#>   2   3  bmi  hyp  chl
#>   2   4  bmi  hyp  chl
#>   2   5  bmi  hyp  chl
#>   3   1  bmi  hyp  chl
#>   3   2  bmi  hyp  chl
#>   3   3  bmi  hyp  chl
#>   3   4  bmi  hyp  chl
#>   3   5  bmi  hyp  chl
#>   4   1  bmi  hyp  chl
#>   4   2  bmi  hyp  chl
#>   4   3  bmi  hyp  chl
#>   4   4  bmi  hyp  chl
#>   4   5  bmi  hyp  chl
#>   5   1  bmi  hyp  chl
#>   5   2  bmi  hyp  chl
#>   5   3  bmi  hyp  chl
#>   5   4  bmi  hyp  chl
#>   5   5  bmi  hyp  chl
#> Class: mids
#> Number of multiple imputations:  5 
#> Imputation methods:
#>      age      bmi      hyp      chl 
#>       ""    "pmm" "logreg"   "norm" 
#> PredictorMatrix:
#>     age bmi hyp chl
#> age   0   1   1   1
#> bmi   1   0   1   1
#> hyp   1   1   0   1
#> chl   1   1   1   0

if (FALSE) { # \dontrun{
# example where we fit the imputation model on the train data
# and apply the model to impute the test data
set.seed(123)
ignore <- sample(c(TRUE, FALSE), size = 25, replace = TRUE, prob = c(0.3, 0.7))

# scenario 1: train and test in the same dataset
imp <- mice(nhanes2, m = 2, ignore = ignore, print = FALSE, seed = 22112)
imp.test1 <- filter(imp, ignore)
imp.test1$data
complete(imp.test1, 1)
complete(imp.test1, 2)

# scenario 2: train and test in separate datasets
traindata <- nhanes2[!ignore, ]
testdata <- nhanes2[ignore, ]
imp.train <- mice(traindata, m = 2, print = FALSE, seed = 22112)
imp.test2 <- mice.mids(imp.train, newdata = testdata)
complete(imp.test2, 1)
complete(imp.test2, 2)
} # }