Multilevel analysis is an appropriate and powerful tool for analyzing hierarchical structure data widely applied from public health to genomic data. In practice, however, we may lose the information on multiple nesting levels in the multilevel analysis since data may fail to capture all levels of hierarchy, or the top or intermediate levels of hierarchy are ignored in the analysis. In this study, we consider a multilevel linear mixed effect model (LMM) with single imputation that can involve all data hierarchy levels in the presence of missing top or intermediate-level clusters. We evaluate and compare the performance of a multilevel LMM with single imputation with other models ignoring the data hierarchy or missing intermediate-level clusters. To this end, we applied a multilevel LMM with single imputation and other models to hierarchically structured cohort data with some intermediate levels missing and to simulated data with various cluster sizes and missing rates of intermediate-level clusters. A thorough simulation study demonstrated that an LMM with single imputation estimates fixed coefficients and variance components of a multilevel model more accurately than other models ignoring data hierarchy or missing clusters in terms of mean squared error and coverage probability. In particular, when models ignoring data hierarchy or missing clusters were applied, the variance components of random effects were overestimated. We observed similar results from the analysis of hierarchically structured cohort data.

A multilevel model has gained popularity as a practical and essential analysis tool in various fields such as epidemiological research, public health research, and social or educational research [

In practice, researchers in the field ignore a hierarchy within the data and prefer an ordinary least square regression model (OLS) that treats all observations as if they are measured at the same level [

To handle the missing issues, several studies proposed imputation methods to fill in the missing and create a complete dataset [

This study aims to evaluate the performance of a model considering all hierarchies, such as an LMM with single imputation, in three-level data and demonstrate that it outperforms other models that do not match the hierarchy of multilevel data. To do so, we compare three models in three-level hierarchical data with missing intermediate-level clusters. The first model is an OLS regression model, which is a single-level model ignoring any hierarchy in the data and treats all observations measured at the same level. Secondly, we consider a two-level LMM considering only level-1 and level-3 and discarding missing intermediate levels. The third model is a three-level LMM with a single imputation that involves all levels of the data hierarchy, so we can assume that the multilevel model matches the design hierarchy in the data by filling in a missing level-2 unit.

In this study, we apply and compare the three models to the Childhood to Adolescence Transition Study (CATS), a motivating case study of this work, in a three-level structure: school (level-3 unit), individual (level-2 unit), and repeated measures observed per individual (level-1 unit) with missing level-2 and level-3 units [

In the context of multilevel data analysis, a LMM is considered as an analysis model that accounts for correlation due to the hierarchical structure of the data. To establish background information on the analysis model for hierarchical structure data, especially for a three-level data structure, we develop a general notation and introduce a brief overview of a three-level model.

Let _{i}_{i}_{ij}_{ij}_{ijk}_{ijk}_{ij}_{i}

where _{ij}_{ij}_{ij}_{i}_{i}_{i}

In this study, we assume that only the regression intercept varies across clusters and hence restrict our attention to a three-level random effect model with a random intercept for each level. It is the simplest type of linear mixed model for multilevel study, although we note that an extended model including random slope can be considered. We revisit this in the discussion. Considering a random effect model with random intercepts only, _{11}=_{ij}_{11}=_{i}

where _{0}(=_{00}) is the intercept, and _{1}(=_{01}),_{2}(=_{01}), and _{3}(=_{10}) are the slope coefficients for the fixed effects at level-3, level-2, and level-1, respectively. Considering random intercepts only in the model, we assume that level-3 and level-2 residuals, _{i}_{ij}

The covariance structure of the response vector _{111},_{112},…,_{L},_{L},_{LM}) is a block-diagonal covariance matrix given by

with an _{(ni} is an _{i}_{i}_{(}_{n}_{i)} is an _{i}_{i}_{i}_{i}_{iM}_{i} is a size of a block-diagonal matrix associates with

To handle the missing level-2 units in a three-level hierarchical data, we adopt a single imputation method introduced in Sandersâ€™ study [

The motivating case study CATS has a three-level structure with the components of school, individual, and its repeated measures per individual, and some measurements at the individual level (level-2; academic numeracy score at baseline) were partially not observed and recorded as missing in the dataset. There were 22.4% missing academic numeracy scores at baseline at level-2, and they were filled in by single imputation that impute a corresponding academic numeracy score measured at level-1 to handle the components with missingness. In other words, we treated each level-1 unit within a missing level-2 unit is nested in a level-2 unit that only contains the level-1 unit itself [

Motivated by the case study of CATS, we focus on an analysis of the simulated data mimicking the CATS data [

The simulated data mimicking the CATS have a three-level hierarchical structure. Individuals are nested within schools, and repeated measures within individuals were collected. The data consist of demographic, educational, and social outcomes as well as mental health outcomes for the CATS study: depressive symptoms, National Assessment Programme – Literacy and Numeracy results (NAPLAN) academic numeracy score, socio-economic status, age, and sex are collected for the study [

The NAPLAN numeracy score is an outcome of interest and is observed at level 2 at wave 1 (potential baseline) and at level 1 in the following waves with missing cases. As we are interested in a case where missingness happens at level-2 only, we discard cases where the NAPLAN numeracy score has missingness at level-1, and the size of data that we used is N = 2,592. The data are unbalanced as the size of level-2 differs, and the number of level-1 units within a level-2 unit varies across individual. The dataset has 163 schools; 54 of 163 schools have a single individual with a single measurement per individual, and the other 109 schools have multiple individuals with a different number of repeated measures. There were 1,142 individuals in the data, and 256 of 1,142 (22.4%) NAPLAN numeracy scores at the individual level are missing.

We conducted a simulation study to evaluate the performance of a three-level LMM with single imputation. In the simulation, we considered complete three-level data sets and incomplete data with various missing rates of level-2 clusters (

We generated complete data sets without missing subjected to the model in _{0}=0.5, _{1}=_{2}=_{3}=0.3, the random effects at level-2 and level-3 of _{u}_{v}_{e}_{ij}_{ij}_{i}_{ij}_{ijk}^{2}). Intraclass correlations were set to 0.398 and 0.602 for level-3 and level-2, respectively. For each case, we generated 500 replicates.

To evaluate the performance of the three-level LMM in the presence of missingness at the level-2 units, we generated missingness from each of the complete 3-level data sets (_{ij}

We use three analysis models to determine the effect of ignorance of hidden levels in the hierarchically structured CATS data: (1) a single-level model (ordinary least square regression) that completely ignores the structure of the hierarchy and treats the data collected at level-1 only (M1), (2) a two-level LMM with a random effect for school (level-3) only and ignoring an individual (level-2) random effect (M2), and (3) a three-level LMM with single imputation considering all levels of hierarchy in the CATS study units (M3). The LMMs we fit for M2 and M3 only have random intercepts, and random slopes are not considered in the data analysis.

Based on a significance level of 5%, depressive symptoms appeared to be significantly meaningful in supporting the effect of mental health on academic outcomes in children from puberty through adolescence across the three models (p < 0.001). Age and NAPLAN numeracy scores observed at baseline were meaningful covariates for explaining the relationship with academic outcome in addition to the mental health outcome. That is, when children are young, have fewer depressive symptoms, and higher NAPLAN numeracy scores at baseline, academic outcomes are likely to be higher. Noticeably, the estimated coefficients of the important covariates do not differ much across the three analysis models.

Considering 30 scenarios (_{i}_{ij}_{ij}

We expected that M3 performed well in the presence of unobserved level-2 clusters, whereas M2 would poorly estimate the coefficient β_{2} of the level-2 associated covariate with missing clusters since M2 ignored the level-2 clusters and partially incorporated the date structure of level-1 and level-3 clusters. Since the M1 did not consider the hierarchical data structure, we expected that it poorly estimated _{1} and β_{2}.

In general, the M3 performed well and better estimated the fixed coefficients in _{2} of the level-2 associated covariate with missing clusters. _{2}, respectively, and compared those values by the three models. The MSEs by the M3 ranged from 0.00039 to 0.00557, while those by the M1 and M2 were 0.0011–0.0639 and 0.0004–0.037, respectively (_{2} by the M3 were approximately 0.814–0.964 (_{2} and their coverage probabilities low as 0.27–0.406 and 0.392–0.498, respectively (_{2} seemed to increase with the missing rate (

We observed MSEs for _{0}, β_{1,} and β_{3} similar to _{0} and β_{1} were around 0.95 from the M3, while we observed small coverage probabilities from M1 (e.g., 0.06–0.302 for β_{1}) and comparable coverage probabilities from M2 (e.g., 0.926–0.96 for β_{1}). The three models yielded similar coverage probabilities for _{3}.

The three models contain different variance components. M3 can estimate the three standard deviations, _{u}_{v}_{e}_{u}_{v}

_{v}

We observed that the estimates for _{u}_{e}_{u}_{e}_{u}_{e}_{u}

In this study, we investigated the effect of missing intermediate level of the hierarchical structure data with varying missing rates, as well as the consequence of ignorance of the level of nesting on the parameter estimation of multilevel analysis. Since missing level-2 units are imputed by the nested level-1 unit’s measurement, the intermediate-level unit has a single observation per unit, and the data can be considered sparse. We compared three models with different levels of hierarchy across varying missing rates at the intermediate-level and various cluster sizes in the top- and intermediate-level clusters. We observed that the three-level LMM with single imputation showed a better performance compared to the other two models, which ignore higher levels of nesting in terms of MSE and coverage. Moreover, lower-level variance components were overestimated, indicating that variance components are affected by ignorance of the intermediate level.

We considered a random effect model for the 2-level and 3-level linear mixed effect models in the simulation study and data application for brevity, and it can be expanded to consider a random slope model and/or interaction terms. However, it requires caution to generalize the model analysis to be more complex because we have a large number of intermediate-level clusters with a single observation, and such sparse clusters might lead to biased estimates in random slope estimation [

Furthermore, we used a three-level LMM with single imputation that replaces the missing by the nesting lower-level unit’s measurements and considering that a single observation is measured per intermediate-level unit. As a future study, other imputation methods for multilevel data can be considered and compared their performances with an approach studied in this work. In practice, it is common that level-1 and level-2 units are both missing, and in such case, the imputation method, such as a multiple imputation handling multiple level’s missingness, should be considered.

No potential conflict of interest relevant to this article was reported.

Conceptualization: SP. Data curation: SP, YC. Formal analysis: SP, YC. Funding acquisition: YC. Methodology: SP. Writing - original draft: SP, YC. Writing - review & editing: SP, YC.

This work was supported by the National Research Foundation of Korea (NRF) grant funded by the Korea government (MSIT) (No. NRF-2021R1C1C1011250).

The MSEs for β_{2}, the coefficient of the level-2 associated covariate, when estimated by the single-level model (···), 2-level LMM (---), or 3-level LMM (─). For each plot of L (=30, 50, 100) level-3 clusters and M (=10, 30) level-2 clusters, each point represents the MSE across 500 simulated data sets over a range of missing rates in level-2 clusters. MSE, mean squared error; LMM, linear mixed effect model.

The coverage probability for β_{2}, the coefficient of the 2-level associated covariate, when estimated by the single-level model (···), 2-level LMM (---), or 3-level LMM (─). For each plot of L (=30, 50, 100) level-3 clusters and M (=10, 30) level-2 clusters, each point represents the coverage probability across 500 simulated data sets over a range of missing rates in level-2 clusters. LMM, linear mixed effect model.

The MSEs for σ_{v} from the 3-level LMM (M3) when the level-3 cluster size changes as L = 30 (···), L = 50 (---), or L = 100 (─). For each case of M = 10 or 30 level-2 cluster sizes, 500 datasets were simulated over a range of missing rates in level-2 clusters. MSE, mean squared error; LMM, linear mixed effect model.

The coverage probability for σ_{v} from the 3-level LMM (M3) when the level-3 cluster size changes as L = 30 (···), L = 50 (---), or L = 100 (─). For each case of M = 10 or 30 level-2 cluster sizes, 500 datasets were simulated over a range of missing rates in level-2 clusters. LMM, linear mixed effect model.

The average estimates for σ_{e}_{e}

Simulation schemes to generate complete 3-level data and incomplete data with missing in level-2 units

–Given level-3 cluster size L (=30, 50, 100) and level-2 cluster size M (=10, 30),
–Generate level-1 cluster size –Given |

–Given each complete data set and missing rate –Randomly select and discard × –Then an incomplete data set consists of |

We note that complete 3-level data sets can be considered as the case of a 0% missing rate. In total, we considered 30 simulation scenarios (three level-3 cluster sizes × two level-2 cluster sizes × five missing rates with a range of 0%, 10%, 25%, 50%, and 75%).

Estimates of coefficients for CATS data inferred by the three models: single-level model (M1), two-level LMM (M2), and three-level LMM (M3)

M1 | M2 | M3 | |||||||
---|---|---|---|---|---|---|---|---|---|

Coef | SE | p-value | Coef | SE | p-value | Coef | SE | p-value | |

(Intercept) | 2.789 | 0.225 | <0.001 | 2.744 | 0.212 | <0.001 | 2.748 | 0.234 | <0.001 |

Depressive symptom |
−0.043 | 0.013 | <0.001 | −0.047 | 0.012 | <0.001 | −0.048 | 0.010 | <0.001 |

Age |
−0.173 | 0.023 | <0.001 | −0.172 | 0.022 | <0.001 | −0.173 | 0.025 | <0.001 |

Sex |
0.037 | 0.041 | 0.371 | 0.018 | 0.038 | 0.633 | 0.021 | 0.044 | 0.901 |

SES |
0.019 | 0.021 | 0.360 | 0.011 | 0.019 | 0.579 | 0.006 | 0.022 | 0.764 |

NAPLAN at baseline |
0.677 | 0.021 | <0.001 | 0.675 | 0.019 | <0.001 | 0.676 | 0.021 | <0.001 |

CATS, Childhood to Adolescence Transition Study; SE, standard error; SES, socio-economic status; NAPLAN, National Assessment Programme – Literacy and Numeracy results.

Covariate at level-1.

Covariate at level-2.

Estimated variance components of random effects and residuals for CATS data inferred by the three models: single-level model (M1), two-level LMM (M2), and three-level LMM (M3)

Variance component | M1 | M2 | M3 |
---|---|---|---|

Level 3: school | - | 0.216 | 0.137 |

Level 2: individual | - | - | 0.325 |

Residual | 1.002 | 0.786 | 0.426 |

CATS, Childhood to Adolescence Transition Study; LMM, linear mixed effect model.