Using Ecometric Data to Explore Sources of Cross-Site Impact Variance in Multi-Site Trials

In 2003, Bloom, Hill, and Riccio (BHR) introduced methods to explain differences in the impact of socio-economic interventions across different locations. Twenty years later, how have these methods advanced and what can researchers do to improve their own BHR analyses? The two chief challenges are inexpensive measurement of local implementation practices and conditions. This article lays out recommendations using both simulated and real-world examples.

Researchers used the Health Professions Opportunity Grants (HPOG) Program to create varying socio-economic simulations with real students and training providers. Data provided from more than 6,600 participants across 37 locations focused on bias, mean square error, and the confidence intervals of mediation coefficients. This article lays out not only what new methods were used but explains how they generally improved the quality of inferences. Findings suggest the amount of time students spent studying by the sixth month of the trial was an important factor in their career progress and long-term degree and credential attainment.

Dive into the full article for more information on new methods and recommendations on how researchers can improve future BHR analyses.