# James E. Pustejovsky

I am a statistician and associate professor in the School of Education at the University of Wisconsin-Madison, where I teach in the Educational Psychology Department and the graduate program in Quantitative Methods. My research involves developing statistical methods for problems in education, psychology, and other areas of social science research, with a focus on methods related to research synthesis and meta-analysis.

### Interests

• Meta-analysis
• Causal inference
• Robust statistical methods
• Education statistics
• Single case experimental designs

### Education

• PhD in Statistics, 2013

Northwestern University

• BA in Economics, 2003

Boston College

# Recent Posts

### Distribution of the number of significant effect sizes

A while back, I posted the outline of a problem about the number of significant effect size estimates in a study that reports multiple outcomes. This problem interests me because it connects to the issue of selective reporting of study results, which creates problems for meta-analysis.

### Approximating the distribution of cluster-robust Wald statistics

$\def\Pr{{\text{Pr}}} \def\E{{\text{E}}} \def\Var{{\text{Var}}} \def\Cov{{\text{Cov}}} \def\cor{{\text{cor}}} \def\bm{\mathbf} \def\bs{\boldsymbol}$ In Tipton and Pustejovsky (2015), we examined several different small-sample approximations for cluster-robust Wald test statistics, which are like $$F$$ statistics but based on cluster-robust variance estimators.

### Implementing Consul's generalized Poisson distribution in Stan

$\def\Pr{{\text{Pr}}} \def\E{{\text{E}}} \def\Var{{\text{Var}}} \def\Cov{{\text{Cov}}} \def\bm{\mathbf} \def\bs{\boldsymbol}$ For a project I am working on, we are using Stan to fit generalized random effects location-scale models to a bunch of count data.

### Implementing Efron's double Poisson distribution in Stan

$\def\Pr{{\text{Pr}}} \def\E{{\text{E}}} \def\Var{{\text{Var}}} \def\Cov{{\text{Cov}}} \def\bm{\mathbf} \def\bs{\boldsymbol}$ For a project I am working on, we are using Stan to fit generalized random effects location-scale models to a bunch of count data.

### Cluster-Bootstrapping a meta-analytic selection model

In this post, we will sketch out what we think is a promising and pragmatic method for examining selective reporting while also accounting for effect size dependency. The method is to use a cluster-level bootstrap, which involves re-sampling clusters of observations to approximate the sampling distribution of an estimator. To illustrate this technique, we will demonstrate how to bootstrap a Vevea-Hedges selection model.

# Working papers

### Conducting power analysis for meta-analysis of dependent effect sizes: Common guidelines and an introduction to the POMADE R package

Sample size and statistical power are important factors to consider when planning a research synthesis. Power analysis methods have been developed for fixed effect or random effects models, but until …

# Recent Publications

### Equivalences between ad hoc strategies and meta-analytic models for dependent effect sizes

Meta-analyses of educational research findings frequently involve statistically dependent effect size estimates. Meta-analysts have often addressed dependence issues using ad hoc approaches that …

### High replicability of newly-discovered social-behavioral findings is achievable.

Failures to replicate evidence of new discoveries have forced scientists to ask whether this unreliability is due to suboptimal implementation of optimal methods or whether presumptively optimal …

### The efficacy of combining cognitive training and non-invasive brain stimulation: A transdiagnostic systematic review and meta-analysis

Over the past decade, an increasing number of studies investigated the innovative approach of supplementing cognitive training (CT) with non-invasive brain stimulation (NIBS) to increase the effects …

### Systematic review of variables related to instruction in augmentative and alternative communication implementation: Group and single-case design

Purpose: This article provides a systematic review and analysis of group and single-case studies addressing augmentative and alternative communication (AAC) intervention with school-aged persons …

### Comparison of competing approaches to analyzing cross-classified data: Random effects models, ordinary least squares, or fixed effects with cluster robust standard errors

Cross-classified random effects modeling (CCREM) is a common approach for analyzing cross-classified data in education. However, when the focus of a study is on the regression coefficients at level …

# Recent Presentations

### Model-Building Considerations in Meta-Analysis of Dependent Effect Sizes

In fields ranging from Education to Economics to Ecology, meta-analysts often encounter complicated data structures, in which some or all primary studies include multiple effect size estimates. These estimates may be correlated because they are based on data from a common sample or a partially overlapping sample, or may be statistically dependent due to use of common study operations.

### Equity-related moderator analysis in syntheses of dependent effect sizes: Conceptual and statistical considerations

Background/Context In meta-analyses examining educational interventions, researchers seek to understand the distribution of intervention impacts, in order to draw generalizations about what works, for whom, and under what conditions. One common way to examine equity implications in such reviews is through moderator analysis, which involves modeling how intervention effect sizes vary depending on the characteristics of primary study participants.

### Calculating Effect Sizes for Single-Case Research: An Introduction to the SingleCaseES and scdhlm Web Applications and R Packages

This workshop will provide an introduction to effect size calculations for single-case research designs, focused on two interactive web applications (or “apps”) and accompanying R packages. I will …

# Software

Power for Meta-Analysis of Dependent Effects

#### lmeInfo

Information Matrices for ‘lmeStruct’ and ‘glsStruct’ Objects

#### simhelpers

Helper package to assist in running simulation studies

#### ARPobservation

Simulate systematic direct observation data

#### clubSandwich

Cluster-robust variance estimation

#### scdhlm

Between-case SMD for single-case designs

#### SingleCaseES

Single-case design effect size calculator

#### wildmeta

Cluster-wild bootstrap for meta-regression