Multiple Hypothesis Testing in Factor Discovery: Adjusting for the Data Snooping Zoo

The Explosion of the Factor Zoo

In his 2011 American Finance Association Presidential Address, John Cochrane famously surveyed the chaotic proliferation of empirical anomalies, declaring that asset pricing faced a "zoo of new factors." Over recent decades, academic literature and quantitative research desks have cataloged hundreds of distinct factors claiming to generate persistent equity outperformance.

In classical statistical analysis, an empirical factor is considered statistically significant if its regression t-statistic exceeds 2.0 (corresponding to a two-tailed significance level of $p \le 0.05$). However, this standard threshold assumes a single hypothesis tested on independent data.

When thousands of researchers test hundreds of thousands of candidate variables across the same historical CRSP and Compustat datasets, a t-statistic of 2.0 guarantees that dozens of false discoveries will appear purely by chance—a process known as p-hacking or data snooping.

The Harvey, Liu, & Zhu Framework: Raising the Significance Hurdle

To address the data snooping crisis, Harvey, Liu, and Zhu (2016) introduced a rigorous multiple hypothesis testing framework to empirical asset pricing. They applied statistical corrections—including Bonferroni, Holm, and False Discovery Rate (FDR) adjustments—to account for the cumulative history of factor testing.

Their findings demonstrated that because hundreds of factors have been mined from historical data, the statistical hurdle for factor discovery must be raised substantially:

• A t-statistic of 2.0 provides virtually no confidence of genuine out-of-sample alpha in a multi-test setting.
• Accounting for multiple testing requires raising the t-statistic threshold to at least 3.0 (corresponding to $p \le 0.0027$).
• Under this elevated threshold, a substantial majority of published academic anomalies fail to achieve statistical significance.

Anchoring Factor Models in Economic Foundations

Systematic factor architectures must be insulated against data snooping. Parallax avoids mining ephemeral anomalies, anchoring its multi-factor model exclusively in core, economically grounded pillars supported by decades of out-of-sample evidence: Value, Quality, Momentum, Defensive, Size, and Tactical—with Size specifically integrated during portfolio construction rather than the overall composite score.

Review our empirical factor architecture in the Parallax Factor Framework Documentation and explore our quantitative research standards on Parallax Technology.