Combining p-values via averaging. / Vovk, Vladimir; Wang, Ruodu.

In: Biometrika, Vol. 107, No. 4, asaa027, 11.06.2020, p. 791-808.

Research output: Contribution to journalArticle



  • Accepted Manuscript

    Accepted author manuscript, 610 KB, PDF document

    Embargo ends: 11/06/21



This paper proposes general methods for the problem of multiple testing of a single hypothesis, with a standard goal of combining a number of p-values without making any assumptions about their dependence structure. An old result by Ruschendorf and, independently, Meng implies that the p-values can be combined by scaling up their arithmetic mean by a factor of 2, and no smaller factor is sufficient in general. A similar result by Mattner about the geometric mean replaces 2 by e. Based on more recent developments in mathematical finance, specifically, robust risk aggregation techniques, we extend these results to generalized means; in particular, we show that K p-values can be combined by scaling up their harmonic mean by a factor of \log K asymptotically as K\to\infty. This leads to a generalized version of the Bonferroni-Holm procedure. We also explore methods using weighted averages of p-values. Finally, we discuss the efficiency of various methods of combining p-values and how to choose a suitable method in light of data and prior information.
Original languageEnglish
Article numberasaa027
Pages (from-to)791-808
Number of pages18
Issue number4
Publication statusPublished - 11 Jun 2020
This open access research output is licenced under a Creative Commons Attribution-NonCommercial-NoDerivs 3.0 Unported License.

ID: 34967919