Refine
Language
- English (3)
Has Fulltext
- yes (3)
Is part of the Bibliography
- yes (3)
Keywords
Institute
Valid online inference is an important problem in contemporary multiple testing research,to which various solutions have been proposed recently. It is well-known that these existing methods can suffer from a significant loss of power if the null p-values are conservative. In this work, we extend the previously introduced methodology to obtain more powerful procedures for the case of super-uniformly distributed p-values. These types of p-values arise in important settings, e.g. when discrete hypothesis tests are performed or when the p-values are weighted. To this end, we introduce the method of super-uniformity reward (SUR) that incorporates information about the individual null cumulative distribution functions. Our approach yields several new 'rewarded' procedures that offer uniform power improvements over known procedures and come with mathematical guarantees for controlling online error criteria based either on the family-wise error rate (FWER) or the marginal false discovery rate (mFDR). We illustrate the benefit of super-uniform rewarding in real-data analyses and simulation studies. While discrete tests serve as our leading example, we also show how our method can be applied to weighted p-values.
Discrete uniform and homogeneous p-values often arise in applications with multiple testing. For example, this occurs in genome wide association studies whenever
a non-parametric one-sample (or two-sample) test is
applied throughout the gene loci. In this paper, we considermultiple comparison procedures for such scenarios
based on several existing estimators for the proportion
of true null hypotheses, 𝜋0, which take the discreteness
of the p-values into account. The theoretical guarantees
of the several approaches with respect to the estimation of 𝜋0 and the false discovery rate control are reviewed. The performance of the discrete procedures is investigated through intensive Monte Carlo simulations considering both independent and dependent p-values. The methods are applied to three real data sets for illustration
purposes too. Since the particular estimator of
𝜋0 used to compute the q-values may influence its performance, relative advantages and disadvantages of the reviewed procedures are discussed. Practical recommendations are given.
Several classical methods exist for controlling the false discovery exceedance (FDX) for large-scale multiple testing problems, among them the Lehmann-Romano procedure (Lehmann and Romano 2005) ([LR] below) and the Guo-Romano procedure (Guo and Romano 2007) ([GR] below). While these two procedures are the most prominent, they were
originally designed for homogeneous test statistics, that is, when the null distribution functions of the p-values Fi, 1 ≤ i ≤ m, are all equal. In many applications, however, the data are heterogeneous which leads to heterogeneous null distribution functions. Ignoring this heterogeneity induces a lack of power. In this paper, we develop three new procedures that incorporate the Fi’s, while maintaining rigorous FDX control. The heterogeneous version of [LR], denoted [HLR], is based on the arithmetic average of the Fi’s, while the heterogeneous version of [GR], denoted [HGR], is based on the geometric average of the Fi’s. We also introduce a procedure [PB], that is based on the Poisson-binomial distribution and that uniformly improves [HLR] and [HGR], at the price of a higher computational complexity. Perhaps surprisingly, this shows that, contrary to the known theory of false discovery rate (FDR) control under heterogeneity, the way to incorporate the Fi’s can be particularly simple in the case of FDX control, and does not require any further correction term. The performances of the new proposed procedures are illustrated by real and simulated data in two important heterogeneous settings: first, when the test statistics are continuous but
the p-values are weighted by some known independent weight vector, e.g., coming from co-data sets; second, when the test statistics are discretely distributed, as is the case for data representing frequencies or counts. Our new procedures are implemented in the R package FDX, see Junge and Döhler (2020).