| Decision | H₀ is true | H₀ is false |
|---|---|---|
| Fail to reject H₀ | TN | Type II error, miss, FN |
| Reject H₀ | Type I error, false alarm, FP | TP |
8 Multiple testing
Multiple testing is especially important in life-science applications such as genomics, transcriptomics, proteomics, and metabolomics, where thousands of hypotheses may be tested simultaneously.
8.1 Error types
Remember from Section 5.4 that the probability of type I and II errors are denoted \(\alpha\) and \(\beta\), respectively;
\[\alpha = P(\textrm{type I error}) = P(\textrm{false alarm}) = P(\textrm{Reject }H_0 \mid H_0 \text{ is true})\] \[\beta = P(\textrm{type II error}) = P(\textrm{miss}) = P(\textrm{Fail to reject }H_0 \mid H_1 \text{ is true})\] and the statistical power
\[\textrm{power} = 1 - \beta = P(\textrm{Reject }H_0 \mid H_1\textrm{ is true}).\]
8.2 Multiple testing
If a single test is performed, we know that
- P(One type I error) = \(\alpha\)
- P(No type I error) = \(1 - \alpha\)
If \(m\) independent tests are performed (e.g. investigate many genes or proteins), the risk of at false alarm (type I error) increases;
- P(No type I errors in \(m\) tests) = \((1 - \alpha)^m\)
- P(At least one type I error in \(m\) tests) = \(1 - (1 - \alpha)^m\)
Two common principles for dealing with multiple testing are control of family-wise error rate or false discovery rate.
- FWER: family-wise error rate, control the probability of one or more false positive \(P(N_{FP}>0)\), e.g. Bonferroni, Holm
- FDR: false discovery rate, control the expected value of the proportion of false positives among hits (rejected hypotheses), \(E[N_{FP}/(N_{FP}+N_{TP})]\) (note: for 0 positives the ratio is defined to be 0), e.g. Benjamini-Hochberg, Storey
8.3 Bonferroni correction
To achieve a family-wise error rate of \(FWER \leq \gamma\) when performing \(m\) tests, declare significance and reject the null hypothesis for any test with \(p \leq \gamma/m\).
A disadvantage of the Bonferroni correction is that it can be overly conservative, especially when many tests are performed.
8.4 Benjamini-Hochbergs FDR procedure
| Decision | H₀ is true | H₀ is false |
|---|---|---|
| Fail to reject H₀ | TN | FN |
| Reject H₀ | FP | TP |
The false discovery rate is the expected proportion of false positives among ‘hits’, i.e. \(\frac{FP}{TP+FP}\).
The Benjamini-Hochberg procedure controls the FDR level, \(\gamma\), underthe assumption of independence (and also under certain forms of positive dependence) when performing \(m\) tests, as follows:
- Sort the p-values \(p_1 \leq p_2 \leq \dots \leq p_m\).
- Find the maximum \(j\) such that \(p_j \leq \gamma \frac{j}{m}\).
- Declare significance for all tests \(1, 2, \dots, j\).
8.5 ‘Adjusted’ p-values
Sometimes an adjusted significance threshold is not reported, but instead ‘adjusted’ p-values are reported.
- Using Bonferroni’s method the ‘adjusted’ p-values are:
\(\tilde p_i = \min(m p_i, 1)\).
A feature’s Bonferroni-adjusted p-value represents the smallest family-wise error rate at which that feature would be deemed significant.
- Benjamini-Hochberg’s ‘adjusted’ p-values are often referred to as \(q\)-values and can be compted after sorting the p-values as follows:
\(q_i = \min(\frac{m}{i} p_i, 1)\)
A feature’s \(q\)-value can be interpreted as the lowest FDR at which the corresponding null hypothesis will be rejected, i.e. the feature will be deemed significant.
In practice, adjusted p-values are usually computed using software such as the R function p.adjust.
Example 8.1 (10000 independent tests)
| p-value | adj p (Bonferroni) | q-value (B-H) |
|---|---|---|
| 1.7e-08 | 0.0002 | 0.0002 |
| 5.8e-08 | 0.0006 | 0.0003 |
| 3.4e-07 | 0.0034 | 0.0011 |
| 9.1e-07 | 0.0091 | 0.0020 |
| 1e-06 | 0.0100 | 0.0020 |
| 2.4e-06 | 0.0240 | 0.0040 |
| 2.3e-05 | 0.2300 | 0.0329 |
| 3.6e-05 | 0.3600 | 0.0450 |
| 0.00022 | 1.0000 | 0.2300 |
| 0.00023 | 1.0000 | 0.2300 |
| 0.00073 | 1.0000 | 0.6636 |
| 0.0032 | 1.0000 | 1.0000 |
| 0.0045 | 1.0000 | 1.0000 |
| 0.0087 | 1.0000 | 1.0000 |
| 0.0089 | 1.0000 | 1.0000 |
| 0.012 | 1.0000 | 1.0000 |
| 0.014 | 1.0000 | 1.0000 |
| 0.045 | 1.0000 | 1.0000 |
| 0.08 | 1.0000 | 1.0000 |
| 0.23 | 1.0000 | 1.0000 |