The trials displayed in a funnel plot may not always estimate the same underlying effect of the same intervention and such heterogeneity between results may lead to asymmetry in funnel plots if the true treatment effect is larger in the smaller trials. For example, if a combined outcome is considered then substantial benefit may be seen only in patients at high risk for the component of the combined outcome which is affected by the intervention.17,18 A cholesterol-lowering drug which reduces coronary heart disease (CHD) mortality will have a greater effect on all cause mortality in high risk patients with established cardiovascular disease than in young, asymptomatic patients with isolated hypercholesterolemia.19 This is because a consistent relative reduction in CHD mortality will translate into a greater relative reduction in all-cause mortality in high-risk patients in whom a greater proportion of all deaths will be from CHD. Trials conducted in high-risk patients will also tend to be smaller, because of the difficulty in recruiting such patients and because increased event rates mean that smaller sample sizes are required to detect a given effect.
Small trials are generally conducted before larger trials are established. In the intervening years standard (control) treatments may have improved, thus reducing the relative efficacy of the experimental treatment. Changes in standard treatments could also lead to a modification of the effect of the experimental treatment. Such a mechanism has been proposed as an explanation for the discrepant results obtained in clinical trials of the effect of magnesium infusion in myocardial infarction.20 It has been argued that magnesium infusion may not work if administered after reperfusion has occurred. By the time the ISIS-4 trial21 (which gave no evidence of a treatment effect) was performed, thrombolysis had become routine in the management of myocardial infarction. However this argument is not supported by subgroup analysis of the ISIS-4 trial, which shows no effect of magnesium even among patients not receiving thrombolysis.22
Some interventions may have been implemented less thoroughly in larger trials, thus explaining the more positive results in smaller trials. This is particularly likely in trials of complex interventions in chronic diseases, such as rehabilitation after stroke or multifaceted interventions in diabetes mellitus. For example, an asymmetrical funnel plot was found in a meta-analysis of trials examining the effect of inpatient comprehensive geriatric assessment programmes on mortality.1323 An experienced consultant geriatrician was more likely to be actively involved in the smaller trials and this may explain the larger treatment effects observed in these trials.1323
Odds ratios are more extreme (further from 1) than the corresponding risk ratio if the event rate is high. Because of this, a funnel plot which shows no asymmetry when plotted using risk ratios could still be asymmetric when plotted using odds ratios. This would happen if the smaller trials were consistently conducted in high-risk patients, and the large trials in patients at lower risk, although differences in underlying risk would need to be substantial. Finally it is, of course, possible that an asymmetrical funnel plot arises merely by the play of chance. Mechanisms which can lead to funnel plot asymmetry are summarised in Table 11.1.
Table 11.1 Potential sources of asymmetry in funnel plots.
1. Selection biases
Publication bias and other reporting biases (see Chapter 3) Biased inclusion criteria
2. True heterogeneity: size of effect differs according to study size Intensity of intervention
Differences in underlying risk
3. Data irregularities
Poor methodological design of small studies (see Chapter 5)
4. Artefact: heterogeneity due to poor choice of effect measure (see Chapter 16)
Funnel plot asymmetry thus raises the possibility of bias but it is not proof of bias. It is important to note, however, that asymmetry (unless produced by chance alone) will always lead us to question the interpretation of the overall estimate of effect when studies are combined in a metaanalysis.
Other graphical methods
Was this article helpful?