The recognition that health care should be based on reliable and up to date evidence has drawn attention to the need for high quality randomised trials addressing important and previously unanswered questions.¹ Clinical trials are widely acknowledged as the design of choice for evaluating medical and surgical treatments and are essential for clinical progress.^2,3 However, in terms of their status and success, there are two very different perceptions of clinical trials. Firstly, there is a conventional wisdom that patients who participate in randomised controlled trials are subject to experimentation and increased risk and this ideology underpins the public perception of sacrifice and altruism. To some extent this is understandable. Well conducted clinical trials are clearly beneficial to society in the long term, and the generation of clinical trial evidence is crucial for efforts to reduce morbidity and mortality and improve the health of populations.⁴ However, individual patients instinctively hope to derive some personal benefit from whichever treatment they are given as part of their health care and, as such, are unlikely to opt for a process that they see as sacrificial rather than beneficial. The proportion of patients consenting to enter clinical trials in maternity care therefore remains small.⁵

Conversely, there is a professional perception that clinical trials represent superior care.⁶ Indeed, Iain Chalmers, co-founder of the Cochrane Collaboration, has repeatedly voiced his preference to enter a trial rather than receive inadequately evaluated routine care (Chalmers I, Report of a Seminar organised by Consumers in NHS Research and MRC Clinical Trials Unit, 2000). In contrast to the public perception, many health professionals recognise that clinical trials are good for us as a society and support the concept of a beneficial trial effect for participants.^6,7 Perhaps unfairly, health professionals have the upper hand in that they have access and exposure to current evidence, the bulk of which seems to support a favourable trial effect.^8,9 Indeed, a number of studies have identified short term beneficial effects to individual trial participants.^10–12

On the whole, previous research has focused on identifying the effect rather than the cause, but a number of potential sources of a trial effect have been reported.⁸ These include the Hawthorne effect whereby merely the process of being observed leads to benefit,¹³ the treatment effect resulting from the treatment under study being superior to the standard alternative, the care effect where the trial protocol requires more comprehensive care,⁸ and the protocol effect. The protocol effect is a simple but potentially significant concept. Clinical guidelines improve the quality of clinical practice,¹⁴ so it is clearly plausible that a similar effect can be achieved by the systemised protocols used in clinical trials that guide clinicians through clear and explicit processes of care, resulting in improved outcomes in trial participants whether in the intervention or the control group. In order to measure a protocol effect, it would be feasible to conduct a controlled trial in which participants were randomly allocated entry to the clinical trial or care following a standardised guideline or protocol. However, clinical trials are not set up to be so introspective and have clear aims in evaluating the effect of the intervention under scrutiny.

The nesting of a clinical trial within the context of care being provided according to an explicit protocol based guideline provided the opportunity for a natural comparison. The aim of this study was to observe a naturally occurring comparison to ascertain whether there is any clinical benefit to enrolling in a clinical trial when the potential advantage of a protocol effect is removed.

METHODS

Sixteen units throughout the Yorkshire region had been managing women with severe pre-eclampsia according to a strict management protocol as part of evidence based regional pre-eclampsia guidelines introduced in 1997 by the Yorkshire Obstetric Critical Care Group (unpublished). The guidelines are reviewed annually by the group and are accepted as best practice. A continuous audit of the guidelines began in 1997.

From July 1998 until November 2001 six of the 16 units were recruited to a multicentre randomised controlled trial of magnesium sulphate for the treatment of pre-eclampsia (Magpie Trial). Trial participants were randomised to receive either magnesium sulphate or a placebo of normal saline. Both those participating in the trial and those who did not continued to be managed using the regional guidelines. Hence, subsequent treatment and management were identical, the only difference being that magnesium sulphate was randomly allocated to some trial participants. As the Magpie Trial Collaborative Group was continuing to analyse Magpie study data for further related research, we were unable to unblind treatment allocation for trial participants to determine whether treatment or placebo was given.

This comparative study seized on a unique opportunity to use the routinely collected continuous audit data to compare women participating in the trial with those who did not participate.

Audit data were recorded by a network of midwives based at the participating units. Data were then collected on site by a regional audit midwife and entered into a central Access database. Incomplete data were retrieved through further site visits by the regional audit midwife, ensuring an accurate and complete data set. The reported blood tests and heparin therapy (thromboprophylaxis) were requirements of the regional guidelines, not of the Magpie trial which required no additional tests or monitoring. Indicators of the process of care (table 1) and clinical outcomes (tables 2 and 3) were compared between the two groups.

Analysis of data

Odds ratios with 95% confidence intervals were calculated for all dichotomous data using EpiInfo 2002 and confidence interval analysis (CIA) software. Where the odds ratio was inappropriate, Fisher’s exact test was used. Means, standard deviations, and mean differences with 95% confidence intervals were calculated for continuous data, again using CIA software. The null hypothesis was that there was no difference in process or outcome measures.

Multiple tests (n = 10) were applied to the process outcomes. Spurious positive results may occur if no correction is used. We applied the most conservative adjustment using the Bonferroni correction which assumes all 10 tests were independent.

RESULTS

Eighty six women from the six trial sites entered the trial (fig 1). A further 322 women from the same trial sites did not enter Magpie but were managed according to the guidelines during the recruitment period for the trial and formed the comparison group. Table 1 shows the comparison of indicators of clinical process between the two groups. There were statistically significant differences in daily blood tests (odds ratio (OR) 6.82, 95% CI 1.62 to 28.72), recording of hourly respirations (OR 3.42, 95% CI 1.69 to 6.92), and thromboprophylaxis (OR 2.21, 95% CI 1.22 to 3.98). The Bonferroni correction uses a significance level of 0.0051. Even with this conservative adjustment, hourly respirations recorded remained significant. Daily blood tests and thromboprophylaxis were close to significance despite the great conservativeness of the correction. There were no significant differences in the other seven process indicators.

• Download figure
• Open in new tab
• Download powerpoint

Figure 1

 Summary of study flow.

Clinical outcomes are shown in tables 2 and 3. Birth weight (mean difference 174.0, 95% CI −97.1 to 445.1) and length of time on the special care baby unit (mean difference 2.57, 95% CI −1.93 to 7.07) were similar in both groups with no significant differences. There was no significant difference in admission to intensive care units (OR 3.81, 95% CI 0.53 to 27.44), incidence of pulmonary oedema (OR 0.93, 95% CI 0.29 to 4.48), and length of time spent on the postnatal guidelines (mean difference 2.45, 95% CI −1.54 to 6.44). Three women from the control group experienced eclamptic fits compared with none in the trial group (p = 0.39). The Magpie trial found that magnesium sulphate when used to treat pre-eclampsia halves the risk of eclampsia.¹⁵

DISCUSSION

The results of this study suggest that compliance with clinical processes is similar for trial participants and non-participants in terms of observations, recording, and clinician involvement. There are, however, two exceptions. Routine daily blood tests and respiration recordings were requirements of the clinical guidelines and should therefore have been performed for all women in both groups. However, women in the trial were more likely to undergo daily blood tests and respiration recordings, suggesting some benefit of trial participation and supporting a positive trial effect in process markers. This may be related to the trial or it may be a chance finding. The sample is small, particularly in relation to hourly respiration recordings in the comparison group. Hourly respiratory rate monitoring was required for all women receiving or potentially receiving magnesium sulphate. This included all women in the trial group and 65 women in the comparison group who were given magnesium sulphate. Despite the differences in measures of process, there were no significant differences in clinical outcomes between the two groups.

This study benefited from robust clinical audit data that enabled a unique comparison of trial participants and non-participants within a protocol environment. Although the eligibility criteria for the regional pre-eclampsia guidelines and the Magpie trial were similar, it cannot be assumed that all the women in the comparison group were eligible for trial entry. No recruitment rate is available for the Magpie trial and only one of the six units participating in the trial from the Yorkshire region recorded trial eligibility and recruitment. Furthermore, it was not possible from the data available to assess severity of pre-eclampsia and this could have influenced selection of potential trial entrants and their subsequent management. There is potential for a Hawthorne effect in this study as clinicians were aware of the ongoing audit of the pre-eclampsia guidelines; however, as all women in both groups were included in the audit, any effect would be equally distributed across both groups. This study focused on an obstetric trial but we have no reason to believe that it is fundamentally different from other clinical trials or doubt the generalisability of our findings.

Previous studies comparing patients in trials with those receiving routine care generally report improved primary outcomes for trial participants.^{10,11,16–18} Only two studies report similar outcomes^19,20 and there is no evidence to suggest that outcomes are worsened by trial participation. This study does not confirm any clear benefit to trial participation. However, it differs from previous studies in two ways. Firstly, all the patients in this study were managed according to a clear and explicit clinical protocol. In a trial environment the protocol ensures that care is consistent and based on available evidence, and strict trial protocols encourage detail and compliance which is known to result in better care.²¹ Clinical guidelines, care pathways and protocols have the potential to simulate the beneficial effects of a trial protocol and there is strong evidence that care significantly improves following their introduction.^14,22 In this study potential differences between trial participants and non-participants may have been diluted by the presence of a protocol. Both groups received care according to the same clinical protocol. As such, the comparison group benefited from enhanced standardisation of care and experienced similar outcomes to the trial participants. This suggests that, where previously benefits to trial participation have been identified, they may be due to the presence of a protocol within the trial and the absence of a protocol outside the trial.

Secondly, our study is one of only a few studies evaluating the trial effect that compares measures of clinical process. Measures of process of care are generally more sensitive indicators of quality of care than outcomes.²³ Most bad outcomes cannot be prevented or are side effects of treatment.²⁴ It may therefore be appropriate to concentrate on process measures when assessing quality of care.

In terms of clinical practice, current evidence suggests that patients are likely to do better in a trial and recruitment should be encouraged. In practice there will never be an appropriate clinical trial for every patient. However, care provided as part of a clear and effectively implemented protocol may enhance standardisation of care and all patients may experience better care if clinical guidelines and protocols were more widely incorporated into routine practice.

Further research is necessary to develop these findings and confirm the existence and nature of a protocol effect. Not all patients have the opportunity to take part in a trial but, perhaps, patients should be encouraged to ask if their treatment is being given as part of an agreed protocol and, if not, to ask why not.