Reporting and methodologic quality of Cochrane Neonatal review group systematic reviews
© Al Faleh and Al-Omran; licensee BioMed Central Ltd. 2009
Received: 02 July 2008
Accepted: 17 June 2009
Published: 17 June 2009
The Cochrane Neonatal Review Group (CNRG) has achieved a lot with limited resources in producing high quality systematic reviews to assist clinicians in evidence-based decision-making. A formal assessment of published CNRG systematic reviews has not been undertaken; we sought to provide a comprehensive assessment of the quality of systematic reviews (both methodologic and reporting quality) published in CNRG.
We selected a random sample of published CNRG systematic reviews. Items of the QUOROM statement were utilized to assess quality of reporting, while items and total scores of the Oxman-Guyatt Overview Quality Assessment Questionnaire (OQAQ) were used to assess methodologic quality. Two reviewers independently extracted data and assessed quality. A Student t-test was used to compare quality scores pre- and post-publication of the QUOROM statement.
Sixty-one systematic reviews were assessed. Overall, the included reviews had good quality with minor flaws based on OQAQ total scores (mean, 4.5 [0.9]; 95% CI, 4.27–4.77). However, room for improvement was noted in some areas, such as the title, abstract reporting, a priori plan for heterogeneity assessment and how to handle heterogeneity in case it exists, and assessment of publication bias. In addition, reporting of agreement among reviewers, documentation of trials flow, and discussion of possible biases were addressed in the review process. Reviews published post the QUOROM statement had a significantly higher quality scores.
The systematic reviews published in the CNRG are generally of good quality with minor flaws. However, efforts should be made to improve the quality of reports. Readers must continue to assess the quality of published reports on an individual basis prior to implementing the recommendations.
As new clinical evidence is accumulating at a phenomenal rate, staying up-to-date with the current state of knowledge can be a challenging task for health care providers. It has been estimated that a physician would need to read 17–20 journal articles a day to keep up to date with his particular area of interest[1, 2]. This is one of many reasons why systematic reviews and meta-analyses have been introduced as a solution to synthesize and summarize evidence of primary studies on a given topic. Meta-analyses of high quality, randomized, controlled trials (RCT) are considered to be the highest level of evidence in the hierarchy of evidence-based medicine for preventive and therapeutic interventions [3, 4].
The Cochrane Collaboration (CC), founded in 1993, is a well-recognized international organization. It aims to help health care professionals make an informed decisions about health care by preparing, maintaining, and promoting the accessibility of systematic reviews on the effects of health care interventions [5, 6]. The Cochrane Neonatal Review Group (CNRG), dedicated to improving outcomes of newborn infants, is one of the 51 collaborative groups registered with the CC, and has achieved a considerable amount with limited resources in synthesizing and providing the highest quality of evidence to neonatal health care providers . Members of the CNRG prepare reviews of the results of RCT's of intervention for treatment and prevention of disease in newborn infants . In preparing their reviews, reviewers follow systematic methods summarized in the Cochrane Reviewer's Handbook and summarized in a checklist developed specifically for the neonatal reviews by the editors of the group. A protocol that outlines the specific scientific objectives and methods of the review must be accepted prior to the start of the review process. All protocols and reviews are published four times a year in the neonatal module of the Cochrane Library and on a National Institute of Child Health and Human Development (NICHD) website for Cochrane neonatal reviews. The number of reviews has steadily increased over the last few years with more than 200 completed reviews currently published.
Compared to systematic reviews published in paper-based journals, Cochrane reviews appear to have greater methodologic rigor and include elements that make them less prone to bias, such as description of inclusion and exclusion criteria, and formal assessment of the trial's quality . In addition, the reviews are more frequently updated. However, Cochrane reviews are not immune to methodologic flaws and the room for continual assessment and improvement in quality still exists. Ensuring publication of systematic reviews with the highest possible quality will likely ensure that the results are less likely influenced by bias, therefore enabling clinicians to be more confident in implementing different interventions in their practice .
As for any research project, assessment of the quality of systematic reviews is essential to judge whether the recommendation warrants a change in practice. This can be done through assessment of both methodologic quality (how well the systematic review was conducted) and reporting quality (how well the methodology and findings are presented. The most commonly used tools to assess quality of systematic reviews are the Quality of Reporting of Meta-analysis statement (QUOROM statement) and the Overview Quality Assessment Questionnaire-OQAQ[13, 14].
The current study was performed to provide a comprehensive assessment of the quality of systematic reviews (both methodologic and reporting quality) published in CNRG and to assess whether the publication of the Quality of Reporting of Meta-analysis statement (QUOROM) published in 1999 is associated with an improvement of in the quality of the reviews under study.
Characteristics of the included CNRG systematic reviews.
Number of included reviews
No. of included studies**
Quality Assessment Instruments
Based on a previous extensive review of published scales and checklists available for quality assessment of systematic reviews, the two instruments selected were methodologic tools that were rigorously developed by Oxman and Guyatt (Overview Quality Assessment Questionnaire [OQAQ]; and the Quality of Reporting of Meta-analysis (QUOROM) statement, published in November 1999.
Quality of Reporting of Meta-analysis (QUOROM) statement
Scores of included CNRG systematic reviews based on elements of the QUOROM statement.
QUOROM statement item *
Title identified as meta-analysis or SR
Structured format used
Data sources reported
Review methods reported
Characteristics of studies
Online Registry of Studies
Hand Search of Journals
Data Abstraction in Duplicate and Independent
Quantitative Data Synthesis
Principal Measure of Effect
Method of Combining Data
Handling Missing Data
Rationale for Subgroups
Assessment of Publication Bias
Quantitative data synthesis
Agreement on selection
Agreement on validity
Present data needed to calculate effect size
Summarize key findings
Discuss internal and external validity
Discuss potential biases
Suggest future research
Overview Quality Assessment Questionnaire (OQAQ)
Scores of included CNRG systematic reviews based on Overview Quality Assessment Questionnaire (OQAQ)
Response to Question *
Search methods used to find evidence stated
Search for evidence reasonable comprehensive
Criteria used for deciding which studies
to include reported
Bias in the selection of studies avoided
Criteria used for assessing validity of included studies reported
Validity of included studies assessed appropriately
Methods used to combine the findings of studies reported
Findings of studies combined appropriately
Conclusions made by authors supported by analysis
Overall quality score
4.5 (0.9), (95% CI: 4.27, 4.77)**
Two reviewers (KA and MA) reviewed the full texts of all included reviews and extracted data independently in a database developed using Microsoft Access 2003 (Microsoft Corporation). Both reviewers have abstracted and analyzed the data independently. Disagreement among reviewers was resolved through consensus or by consulting a third expert adjudicator.
In addition to the items in the OQAQ and QUOROM statement, the review's demographic data were collected, including: 1) the last name of the 1st author, 2) the subspecialty, 3) the epidemiologic affiliation of the 1st author (whether he/she is qualified with a degree in epidemiology or research methodology), and 4) the number of included studies. Stratification based on relation to the QUOROM statement publication date was entertained to assess any potential effect of the QUOROM statement on overall quality of included reviews; however, we found that most reviews published prior to the QUOROM statement publication were updated at a later stage (we assessed the most recent version of the review).
To enhance the reviewer's inter-rater agreement, we evaluated 5 reviews (not included in the study sample) as a pilot testing of the database prior to starting the data abstraction process. Proper scoring of each item in the database was discussed in detail.
The primary analysis of our data was descriptive. The proportion of reports that met each criterion was determined and tabulated. Data on each item is presented as counts and percentages. To determine whether the publication of the QUOROM statement was associated with an improvement of quality of published CNRG reviews, we compared the mean and standard deviation of the overall scientific quality scores (question 10 in OQAQ) of each pre- and post-QUOROM groups using a two sample Student t-test with Minitab software version 14. Since the number of reviews exceeded 30, and the data were normally distributed, a central limit theorem was assumed.
Sixty-one CNRG systematic reviews were evaluated in our study. Table 1 summarizes the demographic data of the included reviews. All reviews in the CNRG addressed interventions of a therapeutic nature. The distribution of topics strongly favored respiratory (31%) and nutritional (23%) interventions. The majority of the reviews (82%) were published after the publication of the QUOROM statement. The median number and inter-quartile ranges of the included trials in each review were 3 (1, 6). Eight reviews (13%) included no studies; therefore, assessment of some elements in the results section of the reviews was unfeasible. This explains the discrepant total numbers that appear in table 2.
Reporting Quality (QUOROM statement elements)
Reports of various sub-items of the QUOROM statement are presented in table 2. Reports ranged from 0–100%. Although all reviews included in the CNRG had the word "review" typed between brackets at the end of the title, none of the titles identified the study as a systematic review or a meta-analysis.
Almost all the CNRG review's abstracts were structured, stated objectives, reported data sources, specified selection criteria adequately, and ended with a conclusion that summarized the review's main findings. On the other hand, important abstract elements of the review's methods (validity assessment, data abstraction, and data synthesis details) and results (mainly characteristics of included studies) were inadequately reported.
As per reporting standards in the Cochrane collaboration, all reviews reported their search terms, databases searched, and any additional sources of data. The majority searched electronic databases, such as Medline and/or the Cochrane Central Register of Controlled Trials (CENTRAL) in the Cochrane Library, citation lists of included studies, and conference proceedings. However, contact with manufacturers (3%) and authors of included studies (16%) to limit the chance of publication bias were inadequately reported. Only a small number of reviews reported restrictions of the search to a certain year, language, or to published studies only.
The selection criteria, validity assessment, and the principle measure of effect were clearly and adequately stated in the methods section of the included reviews. However, most of the reviews were deficient in adequate reporting of clinical and statistical assessment of heterogeneity (21%), and providing a rational for their planned subgroup analysis (11%). None of the included reviews mentioned a priori plans of assessment of publication bias. The statistical effect model planned a priori was reported in most reviews; however, reviewers either chose a fixed effect model (64%) or reported neither fixed nor random models without adequate explanation of the rationale and whether their choice would be altered by the subsequent heterogeneity assessment of the included studies.
Although the majority of the reviews reported a two reviewer selection and data abstraction processes, only 3 (5%) reported agreement levels among the reviewers.
All reviews reported a detailed description of the included and excluded studies, presented summarized results, and the numbers needed to calculate the effect sizes. None of the reviews presented a chart describing the trials flow.
A summary of the review's key findings and a suggestion for a future research agenda were adequately reported in the discussion section (98%); however, a discussion of the internal and external validity issues and potential bias (5%) in the review process and methods requires further improvement.
Methodologic Quality (OQAQ items)
Table 3 presents a summary of OQAQ items scores (items 1–9) of the included reviews. Items that were adequately reported include explicit mention of search methods, comprehensiveness of search strategy, reporting of selection criteria of included studies, methods and appropriateness of validity assessment criteria used, and a conclusion that is supported by the analysis presented in the review. Only 26% of included reviews explicitly addressed how selection bias (particularly publication bias) was avoided during the review process.
Effect of QUOROM statement publication on overall quality scores
A comparative assessment of CNRG systematic reviews based on pre- and post-QUOROM statement status
N = 17
N = 44
3.00 – 5.00
3.00 – 6.00
3.34 – 4.20
4.58 – 5.08
Mean difference = -1.03 (95% CI: -1.49, -0.56), p-value < 0.0001*
Our review is the first report to assess the methodologic and reporting quality of systematic reviews done in the neonatal field; our findings should be quite reassuring to neonatal practitioners. Overall, we found through the use of available instruments that reviews published in CNRG are generally of good quality; however, with minor flaws. The Cochrane reviews in general included elements that render them less prone to bias, such as a well-structured format, description of inclusion and exclusion criteria, and formal assessment of the trial's quality.
As mentioned above, we found that CNRG systematic reviews scored high with regards to clearly stating their objectives, comprehensiveness of search strategy, explicit detailed inclusion criteria, formal quality assessment of included trials, summarizing the key findings, and suggestion of future research topics. However, room for improvement still exists in some areas, such as the title, abstract reporting, a priori plan for heterogeneity assessment and how to handle heterogeneity in case it exists, assessment of publication bias, reporting of agreement among reviewers, documentation of trials flow, and discussion of possible biases in the review process.
As for most medical practitioners, health care professionals practicing in the neonatal field consider the results of systematic reviews published in the Cochrane Library of the highest level of evidence. Although the overall quality of CNRG systematic reviews was judged to be good, greater than one-third (41%) of the included reviews scored 3–4 and were considered to have major flaws, which poses an important question to the validity of the recommendations. Therefore, readers must carefully and critically appraise published reports prior to adopting a recommendation as the basis for change in practice.
We found that the quality scores of included reviews significantly improved after the publication of the QUOROM statement. This observation is prone to many biases and could simply represent improvement of research methods by time trends or improvement in the reviewer's methodological skills as they undertake more reviews.
The final quality scores reported in this review are quite comparable to the recently published review of quality of systematic review in the Cochrane Musculoskeletal Group (CMSG) and higher than reported scores of systematic reviews published in anesthesia, critical care, emergency medicine, and general surgery [3, 18, 19], which reinforces the notion that Cochrane reviews, in general, appear to have a high methodologic quality and "frequently" updated.
Although we took all measures possible to enhance the validity of our results, it is important to remind the reader that our results are of an observational nature and are prone to bias. An important observation we and the authors of previous reports have noted [11, 19] is the difficulty in applying the measurement tools (QUOROM statement and OQAQ), hence these instruments were not subjected to extensive validation testing hence these instruments were not subjected to extensive validation testing and lack clear and detailed guidance of their application. This is particularly important in applying the overall quality score (Q10 in OQAQ), which we found to be very subjective.
In conclusion, systematic reviews published in CNRG are generally of good quality with minor flaws. However, efforts should be made to improve abstract reporting, handling of heterogeneity, assessment of publication bias, and documentation of reviewer's agreement and trials flow. Readers must continue assessing the quality of published reports on an individual basis prior to implementing their recommendations.
This project was partially funded by the Scientific Research Deanship and at the King Saud University, Riyadh, Kingdom of Saudi Arabia. We thank the Evidence-based Medicine and Neonatal Research Chairs for their support in the completion of this project
- Davidoff F, Case K, Fried PW: Evidence-based medicine: why all the fuss?. Ann Intern Med. 1995, 122: 727-View ArticlePubMedGoogle Scholar
- Haynes RB: Where's the meat in clinical journals?. ACP Journal Club. 1993, 199: A23-24.Google Scholar
- Dixon E, Hameed M, Sutherland F, Cook DJ, Doig C: Evaluating meta-analyses in the general surgical literature: a critical appraisal. Ann Surg. 2005, 241: 450-459. 10.1097/01.sla.0000154258.30305.df.View ArticlePubMedPubMed CentralGoogle Scholar
- Guyatt GH, Sackett DL, Sinclair JC, Hayward R, Cook DJ: Users' guides to the medical literature. IX. A method for grading health care recommendations. Evidence-Based Medicine Working Group. JAMA. 1995, 274: 1800-1804. 10.1001/jama.274.22.1800.View ArticlePubMedGoogle Scholar
- Bero L, Rennie D: The Cochrane Collaboration. Preparing, maintaining, and disseminating systematic reviews of the effects of health care. JAMA. 1995, 274: 1935-1938. 10.1001/jama.274.24.1935.View ArticlePubMedGoogle Scholar
- Jadad AR, Haynes RB: The Cochrane Collaboration – advances and challenges in improving evidence-based decision making. Med Decis Making. 1998, 18: 2-9. 10.1177/0272989X9801800102.View ArticlePubMedGoogle Scholar
- Davis PG: Cochrane reviews in neonatology: Past. Present and future. Semin Fetal Neonatal Med. 2006, 11: 111-116. 10.1016/j.siny.2005.11.004.View ArticlePubMedGoogle Scholar
- Sinclair JC, Haughton DE, Bracken MB, Horbar JD, Soll RF: Cochrane neonatal systematic reviews: a survey of the evidence for neonatal therapies. Clin Perinatol. 2003, 30: 285-304. 10.1016/S0095-5108(03)00025-3.View ArticlePubMedGoogle Scholar
- Higgins JPT: GSe, Cochrane Handbook for Systematic Reviews of Interventions. The Cochrane Collabortion. 2008Google Scholar
- Jadad AR, Cook DJ, Jones A, Klassen TP, Tugwell P, Moher M, et al: Methodology and reports of systematic reviews and meta-analyses: a comparison of Cochrane reviews with articles published in paper based journals. JAMA. 1998, 280: 278-280. 10.1001/jama.280.3.278.View ArticlePubMedGoogle Scholar
- Shea B, Bouter LM, Grimshaw JM, Francis D, Ortiz Z, Wells GA, et al: Scope for improvement in the quality of reporting of systematic reviews. From the Cochrane Musculoskeletal Group. J Rheumatol. 2006, 33: 9-15.PubMedGoogle Scholar
- Moher D, Soeken K, Sampson M, Ben-Porat L, Berman B: Assessing the quality of reports of systematic reviews in pediatric complementary and alternative medicine. BMC Pediatr. 2002, 2: 3-10.1186/1471-2431-2-3.View ArticlePubMedPubMed CentralGoogle Scholar
- Moher D, Cook DJ, Eastwood S, Olkin I, Rennie D, Stroup DF: Improving the quality of reports of meta-analyses of randomized controlled trials: the QUOROM statement. Quality of Reporting of Meta-analyses. Lancet. 1999, 354: 1896-1900. 10.1016/S0140-6736(99)04149-5.View ArticlePubMedGoogle Scholar
- Oxman AD, Guyatt GH: Validation of an index of the quality of review articles. J Clin Epidemiol. 1991, 44: 1271-1278. 10.1016/0895-4356(91)90160-B.View ArticlePubMedGoogle Scholar
- Pagano MGK: Principles of Biotastistic. 1993, Belmont, California: Duxbury PressGoogle Scholar
- Oxman AD, Guyatt GH, Singer J, Goldsmith CH, Hutchison BG, Milner RA, et al: Agreement among reviewers of review articles. J Clin Epidemiol. 1991, 44: 91-98. 10.1016/0895-4356(91)90205-N.View ArticlePubMedGoogle Scholar
- Villar J, Mackey ME, Carroli G, Donner A: Meta-analyses in systematic reviews of randomized controlled trials in perinatal medicine: comparison of fixed and random effects models. Stat Med. 2001, 20: 3635-3647. 10.1002/sim.1096.View ArticlePubMedGoogle Scholar
- Choi PT, Halpern SH, Malik N, Jadad AR, Tramer MR, Walder B: Examining the evidence in anesthesia literature: A critical appraisal of systematic reviews. Anesth Analg. 2001, 92: 700-709. 10.1097/00000539-200103000-00029.View ArticlePubMedGoogle Scholar
- Delaney A, Bagshaw SM, Ferland A, Manns B, Laupland KB, Doig CJ: A systematic evaluation of the quality of meta-anaylses in the critical care literature. Crit Care. 2005, 9: R575-R582. 10.1186/cc3803.View ArticlePubMedPubMed CentralGoogle Scholar
- The pre-publication history for this paper can be accessed here:http://www.biomedcentral.com/1471-2431/9/38/prepub
This article is published under license to BioMed Central Ltd. This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.