- Research article
- Open Access
- Open Peer Review
Randomized controlled trials in pediatric complementary and alternative medicine: Where can they be found?
BMC Pediatricsvolume 3, Article number: 1 (2003)
The safety and effectiveness of CAM interventions are of great relevance to pediatric health care providers. The objective of this study is to identify sources of reported randomized controlled trials (RCTs) in the field of pediatric complementary and alternative medicine (CAM).
Reports of RCTs were identified by searching Medline and 12 additional bibliographic databases and by reviewing the reference lists of previously identified pediatric CAM systematic reviews.
We identified 908 reports of RCTs that included children under 18 and investigated a CAM therapy. Since 1965, there has been a steady growth in the number of these trials that are being published. The four journals that published the most reported RCTs are The American Journal of Clinical Nutrition, Pediatrics, Journal of Pediatrics, and Lancet. Medline, CAB Health, and Embase were the best database sources for identifying these studies; they indexed 93.2%, 58.4% and 42.2 % respectively of the journals publishing reports of pediatric CAM RCTs.
Those working or interested in the field of pediatric CAM should routinely search Medline, CAB Health and Embase for literature in the field. The four core journals identified above should be included in their collection.
Complementary and alternative medicine (CAM) is defined as a broad domain of healing resources that encompasses all health systems, modalities and practices and their accompanying theories and beliefs, other than those intrinsic to the politically dominant health system of a particular society or culture in a given historical period .
The safety and effectiveness of CAM interventions are of great relevance to pediatric health care providers. Two recent surveys indicate a use of pediatrics CAM in the United States, finding a 12% and 21% overall use, respectively [2, 3]. Factors such as ongoing medical problems and parents' use of CAM are associated with pediatric use [3, 4]. Ideally, evidence-based practitioners can approach children and their guardians regarding their use of CAM and on a case-by-case basis to help identify relevant evidence regarding therapies that patients use or wish to consider.
Such discussions are likely to be more rewarding when practitioners and families can advocate for the use of CAM therapies for which there is stronger evidence, such as from the results of randomized trials of effective herbal therapies . Similarly, interventions for which there is little of no evidence might be best avoided .
We undertook a program of research to discover whether such evidence, in the form of published systematic reviews and randomized controlled trials involving children, existed [7, 8]. Based on the large number of randomized controlled trials we did find, we identified a need for a guide for evidence-based practitioners who wish to determine whether sound evidence exists.
The objective of the research presented here was to identify where health care practitioners can locate reported RCTs of pediatric CAM; specifically, in which journals these reports are published and through which databases those reports may be identified.
Identification of Candidate Studies
We searched for reports of RCTs in 13 bibliographic databases, using either a detailed search strategy based on the main subject search of the Cochrane Collaboration's Complementary Medicine Field subject search,  or a modification of that search tailored to the capabilities of particular databases . The databases searched are listed in Table 1, and the search strategies are available from the authors upon request. We also identified reports of RCTs from the cited references of 49 pediatric CAM systematic reviews. The reviews were previously identified by the Chalmers Research Group in a study that complements this project .
Bibliographic records were imported into a Reference Manager 9.0™ database and duplicate items were consolidated. Bibliographic records were screened by KC to determine if: the study included children under 18; a CAM therapy was investigated (following the definition of CAM adopted by the Cochrane Complementary Medicine Field Registry of Randomized Controlled Trials and presented in their Registry Guidelines) ; and the article was a report of an RCT. Where a final determination of eligibility could not be made based on information contained in the bibliographic record, the complete article was retrieved and reviewed. After about the first 100 records had been assessed, every tenth record was reviewed by MS as a quality measure. Inter-observer agreement was almost perfect, with Kappa = 0.97  without the reviewers undertaking any discussion designed to achieve consensus. A total of 908 reports of RCTs met the criteria for inclusion in the study (see Figure 1).
Studies published in languages other than English were included if: (1) the article was written in a language we were able to read (German, French, and Spanish); or (2) we were able to ascertain, from the English abstract, whether the study met the inclusion criteria. Only journal-published articles were considered. Theses, conference proceedings, book chapters, and other grey literature were not considered.
A post hoc analysis was undertaken to assess whether the apparent decline in the number of reports of trials in complementary and alternative medicine in the final years of this study was due to a delay in the indexing of these articles. For this analysis, the number of Medline records retrieved by search strategy 1 was broken down by year, ending with September Week 3 2002. Simple counts were taken – the individual items were not evaluated against the study inclusion criteria.
For each eligible publication, the date of publication, authors, language of publication, and journal name were extracted from the Reference Manager database or from inspection of the article. We consulted the reference material for each of the 13 databases searched to determine whether or not the database indexed each of the journals that had published one or more eligible reports of RCTs.
A total of 908 reports of RCTs met the criteria for inclusion in the study.
Year of publication
The earliest reports of RCTs we found were published in 1965, and there was little growth in this literature until 1975. From 1975, there was significant steady growth for over twenty years, with a peak of 89 articles published in 1997 (see Figure 2). There is the beginning of a decline in the next two years, followed by a sudden drop to 45 reports of RCTs in 2000 and 2 in 2001. This decline is probably an artifact of indexing delay, as this research was conducted in the winter of 2001. Post hoc analysis conducted in September 2002 suggests that growth in Medline of records meeting the criteria of search strategy 1 (but not formally screened according to study inclusion criteria) continued to grow through 2000, and again shows a dip for the previous and current year (2001 and 2002).
92.2% of reports of RCTs published in this field are written by more than one author. The largest number of authors for a single paper was 25 , while almost half of all papers (49.3%) have four or fewer authors. The five most prolific authors have published at least 15 reports of RCTs each. The vast majority (57.8%) of all authors cited in the 908 studies authored a single paper and 70% of authors published four or fewer CAM trials.
Language of Publication
Most reports of RCTs were published in English (93.2%), although 12 other languages were represented. The second most common language of publication was Chinese, with 18 reports of RCTs (2%) and a concentration of studies in herbal medicine. There were more than five reports of RCTs in each of German, French, Spanish and Italian, each with trials in a variety of topics. Seven languages were represented by one or two reports of RCTs (Table 2). The language restrictions resulted in the exclusion of 41 publications as we were unable to confirm that they were reports of RCTs in pediatric CAM; 23 of these were published in Russian, 6 written in Chinese, 4 in Ukrainian, 3 in Japanese, 2 in Italian, and one each in Hungarian, Czech and Swedish.
The reports of RCTs originated from 308 different journals. These journals were ranked by the number of reports of RCTs published. As a method to examine the dispersion of the literature, we divided the journals into four zones, each representing approximately one quarter of the articles published.
Of the total number of reports of RCTs, 217 (23.9%) were identified as having been published in the top four journals (1.3% of the total of 308 journals that published at least one relevant reported RCT). In Zone 2, 24 (7.8%) journals published 27.1% of the total number of reports of RCTs identified. Zone 3 encompasses 96 (31.2%) journals and contains 28.7% of the total reports of RCTs identified. Zone 4 contains 184 journals (59.7%), and 20.3% of the total reports of RCTs identified. The top journals, located in Zones 1 and 2, are displayed in Table 3. The names and rankings of journals in Zones 3 and 4 are available from the authors on request.
For the 308 journals that published reports of pediatric CAM RCTs, the number of journals indexed by each of the databases is shown in Table 4, along with the unique contribution of coverage provided by each database.
We identified over 900 reports of RCTs using a CAM intervention in children. We found a steady growth in the literature. We found fewer articles in most recent years, likely due to a lag in indexing. There was an apparent decline in the frequency of reporting from 1998 to 2001. We expected that the indexing of reports of RCTs published during this period would not have been completed by the time we conducted our research in 2001, but post hoc examination of the year by year number of items retrieved from Medline by search strategy 1 shows that some articles in this area can take more than one year to be indexed in bibliographic databases. Those wishing to obtain all journal articles describing randomized controlled trials of pediatric complementary and alternative medicine may wish to consider hand-searching the most recent years of key journals unless it can be determined that recent issues have been indexed by the bibliographic databases.
Our results indicate that most reports (93%) of pediatric CAM RCTs are published in English. This finding has been reported elsewhere , although not specifically for pediatrics. Although we had to exclude relatively few trials (n = 41) on the basis of language, and the databases we used provide content for journals published in many languages besides English, it is possible that we missed important clusters of non-English reports. In particular, Tang et al.  describe a large number of RCTs of traditional Chinese medicine conducted in China, (approximately 7500 published prior to 1997) as being for the most part inaccessible to Western doctors. However, the journal cited by Tang as having published ten times as many RCTs as any other they examined (Zhongguo Zhong Xi Yi Jie He Za/ Chinese Journal of Integrated Traditional and Western Medicine) is indexed in Medline with entries dating back to 1992. Of 1568 articles from that journal indexed, 284 are tagged as randomized controlled trials in Medline. Of these, only 87 relate to childhood age groups, and all of these have abstracts, thus would be have been picked up by our search and eligible for inclusion in our study. Still, Tang's review underscores the risks of over-reliance on database searching to identify all trials in a given area. For those wishing to search exhaustively, there is potentially great value to involving subject experts who can identify the locations of specialty literatures, such as the literature reporting trials of traditional Chinese medicine described by Tang et al.
The 908 reports were identified in more than 300 journals, yet almost a quarter (24%) of them were found in just four journals (American Journal of Clinical Nutrition, Pediatrics, Journal of Pediatrics, and The Lancet). By expanding the core journal list to 28 journals, it is possible to account for slightly more than half of all reports of pediatric CAM RCTs we found. These results suggest that most reports of randomized controlled trials of CAM are 'mainstream' medical journals. This finding is similar to that reported by Pittler and colleagues  who investigated the relationship between the statistical directions of CAM RCT results and where they were likely to be published. They found that mainstream medical journals with an impact factor greater or equal to 1 published an equal number of CAM trials with positive and negative results, while lower impact factor journals published more CAM trials with positive findings.
In terms of database coverage, Medline provided the best coverage of the Zone 1 and 2 journals, indexing all 28. Embase and CAB Health indexed 85.7% and 92.3% respectively. CINAHL and AMED indexed relatively few of the journals in Zones 1 and 2, (25% and 14% respectively) but they did provide unique material not available from the other databases. Together, these five databases gave complete coverage of all journals and reports of RCTs identified in this study. Two other databases, CCTR and IBIDS, provided relatively good coverage of these first two zones (65.3% and 57.1%), but because of overlapping coverage with other databases, provided no unique coverage.
In terms of database selection, a search of Medline alone could potentially identify 97.7% of these reports and a search combining Medline, Embase and CAB Health could achieve 99.4% retrieval, assuming a comprehensive search strategy is used. Given the recent development of the CCTR we were impressed with its coverage although its unique contribution is only marginal in terms of the cumulative increase of identified reports of RCTs resulting from searching this source. In an effort to further enhance the CCTR's coverage we have forwarded the database from this study to the Cochrane Collaboration's CAM field so that the information can be added to the CCTR.
Searching Medline, along with any two of Embase, CCTR, or CAB Health, would appear to constitute a reasonable approach for those wishing to search this topic comprehensively. More exhaustive searching, of the type undertaken for a systematic review, would ideally be supplemented by CINAHL and AMED, as well as selective hand searching and other traditional bibliographic techniques .
Those wanting to keep abreast of the latest pediatric CAM reports of RCTs by scanning journals as they are published, may be able to maximize their time by focusing their attention on four journals. Information specialists are likely to be able to track most new evidence by searching three databases.
Shortly after we completed the retrieval work for this project, the National Library of Medicine introduced a complementary medicine subset to its Medline database . Searches can be limited to this subset through PubMed (available at http://www.pubmed.org, and this feature is likely to become available through database vendors. This may prove to be a useful tool for clinicians wishing quick access to some of the best literature in this area. PubMed includes Medline, is available at no cost to those with Internet access and can be searched without complex syntax. As a case study, we entered the word random* in the search box, applied the limits of age "All Child: 0–18 years" and subset "Complementary Medicine" and publication type of "Randomized Controlled Trial" retrieved 1858 citations. A clinician replacing the search string random* with a subject term massage would find 56 citations as a starting point, the term chiropract* with the same limits would yield 29, st johns wort or st john's wort would yield 8 reports and herbal would yield 140 citations.
A limitation of this study is that our figures on the cumulative percent of trials accounted for (Table 4) assume that all trials published in that journal were indexed in the database. The actual indexing rate may be less than 100%, and the indexing status of each report of an RCT in each database was not verified in this study. In addition, researchers should be aware that indexing practices and search capabilities available with particular databases vary, thus there is some advantage to building redundancy in coverage into a search strategy that has high recall as an objective.
It should be noted that the Cochrane Controlled Trials Registry is a special case among the databases we searched. As its name suggests, it is a database of controlled trials, including randomized controlled trials, and is constructed from trials found in other databases and identified through hand searching of journals [19, 20]. At the time our searches were conducted, it was searchable only through its own proprietary interfaces. It has subsequently become available through at least one database aggregator, Ovid. Had such an interface been available at the time of our searching, we would have searched it using search strategy 1. This could potentially have increased its unique contribution beyond what is seen in Table 4. In a recent study in which known item searching techniques were used to determine if reports of randomized controlled were indexed in various databases, the CCTR performed extremely well, and was identified by the authors as a valuable but underused source for meta-analysts .
We used a strict definition of CAM, following the Registry Guidelines adopted by the Cochrane Complementary Medicine Field Registry of Randomized Controlled Trials, thereby excluding some psychological interventions that might be considered CAM by some (such as cognitive therapy) while including others (such as biofeedback). Our view is that some psychological interventions, such as cognitive behavior therapy, might well be considered a 'standard of care' in many Canadian and American settings.
We did not specifically follow a mega-dose definition for classifying nutritional therapies as CAM or conventional medicine. This may have overstated the contribution of the American Journal of Clinical Nutrition. We attempted to follow the Registry Guidelines for the Cochrane Complementary Medicine Field Registry of Randomized Controlled Trials but erred on the side of over-inclusion of therapies as CAM if the author's self-classification was not clear from the title, abstract and indexing information.
It is clear from this study that there is a large body of evidence from randomized controlled trials in pediatric complementary and alternative medicine. The quality of that literature is also relevant [22–24], although a recent paper suggests that quality measures may be less important than previously reported in obtaining unbiased estimates of intervention effectiveness in meta-analysis . Schulz and colleagues reviewed 250 reports of RCTs and found that those with inadequate allocation concealment, compared to reports in which this information was adequately reported, exaggerated the estimates of an intervention's effectiveness by 30%, on average. Recently, Moss and colleagues  reported that in a review of 134 pediatric surgical RCTs, less than half of the trials (46%) reported any details of the method of randomization used. Tang et al. notes that although improving, methodological problems remain in the large body of trials of traditional Chinese medicine . We previously reported on the quality characteristics of a subset of the trials found in this project .
RCT evidence in pediatric CAM is being published in a substantial number of journals. This study identifies the primary sources of reports of RCTs in the field of pediatric CAM-those journals found in Zone 1. These journals are a necessary core for any collection in this field. The journals in Zone 2 are also important sources of evidence, and are also recommended resources to be included in a pediatric CAM collection. Journals in Zone 3 would be a useful addition for those with larger collection budgets, and are good resources for those needing a comprehensive collection. The journals within Zone 4 contain only one report of an RCT each identified during the time period studied. While not prime candidates for a collection focusing on complementary and alternative medicine, they should be considered by systematic reviewers, who may identify potentially relevant studies from them through the relevant bibliographic databases. This study also indicates that those seeking pediatric CAM evidence require access to at least three indexing databases. These are Medline, Embase, and CAB Health.
– Randomized Controlled Trial
– Complementary and Alternative Medicine
– Cumulative Index to Nursing and Allied Health
– Allied and Complementary Medicine
– The Cochrane Controlled Trials Register
– International Bibliographic Information on Dietary Supplements
– Manual Alternative and Natural Therapy
– The National Center for Complementary and Alternative Medicine
–AGRICultural OnLine Access
Panel on Definition and Description CRMCA1: Defining and describing complementary and alternative medicine. Alternative Therapies. 1997, 3: 49-57.
Ottolini MC, Hamburger EK, Loprieato JO, Coleman RH, Sachs HC, Madden R, et al: Complementary and alternative medicine use among children in the Washington, DC area. Ambul Pediatr. 2001, 1: 122-125.
Sawni-Sikand A, Schubiner H, Thomas RL: Use of complementary/alternative therapies among children in primary care pediatrics. Ambul Pediatr. 2002, 2: 99-103.
Heuschkel R, Afzal N, Wuerth A, Zurakowski D, Leichtner A, Kemper K, et al: Complementary medicine use in children and young adults with inflammatory bowel disease. Am J Gastroenterol. 2002, 97: 382-388. 10.1016/S0002-9270(01)04036-9.
Ernst E: Herbal medicines: where is the evidence?. BMJ. 2000, 321: 395-396. 10.1136/bmj.321.7258.395.
Complementary medicine: time for critical engagement. Lancet. 2000, 356: 2023-10.1016/S0140-6736(00)03392-4.
Moher D, Soeken K, Sampson M, Ben Porat L, Berman B: Assessing the quality of reports of systematic reviews in pediatric complementary and alternative medicine. BMC Pediatr. 2002, 2: 3-10.1186/1471-2431-2-3.
Moher D, Sampson M, Campbell K, Beckner W, Lepage L, Gaboury I, et al: Assessing the quality of reports of randomized trials in pediatric complementary and alternative medicine. BMC Pediatr. 2002, 2: 2-10.1186/1471-2431-2-2.
The Cochrane Library. Oxford, Update Software. 2001, 1
McKibbon A: PDQ Evidence-based principles and practice. Hamilton: BC Decker Inc. 1998
Cochrane Complementary Medicine Field Registry Guidelines. 2001
Landis JR, Koch GG, Freeman JL, Freeman DHJ, Lehnen RC: A general methodology for the analysis of experiments with repeated measurement of categorical data. Biometrics. 1977, 33: 138-158.
Visalli N, Cavallo MG, Signore A, Baroni MG, Buzzetti R, Fioriti E, et al: A multi-centre randomized trial of two different doses of nicotinamide in patients with recent-onset type 1 diabetes (the IMDIAB VI). Diabetes Metab Res Rev. 1999, 15: 181-185. 10.1002/(SICI)1520-7560(199905/06)15:3<181::AID-DMRR31>3.3.CO;2-8.
Moher D, Pham , Klassen TP, Schulz KF, Berlin JA, Jadad AR, et al: What contributions do languages other than English make on the results of meta-analyses?. J Clin Epidemiol. 2000, 53: 964-972. 10.1016/S0895-4356(00)00188-8.
Tang JL, Zhan SY, Ernst E: Review of randomised controlled trials of traditional Chinese medicine. BMJ. 1999, 319: 160-161.
Pittler MH, Abbot NC, Harkness EF, Ernst E: Location bias in controlled clinical trials of complementary/alternative therapies. J Clin Epidemiol. 2000, 53: 485-489. 10.1016/S0895-4356(99)00220-6.
Counsell C: Formulating questions and locating primary studies for inclusion in systematic reviews. In Systematic reviews: Synthesis of best evidence for health care decisions. Edited by: Mulrow C, Cook D. 1998, Philadelphia: American College of Physicians
Nahim AM: Complementary medicine – new PubMed subset. NLM Technical Bulletin. 2001, 318: e7-
The Cochrane Collaboration Handbook 4.1.1. In The Cochrane Library. Edited by: Clarke M, Oxman AD. 2000, Oxford: Update Software, 1
Avenell A, Handoll HHG, Grant AM: Lessons for search strategies from a systematic review, in The Cochrane Library, of nutritional supplementation trials in patients after hip fracture. Am J Clin Nutr. 2001, 73: 505-510.
Sampson M, Barrowman NJ, Moher D, Klassen TP, Pham B, Platt R, et al: Should meta-analysts search Embase in addition to Medline?. Journal of Clinical Epidemiology. 2003.
Kjaergard LL, Villumsen J, Gluud C: Quality of randomised clinical trials affects estimates of intervention efficacy. 7th Annual Cochrane Colloquium Abstracts, in Rome. October 1999
Moher D, Pham B, Jones A, Cook DJ, Jadad AR, Moher M, et al: Does quality of reports of randomised trials affect estimates of intervention efficacy reported in meta-analyses?. Lancet. 1998, 352: 609-613. 10.1016/S0140-6736(98)01085-X.
Schulz KF, Chalmers I, Hayes RJ, Altman DG: Empirical evidence of bias: dimensions of methodological quality associated with estimates of treatment effects in controlled trials. JAMA. 1995, 273: 408-412. 10.1001/jama.273.5.408.
Balk EM, Bonis PA, Moskowitz H, Schmid CH, Ioannidis JP, Wang C, et al: Correlation of quality measures with estimates of treatment effect in meta-analyses of randomized controlled trials. JAMA. 2002, 287: 2973-2982. 10.1001/jama.287.22.2973.
Moss RL, Henry MC, Dimmitt RA, Rangel S, Geraghty N, Skarsgard ED: The role of prospective randomized clinical trials in pediatric surgery: state of the art?. J Pediatr Surg. 2001, 36: 1182-1186. 10.1053/jpsu.2001.25749.
The pre-publication history for this paper can be accessed here:http://www.biomedcentral.com/1471-2431/3/1/prepub
We thank Lynn McCleary for her helpful editorial advice.
Margaret Sampson participated in the design of study, development of search strategy, data extraction, statistical analysis, and writing of the manuscript. Kaitryn Campbell executed the search strategy, evaluated all trials for eligibility, performed data extraction, statistical analysis and writing of the manuscript. Isola Ajiferuke participated in the design of the study, supervised the statistical analysis and participated in the preparation of manuscript. David Moher conceived of the study, secured funding, participated in the design of the study, advised in the statistical analysis and participated in the preparation of the manuscript.