Randomized controlled trials in pediatric complementary and alternative medicine: Where can they be found?

Background The safety and effectiveness of CAM interventions are of great relevance to pediatric health care providers. The objective of this study is to identify sources of reported randomized controlled trials (RCTs) in the field of pediatric complementary and alternative medicine (CAM). Methods Reports of RCTs were identified by searching Medline and 12 additional bibliographic databases and by reviewing the reference lists of previously identified pediatric CAM systematic reviews. Results We identified 908 reports of RCTs that included children under 18 and investigated a CAM therapy. Since 1965, there has been a steady growth in the number of these trials that are being published. The four journals that published the most reported RCTs are The American Journal of Clinical Nutrition, Pediatrics, Journal of Pediatrics, and Lancet. Medline, CAB Health, and Embase were the best database sources for identifying these studies; they indexed 93.2%, 58.4% and 42.2 % respectively of the journals publishing reports of pediatric CAM RCTs. Conclusions Those working or interested in the field of pediatric CAM should routinely search Medline, CAB Health and Embase for literature in the field. The four core journals identified above should be included in their collection.


Background
Complementary and alternative medicine (CAM) is defined as a broad domain of healing resources that encompasses all health systems, modalities and practices and their accompanying theories and beliefs, other than those intrinsic to the politically dominant health system of a particular society or culture in a given historical period [1].
The safety and effectiveness of CAM interventions are of great relevance to pediatric health care providers. Two recent surveys indicate a use of pediatrics CAM in the United States, finding a 12% and 21% overall use, respectively [2,3]. Factors such as ongoing medical problems and parents' use of CAM are associated with pediatric use [3,4]. Ideally, evidence-based practitioners can approach children and their guardians regarding their use of CAM and on a case-by-case basis to help identify relevant evidence regarding therapies that patients use or wish to consider.
Such discussions are likely to be more rewarding when practitioners and families can advocate for the use of CAM therapies for which there is stronger evidence, such as from the results of randomized trials of effective herbal therapies [5]. Similarly, interventions for which there is little of no evidence might be best avoided [6].
We undertook a program of research to discover whether such evidence, in the form of published systematic reviews and randomized controlled trials involving children, existed [7,8]. Based on the large number of randomized controlled trials we did find, we identified a need for a guide for evidence-based practitioners who wish to determine whether sound evidence exists.
The objective of the research presented here was to identify where health care practitioners can locate reported RCTs of pediatric CAM; specifically, in which journals these reports are published and through which databases those reports may be identified.

Identification of Candidate Studies
We searched for reports of RCTs in 13 bibliographic databases, using either a detailed search strategy based on the main subject search of the Cochrane Collaboration's Complementary Medicine Field subject search, [9] or a modification of that search tailored to the capabilities of particular databases [10]. The databases searched are listed in Table 1, and the search strategies are available from the authors upon request. We also identified reports of RCTs from the cited references of 49 pediatric CAM systematic reviews. The reviews were previously identified by the Chalmers Research Group in a study that complements this project [7].
Bibliographic records were imported into a Reference Manager 9.0™ database and duplicate items were consolidated. Bibliographic records were screened by KC to determine if: the study included children under 18; a CAM therapy was investigated (following the definition of CAM adopted by the Cochrane Complementary Medicine Field Registry of Randomized Controlled Trials and presented in their Registry Guidelines) [11]; and the article was a report of an RCT. Where a final determination of eligibility could not be made based on information contained in the bibliographic record, the complete article was retrieved and reviewed. After about the first 100 records had been assessed, every tenth record was reviewed by MS as a quality measure. Inter-observer agreement was almost perfect, with Kappa = 0.97 [12] without the reviewers undertaking any discussion designed to achieve consensus. A total of 908 reports of RCTs met the criteria for inclusion in the study (see Figure 1). Studies published in languages other than English were included if: (1) the article was written in a language we were able to read (German, French, and Spanish); or (2) we were able to ascertain, from the English abstract,  [9]. **Strategy 2: ((SR-COMPMED and CHILD*:ME) not (ADULT*:ME and ADOLESCENCE:ME)) ***Strategy 3: A combination of the following truncated or non-truncated terms with the appropriate syntax were used: randomized, controlled, child, pediatric, complementary medicine, and alternative medicine.

Figure 1
Study flowchart. Note: For some reports, more than one exclusion criteria was noted, therefore numbers do not add up to 2061. A post hoc analysis was undertaken to assess whether the apparent decline in the number of reports of trials in complementary and alternative medicine in the final years of this study was due to a delay in the indexing of these articles. For this analysis, the number of Medline records retrieved by search strategy 1 was broken down by year, ending with September Week 3 2002. Simple counts were taken -the individual items were not evaluated against the study inclusion criteria.

Data Extraction
For each eligible publication, the date of publication, authors, language of publication, and journal name were extracted from the Reference Manager database or from inspection of the article. We consulted the reference material for each of the 13 databases searched to determine whether or not the database indexed each of the journals that had published one or more eligible reports of RCTs.

Results
A total of 908 reports of RCTs met the criteria for inclusion in the study.

Year of publication
The earliest reports of RCTs we found were published in 1965, and there was little growth in this literature until 1975. From 1975, there was significant steady growth for over twenty years, with a peak of 89 articles published in 1997 (see Figure 2). There is the beginning of a decline in the next two years, followed by a sudden drop to 45 reports of RCTs in 2000 and 2 in 2001. This decline is probably an artifact of indexing delay, as this research was conducted in the winter of 2001. Post hoc analysis conducted in September 2002 suggests that growth in Medline of records meeting the criteria of search strategy 1 (but not formally screened according to study inclusion criteria) continued to grow through 2000, and again shows a dip for the previous and current year (2001 and 2002).
Authorship 92.2% of reports of RCTs published in this field are written by more than one author. The largest number of authors for a single paper was 25 [13], while almost half of all papers (49.3%) have four or fewer authors. The five most prolific authors have published at least 15 reports of RCTs each. The vast majority (57.8%) of all authors cited in the 908 studies authored a single paper and 70% of authors published four or fewer CAM trials.

Language of Publication
Most reports of RCTs were published in English (93.2%), although 12 other languages were represented. The second most common language of publication was Chinese, with 18 reports of RCTs (2%) and a concentration of studies in herbal medicine. There were more than five reports of RCTs in each of German, French, Spanish and Italian, each with trials in a variety of topics. Seven languages were represented by one or two reports of RCTs ( Table 2). The language restrictions resulted in the exclusion of 41 publications as we were unable to confirm that they were reports of RCTs in pediatric CAM; 23 of these were published in Russian, 6 written in Chinese, 4 in Ukrainian, 3 in Japanese, 2 in Italian, and one each in Hungarian, Czech and Swedish.

Journal
The reports of RCTs originated from 308 different journals. These journals were ranked by the number of reports of RCTs published. As a method to examine the dispersion of the literature, we divided the journals into four zones, each representing approximately one quarter of the articles published.
Of the total number of reports of RCTs, 217 (23.9%) were identified as having been published in the top four journals (1.3% of the total of 308 journals that published at least one relevant reported RCT). In Zone 2, 24 (7.8%) journals published 27.1% of the total number of reports of RCTs identified. Zone 3 encompasses 96 (31.2%) journals and contains 28.7% of the total reports of RCTs identified. Zone 4 contains 184 journals (59.7%), and 20.3% of the total reports of RCTs identified. The top journals, located in Zones 1 and 2, are displayed in Table 3. The names and rankings of journals in Zones 3 and 4 are available from the authors on request.
For the 308 journals that published reports of pediatric CAM RCTs, the number of journals indexed by each of the databases is shown in Table 4, along with the unique contribution of coverage provided by each database.

Discussion
We identified over 900 reports of RCTs using a CAM intervention in children. We found a steady growth in the literature. We found fewer articles in most recent years, likely due to a lag in indexing. There was an apparent decline in the frequency of reporting from 1998 to 2001. We expected that the indexing of reports of RCTs published during this period would not have been completed by the time we conducted our research in 2001, but post hoc examination of the year by year number of items retrieved from Medline by search strategy 1 shows that some articles in this area can take more than one year to be indexed in bibliographic databases. Those wishing to obtain all The rate of publication seems to be dropping even in the final years shown here. We believe this to represent a lag between the time of publication and the time the journal articles are indexed in the bibliographic databases, rather than a true decline in the rate of publication of reports of randomized controlled trials in pediatric complementary and alternative medicine. The 908 reports were identified in more than 300 journals, yet almost a quarter (24%) of them were found in just four journals (American Journal of Clinical Nutrition, Pediatrics, Journal of Pediatrics, and The Lancet). By expanding the core journal list to 28 journals, it is possible to account for slightly more than half of all reports of pediatric CAM RCTs we found. These results suggest that most reports of randomized controlled trials of CAM are 'mainstream' medical journals. This finding is similar to that reported by Pittler and colleagues [16] who investigated the relationship between the statistical directions of CAM RCT results and where they were likely to be published. They found that mainstream medical journals with an impact factor greater or equal to 1 published an equal number of CAM trials with positive and negative results, while lower impact factor journals published more CAM trials with positive findings.
In terms of database coverage, Medline provided the best coverage of the Zone 1 and 2 journals, indexing all 28. Embase and CAB Health indexed 85.7% and 92.3% respectively. CINAHL and AMED indexed relatively few of the journals in Zones 1 and 2, (25% and 14% respectively) but they did provide unique material not available from the other databases. Together, these five databases gave complete coverage of all journals and reports of RCTs identified in this study. Two other databases, CCTR and IBIDS, provided relatively good coverage of these first two zones (65.3% and 57.1%), but because of overlapping coverage with other databases, provided no unique coverage.
In terms of database selection, a search of Medline alone could potentially identify 97.7% of these reports and a search combining Medline, Embase and CAB Health could achieve 99.4% retrieval, assuming a comprehensive search strategy is used. Given the recent development of the CCTR we were impressed with its coverage although its unique contribution is only marginal in terms of the cumulative increase of identified reports of RCTs resulting from searching this source. In an effort to further enhance the CCTR's coverage we have forwarded the database from this study to the Cochrane Collaboration's CAM field so that the information can be added to the CCTR.
Searching Medline, along with any two of Embase, CCTR, or CAB Health, would appear to constitute a reasonable approach for those wishing to search this topic comprehensively. More exhaustive searching, of the type undertaken for a systematic review, would ideally be supplemented by CINAHL and AMED, as well as selective hand searching and other traditional bibliographic techniques [17].
Those wanting to keep abreast of the latest pediatric CAM reports of RCTs by scanning journals as they are published, may be able to maximize their time by focusing their attention on four journals. Information specialists are likely to be able to track most new evidence by searching three databases.
Shortly after we completed the retrieval work for this project, the National Library of Medicine introduced a complementary medicine subset to its Medline database [18]. Searches can be limited to this subset through 1 The incremental percentage column shows the increase in number of additional journals identified, with each additional database search. For example, searching the CAB Health database, given that the Medline and Embase databases had already been searched, would yield relevant reports of RCTs from six additional journals. Databases are listed in order of the number of journals indexed that were not indexed by databases already considered, thus providing a clearer picture of the cumulative coverage if databases were searched in the order listed, which is the most efficient order for this collection. The cumulative % column shows the percent of journals that published reports of pediatric CAM RCTs that would be covered by searching the databases in the order listed. 2 This figure assumes that all trials published in that journal were indexed in the database. The actual indexing rate may be less than 100%, and the indexing status of each report of an RCT in each database was not verified.
PubMed (available at http://www.pubmed.org, and this feature is likely to become available through database vendors. This may prove to be a useful tool for clinicians wishing quick access to some of the best literature in this area. PubMed includes Medline, is available at no cost to those with Internet access and can be searched without complex syntax. As a case study, we entered the word random* in the search box, applied the limits of age "All Child: 0-18 years" and subset "Complementary Medicine" and publication type of "Randomized Controlled Trial" retrieved 1858 citations. A clinician replacing the search string random* with a subject term massage would find 56 citations as a starting point, the term chiropract* with the same limits would yield 29, st johns wort or st john's wort would yield 8 reports and herbal would yield 140 citations. A limitation of this study is that our figures on the cumulative percent of trials accounted for (Table 4) assume that all trials published in that journal were indexed in the database. The actual indexing rate may be less than 100%, and the indexing status of each report of an RCT in each database was not verified in this study. In addition, researchers should be aware that indexing practices and search capabilities available with particular databases vary, thus there is some advantage to building redundancy in coverage into a search strategy that has high recall as an objective.
It should be noted that the Cochrane Controlled Trials Registry is a special case among the databases we searched. As its name suggests, it is a database of controlled trials, including randomized controlled trials, and is constructed from trials found in other databases and identified through hand searching of journals [19,20]. At the time our searches were conducted, it was searchable only through its own proprietary interfaces. It has subsequently become available through at least one database aggregator, Ovid. Had such an interface been available at the time of our searching, we would have searched it using search strategy 1. This could potentially have increased its unique contribution beyond what is seen in Table 4. In a recent study in which known item searching techniques were used to determine if reports of randomized controlled were indexed in various databases, the CCTR performed extremely well, and was identified by the authors as a valuable but underused source for meta-analysts [21].
We used a strict definition of CAM, following the Registry Guidelines adopted by the Cochrane Complementary Medicine Field Registry of Randomized Controlled Trials, thereby excluding some psychological interventions that might be considered CAM by some (such as cognitive therapy) while including others (such as biofeedback). Our view is that some psychological interventions, such as cognitive behavior therapy, might well be considered a 'standard of care' in many Canadian and American settings.
We did not specifically follow a mega-dose definition for classifying nutritional therapies as CAM or conventional medicine. This may have overstated the contribution of the American Journal of Clinical Nutrition. We attempted to follow the Registry Guidelines for the Cochrane Complementary Medicine Field Registry of Randomized Controlled Trials but erred on the side of over-inclusion of therapies as CAM if the author's self-classification was not clear from the title, abstract and indexing information.
It is clear from this study that there is a large body of evidence from randomized controlled trials in pediatric complementary and alternative medicine. The quality of that literature is also relevant [22-24], although a recent paper suggests that quality measures may be less important than previously reported in obtaining unbiased estimates of intervention effectiveness in meta-analysis [25]. Schulz and colleagues reviewed 250 reports of RCTs and found that those with inadequate allocation concealment, compared to reports in which this information was adequately reported, exaggerated the estimates of an intervention's effectiveness by 30%, on average. Recently, Moss and colleagues [26] reported that in a review of 134 pediatric surgical RCTs, less than half of the trials (46%) reported any details of the method of randomization used. Tang et al. notes that although improving, methodological problems remain in the large body of trials of traditional Chinese medicine [15]. We previously reported on the quality characteristics of a subset of the trials found in this project [8].

Conclusions
RCT evidence in pediatric CAM is being published in a substantial number of journals. This study identifies the primary sources of reports of RCTs in the field of pediatric CAM-those journals found in Zone 1. These journals are a necessary core for any collection in this field. The journals in Zone 2 are also important sources of evidence, and are also recommended resources to be included in a pediatric CAM collection. Journals in Zone 3 would be a useful addition for those with larger collection budgets, and are good resources for those needing a comprehensive collection. The journals within Zone 4 contain only one report of an RCT each identified during the time period studied. While not prime candidates for a collection focusing on complementary and alternative medicine, they should be considered by systematic reviewers, who may identify potentially relevant studies from them through the relevant bibliographic databases. This study also indicates that those seeking pediatric CAM evidence require access to at least three indexing databases. These are Medline, Embase, and CAB Health.

Competing interests
None declared.

Authors' contributions
Margaret Sampson participated in the design of study, development of search strategy, data extraction, statistical analysis, and writing of the manuscript. Kaitryn Campbell executed the search strategy, evaluated all trials for eligibility, performed data extraction, statistical analysis and writing of the manuscript. Isola Ajiferuke participated in the design of the study, supervised the statistical analysis and participated in the preparation of manuscript. David Moher conceived of the study, secured funding, participated in the design of the study, advised in the statistical analysis and participated in the preparation of the manuscript.