- Open Access
- Open Peer Review
Potential biases in the classification, analysis and interpretations in cross-sectional study: commentaries – surrounding the article "resting heart rate: its correlations and potential for screening metabolic dysfunctions in adolescents"
BMC Pediatricsvolume 14, Article number: 117 (2014)
Resting heart rate reflects sympathetic nerve activity. A significant association between resting heart rate (HR) and all causes of cardiovascular mortality has been reported by some epidemiologic studies. Despite suggestive evidence, resting heart rate (RHR) has not been formally explored as a prognostic factor and potential therapeutic outcome and, therefore, is not generally accepted in adolescents.
The core of the debate is the methodological aspects used in "Resting heart rate: its correlations and potential for screening metabolic dysfunctions in adolescents"; the points are: cutoff used for cluster RHR, two different statistical models used to analyze the same set of variables, one for continuous data, and another for categorical data; interpretation of p-value < 0.05, sampling process involving two random stages, analysis of design effect and the parameters of screening tests.
Aspects that must be taken into account for evaluation of a screening test to measure the potential for discrimination for a common variable (population with outcome vs. no outcome population), the main indicators are: sensitivity, specificity, accuracy, positive predictive value and negative predictive value. The measures of argumentation equality (CI) or difference (p-valor) are important to validate these indicators but do not indicate quality of screening.
Recently, Fernandes et al. published an article aimed at analyzing the potential effects of screening and resting heart rate (RHR) on cardiometabolic risk in adolescents  in this respected journal. We read the manuscript with great interest, since RHR reflects sympathetic nerve activity [2, 3], and it is an easily accessible clinical measurement. A significant association between resting HR and all-causes of cardiovascular mortality has been reported in some epidemiological studies [2, 4–6].
After studying the article, we decided to take the opportunity to propose a healthy debate on the methodological aspects used by Fernandes et al. . With this debate, we hope to contribute to the enrichment of the reader, especially with regard to statistical analysis and interpretation of results.
The aim of this article is to present a critical appraisal of methodological aspects of the article "Resting heart rate: its correlations and potential for screening metabolic dysfunctions in adolescents" presented by BMC Pediatrics.
First, with regard to the manuscript methodology, what drew our attention was the cutoff used for cluster RHR. We see that the authors used cutoffs developed by the group of the first author (Fernandes RA) . These cutoff points were developed by percentile distribution of a sample composed only of children and adolescent males and the study published in this journal is composed only of adolescents of both sexes. This decision introduced classification bias into the study, though it was not recognized as a study limitation: children are biologically different than adolescents because they have not gone through puberty, and there are important and significant differences between the sexes concerning the cardiovascular system .
Boys had higher pooled prevalence than girls [9, 10]. There are possible explanations for differences between the sexes: 1) the boys had a higher accumulation of visceral fat and intra-abdominal fat than girls , and visceral fat has been associated with higher sympathetic activity [12, 13] This activation is a key mechanism underlying the effect of intra-abdominal fat accumulation on the development of hypertension . For example, increased sympathetic flow may increase sodium re-absorption and subsequent increased peripheral vascular resistance resulting in increased blood pressure . Also, this increased sympathetic activation can be caused by increased testosterone concentrations in males. Testosterone, acting as a mediator of the androgen receptor gene function , has been associated not only with increased visceral fat but also with greater vasomotor sympathetic tone and blood pressure in adolescent boys, compared to girls . Therefore, we believe that the cutoffs used are not appropriate for the above and highlight the need for the scientific community to develop better diagnostic criteria and methodological quality appropriate for each sex and age of this important indicator of the cardiovascular system.
According to the title of the article, the authors’ objective was to analyze the impact of RHR for screening metabolic dysfunctions and also to identify its significance in adolescents.. For this, they used two different statistical models in order to analyze the same set of variables, one for continuous data, and another for categorical data. We found this odd, since assumptions for statistical models are quite distinct (binary logistic regression model vs. linear regression model). So we raise the following questions: "Were the linear models used because no association was found with categorical variables? Why were the two models used? Why analyze variables with continuous data and then analyze these variables with categorical data, sequentially?" We performed these questions, because according the objectives; the authors wanted determine the correlation between RHR and metabolic dysfunctions and also the potential power of screening the RHR. What is not clear is the use of logistic regression to meet those aims. In some instances we recommended that the authors state why they have used these tests and provide a reference for a definitive description for readers .
With regard to OR estimates using binary logistic regression, the literature shows that the use of OR (estimated with logistic regression) as a measure of effect in the cross-sectional studies has limitations: OR overestimates RP/RR according to increases of prevalence/incidence of outcome; between 5% and 10% OR has good approximation with RP/RR, after that the risk value is very distorted and it serves more to show the association direction (risk or protection) and not its magnitude; this topic was widely discussed in the nineties by experts [18–20], and confirms that OR overestimates the magnitude of the associations between exposures and outcomes, particularly in high prevalence [21, 22]. The mathematical model for logistic regression was developed in the 1970s and 1980s to analyze case–control studies and used as a proxy for relative risk [23, 24], where it is not possible to estimate prevalence, another important methodological factor neglected by the authors.
The authors say they used a sampling process involving two random stages (schools in the first stage and individual classes in the second stage), but give no further details of this process, for example, whether the complex sample has good accuracy. When using complex samples the design effect (deff) helps to estimate how accurate the sample was [25–27]. When the sampling process is not accurate the analyses need to be adjusted for the complexity of the sample, and the lack of this setting also impacts the associations . Therefore, the impact of risk factors estimated by the logistic models, even without statistical significance, may not be exactly the absence shown by adjusting the primary sampling unit.
We found the use of RHR to screen for alterations in glucose and triglycerides interesting but, according to the data presented, we believe that there is no evidence for this. Accuracy (AUC) for high glucose was 0.611 (95% CI 0.534–0.688) and high triglycerides, 0.618 (95% CI 0.531–0.705), both with p-values < 0.05, but with low discrimination power—note the lower confidence bound in some cases is very close to 0.50 (random event). In other words, if we consider random variations within the CI bounds of AUC, determining the presence or absence of high glucose and high triglycerides will be as precise as playing a game of heads or tails. With regard to the accuracy of results, Swets  suggested operational cut-off points: the test can be non-informative/test equal to chance (0.5AUC < 0.7); moderately accurate (0.7 > AUC ≤ 0.9); highly accurate (0.9 > AUC < 1.0); and perfect discriminatory tests (AUC = 1.0).
Nowadays a "p-value < 0.05" or significant association is commonly employed to illustrate the importance of latest scientific finding. We emphasize, however, that statistical significance is neither a necessary nor a sufficient condition for proving a scientific result . P-values are often used to emphasize the certainty of data, but they are only a passive read-out of a statistical test and do not take into account how well an experiment was designed, for example . Goodman , in his "The P Value Fallacy" explains about the apparent inconsistency in much medical research, where by studies are designed according to a Neyman-Pearson statistical approach (eg. based on formal decision making and long-run evaluation of the inferential procedures), fixing statistical parameters as significance level and power, but are then analyzed by using a Fisherian point of view (eg. computing p-values and making inference based on its value, in comparison to common thresholds).
We must remember that the screening is conceptually defined as tests performed on apparently healthy people to identify those at an increased risk of a disease or disorder . According to the literature, for screening to be accurate, a good screening test must have high sensitivity (few false-negative results) and a high specificity (few false-positive results)  and even very good tests have poor positive predictive value when applied to low-prevalence populations .
We would like to emphasize that Fernandes et al.  have provided an important scientific contribution with their study on RHR, and that criticism is an integral part of scientific progress. As the pediatrician John Locke said, "…every step the mind takes in its progress towards knowledge makes some discovery, which is not only new, but the best too, for the time at least".
The main indicators that must be taken into account for evaluation of a screening test to measure the potential for discrimination for a common variable (population with outcome vs. no outcome population) are: sensitivity, specificity, accuracy, positive predictive value and negative predictive value. The measures of argumentation equality (CI) or difference (p-valor) are important to validate these indicators but do not indicate quality of screening.
We believe the statistical methodologies employed in support of science should consider the objectives of the paper, type of data available (with the least possible transformations) and statistical assumptions in order to answer scientific hypotheses. The interpretation of statistical data has to be made very carefully, otherwise science loses its footing and becomes a relentless pursuit of the "p-value < 0.05".
Resting heart rate
Fernandes RA, Vaz Ronque ER, Venturini D, Barbosa DS, Silva DP, Cogo CT, Carnelossi MS, Batista MB, Coelho-E-Silva MJ, Sardinha LB, Cyrino ES: Resting heart rate: its correlations and potential for screening metabolic dysfunctions in adolescents. BMC Pediatr. 2013, 13 (1): 48-10.1186/1471-2431-13-48.
Bemelmans RH, van der Graaf Y, Nathoe HM, Wassink AM, Vernooij JW, Spiering W, Visseren FL, group obotSs: The risk of resting heart rate on vascular events and mortality in vascular patients. Int J Cardiol. 2013, 168 (2): 1410-1415. 10.1016/j.ijcard.2012.12.043.
Grassi G, Vailati S, Bertinieri G, Seravalle G, Stella ML, Dell’Oro R, Mancia G: Heart rate as marker of sympathetic activity. J Hypertens. 1998, 16 (11): 1635-1639. 10.1097/00004872-199816110-00010.
Jouven X, Empana JP, Schwartz PJ, Desnos M, Courbon D, Ducimetière P: Heart-rate profile during exercise as a predictor of sudden death. N Engl J Med. 2005, 352 (19): 1951-1958. 10.1056/NEJMoa043012.
Palatini P, Julius S: Elevated heart rate: a major risk factor for cardiovascular disease. Clin Exp Hypertens. 2004, 26 (7–8): 637-644.
Pocock SJ, Wang D, Pfeffer MA, Yusuf S, McMurray JJ, Swedberg KB, Ostergren J, Michelson EL, Pieper KS, Granger CB: Predictors of mortality and morbidity in patients with chronic heart failure. Eur Heart J. 2006, 27 (1): 65-75.
Fernandes RA, Freitas Júnior IF, Codogno JS, Christofaro DG, Monteiro HL, Roberto Lopes DM: Resting heart rate is associated with blood pressure in male children and adolescents. J Pediatr. 2011, 158 (4): 634-637. 10.1016/j.jpeds.2010.10.007.
Spear B: Children and Adolescent Grrowth and Development. Adolescent Nutrition: Assessment and Management. Edited by: Rickert V. 1995, New York: Chapman & Hall Publishing, Volume 1
Muntner P, He J, Cutler JA, Wildman RP, Whelton PK: Trends in blood pressure among children and adolescents. J Am Med Assoc. 2004, 291 (17): 2107-2113. 10.1001/jama.291.17.2107.
Ostchega Y, Carroll M, Prineas RJ, McDowell MA, Louis T, Tilert T: Trends of elevated blood pressure among children and adolescents: data from the national health and nutrition examination survey 1988–2006. Am J Hypertens. 2009, 22 (1): 59-67. 10.1038/ajh.2008.312.
Pausova Z, Mahboubi A, Abrahamowicz M, Leonard GT, Perron M, Richer L, Veillette S, Gaudet D, Paus T: Sex differences in the contributions of visceral and total body fat to blood pressure in adolescence. Hypertension. 2012, 59 (3): 572-579. 10.1161/HYPERTENSIONAHA.111.180372.
Esler M, Straznicky N, Eikelis N, Masuo K, Lambert G, Lambert E: Mechanisms of sympathetic activation in obesity-related hypertension. Hypertension. 2006, 48 (5): 787-796. 10.1161/01.HYP.0000242642.42177.49.
Alvarez GE, Beske SD, Ballard TP, Davy KP: Sympathetic neural activation in visceral obesity. Circulation. 2002, 106 (20): 2533-2536. 10.1161/01.CIR.0000041244.79165.25.
Huggett RJ, Burns J, Mackintosh AF, Mary DA: Sympathetic neural activation in nondiabetic metabolic syndrome and its further augmentation by hypertension. Hypertension. 2004, 44 (6): 847-852. 10.1161/01.HYP.0000147893.08533.d8.
Weise M, Eisenhofer G, Merke DP: Pubertal and gender-related changes in the sympathoadrenal system in healthy children. J Clin Endocrinol Metab. 2002, 87 (11): 5038-5043. 10.1210/jc.2002-020590.
Pausova Z, Abrahamowicz M, Mahboubi A, Syme C, Leonard GT, Perron M, Richer L, Veillette S, Gaudet D, Paus T: Functional variation in the androgen-receptor gene is associated with visceral adiposity and blood pressure in male adolescents. Hypertension. 2010, 55 (3): 706-714. 10.1161/HYPERTENSIONAHA.109.146720.
Greenhalgh T: How to read a paper. Statistics for the non-statistician. I: different types of data need different statistical tests. BMJ. 1997, 315 (7104): 364-366. 10.1136/bmj.315.7104.364.
Lee J, Chia KS: Estimation of prevalence rate ratios for cross sectional data: an example in occupational epidemiology. Br J Ind Med. 1993, 50 (9): 861-862.
Axelson O, Fredriksson M, Ekberg K: Use of the prevalence ratio v the prevalence odds ratio as a measure of risk in cross sectional studies. Occup Environ Med. 1994, 51 (8): 574-10.1136/oem.51.8.574.
Axelson O, Fredriksson M, Ekberg K: Use of the prevalence ratio v the prevalence odds ratio in view of confounding in cross sectional studies. Occup Environ Med. 1995, 52 (7): 494-10.1136/oem.52.7.494.
Barros AJ, Hirakata VN: Alternatives for logistic regression in cross-sectional studies: an empirical comparison of models that directly estimate the prevalence ratio. BMC Med Res Methodol. 2003, 3: 21-10.1186/1471-2288-3-21.
Polderman J, Gurgel RQ, Barreto-Filho JAS, Roelofs R, Ramos REDO, De Munter JS, Wendte JF, Agyemang C: Blood pressure and BMI in adolescents in Aracaju, Brazil. Public Health Nutri. 2011, 14 (6): 1064-1070. 10.1017/S1368980010003666.
Breslow N, Day N: Statistical Methods in Cancer Research. The Analysis of Case–Control Studies. 1980, Lyon: IARC Scientific Publications
Schlesselman J: Case–Control Studies: Design, Conduct, Analysis. 1982, New York: Oxford University Press
Kish L: Survey Sampling. 1965, New York: John Wiley & Sons
Korn EL, Graubard BI: Epidemiologic studies utilizing surveys: accounting for the sampling design. Am J Public Health. 1991, 81 (9): 1166-1173. 10.2105/AJPH.81.9.1166.
Heo M, Kim Y, Xue X, Kim MY: Sample size requirement to detect an intervention effect at the end of follow-up in a longitudinal cluster randomized trial. Stat Med. 2010, 29 (3): 382-390.
Barros A, Bertoldi A: Inequalities in utilization and access to dental services: a nationwide assessment. Cien Saude Colet. 2002, 7 (4): 709-717.
Swets JA: Measuring the accuracy of diagnostic systems. Science. 1988, 240: 1285-1293. 10.1126/science.3287615.
Brumfiel B: Significant adjective. Nature. 2008, 455 (7216): 1027-1028.
Ziliak S, Mccloskey D: The Cult of Statistical Significance: How the Standard Error Costs Us Jobs, Justice, and Lives: University of Michigan Press. 2008
Goodman SN: Toward evidence-based medical statistics. 1: the P value fallacy. Ann Intern Med. 1999, 130 (12): 995-1004. 10.7326/0003-4819-130-12-199906150-00008.
Cuckle H: Principles of screening. Obstetr Gynaecol. 2004, 6 (1): 21-25. 10.1576/toag.22.214.171.124976.
Nielsen C, Lang RS: Principles of screening. Med Clin North Am. 1999, 83 (6): 1323-1337. 10.1016/S0025-7125(05)70169-3. v
Grimes DA, Schulz KF: Uses and abuses of screening tests. Lancet. 2002, 359 (9309): 881-884. 10.1016/S0140-6736(02)07948-5.
The pre-publication history for this paper can be accessed here:http://www.biomedcentral.com/1471-2431/14/117/prepub
The remaining authors state no competing interest.
ACFM and AJFC made substantial contributions to the conception and interpretation of the material; ACFM, HBC, AJFC, LAM were involved in drafting the manuscript and revising it critically for important intellectual content and approval of the version to be published.
Augusto César Ferreira de Moraes, Alex Jones Flores Cassenote contributed equally to this work.