Stool frequency recording in severe acute malnutrition (‘StoolSAM’); an agreement study comparing maternal recall versus direct observation using diapers

Abstract

Background

Approximately 50% of the deaths of children under the age of 5 can be attributed to undernutrition, which also encompasses severe acute malnutrition (SAM). Diarrhoea is strongly associated with these deaths and is commonly diagnosed solely based on stool frequency and consistency obtained through maternal recall. This trial aims to determine whether this approach is equivalent to a ‘directly observed method’ in which a health care worker directly observed stool frequency using diapers in hospitalised children with complicated SAM.

Methods

This study was conducted at ‘Moyo’ Nutritional Rehabilitation Unit, Queen Elizabeth Central Hospital, Malawi. Participants were children aged 5–59 months admitted with SAM. We compared 2 days of stool frequency data obtained with next-day maternal-recall versus a ‘gold standard’ in which a health care worker observed stool frequency every 2 h using diapers. After study completion, guardians were asked their preferred method and their level of education.

Results

We found poor agreement between maternal recall and the ‘gold standard’ of directly observed diapers. The sensitivity to detect diarrhoea based on maternal recall was poor, with only 75 and 56% of diarrhoea cases identified on days 1 and 2, respectively. However, the specificity was higher with more than 80% of children correctly classified as not having diarrhoea. On day 1, the mean stool frequency difference between the two methods was −0.17 (SD; 1.68) with limits of agreement (of stool frequency) of −3.55 and 3.20 and, similarly on day 2, the mean difference was −0.2 (SD; 1.59) with limits of agreement of −3.38 and 2.98. These limits extend beyond the pre-specified ‘acceptable’ limits of agreement (±1.5 stool per day) and indicate that the 2 methods are non-equivalent. The higher the stool frequency, the more discrepant the two methods were. Most primary care givers strongly preferred using diapers.

Conclusions

This study shows lack of agreement between the assessment of stool frequency in SAM patients using maternal recall and direct observation of diapers. When designing studies, one should consider using diapers to determining diarrhoea incidence/prevalence in SAM patients especially when accuracy is essential.

Trial registration number

ISRCTN11571116 (registered 29/11/2013).

Background

Malnutrition in all its forms (fetal growth restriction, suboptimum breastfeeding, stunting/wasting and micronutrient deficiencies) is widespread and is directly or indirectly responsible for approximately 45% of deaths in children aged under 5 years that occur globally [1]. In low-income countries, especially those affected by HIV/AIDS and tuberculosis, mortality in children with severe acute malnutrition (SAM) remains high and tackling this is an international priority [2].

Apart from co-morbidities such as HIV or tuberculosis, diarrhoea is common among children with severe malnutrition [3,4,5,6] leading to prolonged admission and high mortality rates [6]. Recent reports from multiple countries noted a prevalence between 50 and 60% in children with SAM [3,4,5,6]. Our clinical impression is that the prevalence of diarrhoea is similar in our department, i.e., the Nutritional Rehabilitation and Research Unit (“MOYO”) at the Department of Pediatrics at Queen Elizabeth Central Hospital (QECH), Blantyre, Malawi. Diarrhoea is strongly associated with adverse clinical outcome [6] 1 and is frequently used as an outcome measure in research studies as well as in daily clinical practice. Therefore, a reliable and valid method of stool frequency assessment is of pivotal importance [7]. Currently most centres rely on maternal recall of stool frequency and consistency to assess the presence and severity of diarrhea [7] (defined as ≥3 loose or watery stools/day [8]. Based on our own observations and unpublished reports from other nutrition rehabilitation centres we postulated that the maternal recall for determining stool frequency is associated with high observer variability and significant reporting bias.

To ensure we and others in Nutritional Rehabilitation and Research Units (NRU) in low income countries are making clinical decisions on the basis of the best possible information, the aim of the current trial was to determine whether stool frequency as assessed by ‘maternal recall’ was equivalent [9] to a ‘directly observed method’ in which a health care worker directly observed stool frequency using diapers in children with SAM. Complementing this, we assessed maternal preference for stool frequency assessment method.

Methods

Characteristics of participants

This study was conducted at ‘Moyo’ NRU at Queen Elizabeth Central Hospital (QECH), the academic teaching hospital at the College of Medicine (COM), University of Malawi. As well as being a referral centre for the whole Southern Region of Malawi, ‘Moyo’ provides inpatient SAM treatment services for the whole of Blantyre urban and rural district, covering a population of approximately 1 million. Participants were children aged 5–59 months admitted with SAM defined as: weight-for-height ≤ −3 Z-scores (WHO growth standards) and/or a mid-upper-arm circumference (MUAC) of <115 mm (non-oedematous malnutrition, “marasmus”), and/or nutritionally induced bilateral pitting oedema (oedematous malnutrition, “kwashiorkor” and “marasmic kwashiorkor”). Oedema was defined as: Oedema level-I (+) is bilateral pitting oedema affecting the ankles/ft; level-II (++) affects both feet, hands, lower arms and lower legs; and level-III (+++) is generalized bilateral pitting oedema including both feet, legs, arms and face.

All patients had complicated SAM; with medical complications like systemic or respiratory infection, gastroenteritis or HIV disease. Those with uncomplicated SAM would have been treated as outpatients in community-based treatment programs in Blantyre district [10].

Design

After taking informed consent, we prospectively enrolled children with SAM into our study lasting the first 3 days of admission. For the StoolSAM study we assessed stool frequency and consistency prospectively in 120 SAM patients (see below). We compared the current MOYO practice of maternal-reported stool frequency (with a picture aid to help accurate recall during the previous night/day, see Fig. 1) versus a ‘gold standard’ in which a health care worker observed stool frequency using diapers.

Fig. 1

Simple picture aid used to help primary care giver recall stool frequency of child with SAM

As ‘gold standard’ (the direct observation method) diapers were assessed for presence and consistency of stools every 2 h during office hours (8 AM-6 PM), 7 days a week, by one of two members of the study team (AB, IP), thereafter we noted the number of diaper-changes made by the caregiver.

For the maternal recall, we used the standard departmental questionnaire administered during the clinicians’ morning ward round, roughly between 9 and 11 am each day. Supporting the questionnaire, primary care givers were shown a picture aid to help accurate recall of stool frequency during the previous 24 h night and day period (see Fig. 1). For simplicity, we here use the common term ‘maternal recall’ as primary care givers were by and large the actual mother of the child. Study patients were assessed both on week as well as weekend days. We did not have the resources to directly and regularly observe diaper-based output throughout the night (6 PM-8 AM). However, primary care givers were allowed to change diapers as needed, enabling the study team to count the diapers used the following morning and tally the total diapers used in each 24 h period. Diapers were observed by the study team to confirm watery stool rather than urine.

After completing the 3-day study, guardians were asked which of the two methods they preferred and what level of education they had obtained. To assess maternal preference, we used a 1–5 scale: 1: strongly prefers diapers, 2: prefers diapers a little, 3: doesn’t mind which method was used, 4: prefers recall method a little, 5: strongly prefers recall method. Guardians were then prompted to explain their choice (free text) on the above-mentioned 5-point scale.

We initially designed, registered and conducted the study as an RCT (see Additional file 1 for study flow chart). First, we aimed to establish whether stool frequency reported by care givers differed if diapers were also being directly assessed by staff health workers; insuring that “maternal reporting” is representative of normal practice even though direct observation is also being conducted. This also allowed us to check if diarrhoea prevalence estimated with maternal recall was the same in group-1 (“recall only” – unbiased by seeing diapers) and group-2 (“recall and diaper observation”). Recognising the limitations of this design (i.e. the assumption that true numbers of diarrhea stools are the same in the two groups) we also performed a post-hoc secondary analysis focusing on the discrepancies in stool frequency obtained by the two methods in each individual child of group-2.

The randomization sequence was computer generated. Allocation concealment was achieved by inserting group labels into sealed, sequentially numbered opaque envelopes. At enrolment, the guardian drew the next numbered envelope and opened it in presence of a study team member to show their assigned group.

Analyses and statistics

Group-1 (baseline – maternal recall only) compared to group-2 (maternal recall with introduced diaper observation)

Difference in stool frequency by maternal recall between group-1 and group-2 were assessed using generalized linear models with a Poisson distribution error for count data. Difference between the 2 groups in diarrhoea prevalence as obtained by maternal recall was assessed with Fisher Exact test.

Group-2 analysis – comparing maternal recall to direct diaper observation

We took our ‘gold standard’ measurement of diarrhoea as having three or more diapers with ‘loose’ or worse consistency during the 14-h daytime observation period. The sensitivity and specificity of using stool frequency assessed by maternal recall to classify children as having diarrhoea or not were estimated with 95% confidence intervals for both day 1 and day 2. We analyzed all patients with sufficient data to be classified. Since maternal recall is always retrospective in nature and diapers prospective, day 2 recall was compared with day 1 diaper data, and similarly, day 3 recall was compared with day 2 diaper data.

We recruited 58 children for group-2 (with both recall and diaper observation): this sample size assumed that a clinically relevant limit of agreement would be ±1.5 stool episodes per day (alpha = 0.025; beta, power = 80%; SD 2.5). These figures were based on a review of recent case notes of patients admitted to our ward. To establish agreement, mean differences and limits of agreement between methods were calculated with the R package MethComp [11] which is based on the Bland-Altman approach [12]. Generalized linear models with Poisson error distribution were used to relate stool frequency and primary care giver education with absolute discrepancy between methods. For analysis, WHZ, WAZ, HAZ and MUAC Z-scores were calculated using the WHO Child Growth Standards R package: igrowup [13]. Data entry was done with Microsoft access. Stata version 12.0 (StataCorp USA), SPSS and R (Version 3.2.3) were used for further analyses. Significance threshold was set at 5% for all statistical tests.

Ethics

The Malawi College of Medicine Research and Ethics Committee approved this study (P.07/13/1429) and all research associated activities were carried out according to Good Clinical Practice guidelines which are based on the Declaration of Helsinki [14].

Results

Between November 2013 and February 2014, 307 children were admitted to ‘Moyo’ Nutritional Rehabilitation and Research Unit (NRU) and 120 children with SAM were enrolled in the “StoolSAM study”, 113 had data for recall analysis; 55 in group-1 (baseline – maternal recall only) and 58 in group-2 (maternal recall with direct diaper observation). Of these patients in group 2, 52 had both recall and diaper observation data for day-1 and 50 for day-2. The children had a median age of 20.5 months and had almost 2 weeks of co-morbidity symptoms (13.5 days) before admission to Moyo; also, all study children were reported to have had a median of 3 days of consistent diarrhoea prior to admission (Table 1). Most primary care givers had only attended primary school (70.2%) or had not received any formal education (3.5%). Kwashiorkor prevalence was high in this cohort (56.9%). The majority, 64% (37/58) of patients, presented with diarrhoea and 33% (19/58) were HIV sero-positive. Most children, 81% (47/58) completed 3 days of stool frequency recording, 3 patients died during recall assessment and 1 patient absconded. The overall mortality rate during the ‘StoolSAM’ trial was 16% (9/58).

Table 1 Patient characteristics of Group-2 at baseline

No differences were found in stool frequency reported by maternal recall between group-1 (baseline – maternal recall only) and group-2 (maternal recall with introduced diaper observation) (On all 3 days, the differences between the 2 groups in reported stool-frequencies were between 0.9 and 1.07 stools/day, 95% Confidence Intervals between: 0.65–1.38, and P values between: 0.25–0.62, > 0.05 showing no difference, see Additional file 1). Furthermore, no difference in diarrhoea prevalence as calculated by maternal recall on day 1 or day 2 were found between the two groups (both day-1 and day-2; p-value =1).

The analysis of group-2 showed poor agreement between maternal recall and the ‘gold standard’ of directly observed diapers by health care workers. The prevalence of diarrhea and diagnostic accuracy of next day maternal recall compared with the diaper-observed method as ‘gold standard’ are summarized Table 2. Overall sensitivity was poor, with only 75 and 56% of cases of diarrhoea detected using the maternal recall data on days 1 and 2, respectively. The specificity was higher with 80% or more of children with no diarrhoea correctly identified each day. Positive predictive values (PPV, (95% CI) ranged from 88% (65–97%) to 74% (54–87%) on days 1 and 2, respectively. Negative Predictive Values (NPV, (95% CI) ranged from 65% (53–75%) to 86% (74–93%) on days 1 and 2, respectively (see Table 2 for Prevalence of diarrhoea, PPV and NPV). On day 1, the mean stool frequency difference between the two methods was −0.17 (SD; 1.68) with calculated 2.5 and 97.5% limits of agreement of −3.55 and 3.20 stool per day and, similarly on day 2, the mean difference was −0.2 (SD; 1.59) with limits of agreement of −3.38 and 2.98 stools per day. These calculated limits are well beyond the pre-specified ‘acceptable’ limits of agreement established at ±1.5 stool per day; and this confirms non-agreement of the 2 methods. The average mean difference in stool frequency between the methods (−0.19) does not support a significant systematic bias towards under- reporting stool frequency by primary care givers.

Table 2 Prevalence of diarrhoea and diagnostic accuracy of next-day maternal recall compared with diaper observation of stool as the ‘gold standard’ method

However, it was clear that the higher the stool frequency, the more discrepant the two methods were (see Figs. 2 and 3); where a strong association was found between increased stool frequency and absolute difference between methods (day 1: 1.33 (95%CI: 1.20 to 1.48; p-value <0.001); day 2: 1.20 (95%CI; 1.03 to 1.41; p-value = 0.017).

Fig. 2

Cross-tabulation of actual diaper observed and recalled stool frequencies for day 1. Dark grey cells indicate concordance between stool frequency by recall and diaper observation; light grey cells indicate acceptable levels of difference (i.e. not greater than limit of agreement of ±1.5 stool counts per day)

Fig. 3

Cross-tabulation of actual diaper observed and recalled stool frequencies for day 2. Dark grey cells indicate concordance between stool frequency by recall and diaper observation; light grey cells indicate acceptable levels of difference (i.e. not greater than limit of agreement of ±1.5 stool counts per day)

The education level of the primary care giver may also influence the deviation between maternal recall and direct diaper observation. The absolute difference between maternal recall and direct diaper observation was significantly associated with years of education on day 1 (0.93; 95% CI 0.87 to 0.99; p-value = 0.03); with higher educated mothers reporting stool frequency counts that showed less discrepancy with direct diaper observation. However, this relationship is unclear as it was not found to be significant on day 2 (0.97; 95% CI 0.90 to 1.03; p-value = 0.3).

Preference between the two methods was obtained for 47 primary care giver/child pairs. Most primary care givers strongly preferred diapers: 95.7% (45/47). The main reasons given were: the accuracy of evaluating the child’s illness (21/47); better hygienic conditions for the child, i.e. bedding and clothes are not soiled, (15/47); less time on laundry and more with the child (8/45).

Discussion

To identify those children with increased risk for mortality it is important to have accurate indicators of diarrhoea regarding both frequency and consistency. Our results clearly show there is non-agreement between maternal recall and a ‘gold standard’ of directly observing stool frequency using diapers (see Methods) in children with complicated SAM. The higher the stool frequency was, the more the two methods differed suggesting that maternal recall becomes increasingly inaccurate when stool frequency becomes most critical clinically. Although practitioners anecdotally report that primary care givers tend to under-report the stool frequency of their child, there was no evidence to support a systematic negative bias in our study. The educational level of the primary care giver may improve accuracy of stool frequency recall; but this tendency was inconsistent and should be re-investigated.

Diarrhoea is notoriously difficult to assess. It is, however, a highly important clinical measurement as it relates strongly to clinical outcome. Being confident that your measurement of stool frequency and consistency is accurate and precise is therefore essential. In the latest WHO update on the management of SAM [15] several research priorities were detailed including assessment and management of diarrhoea. Although not designed to detect improved care, the results from this small trial suggest that implementing this new method could be both useful for improving patient care and for enhancing the informative quality of diarrhoea outcome in future clinical studies.

There are several limitations in this study. First of all, with 113 enrolled patients and 58 available for method comparison, our sample size is small. Second, we were unable to directly observe diapers 24 h per day; and doing so for 10 h per day was the nearest we could get to the ‘gold standard’. The design of the trial might have biased the outcomes; having nappies could have influenced recall and running a study on stool frequency could have heightened the general awareness to diarrhoea; the ‘Hawthorne effect’ [16]. This could have influenced the recall performance of our baseline group-1. In addition, using our picture aid (Fig. 1) could also have favourably influenced maternal recall. This entails that in ‘normal’ practice, the anecdotal reports by physicians of under-reporting a sick child’s stool frequency may still be a real cause for concern. Conversely, there is no one definite “gold standard” method (other than continuous observation by a health professional). The diaper method may also underestimate the true number of abnormally loose/watery stools. Considering that the established clinical cut off to categorize a child as having diarrhoea or not is 3 or more loose/watery stools per day, the margin for error is small.

We did not look whether the choice of method would influence clinical practice or if it would change the number of days of admission to the ward. Although collecting diapers 2-hourly is a huge demand on personnel, when stool frequency is the primary outcome in a SAM research project, we argue that the direct observation of diapers should be used. Therefore, the method described in this paper might contribute to better outcomes by accurately estimating frequency of diarrhoeal stools (in an in-patient setting with low resources).

In many ways, Moyo NRU is a typical ward in a poor resource setting, but in light of the research conducted in this NRU, its resources are superior to many other NRU’s across the country and elsewhere in Sub-Saharan Africa. Because Moyo NRU is a relatively well-equipped NRU, questions could arise over generalizability of the results. Nevertheless, it is likely that recording of defecation frequency in any NRU, in less well-supported settings, is highly unreliable. We call for other nutritional rehabilitation units in middle- and low-income countries to also investigate this issue as the benefits in doing so could be significant.

High-income countries do rely on stool frequency recalled by parents (19) but the usage of diapers is ubiquitous. In the high-come setting, the cost of diapers is not a concern when designing trials. In Malawi, the cheapest diapers cost 200 MWK (0,45 USD). The mean number of diapers used in this study was 15 so per child this adds to less than 2 USD per day of study. This is above the daily budget of many families of children admitted with complicated SAM. Another issue with regards to using disposable diapers is the waste associated with it and the energy costs to make diapers.

Conclusions

In conclusion this study shows non-agreement between the assessment of stool frequency in admitted SAM patients using maternal recall and direct observation of diapers by health care staff. It shows that relying only on maternal recall is inaccurate in measuring the incidence/prevalence of diarrhoea, as the reported number of loose/watery stools may not reflect the actual number. The use of diapers to record stool frequency in SAM patients might be a promising method but more studies in other low-income settings are needed before recommending more widespread use. Clinically, we recognize a different cost/benefit balance than in resource rich settings especially given that SAM is only really common in resource poor settings. We suggest to selectively use diapers for high-risk children (e.g. for those with complicated SAM, admitted to a NRU), where stool frequency monitoring is crucial. The results of this trial should influence the design of future trials in malnutrition research; especially where stool frequency is the primary outcome.