This article has Open Peer Review reports available.
Lessons from SARS: A retrospective study of outpatient care during an infectious disease outbreak
© Nasef et al; licensee BioMed Central Ltd. 2010
Received: 23 November 2009
Accepted: 20 July 2010
Published: 20 July 2010
During severe acute respiratory syndrome (SARS) outbreak in Toronto, outpatient clinics at SickKids Hospital were closed to prevent further disease transmission. In response, a decision was made by the neonatal neuro-developmental follow up (NNFU) clinic staff to select patients with scheduled appointments to have a mail/telephone assessment using Ages and Stages Questionnaire (ASQ) or to postpone/skip their visit. The objective of this study was to compare the developmental assessment and its outcome in two groups of NNFU clinic patients, SARS versus non-SARS, over three standard clinic appointments.
We compared the diagnostic accuracy (identification of developmental delay), and patient management (referral for therapy or communication of a new diagnosis) of the strategies used during SARS, April/May 2003, to the standard assessment methods used for patients seen in April/May 2005 (non-SARS). In all cases data were obtained for 3 patient visits: before, during and after these 2 months and were compared using descriptive statistics.
There were 95 patients in the SARS group and 99 non-SARS patients. The gestational age, sex, entry diagnosis and age at the clinic visit was not different between the groups. The NNFU clinic staff mailed ASQ to 27 families during SARS, 17 (63%) were returned, and 8 of the 17 were then contacted by telephone. Criteria used to identify infants at risk selected for either mailed ASQ or phone interviews were not clearly defined in the patients' charts. There was a significant under identification of developmental delay during SARS (18% versus 45%). Of those who responded to the mailed questionnaire, referrals for therapy rates were similar to non-SARS group. The lost to follow up rate was 24% for the SARS group compared with 7% for non-SARS. There was no difference in the overall rate of developmental delay in the two groups as identified at the 'after' visit.
Poor advanced planning led to a haphazard assessment of patients during this infectious disease outbreak. Future pandemic plans should consider planning for outpatient care as well as in hospital management of patients.
Care in neonatal intensive care units has progressed over the past years resulting in higher number of surviving preterm extreme low birth weight infants as well as high risk full term infants [1–4]. These infants are at an increased risk for developmental delay that can arise at any time along their developmental trajectory . Neonatal neuro-developmental follow up (NNFU) clinics were designed for effective monitoring and assessment of these infants at regular time intervals. Regular attendance at scheduled appointments by children with their parents or care givers and sensitive feedback of the assessment to the family are important factors in early prediction of neuro-developmental delay in high risk infants .
In spite of efforts made to accommodate care givers there is still a significant number of missed appointments and loss to follow up at most neonatal follow up clinics due to factors related to the parents or care givers . Thus, alternative approaches for obtaining outcome data, including home visits by trained lay interviewers, telephone interviews, in-person interviews in a clinic, and mailed questionnaires have been evaluated .
Infectious diseases such as SARS (Severe Acute Respiratory Syndrome) are an important cause of changing routine health practice due to closure of non emergency health services to control disease spread [9–11]. SARS spread to the Greater Toronto Area on February 23rd, 2003 following which there were two phases of the outbreak. The first outbreak was believed to be over after passage of two incubation periods in early May. However, on May 20, 2003 new cases of respiratory illness occurred in a rehabilitation facility and a second outbreak was identified [12, 13]. A provincial emergency was declared throughout the two outbreaks from the end of March to the middle of June. As part of government imposed containment directives, all elective outpatient hospital clinics were closed between March 2003 and June 2003 [14, 15]. Hospital staff was expected to continue to come to work for regularly scheduled shifts.
High risk preterm and full term infants are regularly scheduled for neuro-developmental assessment in our NNFU clinic to provide diagnosis and referral for therapy. As a consequence of SARS all outpatient clinics were closed. Concerns about missed appointments in the NNFU clinics and the impact this may have on patients and their families led the NNFU team at The Hospital for Sick Children and Mount Sinai Hospital in Toronto to use an alternative method of assessment. A decision was made to use a mailed questionnaire and a telephone interview rather than the standard face to face patient assessment. The Ages and Stages questionnaire (ASQ) was mailed to parents for them to complete.
The objective of this retrospective study was to compare the developmental assessment and its' outcome in two groups of NNFU clinic patients, SARS versus non-SARS over an assessment trajectory of 3 booked clinic appointments (labeled before, during and after according to the time of clinic closure during SARS). The outcome measures explored included diagnostic accuracy, appropriate referral and return to follow up rate.
This was a retrospective study comparing the management of 2 groups of patients in the NNFU program at The Hospital for Sick Children and Mount Sinai Hospital in Toronto, Canada. Charts for all infants enrolled in the program during the SARS outbreak (SARS group) were reviewed for the accuracy of developmental assessment for the period before, during and after the SARS containment period. The comparator was patients enrolled in the program two years later who were identified as the non-SARS group. The study was approved by the Research Ethics Boards at Mount Sinai Hospital and The Hospital for Sick Children.
Infants were identified as eligible for neuro-developmental follow up in our program if either preterm (≤27 weeks gestation) or having other pre or post natal risk factors for adverse neuro-developmental outcome. SARS group infants were identified as those whose NNFU appointments were planned but subsequently cancelled during the period of April and May 2003 due to SARS. Non-SARS group infants had planned appointments in the corresponding time period of April and May 2005.
Data were abstracted from the NNFU chart including demographics, reason for referral and results of developmental assessment. The SARS period was defined as April to May 2003 and was labeled as 'during' visit. The visit immediately prior to this was labeled 'before' and visit 3 as 'after'. The non-SARS group assessments were matched for the corresponding time periods in 2005. Data collection included the type of assessment used and identification of a developmental delay in any of: gross motor, fine motor, language and socio-adaptive skills. Other data regarding referral for further therapy or further testing and the delivery of a new diagnosis were also reviewed.
The developmental assessment visits at our NNFU are scheduled at 4, 8, 12 and 20 months corrected age. Neuro-developmental assessments are performed by an inter-disciplinary team of physicians, nurses and therapists (occupational, physical and speech). Hearing, vision, neuromuscular and cognitive development are assessed using standardized tools. During the SARS period (during visit), as a replacement for a usual visit, the ASQ , a standardized parent completed survey assessing five domains of development (gross motor, fine motor, problem solving, socio-adaptive and language) was mailed to parents. Upon its' return there was an attempt to make a follow up telephone call to confirm the report. A developmental delay was defined as testing below the cutoff in any domain.
Data analysis and statistical tests
Data were analyzed to compare the rate of reported developmental delay as well as the rate of referral for further therapy or further testing between SARS group and non-SARS group during each of the three studied visits. The rate of new diagnoses communicated with a family was also compared.
Statistical analysis was performed using Microsoft Access and Excel (Microsoft Corp., Chicago, IL, USA). Student t test was used to compare demographic characteristics, while Chi Square independent test or Fischer's exact test was used to compare categorical developmental data (developmental delay, yes or no). A p < 0.05 was considered statistically significant.
n = 95
n = 99
Gestational Age (weeks)
32.1 ± 5.1
32.7 ± 5.6
Birth Weight (grams)
1885 ± 1145
1980 ± 1138
Neonatal drug withdrawal
Identification of Developmental Delay
Rate of Identification of Developmental Delay and Referral for Therapy
Referral for therapy
Rate of referral for therapy/further testing
The rate of referral for therapy or testing was not significantly different between the SARS and non-SARS group over the span of the three visits. The apparent difference in the during visit (12% versus 26%) did not reach statistical significance.
Rate of new diagnosis
Of the SARS group, the follow up team identified 4 infants with cerebral palsy as well as 2 infants with autism in the visit after SARS. There were no new neurological diagnoses made remotely during the SARS period. None of the newly diagnosed children with cerebral palsy or autism in the after visit had been evaluated during the SARS period. In contrast to the SARS group, the follow up team was able to identify new diagnoses among the non-SARS group throughout the three visits using direct face to face assessment, 4 at the during visit and 1 at the after visit.
Loss to follow up
Data for the period prior to SARS (before visit) were available for all infants while data for 73 infants were available following SARS (after visit) representing a 24% (22/95) loss to follow up rate over the three visits compared to 5% (5/99) for non-SARS group over the same period.
This study clearly illustrates that in absence of a priori planning, patient care and contact was unorganized and provided haphazardly in a non-urgent care clinic during the time of an infectious disease outbreak. Other reports have focused on inpatient care, where non urgent hospital utilization and hospital transfers were restricted, but this is the first such report of the impact on a hospital based clinic from the SARS time period [17, 18].
There were 95 patients with cancelled appointments during the SARS period and the NNFU team made contact with only 27 of these families despite clinic staff being required to attend work. There were no established criteria defining who should be contacted or how they should be evaluated during the crisis but rather clinicians made an intuitive decision as to who may require assistance. The response to the mailed questionnaire was poor (17/27). The eight families who received a follow up telephone call all returned the call but again there was no indication that any specific criteria were used in planning who should receive a telephone follow up in addition to the mailed questionnaire. This illustrates a haphazard approach during the time of clinic closure.
The loss to follow up rate was much greater over the SARS period compared to the non-SARS period 2 years later. It appears from this study that to maintain patient engagement in a clinic, there is a requirement not only for a mail out but also a telephone call. As this is a retrospective study we were not able to ascertain the reasons for not returning to the clinic however some parents expressed lingering concerns about the risk of exposure to an infectious disease in an outpatient clinic.
Although our results suggest that making a new neurological diagnosis may require a face to face assessment of the patient our study was not designed to look at this question and further research is required to determine if this is in fact true. If so, such an assessment may be facilitated through a telehealth assessment as opposed to a telephone interview.
The validity of using the parent completed ASQ system as an alternative approach to evaluate developmental outcome has been extensively evaluated [19–22] and has the advantage of being a cost-effective and simple screening tool for long term follow up of preterm high risk infants . Some literature has suggested that parents may be unable to complete questionnaires satisfactorily due to reading, organizational, mental health and cognitive disabilities . A limitation of this study is that we did not obtain socio-economic data of the parents as a determinant of the rate of ASQ reply and the accuracy of reporting developmental delay in their infants. It has been previously shown  however that the accuracy of parent reporting is not influenced by socio-demographic factors or maternal educational level. A further limitation is the potential difference in populations between the SARS and non-SARS groups as they were recruited in different time periods which may have affected the accuracy of our result. Future studies may compare alternative assessment types at the same time period.
Our rate of detection of developmental delay may have been ameliorated by adding other screening tools during the telephone contact. Further research should be performed to test the accuracy of different assessment systems when the gold standard of face to face assessment cannot be provided.
This study underscores the need for advance planning as to how best to provide outpatient care or assessment to families during an infectious disease outbreak. If there is no a priori planning there is a risk of inconsistent patient management as was illustrated here. This applies not only to NNFU clinics but to all clinics providing care for patients with chronic health conditions. Well planned mail and telephone contact may be of assistance in providing patient care during the time of infectious disease containment measures particularly if staff is deployed to work in other areas. Such contact may also help with maintaining the engagement of families and return to clinic. Consideration should be given for remuneration of physician and other health care providers for this telephone assessment. If staffing was adequate a telehealth or internet based  assessment may be more appropriate to diagnose certain types of conditions. After the crisis has passed, workload adjustments may be required to enable rapid rescheduling of face to face assessments. These reflections are timely in view of the current focus of hospitals and public health authorities' planning for anticipated influenza pandemics. In conclusion, non-urgent care clinics should have a priori planning for identification, contacting and assessment of their high risk patients during the periods of clinic closure due to infectious diseases outbreaks.
- Hack M, Flannery DJ, Schluchter M, Cartar L, Borawski E, Klein N: Outcomes in young adulthood for very-low-birth-weight infants. N Engl J Med. 2002, 346 (3): 149-157. 10.1056/NEJMoa010856.View ArticlePubMedGoogle Scholar
- Saigal S, Stoskopf B, Pinelli J, Streiner D, Hoult L, Paneth N, Goddeeris J: Self-perceived health-related quality of life of former extremely low birth weight infants at young adulthood. Pediatrics. 2006, 118 (3): 1140-1148. 10.1542/peds.2006-0119.View ArticlePubMedGoogle Scholar
- Bhutta AT, Cleves MA, Casey PH, Cradock MM, Anand KJ: Cognitive and behavioral outcomes of school-aged children who were born preterm: a meta-analysis. JAMA. 2002, 288 (6): 728-737. 10.1001/jama.288.6.728.View ArticlePubMedGoogle Scholar
- Doyle LW: Evaluation of neonatal intensive care for extremely low birth weight infants in Victoria over two decades: II. Efficiency. Pediatrics. 2004, 113 (3 Pt 1): 510-514. 10.1542/peds.113.3.510.View ArticlePubMedGoogle Scholar
- Colvin M, McGuire W, Fowlie PW: Neurodevelopmental outcomes after preterm birth. BMJ. 2004, 329 (7479): 1390-1393. 10.1136/bmj.329.7479.1390.View ArticlePubMedPubMed CentralGoogle Scholar
- Hille ET, den Ouden AL, Stuifbergen MC, Verrips GH, Vogels AG, Brand R, Gravenhorst JB, Verloove-Vanhorick SP: Is attrition bias a problem in neonatal follow-up?. Early Hum Dev. 2005, 81 (11): 901-908. 10.1016/j.earlhumdev.2005.07.006.View ArticlePubMedGoogle Scholar
- Campbell MK, Halinda E, Carlyle MJ, Fox AM, Turner LA, Chance GW: Factors predictive of follow-up clinic attendance and developmental outcome in a regional cohort of very low birth weight infants. Am J Epidemiol. 1993, 138 (9): 704-713.PubMedGoogle Scholar
- McCormick MC, Workman-Daniels K, Brooks-Gunn J, Peckham GJ: When you're only a phone call away: a comparison of the information in telephone and face-to-face interviews. J Dev Behav Pediatr. 1993, 14 (4): 250-255. 10.1097/00004703-199308010-00007.View ArticlePubMedGoogle Scholar
- Tsai MC, Arnold JL, Chuang CC, Chi CH, Liu CC, Yang YJ: Impact of an outbreak of severe acute respiratory syndrome on a hospital in Taiwan, ROC. Emerg Med J. 2004, 21 (3): 311-316. 10.1136/emj.2003.011122.View ArticlePubMedPubMed CentralGoogle Scholar
- Stukel TA, Schull MJ, Guttmann A, Alter DA, Li P, Vermeulen MJ, Manuel DG, Zwarenstein M: Health impact of hospital restrictions on seriously ill hospitalized patients: lessons from the Toronto SARS outbreak. Med Care. 2008, 46 (9): 991-997. 10.1097/MLR.0b013e3181792525.View ArticlePubMedGoogle Scholar
- Krumkamp R, Duerr HP, Reintjes R, Ahmad A, Kassen A, Eichner M: Impact of public health interventions in controlling the spread of SARS: modelling of intervention scenarios. Int J Hyg Environ Health. 2009, 212 (1): 67-75. 10.1016/j.ijheh.2008.01.004.View ArticlePubMedGoogle Scholar
- Ofner-Agostini M, Wallington T, Henry B, Low D, McDonald LC, Berger L, Mederski B, Wong T: Investigation of the second wave (phase 2) of severe acute respiratory syndrome (SARS) in Toronto, Canada. What happened?. Can Commun Dis Rep. 2008, 34 (2): 1-11.PubMedGoogle Scholar
- Spurgeon D: Toronto succumbs to SARS a second time. BMJ. 2003, 326 (7400): 1162-10.1136/bmj.326.7400.1162-a.View ArticlePubMedPubMed CentralGoogle Scholar
- Silverman A, Simor A, Loutfy MR: Toronto Emergency Medical Services and SARS. Emerg Infect Dis. 2004, 10 (9): 1688-1689.View ArticlePubMedPubMed CentralGoogle Scholar
- Borgundvaag B, Ovens H, Goldman B, Schull M, Rutledge T, Boutis K, Walmsley S, McGeer A, Rachlis A, Farquarson C: SARS outbreak in the Greater Toronto Area: the emergency department experience. CMAJ. 2004, 171 (11): 1342-1344.View ArticlePubMedPubMed CentralGoogle Scholar
- Bricker DD, Squires J, Mounts L: Ages & stages questionnaires: a parent-completed, child-monitoring system. 1999, Baltimore, Md.: Paul H. Brookes, 1 v. (various pagings), 2Google Scholar
- Haines CJ, Chu YW, Chung TK: The effect of Severe Acute Respiratory Syndrome on a hospital obstetrics and gynaecology service. BJOG. 2003, 110 (7): 643-645. 10.1046/j.1471-0528.2003.03007.x.View ArticlePubMedGoogle Scholar
- Schull MJ, Stukel TA, Vermeulen MJ, Zwarenstein M, Alter DA, Manuel DG, Guttmann A, Laupacis A, Schwartz B: Effect of widespread restrictions on the use of hospital services during an outbreak of severe acute respiratory syndrome. CMAJ. 2007, 176 (13): 1827-1832.View ArticlePubMedPubMed CentralGoogle Scholar
- Squires J, Bricker D, Potter L: Revision of a parent-completed development screening tool: Ages and Stages Questionnaires. J Pediatr Psychol. 1997, 22 (3): 313-328. 10.1093/jpepsy/22.3.313.View ArticlePubMedGoogle Scholar
- Klamer A, Lando A, Pinborg A, Greisen G: Ages and Stages Questionnaire used to measure cognitive deficit in children born extremely preterm. Acta Paediatr. 2005, 94 (9): 1327-1329.View ArticlePubMedGoogle Scholar
- Lindsay NM, Healy GN, Colditz PB, Lingwood BE: Use of the Ages and Stages Questionnaire to predict outcome after hypoxic-ischaemic encephalopathy in the neonate. J Paediatr Child Health. 2008, 44 (10): 590-595. 10.1111/j.1440-1754.2008.01388.x.View ArticlePubMedGoogle Scholar
- Skellern CY, Rogers Y, O'Callaghan MJ: A parent-completed developmental questionnaire: follow up of ex-premature infants. J Paediatr Child Health. 2001, 37 (2): 125-129. 10.1046/j.1440-1754.2001.00604.x.View ArticlePubMedGoogle Scholar
- Bricker D, Squires J, Kaminski R, Mounts L: The validity, reliability, and cost of a parent-completed questionnaire system to evaluate at-risk infants. J Pediatr Psychol. 1988, 13 (1): 55-68. 10.1093/jpepsy/13.1.55.View ArticlePubMedGoogle Scholar
- Johnson S, Marlow N, Wolke D, Davidson L, Marston L, O'Hare A, Peacock J, Schulte J: Validation of a parent report measure of cognitive development in very preterm infants. Dev Med Child Neurol. 2004, 46 (6): 389-397. 10.1017/S0012162204000635.View ArticlePubMedGoogle Scholar
- Etter JF: Comparing the efficacy of two Internet-based, computer-tailored smoking cessation programs: a randomized trial. J Med Internet Res. 2005, 7 (1): e2-10.2196/jmir.7.1.e2.View ArticlePubMedPubMed CentralGoogle Scholar
- The pre-publication history for this paper can be accessed here:http://www.biomedcentral.com/1471-2431/10/51/prepub
This article is published under license to BioMed Central Ltd. This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.