Delivery of good quality health care has considerable potential to reduce childhood deaths in low-income countries where mortality is high [1, 2]. However, both anecdotal and empirical evidence suggest that the quality of care offered in many facilities, both primary and referral, is generally poor [3–8]. Valid and reliable performance measures (indicators) can be used to evaluate quality of care and help health workers to improve it [9]. In some high income countries, quality of care measures are increasingly recognized as a priority to help foster improvement and promote accountability, and substantial investments have been made in their development [10–12]. In contrast, relatively little effort has been put into developing such measures for low-income countries [13].

Recently, the WHO revised an earlier tool for assessing the quality of care in first-referral health care facilities (rural or district level hospitals) in developing countries, based on multi-country experience and informed by discussions at a global WHO meeting in Bali in 2007 [14]. The WHO tool is based on the classical quality of care framework involving structure, process and outcome [15]. Amendments can be made to this tool by countries to suit their local needs. Uptake of the tool will probably depend on "buy in" and support from influential persons in each country. Using this tool, structure and processes of care are rated on a simple semi-qualitative scale (the latter based on a convenience sample of 5 cases) making it useful for rapid appraisal of hospitals. However, such scores are of limited value if the aim is to provide an objective comparison between hospitals within a region or within hospitals at different periods - particularly if different observers use the tool.

The debate about which quality measures are best continues [16]. Proponents of process measures argue that they directly measure actions that are within the control of health workers [17]. However, without appropriate drugs and equipment, health workers may be unable to offer the correct care, making structure a limiting factor in quality improvement [18]. Outcome measures may be intuitively the most useful indicators of quality, but their opponents argue that they are subject to considerable confounding and may require large samples for sufficiently precise measurement.

To decide which measures to use, expert judgement, supported by evidence where available, is often used. Experts, however, differ in opinion and a family of consensus methods such as the nominal group technique (NGT), the Delphi technique, and the RAND/UCLA appropriateness method (a hybrid of the former two), are often used to facilitate communication and avoid the negative social influences associated with group processes [19]. This article reports the results of a Delphi study that aimed to determine which measures of quality of admission care for children in first level referral hospitals in low-income countries were widely supported, using explicit methods. Performance measures can be used to assess quality at different levels - clinical, organisation or population. This study considered measures pertinent to the clinical level. Further, the study aimed to identify process measures which could give quantitative measures of performance and performance change. We had a particular focus on the African setting and also examined the likely acceptability of internationally suggested indicators to an influential Kenyan audience. Ethical approval was obtained from the KEMRI National Ethical Review Committee.

This study sought to assess the perceived value and validity of a set of indicators of quality of inpatient care proposed for children and newborns admitted to hospital in low-income countries. It used a transparent and inexpensive method that combines scientific evidence and expert opinion. Additionally, we intended to investigate how well recommendations typically issued by international organisations such as the WHO might be accepted by local experts, as such views might influence their local credibility. The Delphi technique proved useful in both respects. The lack of an obligation to meet face to face significantly improved the feasibility of the study as cost was not a constraint on either the size or composition of the international expert panel [36]. Most of the indicators proposed were considered: reliable and relevant to low-income settings; within the capabilities of resource constrained health systems to improve; able to identify areas urgently in need of attention and; to be feasible targets of data collection.

However, one of our anticipated project outputs was identification of a parsimonious set of indicators (about 20-30) that we felt might realistically form the basis of routine and widespread reporting to ministries of health in low-income countries. In this respect it can be argued that the process failed, with support for a large number of indicators of quality of care, making implementation of routine, national quality assessment incorporating all of these potentially more difficult. The demonstrated support for a large number of indicators might be interpreted as an endorsement of the scope and content of the original WHO assessment tool and, by extension, a desire to ensure that any assessment tool for hospital inpatient care should span the resources, assessment tasks and management required to provide effective care for multiple, important diseases. Alternatively it may reflect in part the numerous indicators presented to the experts. Presenting a large initial set of indicators was meant to minimize the risk of missing potentially important issues and prevent a small number of investigators (SN, ME) imposing their priorities at the outset. Interestingly some of the national panel, when prompted to prioritise indicators for reporting to the ministry of health, were more likely than members of the international panel to 'downgrade' indicators (although only to an uncertain status) producing a potentially shorter list and resulting in less apparent consensus in their final round.

It is reassuring that there were high levels of agreement between the local and international panels. Worth noting is the high agreement achieved on the sets of indicators for care of sick neonates and malnourished children. This may have been influenced by increasing recent appreciation globally of the high case fatality rates in these groups. Within panels however, there was somewhat more divergence (larger variance) around structural elements of care perhaps reflecting different experts' experiences of resource environments or the poor scientific evidence available supporting their contribution to better outcomes.

Responses on the feasibility of data collection for assessment of an indicator under current conditions in low-income settings varied, suggesting many panellists were concerned that assessment of many important process indicators requiring data from routine review of medical charts may not be possible currently. There was however optimism that measures could be instituted to improve quality of data. This concern is echoed more generally internationally [37], supported by experience nationally [38, 39]. However, an unexpected finding was that indicators for neonatal care were accorded generally higher scores on feasibility. Our experience of working in district hospitals in Kenya [7] and from published data elsewhere [8] seem contrary to these views. It is possible, therefore, that the views expressed by the experts on feasibility are overoptimistic and unduly influenced by their desire to promote certain indicators. Before any indicators are widely adopted they should be tested for feasibility and ability to detect significant change in performance (sensitivity) [40]. We are in the process of evaluating the feasibility of the accepted set of indicators using data from 8 Kenyan district hospitals. More generally, improving the quality of care within a health system will need improvements in information system.

As our goal was to develop a tool that might be routinely used to provide quantitative measures of quality at scale within the capacity of an existing health system, our focus was case record review. However, there are alternative methods of collecting data on the process of care such as direct observation or use of vignettes. Direct observation may influence care, and would be time consuming and costly if a sample size sufficient to produce a quantitative estimate is desired [41]. Vignettes, perhaps best for assessing health worker knowledge, have been used but similarly require organised access to multiple health workers and so may be hard to implement at scale in African settings [42].

There are potential limitations in our study that warrant mention. First, the results reported here represent the opinions of only a few, non-randomly selected individuals. However, our international panel had members with wide and long clinical and quality improvement experience in low-income settings. The local panel consisted largely of experts drawn from an academic referral centre. However, the local experts are influential, frequently called upon by government to provide expert technical advice and are responsible for training of health workers. They thus represent an important constituency in brokering the acceptance or rejection of international recommendations at a country level.

Second, we did not grade or present the strength of evidence linking indicators to outcomes in the questionnaires. This was in part due to the scale and scope of the task with considerable implications on workload for both the researchers and expert panellists. Moreover, the indicators were derived from the practices or resource implications of existing WHO recommendations for hospital care [28]. Third, our definition of consensus, though based on published studies, remains somewhat arbitrary [30, 43]. Altering the definition of consensus, for example by calculating mean scores across attributes for each indicator, did not result in significant change in those accepted by the predefined criteria (data not shown). The post hoc analysis (consensus criterion 2) may be useful if the intention is to reduce the list of indicators. Fourth, the process did not involve experts meeting face to face. While this reduced the cost, many may argue that an opportunity for experts to meet would have stimulated useful discussion on contentious issues [31]. We did however encourage experts to make comments which we then summarised and fed back to the group through a moderator.

Methodologically, some valuable lessons were learnt. First, Delphi studies are time consuming with an average turnabout time between rounds of approximately 1.5 months for emailed questionnaires. Second, the method of delivering the instructions may have an effect on the results of the process. The local panel received instructions by word of mouth and were more likely to prioritise indicators in round 2 compared to the international experts who were given written instructions. Finally, there is scope for refining definitions of consensus by allowing the participants to decide on an appropriate definition instead of imposing one.

Measurement of the quality of care is a prerequisite for determining whether quality of care is improving. Although there remains significant challenges in defining such measures, this study represents the first attempt at a transparent, consensus based, international approach to identify indicators of quality hospital care for children and newborns suitable for use in low-income settings. For process based measures, feasibility of data collection remains a concern and should be further evaluated. This study, based on a transparent process, helps formally define widely acceptable, quantitative indicators and provides a platform for further debate and continuing indicator development that should include the review and updating of indicators as priorities and technologies change and interventions to improve quality of care are scaled up. Such reviews may be more likely if a relatively inexpensive process such as the one described here is used.