This study evaluated the applicability of the Psycat and its validity for identifying EBP in routine PCH services. Regarding applicability, participation rates were low to moderate, and most participating parents and PCH professionals were satisfied with the Psycat. Furthermore, the Psycat proved to be efficient, requiring a mean number of 16 items to correctly identify a child with EBP. Regarding validity, the Psycat was found to be valid to identify children with EBP in routine PCH care.
Applicability: participation rate
The mean participation rate of parents was 41%, but we found differences between the two samples. The participation rate was low (29%) in the application-validation sample and satisfactory (69%) in the application sample, although both samples comprised data that was collected in a routine PCH setting. A likely explanation is that the method of data collection caused the low participation rate in the application-validation sample (29%). That method gave a large burden on parents: they had to complete the Psycat and an additional CBCL questionnaire consisting of 120 items serving as criterion, via different media (paper and computers). The much higher participation rate in the application sample suggest the Psycat as the only instrument offered to be much more acceptable. These findings suggest that the Psycat is applicable.
Applicability: satisfaction
Most participating parents and PCH professionals were satisfied with Psycat, which aligns with other studies which also found adaptive tests to be well accepted by respondents [23,24,25].
Our findings indicate that professionals may underestimate the satisfaction of parents with Psycat, as they expected many parents to judge the Psycat as unclear (Table 3), while parents themselves actually reported the Psycat to be clear (Table 2). In general, these findings suggest that the Psycat is ready for further trials in routine use in PCH practice.
Applicability: efficiency
The Psycat was an efficient procedure, requiring a mean number of 16 items to correctly identify a child, which is however four more items than in the simulation study [19]. This discrepancy can be explained in two ways. First, a simulation study may result in overfitting of the mean number of items, however this explanation is rather unlikely because separate samples were used to determine the cut-off point and to assess the validity and efficiency (mean number of items) in the simulation study. Second, this discrepancy may be due to small differences in design. Additional research is necessary to investigate this more in depth.
Validity
We found mostly good validity indices for the Psycat in routine PCH use. However, sensitivity of the Psycat internalizing scale was low (0.46), at rather high specificity (0.93). An explanation might be that internalizing behavior are more difficult to observe for parents [26, 27]. This explanation is rather unlikely however, because the literature shows that the criterion (CBCL) for internalizing behaviors is not affected by this measurement bias [22]. A more likely explanation is the small number of participants who met the criterion (CBCL), leading to a low statistical power of our validation study in this respect. As a consequence, the validity indices from this study on emotional problems need to be interpreted with caution.
The mostly good validity indices for the Psycat in routine PCH confirm the findings of the previous simulation study regarding the Psycat of Vogels et al. [19] and other (simulation) studies regarding CATs aimed to measure quality of life [16, 18] or mental health [17] in the field of pediatric mental health care. Gardner’s CAT assessed the same construct (mental health in children) as the Psycat. Gardner et al. [17] showed that CAT based on the Pediatric Symptom Checklist (PSC) for parents in a simulation study reproduced the results of the full scale PSC with greater efficiency. There are however some differences with respect to the Psycat. Gardner used the full PSC as criterion for EBP, while the Psycat used the widely validated CBCL. Furthermore, Gardner used only PSC items to develop the CAT, we use items from four questionnaires. Lastly, the Psycat was studied in routine PCH, whereas Gardner study was limited to a simulation study. Our study therefore provides a stronger argument for the validity and feasibility of CAT-based procedures in the field of mental health of children.
Strengths and limitations
Our study has a number of strengths, such as its community-based nature and embedding in routine PCH practice. Moreover, we were able to assess various aspects of applicability, including the parent- and the professional perspective.
However, some limitations should also be considered. First, we used the very well-validated CBCL questionnaire as the validation criterion, and not clinical assessments such as psychiatric interviews. Clinical assessments may provide additional information, but could not be used in this study because of their complexity and high costs. Second, the participation rate regarding the validation was low, which may have led to selective non-response among parents of children with EBP. This may affect prevalences and perhaps also validity indices. Finally, some of the items predicting the criterion were part of the criterion itself, i.e. the CBCL items. However, this regards relatively few of the items included in the CAT and it has a small effect on conclusions regarding validity.
Implications
Our findings indicate that the Psycat is an efficient instrument that could be validly used to help parents and professionals identify EBP in routine PCH settings. These promising findings suggest that the Psycat is suitable for further trials in routine use in PCH services. It provides an efficient and valid procedure for the assessment of EBP, overall and more specific, for routine use in PCH.
Our study also showed that parents in a real-life setting needed more items to identify a child with EBP than they needed in a simulation setting. This implies that simulation studies of adaptive tests may provide an over-optimistic picture of its performance in real-life healthcare settings. This latter aspect and the low to moderate response rates definitely require additional research on the performance of CAT in both PCH services and other healthcare settings.