Impact

This section summarises the impact of these changes on how you can interpret statistics within the NHS Talking Therapies Monthly Statistics Including Employment Advisors publication series.

When trying to understand the impact of the changes to the submission windows, it is important to understand that the IAPT dataset is a large relational dataset that collects many millions of rows of data each month. In each case it should still be considered a trade-off between timeliness and completeness when assessing the impact of the change.

New referrals are a good proxy for understanding volumes of data as the referral table is required as part of any submission to the dataset and as such gives a good indication of overall coverage.

There were 144,074 new referrals (M001) reported in the May 2023 provisional window. This is compared to 146,659 new referrals in the May 2023 performance window. This represents a difference of 2,585 or 1.79%. This change is reported to be not significant based on Wilcoxon Signed-Rank Test (p value = 0.00627).

The majority of metrics in the publication are impacted to some degree. For some metrics, especially those where the national count is a small number, or where the metric is particularly volatile, the percentage change between submissions is more pronounced. This is illustrated in Table 1 which shows the top 10 percentage differences for metrics at England level in May 2023.

Table 1: Top 10 percentage differences for metrics at England level, May 2023

Table 2 below displays those metrics which show the biggest differences in absolute values. As illustrated by the table, just because the differences appear to be large, the percentage changes in these metrics are relatively small. It should be noted too that these metrics generally cover care contact appointments and employment statuses, both of which are recorded in large volumes in the IAPT dataset. There are around 650,000 care contacts recorded each month within NHS Talking Therapies and therefore while there is a difference of 13,481 for number of appointments in the Reporting period, this only represents a difference of 2%.

The aim would be for these differences to reduce as more providers submit early and increase the quality of their submissions.

Table 2: Top 10 numerical differences for metrics at England level, May 2023

On average, for most metrics there is a difference of between zero and five percent. 260 out of the 296 measures analysed showed a percentage difference between 0% and 5% at national level in May 2023. There are several metrics that have a difference of greater than five percent and as such it is important to understand where changes are driven by the volatility of measures, provider behaviours or where the percentage change is related to other factors. To aid users in understanding this, an appendix has been added to show which metrics are significantly different based on Wilcoxon Signed-Rank Test and can be found as Appendix A.

At provider level there are disparities in the submission patterns between groups of providers and individual providers. When looking at differences between submissions for individual providers it is also important to recognise that there are other underlying reasons that a submission may differ between the provisional and performance window. For example, there may be IT issues which prevent a successful provisional submission. Equally, differences between the provisional and performance window are likely to reduce over time as the focus of the publication moves towards the data submitted in the first month following activity.

Despite this, based on the May 2023 data there does appear to be differences between groups of providers. Generally, most NHS Trusts show a smaller difference when comparing data in the provisional and performance windows. For NHS Trusts 249 out of the 296 measures analysed showed a percentage difference between 0% and 2% at national level in May 2023. In contrast, data for independent providers differs more widely. For Independent Healthcare Providers only 50 out of the 296 measures analysed showed a percentage difference between 0% and 2% at national level in May 2023.

A breakdown of M001, People in contact with services at the end of May 2023 is shown in Table 3 for each Organisation Type. Organisations have been grouped and the total percentage change for all providers in the group between the provisional and performance windows is shown

Table 3: Percentage difference for selected organisation types for M001, People in contact with services at the end of May 2023 between the provisional and performance windows.

This situation is reflected across most of the metrics. M001 has been used here as a proxy to indicate overall differences in the submissions as it is a key metric in the monthly publication. There will be other areas where differences maybe more pronounced, but these will likely differ between individual providers.