Impact

This section summarises the impact of these changes on how you can interpret statistics within the Mental Health Services Monthly Statistics publication series. 

When trying to understand the impact of the changes to the submission windows, it is important to understand that the MHSDS is a large relational dataset that collects many millions of rows of data each month. MHSDS covers a wide variety of service areas and as such there are nuances to the data available in the month following activity. In each case it should still be considered a trade-off between timeliness and completeness when assessing the impact of the change.

New referrals are a good proxy for understanding volumes of data as the referral table is required as part of any submission to the dataset and as such gives a good indication of overall coverage.

There were 413,807 new referrals (MHS32) reported in the May 2023 provisional window. This is compared to 422,190 new referrals in the May 2023 performance window. This represents a difference of 8,383 or 2.03%. This change is reported to be significant based on Wilcoxon Signed-Rank Test.

When looking at restraints data, there are large differences in the data submitted in the provisional and performance windows. Table 1 below demonstrates the differences between the provisional and performance data for restraints in May 2023.

Each row has a provider code and provider name of 'England'. 

All metrics in the publication are impacted to some degree. For some metrics, especially those where the national count is a small number, or where the metric is particularly volatile, the percentage change between submissions is more pronounced. This is illustrated in Table 2 which shows the top 10 percentage differences for metrics at England level in May 2023. Table 2 also suggests that there could be some issues in interpreting data regarding short term orders. Short term orders often start within a police setting and as such that could be a possible cause for a delay in data.

Each row has a provider code and provider name of 'England'. 

Table 3 below displays those metrics which show the biggest differences in absolute values. As illustrated by the table, just because the differences appear to be large, the percentage changes in these metrics are relatively small. It should be noted too that these metrics generally cover bed days, contacts, and referrals. It should be expected that these items all flow in large numbers in the MHSDS. There are around 2.5 million contacts recorded each month within MHSDS and therefore while there is a difference of 55,869 for Contacts in the Reporting period, this only represents a difference of 2.29%.

The aim would be for these differences to reduce as more providers submit early and increase the quality of their submissions.

Each row has a provider code and provider name of 'England'.

On average, for most metrics there is a difference of between one and five percent. 200 out of the 300 measures analysed showed a percentage difference between 0% and 5% at national level in May 2023. This number rises to 237 out of the 300 measures when differences between -5% and 5%.

There are several metrics that are outside of this range and as such it is important to understand where changes are driven by the volatility of measures, provider behaviours or where the percentage change is related to other factors, such as difficulty of recording in the case of restraints. To aid users in understanding this, an appendix has been added to show which metrics are significantly different based on Wilcoxon Signed-Rank Test and can be found as Appendix A.

At provider level there are disparities in the submission patterns between groups of providers and individual providers. When looking at differences between submissions for individual providers it is also important to recognise that there are other underlying reasons that a submission may differ between the provisional and performance window. For example, there maybe IT issues which prevent a successful provisional submission. Equally, differences between the provisional and performance window are likely to reduce over time as providers are given more time to submit in the window following the activity month and as the focus of the publication moves towards the data submitted in the first month following activity.

Despite this, based on the May 2023 data there does appear to be differences between groups of providers. Generally, most NHS Trusts show a difference of around 1% when comparing data in the provisional and performance windows. In contrast, data for independent providers differs more widely with differences between -81% (with a total decrease of 75) and 225% (with a total increase of 165). This exemplifies the differences in the submissions but also the fact that generally, numbers submitted by independent providers are generally small.

A breakdown of MHS01, People in contact with services at the end of May 2023 is shown in Table 4 for each Organisation Type. Organisations have been grouped and the total percentage change for all providers in the group between the provisional and performance windows is shown.

Note: Numbers have been rounded to the nearest 5. Percentages have been rounded to the nearest single decimal place.

This situation is reflected across most of the metrics. MHS01 has been used here as a proxy to indicate overall differences in the submissions as it is a key metric in the monthly publication. There will be other areas where differences maybe more pronounced, but these will likely differ between individual providers. It is also worth noting that the providers groups showing the biggest percentage differences in Table 4 only make up a small percentage of the total records submitted to MHSDS. Overall, the majority of data flowing to MHSDS comes from NHS Trusts where the differences are smaller.