Announcement of methodological changes to the Summary Hospital-level Mortality Indicator (SHMI)

The Summary Hospital-level Mortality Indicator (SHMI) has been published by NHS Digital since September 2011. It is a complex indicator and there are a range of academic views on various aspects of the methodology. The SHMI is subject to continuous evaluation and the methodology is now due for review.

A SHMI Engagement Group consisting of stakeholders and users of the indicator and a Technical Sub-Group consisting of academic experts have been convened to lead this work. They report to the National Medical Director at NHS England, Professor Stephen Powis.

As part of the review, several tactical improvements to the methodology have been recommended as well as some additional breakdowns of the data and presentational improvements. A longer term, more fundamental review of the methodology is also underway, and users will be kept informed as this work progresses. The purpose of this document is to announce the tactical improvements that will be implemented from the next SHMI publication onwards.

The following changes to the methodology will be made from the May 2019 SHMI publication, which will cover discharges in the period January 2018 – December 2018. Full details of the updated methodology are available in the methodology specification document which is available to download from the SHMI homepage. 

The SHMI is calculated using Hospital Episode Statistics (HES) data and small number suppression is applied where necessary for the purposes of disclosure control. Currently the SHMI and accompanying contextual indicators are suppressed using a methodology described in a now superseded version of the HES Analysis Guide. Counts of 1-5 inclusive are suppressed and secondary suppression is then applied where necessary to prevent the calculation of suppressed values from other values in the data.

The HES disclosure control methodology has recently been updated to a methodology based on rounding¹ . This methodology will be applied to the SHMI and contextual indicator data from the May 2019 publication as follows:

• all counts that are greater than 0 and less than or equal to 7 will be replaced with the special character ‘*’. All other counts will be rounded to the nearest 5
• the number of expected deaths is not a count of individuals and is a purely statistical construct. If this value is less than 7.5 then it will be replaced with the special character ‘*’. Otherwise it will be rounded to the nearest 5. This is to prevent back calculation of the unrounded count of observed events from the number of expected deaths and the SHMI value
• percentages will be calculated using the rounded values when the rounded denominator is greater than or equal to 400. Otherwise, the percentage will be replaced with the special character ‘*’

Disclosure control is not required for complex calculations such as the SHMI value, the SHMI banding or the SHMI control limits and these will be calculated using the unrounded values. However, these values will only be calculated if the count of finished provider spells is greater than or equal to 30 and the observed and expected number of deaths have not been replaced with the special character ‘*’. The condition on the number of finished provider spells is for reasons of statistical robustness rather than disclosure control² .

No disclosure control will be applied to the national level contextual indicator results.

Trusts will continue to have access to unsuppressed data for their own organisation via the Clinical Indicator Previewer and the SHMI Extract Service

The SHMI is currently composed of 140 different diagnosis groups and these are aggregated to calculate the overall SHMI value for each trust. Each ICD-10 diagnosis code is assigned to a Clinical Classifications Software (CCS) category using a mapping produced by the Agency for Healthcare Research and Quality (AHRQ)³ . CCS categories are then further aggregated into the 140 diagnosis groups which make up the SHMI to ensure that there is a sufficient number of events in each group.

Due to concerns that CCS categories have been combined in a way that may not be clinically appropriate, two new diagnosis groups will be created:

• livebirths (CCS category 218) will be removed from diagnosis group 106 (pregnancy related conditions) and included in a new diagnosis group (group 141)
• Non-Hodgkin’s lymphoma (CCS category 38) will be removed from diagnosis group 26 (Hodgkin’s disease and non-Hodgkin’s lymphoma) and included in a separate diagnosis group (group 142)

There are seasonal variations in death rates for some conditions. This has little impact on the SHMI because it includes discharges in a rolling twelve-month period. However, seasonality does have the potential to impact on other analyses derived from the SHMI, for example the Variable Life-Adjusted Display (VLAD) charts which are provided to trusts to allow the visualisation of trends over time.

Therefore, an adjustment for seasonality will be included in the logistic regression models which are used to calculate the expected number of deaths. This will be achieved by including a categorical variable representing the month of the patient’s admission.

An adjustment for birthweight will be included in the logistic regression models for individuals aged less than one year for the following diagnosis groups:

• Group 115 – cardiac and circulatory congenital anomalies
• Group 116 – digestive congenital anomalies, genitourinary congenital anomalies, nervous system congenital anomalies, other congenital anomalies
• Group 117 – short gestation, low birth weight, fetal growth retardation
• Group 118 – intrauterine hypoxia and birth asphyxia, respiratory distress syndrome, hemolytic jaundice and perinatal jaundice, birth trauma
• Group 119 – other perinatal conditions
• Group 141 – livebirths

This will improve the accuracy of the statistical models for these diagnosis groups and therefore the accuracy of the overall SHMI.

The SHMI does not make any adjustment for deprivation. Instead, two contextual indicators are published to support the interpretation of the SHMI. These present the percentage of provider spells and deaths reported in the SHMI which fall under each deprivation quintile. Deprivation quintile 1 is the most deprived group and deprivation quintile 5 is the least deprived group. The deprivation quintile is currently calculated from the Index of Multiple Deprivation (IMD) Overall Rank field in the HES dataset. This uses the 2010 version of the IMD.

The methodology for both deprivation contextual indicators will be updated so that the latest version (currently the 2015 version) of the IMD is used to derive the deprivation quintile. Rather than using the IMD Overall Rank field from the HES dataset, the IMD rank will be obtained by linking HES data to the English Indices of Deprivation 2015 dataset using the Lower Super Output Area⁴ 2011 (LSOA11) field.

The combined impact of the changes is very small. The following analysis is based on data for discharges in the period October 2017 – September 2018.

At trust level, the change in the SHMI value and expected number of deaths is less than 1 per cent for all trusts. No trusts change SHMI banding. The number of finished provider spells and observed deaths is not affected by the changes to the methodology, except that these counts are now displayed to the nearest five.

For diagnosis groups 26 (Hodgkin’s disease) and 106 (pregnancy related conditions), the number of finished provider spells, observed deaths and expected deaths has decreased because some of the activity previously included in these groups is now included in the newly created diagnosis groups 141 (livebirths) and 142 (nonHodgkin’s lymphoma).

For the other diagnosis groups, the number of finished provider spells and observed deaths is not affected by the changes to the methodology, except that these counts are now displayed to the nearest five. There are only 22 trust and diagnosis group combinations (out of a total of more than 18,000) where the number of expected deaths changes by more than 5, and only one where the number of expected deaths changes by more than 10 (this still only represents a change of less than 2 per cent). As explained above, the impact of these changes on the overall SHMI is very small.

The success of the case-mix adjustment in predicting whether a patient died or survived can be evaluated using the c statistic for each logistic regression model. The c statistic is the probability of estimating a lower risk of death for a randomly selected patient who survived compared to a randomly selected patient who died and can take values in the range 0.5 – 1.0. Using the current methodology, three diagnosis groups (all relating to perinatal conditions) have c statistics of less than 0.6, indicating a poor model fit. Using the updated methodology, the c statistic for all three of these models increases to 0.7 or higher (indicating a reasonable predictive ability). The c statistic increased by at least 0.01 for 9 of the current 140 diagnosis groups, changed by less than 0.01 for 130 diagnosis groups and decreased by at least 0.01 for one diagnosis group (this decrease was for group 106 – pregnancy related conditions, and is due to the removal of livebirths from this diagnosis group).

Only the deprivation contextual indicators are affected by the change to using the latest version of the IMD. This has a very small impact on the national results for these indicators (a change of less than one percentage point). However, for some trusts the impact of the change is greater due to changes in the IMD for the local areas served by the trust. The change will mean that the indicator will reflect the latest deprivation data and so will be more useful in aiding the interpretation of the SHMI.

For the diagnosis breakdown data and the contextual indicator results, the number of values replaced with the special character ‘*’ for the purposes of disclosure control has increased. This is because counts of 1-7 (inclusive) will be suppressed, whereas the current methodology suppresses counts of 1-5 (inclusive). Some larger values are currently suppressed as a result of secondary suppression, which is applied where necessary to prevent the calculation of suppressed values from other values in the data. Following the introduction of the new disclosure control methodology, this suppression of larger values will no longer be required as all counts will rounded to the nearest five.

For the SHMI contextual indicators, percentages will be calculated using the rounded values and will only be calculated when the rounded denominator is greater than or equal to 400. This means that users can assume that the calculated percentage is within one percentage point of the ‘true’ percentage.

In addition to the methodological changes described above, some new SHMI data breakdowns will be available from the May 2019 SHMI publication.

Trusts may be located at multiple sites and may be responsible for one or more hospitals. In addition to the SHMI data at trust level, a breakdown of the SHMI by site of treatment will be published from the May 2019 release onwards. This new breakdown is being published as an experimental statistic. Experimental statistics are official statistics which are published in order to involve users and stakeholders in their development and as a means to build in quality at an early stage.

The range of SHMI values is considerably greater at site level than at trust level. There are several factors which contribute to this. These include some sites having particular specialisms and service models (for example, dialysis, maternity and end of life care) and also some inconsistencies in how trusts have defined their ‘sites’. A site type classification will be included in the data to assist interpretation and has been derived by linking HES and Care Quality Commission (CQC) reference data. The alignment between codes in the respective reference tables is known to be imperfect, and this remains work in progress, but is considered to be of sufficient accuracy for inclusion in this breakdown.

A small number of trusts carry out all activity at a single site. In such cases, the SHMI value and counts of provider spells, observed deaths and expected deaths will be the same in both the site level and trust level data. However, the SHMI banding (‘higher than expected’, ‘as expected’ or ‘lower than expected’) may be different. The reason for this difference is a result of the greater variability in the data at site level (as described above), which in turn affects the calculation of the control limits used to define the SHMI banding. As the control limits at site and trust level may therefore not be the same, this leads to the same SHMI value potentially having a different categorisation at trust and site level.

Provider spells may consist of one or more episodes (a single period of care under one consultant). This breakdown will use the site from the first episode in the provider spell. A new contextual indicator on the percentage of provider spells where the site of treatment for the last episode is different to the first will be included as part of the SHMI publication and is designed to aid in the interpretation of the site level data.

As part of each SHMI publication, the number of finished provider spells, observed deaths and expected deaths broken down by diagnosis group is provided for each trust. From the May 2019 publication, SHMI values, SHMI bandings and control limits (indicating whether the SHMI is ‘higher than expected’, ‘as expected’ or ‘lower than expected’) will also be included as part of this breakdown for the following diagnosis groups:

Group 2 – septicaemia (except in labour), shock 

Group 15 – cancer of bronchus, lung

Group 30 – secondary malignancies 

Group 37 – fluid and electrolyte disorders

Group 57 – acute myocardial infarction

Group 73 – pneumonia (excluding TB/STD)

Group 74 – acute bronchitis

Group 96 – gastrointestinal haemorrhage 

Group 101 – urinary tract infections 

Group 120 – fracture of neck of femur (hip)

These groups have been chosen because they have high numbers of deaths and statistical models that are considered to have sufficiently explained the expected variation in outcomes due to the case-mix adjustment (they are the same diagnosis groups for which NHS Digital generates VLAD charts which are provided to trusts via the SHMI Extract Service).

From May 2019 onwards, the SHMI will be published on a monthly basis rather than every quarter. Each publication will include discharges in a rolling twelve-month period. For example, the May 2019 publication will include discharges in the period January 2018 – December 2018 and the June 2019 publication will include discharges in the period February 2018 – January 2019.

Trusts will continue to be provided with pre-release access to their own SHMI data for the purpose of quality assurance on a quarterly basis (for the May, August, November and February releases).

Questions and feedback on the publication are welcomed and should be sent to e[email protected] or alternatively call 0300 303 5678.