Publication, Part of Patient Reported Outcome Measures (PROMs)

Provisional Patient Reported Outcome Measures (PROMs) in England - for Hip and Knee Replacement Procedures (April 2021 to March 2022)

Official statistics

Publication Date:: 8 Jun 2023
Geographic Coverage:: England
Geographical Granularity:: Country, Strategic Health Authorities, Hospital Trusts, Primary Care Trusts, Independent Sector Health Care Providers, Clinical Commissioning Groups, NHS Trusts, Primary Care Organisations, County, Care Trusts, Hospital and Community Health Services
Date Range:: 01 Apr 2021 to 31 Mar 2022

Data Quality Statement

Introduction

This page aims to provide users with an evidence-based assessment of the quality of the statistical output of the Provisional Patient Reported Outcome Measures publication for April 2021 to March 2022 data by reporting against those of the nine European Statistical System (ESS) quality dimensions and principles appropriate to this output.

More information regarding the background to PROMs and information relating to data quality measures can be found in the files via the "Resources" section (this can be found on the "Overview" tab).

Relevance

Health providers and commissioners use PROMs publications and statistics to improve the quality of health care offered to patients. Patients (and others involved in managing patient care such as GPs and carers) can use PROMs data to help decide where to receive treatment using the interactive publication tools.

Academic researchers use these data to inform research on PROMs-eligible procedures. Patients, carers and other organisations that support patients (including GP practices and charities) may also find these statistics helpful in making informed choices about elective procedures and providers.

NHS Digital is keen to gain a better understanding of the users of this publication and of their needs; feedback is welcome and may be sent to [email protected] (please include ‘PROMs’ in the subject line).

Accuracy and Reliability

Accuracy is the proximity between an estimate and the unknown true value. Reliability is the closeness of early estimates to subsequent estimated values.

Strengths and limitations of the data set

Strengths

PROMs are a detailed, record-level dataset that can be used, alongside other datasets, in order to research possible factors, such as surgical approach or implant type, which may affect a patient's outcome.

Data quality and completeness for PROMs are high, especially for patient demographics and identifiers. This ensures that the risk of linking a patient's questionnaire to the wrong hospital episode is very low.

Limitations

Care should be taken when making comparisons using PROMs data. Whilst it is fair to compare scores across organisations for the same procedure and measure, it is not a valid use of PROMs data to compare scores for the same measure across procedures. For example, making a decision to reduce funding for knee replacement procedures in favour of funding more hip replacements based on the EQ-5D is not a valid use of PROMs data.

Currently PROMs covers only two elective procedures. Although they are high volume, they represent only a small percentage of a typical hospital’s elective caseload. Care should be taken to not judge an organisation's overall performance based on PROMs data alone.

Data Linkage

NHS Digital links pre-operative PROMs questionnaires to administrative data about the related inpatient hospital procedure held in the Hospital Episodes Statistics: Admitted Patient Care dataset (HES: APC). Pre-operative questionnaires collect personal information about the patient’s postcode, date of birth and sex. This information is used to support patient-level linkage with HES (other administrative information such as NHS number is also used for data linkage).

The HES: APC contains a range of variables (such as the Indices of Multiple Deprivation) that are used in the PROMs Casemix adjustment methodology. Whilst PROMs records that cannot be linked with HES: APC episodes are used in much of our reporting, only records that link to HES: APC can be used to make comparisons between provider-level and England-level outcomes.

When linking to HES: APC, matching rules are used to identify and rank potential matches between pre-operative questionnaires and hospital episodes. Three kinds of error may be introduced during linkage:

A questionnaire may be linked to an unrelated episode. This is rare as it can only happen when a patient has two or more PROMs-eligible procedures within a very short space of time (for example, a primary procedure, followed shortly after by an elective revision procedure).
A questionnaire may not link successfully to a related episode. This is more common and can have a number of different causes such as poorly written or incorrect identifiers on the PROMs questionnaire or poorly coded HES episodes
Very rarely, a questionnaire may be linked to the wrong patient: this could potentially happen if two patients with the same sex, date of birth (where this is not 1 January) and postcode had the same PROMs-eligible procedure at the same hospital within a short period of time.

For activity in April 2021 changes needed to be made in the linkage methodology for PROMS data collected after that point in order to accommodate changes to the structure of the HES data set further details regarding these nature of these changes can be found in the updated PROMs Guide published in the coming weeks.

Timeliness and Punctuality

Timeliness refers to the time gap between publication and the reference period. Punctuality refers to the gap between planned and actual publication dates.

Timeliness

Analysis in this report is based on provisional data for:

Pre-operative patient questionnaires completed between 1 April 2021 and 31 March 2022 and any associated in-patient hospital episodes and post-operative questionnaires;
Episodes of inpatient hospital care where the episode started between 1 April 2021 and 31 March 2022 and which included one of the two surgical procedures eligible for PROMs

To address a user need for more timely finalised data and following a review of changes in historical data in the calculated metrics across the fifteen month period a change has been implemented for the annual datasets are now finalised six months after the end of the reporting period covered (previously this was fifteen months after). This delay is needed:

to allow sufficient recovery time after surgery before post-operative questionnaires are completed and;
to maximise the number of post-operative questionnaires returned.

A further four weeks (approximately) is necessary for data processing, analysis and production and checking of the annual publication.

To add to the value of the PROMs questionnaire data, they are linked routinely with Hospital Episode Statistics (HES) episode-level information. In 2021 significant changes were made to the processing of HES data and its associated data fields which are used to link the PROMs-HES data.

The completion of the development and assurance work in our operational processes needed to update the linkage methodology was longer than expected which has caused a subsequent delay in the timeliness of our PROMS publication series. This provisional publication which normally would have been published in August 2022 has instead been published 9 months later than expected and will be followed by publication of the finalised publication of PROMS data which normally would have been published in February 2023 for the time period in July 2023. This will bring our publication reporting sequence of PROMS data up to date for users of these statistics.

Punctuality

This publication was published on the pre-announced release date.

The interrogation tool based on Microsoft Power BI will be available within one hour of the 9:30am publication release. This is because we have no control over when Microsoft refreshes will occur and cannot risk being in breach of our statistical governance regulations, by anticipating a delay and releasing these tools earlier, only for the delay to be shorter than expected and the data become available earlier than they should.

Accessibility and Clarity

Accessibility is the ease with which users are able to access the data, also reflecting the format in which the data are available and the availability of supporting information. Clarity refers to the quality and sufficiency of the metadata, illustrations and accompanying advice.

The HES dataset used in the PROMs publication has been collected primarily for official administrative purposes. Information about the administrative source and its use for statistical purposes is included in NHS Digital’s Statement of Administrative Sources.

This publication is being made available on the world-wide-web as a combination of web pages and downloadable reports and data files. The publication may be requested in large print or other formats through the NHS Digital’s contact centre: [email protected] (please include ‘PROMs’ in the subject line).

Coherence and Comparability

Coherence is the degree to which data that are derived from different sources or methods, but refer to the same topic, are similar. Comparability is the degree to which data can be compared over time and domain.

Other official statistics published by NHS Digital that report on extracts of HES inpatient data allocate episodes to time periods based on episode end date. PROMs publications, however, use the episode start date to assign records to time periods, as this date more closely represents the date of the PROMs operation.

A small number of HES episodes record more than one eligible PROMs procedure. PROMs publications report on eligible procedures, whereas other publications from HES data report on episodes. In practice, this means that if a patient undergoes both a hip and knee replacement in a single hospital episode, their hospital records this as one episode for both procedures. However, the distinction between the two is made within the PROMs analysis of the linked PROMs questionnaires.

Trade-offs between output quality components

The decision to finalise the dataset around six months after the end of the financial year when the PROMs procedures have taken place is a balance between timeliness and completeness. Patients can complete and return their post-operative questionnaire many months after it has been sent to them and often they are only returned after a series of reminders or duplicate questionnaires are issued. The decision to take six months was taken after an analysis of the time taken by patients to return questionnaires showed that the number being returned after this point and its impact on the calculated metrics was small.

Assessment of user needs and perceptions

To report to feedback and suggestions about this publication, please email our enquiries mailbox at [email protected] (please include ‘PROMs’ in the subject line).

Performance, cost and respondent burden

The PROMs programme invites all NHS-funded patients going through a PROMs-eligible procedure to complete pre- and post-operative questionnaires, which are linked, where possible, to administrative data about their procedure-related health care.

Pre-operative questionnaires may be administered on the day the patient is admitted for treatment or at a pre-assessment clinic beforehand. Paper-based questionnaires are predominantly used, but questionnaires can be delivered electronically. Typically, the questionnaire will take around 5 - 10 minutes to complete. These questionnaires are collected by providers after completion and sent to their data supplier. The data supplier then submits the pre-operative data as record-level xml files to NHS Digital.

Where pre-operative questionnaires link to a HES episode, NHS Digital sends the data supplier a file containing the PROMs Serial Number, the procedure, and the operation date. This lets data suppliers calculate when to send post-operative questionnaires to patients. Post-operative questionnaires should be sent out six months after the procedure date. Data suppliers can send out up to two follow-up mailings to patients that have not returned post-operative questionnaires, encouraging them to do so. Patients return completed post-operative questionnaires to the data supplier using a pre-paid envelope. The data supplier then submits the post-operative data to NHS Digital.

On receipt of the data, NHS Digital performs a number of validation checks and the data becomes available for NHS Digital analyses and publications. NHS Digital links pre-operative questionnaires with administrative data about patients’ hospital stays (for relevant procedures) held in the Hospital Episode Statistics (Admitted Patient Care) dataset.

Confidentiality, transparency and security

The use of PROMs data are covered by consent which specifies what the data can be used for, what other datasets (such as HES) that can link to PROMs and how long the data can be kept. This includes retaining the patient identifiers (NHS number, date of birth and postcode) for no longer than 24 months after completion of the pre-operative questionnaire.

Data are subject to disclosure control before being released. More information about the disclosure control rules can be found in the Footnotes located in CSV Data Pack accompanying this report.

Data are released under the Open Government Licence, which encourages the re-use of our statistics as long as certain conditions are observed.

Last edited: 7 June 2023 3:06 pm