Interchange data quality user guide

The Interchange data quality report provides organisations who submit CDS data to SUS with near immediate feedback regarding the quality of the data in each interchange. 

This document provides information about how the report is built and distributed, guidance on how to register to receive the report and recommendations for consuming the report in a resilient way.

The intention is that, overtime, the number of data quality checks covered by the report will grow.  To avoid having to update this document on a regular basis, details of the data quality checks that we perform will be available on the SUS Guidance web page.

Each time SUS+ processes an interchange of CDS data it checks to see if the sender of the data has a registered MESH mailbox through which to receive feedback.  If it finds a MESH mailbox registered for the sender, SUS+ assesses the quality of the data submitted and loads the results into the registered mailbox.  The challenges to the records will be returned to the Mesh mailbox associated with the EDI address of the sender.  This means that the sender (eg provider, third party) will receive the report.

Where an interchange contains data for multiple providers, SUS+ will generate and distribute Data Quality reports to the sender of the providers that appear in the interchange. 

At present the data quality report is only available to the sender of the data.  In the fullness of time the intention is that the same processing will be used to feed data to a revised national data quality dashboard system and possibly to provide data quality data to commissioners as part of their data extracts.

The intention of the Data Quality report is to support senders in identifying and acting upon potential data quality issues in the data that they supply to SUS+.  As such the report may contain patient identifiers and should be treated as Patient Confidential Data.  The  provider organisation is responsible for ensuring that suitable governance including appropriate approvals by the provider’s Caldicott Guardian are in place to handle this data.

The standard transport mechanism adopted by NHS England for the movement of Patient Confidential Data is the Message Exchange for Social Care and Health (MESH).   

In summary, or the purposes of the data quality report, MESH operates in a similar way to a mail system between NHS England and providers. 

• each provider registers for a MESH mailbox (unless they already have one) with the MESH team.
• once the MESH mailbox is set up the provider raises a helpdesk call with the National Service Desk (log in required), or call 0300 303 5035) to ask the SUS Service team to start sending data quality reports.  This request needs to provide the MESH mailbox identifier and the cds interchange sender id (EDI address) for which reports should be sent
• NHS England will complete due diligence to ensure that the MESH mailbox and cds interchange sender id (EDI address) have a bona fide relationship
• the MESH mailbox and cds interchange sender id (EDI address) will then be added to the SUS+ reference data and SUS+ will begin depositing data quality reports in the mailbox with a workflow id of  SUS_CDS_FEEDBACK

The preferred operating pattern for the MESH system is to have one mailbox per organisation and to use WorkflowID to distinguish types of data within the mailbox.  The mailbox that you wish to use to receive data quality reports must be enabled to receive data.

There may be some confusion where XML middleware suppliers have set up MESH mailboxes for the purposes of transmitting CDS interchanges to the SUS system.  In some cases the middleware supplier may have registered a MESH mailbox in the name of your organisation.  It is possible to use the same mailbox to both send CDS Interchanges to the SUS system and to receive Interchange DQ reports from SUS.  For this to work the MESH mailbox must be terminated within the hospital providers infrastructure. 

If a MESH mailbox is registered for the provider but is not terminated within the provider’s infrastructure the preferred way forward is to obtain a new mailbox that can be installed at the provider’s site without disrupting the flow of CDS interchanges.

Files that are put in a MESH mailbox are retained for 5 days and then deleted.  SUS+ will not be able to re-generate Interchange Data Quality reports.

The file that is delivered is a zip of several constituent files, a JSON file and one or more CSV files.

The detailed format of all of these files is published alongside this document on the SUS webpages.

Files will only be generated to reflect data quality issues identified.  If no data quality issues are identified no files will be generated.

The JSON file contains all of the data quality issues found in the interchange.  JSON is a data transmission protocol that supports iterative automated processing with a much reduced risk of changes to the files structure breaking local ETL processes.

In addition to the JSON format the data is also presented in CSV format.  The CSV files are split to reflect the data items required for the different classes of data quality issue.  At the time of writing there are three CSVs generated:

• CQUIN
• PCD Leakage issues
• derivation differences

Data quality Issues (present, default, valid)

• primary diagnosis (Icd)
• emergency care diagnosis (Snomed Ct)
• site code (of treatment)
• Primary Procedure (Opcs)
• postcode

Each CSV record contains a standard set of record identifiers to assist the sender to locate the record that contains the data quality issue.

It is the nature of CSV files that if additional fields are added to the format the local ETL process must be amended to accommodate the new field (even if the new field is of no interest to the user).  For this reason, in the event that a new field must be added to a CSV file both the original and the revised format of the data will be included in the ZIP file. Announcements will be made via the What’s New page, Show and Tells and the SUS User Group.  Both the old and the new format files will be made available for a limited period of time to allow senders who have chosen to use the CSV format to amend their local processes.  The old format will then be deprecated and only the new format will be provided.