Delta extracts

Delta extracts allow you to set a baseline and to then request that you receive only new or amended records in subsequent extracts. Delta extracts are completed more quickly than a full extract for the same period and they return fewer records making them more efficient both for the central system and on local ETL processes.

At present delta extracts are only supported for accident and emergency, emergency care and outpatient data sets. The admitted patient care data set is not yet supported.

Any outpatient A&E or Emergency Care Data Set (ECDS) extract can be selected to act as a baseline for a set of delta extracts. You simply set up and run your extract as normal. When the results appear in the portal you will see a delta triangle next to any extract that the system could generate a delta extract for.

When you are ready to see new, amended or deleted records for your baseline click the ‘delta’ symbol and the system will generate files with the same parameters you originally requested but only containing new or amended records.

In addition to your normal data files you will see a Delta file. This file contains a list of Generated Unique IDs (GUIDs) for all records in the baseline file that have been added, amended or removed. Your ETL should first delete all of the records associated with the GUIDs in the Delta file and then add the records in the data files.

You will notice that there is a delta symbol next to the delta extracts themselves. This allows you to run ‘deltas on deltas’ and is the most efficient way to use this facility. Instead of going back to the baseline every time you want to see changes (and getting a list of all changes since the baseline was executed) with deltas on deltas you only see the changes since the last delta. You still see all of the changes and the file sizes you have to process are kept to a minimum.

We will only be giving you updated records when the record has been resubmitted by the hospital provider. Most of the time this will be all of the changes however very occasionally there will be changes to reference data that will impact the data on your extracts. In these cases you will have to re-run the full baseline to make sure that you pick up all of the data and then run subsequent deltas on the new baseline.

Some example changes that would require a new baseline are:

• changing the list of legally restricted codes
• changing the list of valid SNOMED codes for ECDS
• changes to the list of organisations subscribed to a DSCRO

If we make a change that means that all delta extracts should be re-baselined then we will let everyone know via show and tells and what’s new. However changes that only impact your organisation are your responsibility. 

Typically we would not expect the running of the baseline jobs to be time critical. This is because once the baseline has been run and consumed it is the top up delta extracts that will be run at critical times to bring your data up to date.

We would suggest that jobs to establish a baseline or to establish a new baseline be run in the quieter times of the month outside of the inclusion period (5 days either side of the published inclusion date). This will guarantee that the bulk of the data that you require will be processed and delivered as quickly as possible ready for your busy processing periods.

Delta extracts can then be run as and when you require. These will be much smaller datasets that will be delivered more quickly by SUS and which should more rapidly load into local systems.

Users can track the progress of submitted interchanges using the interchange tracking tab. All times displayed on tracker are in Coordinated Universal Time (UTC).