Guidance for submitting participant data (cohort file)

NHS England’s Participant Validation Engine enables users to upload their cohort of participants to allow for validation, tracing and retention of those participants’ details for future linkage to NHS England datasets listed within a particular Data Sharing Agreement (DSA).

The method of submitting a cohort file is via an upload to NHS England’s Secure Electronic File Transfer (SEFT).   

Once a file has been uploaded, the participants are initially validated to ensure that the data items provided are in the correct format and listed within the DSA.

If the file is successfully validated the participants’ details will be matched to the Master Person Service (MPS) held by NHS England. Once matched, the participants’ details will be stored in a cohort table for linkage to NHS England datasets.

Users receive email notifications when their file passes or fails validation, and receive a report on what validation failures occurred and which participants failed matching.

Check this list of reminders before you submit your cohort file.

Note the following points before you create your cohort file:

1. Once your organisation’s Data Sharing Agreement (DSA) is active, the Secure Electronic File Transfer (SEFT) team will email the individual within your team identified as the 'Cohort Submitter' or ‘Participant Data Provider’ to inform them that the cohort associated with your DSA can be uploaded to SEFT. Guidance for using SEFT will be provided within the email and can also be found at the end of this guidance.

2. Cohort files must be submitted as ‘Comma Separated Value’ (.CSV) only. Ensure you do not include any commas in the values provided in the rows. We cannot support any other file formats, such as files submitted as .XLSX.

3. The cohort filename must contain the first part of your DSA number, followed by free text of your choosing, such as NIC-123456[anyothertext].csv.

As an example: ‘NIC-123456_October Upload.csv’.

Where you have multiple agreements, ensure you have selected the correct NIC number.

4. Ensure you have read through the SEFT guidance and are aware of which folders to use for uploading your cohort submission and for retrieving your data.

5. Check that all data items that your Data Sharing Agreement allows you to submit for your participants have been collated.

1. The cohort submission template is provided as an accompanying file to this guidance. It serves as an example as to what a correctly formatted cohort file should look like. 

2. Remove the example data in the template and add your participants’ details. Alternatively create your own CSV cohort file using the template as a guide. Include the headers as per the template.

3. The template contains column headers and within those columns are examples of the data that can be entered. 

4. Although the template has most of its fields completed, you may not be able to submit all of these fields. To determine which fields can be submitted in line with your Data Sharing Agreement see ‘Checking against the Data Sharing Agreement’.

5. Save the CSV cohort file with a filename as per point 3 in ’Preliminary checks’. Make sure you include the first part of your DSA number.

The file formatting table below provides guidance on the requirements of the cohort file(s). These must be adhered to, as this ensures the cohort can progress successfully through the cohort validation process. 

1. We strongly advise that you submit as many identifiers as possible (agreed in the DSA), one of these needs to be DOB in your cohort.

The best quality tracing is provided when NHS number and DOB are available.

Other combinations of identifiers (such as DOB, NHS number and Family name) will also generate a trace but the quality will depend on the data (see ‘Tracing/matching process’ for more details). Too little identifier information could result in your file submission being accepted, but your cohort may not be traced.

2. The purpose of the ‘Status’ field is to confirm whether the participant should be added to or deleted from the cohort. A file must contain ADD or DELETE in the 'Status' column for every row of data. You must not submit the same participant with a DELETE and ADD status in the same file.

3. The file must contain a ‘UNIQUE_REFERENCE’ for each row of data (so if you DELETE and ADD the same record in a file it will fail because it would have the same ‘UNIQUE_REFERENCE’).

4. There is a maximum cohort size of 1 million participants/rows in the cohort file. If you submit more than 1 million your file will be rejected. 

5. PERSON_MIN_DATE specifies the earliest date from which you would like data about the participant and PERSON_MAX_DATE specifies the latest date from which you would like data about the participant.

You must complete the ‘Cohort Submission Template’ according to one of the options below.

If date minimisation required:

If your data sharing agreement requires your extracts to be minimised by date, each participant needs at least one date filled in (either Person_Min_Date or Person_Max_Date) to be included in the extract. You can either:

• fill in just Person_Min_Date - the date entered cannot be later than the ‘PERSON MAX DATE’ if you are using both fields
• fill in both Person_Min_Date and Person_Max_Date

If for any reason, there are participants without any dates, these will automatically be excluded from the extract.

Date minimisation not required:

In the case that you do not require date minimisation you should not include dates for any participant in the ‘Cohort Submission Template’.

If mandatory data items are not provided then the file will either be rejected or it will not be possible to match any participants.

When uploading your cohort, please refer to Annex B ‘Additional Technical Information’ in your Data Sharing Agreement (DSA) to ensure the identifiers (such as Family name, DOB, NHS number) you will upload in your file are within the scope of your DSA. 

In the example above, the Data Sharing Agreement allows for the 7 data items listed to be included in the cohort file. You may choose not to submit one or more of those data items but the matching process may be adversely affected, depending on the data item(s) in question. This may result in participants failing to trace and therefore not being included in your cohort list.

In this example only 3 data items can be provided in line with the DSA. If, for example, you do include something else, such as Postcode, this is not allowed under the DSA and the system would remove the Postcode data and it would not be used in the matching.

1. To upload your cohort, you must access your Cohort Submitter SEFT Account. This account contains your initials at the end of your User ID - such as NIC-12345-ABCDE_XX. Our system will only accept files dropped into this location.

2. The system will validate your submitted file. If the file fails any checks you will receive an email advising of the failure and directing you to the submitter’s SEFT account to view the associated error file named ‘Participant-Data-Validation-Errors'.

Note that you may have up to 2% of the participants fail validation checks for the file to continue to the matching stage. If, however, 2% or more of the participants fail validation then the whole file will be rejected. You will need to rectify the errors and submit a new file. On average it will take 1 hour for validation/matching to take place, but this can take up to 12 hours, depending on the size of the cohort and data items supplied for matching.

3. After your file has been successfully validated and participants matched, you will receive a link directing you to download a ‘Participant-Data-Summary-File’. This file will provide you with key information about the submission. This will be placed in the download folder of the Cohort Submitter’s SEFT account in the form of a multi worksheet tab Excel file. 

4. Check for outstanding validation errors in the ‘Errors’ tab of the summary file. You can correct those records and resubmit another file. 

5 To add new participants to an existing cohort at a later date, repeat the process above and submit a new file with new participants marked with a status of ADD.

6. To amend the details of a participant in your cohort that has already been uploaded , submit a new file that includes the updated participant’s details and mark their status as ADD. The existing record will then be updated with the new values supplied. You can do this as a standalone file or include these amended details in a file alongside new participants to be added to the cohort.

7. If you are the Data Recipient of the extracts linked to your cohort, the SEFT account that you will need to access to download your data is the one without your initials at the end of your User ID - such as NIC-12345-ABCDE. 

After successful validation we will send the list of participants to be ‘Traced’ or matched against our Master Person Service (MPS). Submit as much identifier information as your DSA allows to give the maximum opportunity for trace success.

MPS methods to trace participants are listed in this table:

Cross check

Only DOB and NHS number are mandatory for this trace step, but if there is not an exact match then Postcode and name information can be used.

Alphanumeric

If there is no match at ‘Cross check’ trace, then we will run an ‘Alphanumeric’ trace.

The minimum required fields for an Alphanumeric trace are DOB, Family name and Gender, but the fields Given name and Postcode can also be used.

Algorithmic

Algorithmic trace uses the supplied identifiers to perform further traces. In this type of tracing method a match needs to be above a set threshold to be returned as a match. 

For more information please refer to the Master Person Service (MPS) user guidance.

If you have any queries or require further clarification, email [email protected] and include the words ‘Participant Validation Engine’ in the subject line.