Suppliers guide to transferring records into the National Document Repository

This guidance is for organisations that transfer documents into the NDR, including: 

• scanning providers 
• GP practices 
• integrated care boards (ICBs) 
• commissioning support units (CSUs) 

In this guidance, these organisations are referred to as ‘you’ or as the ‘data transferer’. 

Make sure you read these two documents. This guidance refers to information in both: 

1. Digitisation of Lloyd George records: guidance for general practices in England. This document will be called the ‘Digitisation of Lloyd George records’ in this guidance.   

2. NHS England Dynamic Purchasing System (DPS) for digitisation of Lloyd George records (for GP practices): service specification. This document will be called the ‘service specification’ in this guidance.  

Lloyd George records must be scanned in accordance with the national scanning standard. This standard can be found in the service specification.  

If you have files with filenames or metadata files which do not meet the required format, ask the PRM team for support.  

Files should be organised into folders labelled with the ODS code. Name each folder using the ODS code of the practice where the patients, whose files are stored in the folder, are registered.

Check you are ready to transfer files into the NDR

Records are transferred into the NDR following these steps: 

The PRM team will give you the user connection and server details.  

An SFTP client is a software application that enables users to securely transfer files to a remote server using the Secure File Transfer Protocol (SFTP). It acts as the interface for users to connect to an SFTP server and upload their files. 

You’ll need to choose an SFTP client to connect to the NDR’s AWS Transfer Family server. For information on choosing and using an SFTP client, see the AWS guidance on transferring files over a server endpoint.

The NDR uses SSH (Secure Shell) keys for bulk uploads. 

SSH keys are authentication credentials used in file transfer tools. They help control access and keep data secure. 

If you're a data transferer, you need to: 

• generate SSH keys 
• share the public key with the PRM team by email to [email protected] 

Read the AWS guidance on creating SSH keys.  

Once your access is confirmed and you've shared your public SSH key with the PRM team, you can connect to the AWS Transfer Family using SFTP with your chosen SFTP client: 

sftp://[USERNAME]@[SERVER_NAME].server.transfer.eu-west-2.amazonaws.com 

• Replace [USERNAME] with your assigned username 
• Replace [SERVER_NAME] with the name of your server 

What you can access 

You’ll be given access to an S3 staging bucket. This bucket is: 

• only accessible by you 
• isolated from other S3 buckets in the AWS account 

You cannot transfer data outside of the staging bucket while uploads are in progress. 

Your access will be removed once your transfers are complete.  

Once the connection is working, you can upload to the S3 staging bucket and transfer patient records at the agreed time. 

Virus scanning 

Each file added to the staging bucket is processed by the virus scanner. 

If a file fails the virus scan, it is: 

• removed from the staging bucket 
• put in the quarantine bucket  
• reported to NHS cyber security  

The virus scan failure is recorded in the report. 

Once your files are successfully uploaded to the staging bucket, the PRM team will remove your access and the record ingestion process will begin.  

A scheduled ingestion process moves scanned records from the staging bucket into the NDR. Only records that meet the criteria and pass validation will be ingested.  

Any files that fail validation remain in the staging bucket and may be deleted as soon as a rejection happens. These files will never be kept for longer than 2 weeks.  

To make sure records can be transferred to the National Document Repository (NDR), we’ve put checks in place to validate the files we accept:  

Metadata file requirements 

To make sure records can be transferred to the National Document Repository (NDR), we use a metadata file to validate each upload. 

You must include a metadata file with every record uploaded to the staging bucket. This file helps the system check that your records meet the criteria for ingestion.  

The required fields and format for the metadata file are given in the service specification document.  

Validation steps for Lloyd George records 

To check and validate the Lloyd George records ingested into the repository, the validation Lambda: 

1. References the metadata

• uses the metadata.csv file to begin validation

2. Checks the number of files

• confirms the number of files matches the total stated in the filename. 

3. Validates filenames

• checks each filename follows the required naming convention.  
• identifies any duplicate filenames 

4. Verifies patient details

• makes sure every file within a single patient record has the same: 
  • patient name 
  • date of birth 
  • NHS number 

5. Cross-checks the NHS numbers

• confirms the NHS number in the metadata file matches the one in the filename 
• checks the NHS number is valid 

6. Retrieves and matches to Personal Demographics Service (PDS) data

• the metadata file must match the data in the PDS for NHS number and 2 out of 3 demographic details (first name, last name, previous names and date of birth) 
• gets patient information from the PDS 
• checks that the patient name and date of birth in the filename match the PDS record. 

7. Checks the virus scan status

• looks for the virus scan results in the staging bucket: 
  • if status is ‘clean’, continue to step 8
  • if status is ‘not found’, it sends a message to the SQS queue and retries after 5 minutes

8. Copies the files

• copies validated files to the Lloyd George staging bucket 
• adds a record to the Lloyd George DynamoDB table

9. Cleans up

• deletes the copied files from the staging bucket 

When files are rejected

• a record of the rejection is added to the report database 
• they are temporarily held in the staging bucket, and may be deleted  
• they will not be held for any longer than 2 weeks 

When files are successfully ingested

• they are removed from the staging bucket 
• a record of successful transfer into the NDR is added to the report database 

At the end of the bulk ingestion process, 3 reports are generated: 

• Success report: records which were successfully transferred into the service 

• Rejection report: records which were rejected / not transferred into the service 

• Summary report: the NHS numbers and total number of records transferred into the service for the GP practice

The GP practice will share the rejection report with the data transferer to resolve any record rejections.    

The following table gives an overview of the main reasons for rejections and how to correct them.

The data transferer will upload corrected files to the staging bucket. The PRM team will move the corrected records from the staging bucket into the service from at the next bulk upload ingestion cycle 

If you need further support on managing and correcting record rejections contact the PRM team on [email protected]. 

Once your records have been successfully transferred into the NDR, your paper records or electronic copies can be destroyed. Your ICB can advise you on when to do this, and how to do this safely and securely.  

Further information on record destruction can be found in the digitisation of Lloyd George records guidance.