NICE Technology Appraisals in the NHS in England (Innovation Scorecard), To December 2022

The Innovation Scorecard is an action taken from The Department of Health’s (DH) paper ‘Innovation, Health and Wealth’ (Dec 2011).

Further information and report can be found at: https://www.gov.uk/government/news/accelerating-adoption-of-innovation-in-the-nhs

More recently, the Accelerated Access Review (AAR) makes a number of references to the development of the Scorecard in particular its role in measuring the uptake of medicines and potential to measure other technologies and in future, the Innovation Scorecard should be the single source of information on the use of innovation in the NHS.

Further details and final report can be found at: https://www.gov.uk/government/organisations/accelerated-access-review

The Innovation Scorecard aims to support and drive NHS compliance with NICE Technology Appraisals (TAs) and was designed to help the NHS identify variation which can then be justified, challenged or acted upon.

It is published by NHS England on behalf of the Office for Life Sciences. The work is informed by collaborative working with colleagues from: the Association of the British Healthcare Industries (ABHI), Cabinet Office, DH, the NHS, NHS England, NICE, Office for Life Sciences, and the pharmaceutical industry. Recent developments in 2021 included:

• Introduction of Secondary Care Medicines Data (SCMD) as supplied by Rx-Info and published by the NHS BSA.
• All medicines are now reported at organisation, regional and national level.
• Addition of new groupings and revision of existing medicine groupings to align with the 5 High Health Gain policy areas for NHS England and NHS Improvement.

Technology appraisals  are recommendations on the use of new and existing medicines and treatments within the NHS, including medicines, medical devices, diagnostic techniques, surgical procedures and health promotion activities.

Some medicines and treatments may be covered by more than one technology appraisal. Each technology appraisal may contain more than one recommendation. NICE classify their recommendations into four categories:

• Recommended - the medicine or treatment is recommended for use:
  • In line with the marketing authorisation from the European Medicines Agency (EMA) or from the Medicines and Healthcare Products Regulatory Agency (MHRA) or
  • In line with how it is used in clinical practice in the NHS
  • or both
• Optimised - the recommendations have a material effect on the use of a medicine or treatment, and it is recommended for a smaller subset of patients than originally stated by the marketing authorisation. This test of materiality takes into account advice from clinical experts on the anticipated use of the technology in routine clinical practice. In some instances, an optimised recommendation is made because the committee considers that a medicine or technology is only a cost-effective treatment option for a specific group of people; for example in people who are resistant to or cannot tolerate other medicines.
• Only in research - The medicine or treatment is recommended for use only in the context of a research study, for example, a clinical trial. Often, particularly in the case of promising new technologies, sufficient clinical evidence has not been collected at the time of the appraisal and so the Appraisal Committee is unable to recommend the technology for use in the NHS until further evidence on its effectiveness is available for re-appraisal.
• Not recommended - the medicine or treatment is not recommended. In most instances, a technology will not be recommended if there is a lack of evidence for its clinical effectiveness or if the technology is not considered to be a cost-effective use of NHS resources, compared with current NHS practice.

The technologies included in an appraisal may not be the only treatment for the condition recommended in NICE guidance, or otherwise available in the NHS. Therefore, if a NICE technology appraisal recommends use of a technology, it is as an option for the treatment of a disease or condition. This means that the technology should be available for a patient who meets the clinical criteria set out in the guidance, subject to the clinical judgement of the treating clinician. The NHS must provide funding and resources when the clinician concludes and the patient agrees that the recommended technology is the most appropriate to use, based on a discussion of all available treatments.

Some medicines and medical technologies previously covered by TAs are now included in clinical guidelines or in best practice guidelines.

NICE Technology Appraisals for Medicines

Further guidance on NICE TAs and those technologies that are considered by NICE guidance other than the TA process is available at: http://www.nice.org.uk/About/What-we-do/Our-Programmes/NICE-guidance/NICE-technology-appraisal-guidance

The Scorecard includes a range of data from a number of sources which includes:

• English Prescribing Dataset (EPD) published by the NHS Business Services Authority
• Secondary Care Medicines Data (SCMD) as supplied by Rx-Info and published by the NHS Business Services Authority
• Hospital Episode Statistics (HES) and Quality Outcomes Framework (QOF) data from NHS England
• Mid-year population estimates from Office for National Statistics
• Defined Daily Doses (DDD) from World Health Organisation (WHO) Collaborating Centre for Drug Statistics Methodology
• Actual Daily Doses (ADD) from NHS England

The Innovation Scorecard is published by NHS England with the first release in January 2013. Initially published on a quarterly basis the publication has been bi-annual since October 2019.

All releases of the Innovation Scorecard are available online and can be accessed at: https://digital.nhs.uk/data-and-information/publications/statistical/nice-technology-appraisals-in-the-nhs-in-england-innovation-scorecard

A process for selecting the medicines and medical technologies for inclusion was developed to ensure a systematic approach for inclusion within the scorecard. The Strategic Metrics Group and Technical Working Groups agreed a series of inclusion and exclusion criteria, and a rationale for any medicines excluded.

Inclusion criteria for medicines

• The medicine has a TA published from 2018 onwards, which is positively appraised by NICE. The date will move forward a year as part of an annual cycle starting with the October publication.
• TAs published prior to 2018, which are deemed to be innovative will be considered for inclusion.
• The medicine has a TA prior to 2018 but is of interest to the NHS. This may include those medicines which do not have widespread use across the whole service, for example, where there is unwarranted variation or where there is known significant therapeutic value.
• The cut-off date for the inclusion of TA’s over the next year is:

Exclusion criteria for medicines

• Where a significant proportion of data is not currently available, for example, those medicines which can be purchased over the counter, or dispensed via routes other than hospital or community pharmacies, etc.
• Where the medicine is not new and therefore, not considered to be innovative as the treatment would be well established already.
• The patient population is small and therefore there is a risk of identification when presenting the data
• Where inclusion of the data available is misleading or of limited value to the NHS

Inclusion criteria for medical devices and diagnostics

• Medical devices and diagnostics are currently excluded from the scorecard.

Inclusion criteria for the estimates report

The inclusion of estimates to show expected uptake versus actual uptake were first included in the scorecard in the September 2014 publication. The estimate approach requires its own criteria to determine which medicines should be selected for that methodology. The “Use of NICE appraised medicines in the NHS in England” report provides historical selection criteria for those medicines included in the estimate approach.

Yes, data volumes can vary between publications due to an issue known as backtracking in the secondary care medicines dataset. Backtracking is a process where pharmacy stock control systems alter historical volumes based on the stock issued. On this basis data should be treated as provisional. Users should always use the figures in the latest publication to ensure they are the most up to date figures available. Further information is provided in the Background Quality Report.

The Innovation, Health and Wealth Strategy included a recommendation that all NHS organisations should publish information which sets out which NICE technology appraisals are included in their local formularies by 1 April 2013.

Formularies are available on individual Trust websites. Please contact the hospitals directly with any queries relating to the formularies.

Compliance is subject to interpretation, and the measurement of compliance is not simple. An important aspect of this is consideration of the circumstances under which a medicine is administered or a medical technology used, and whether this aligns with the precise recommendations within the NICE guidance. The ”Innovation, Health and Wealth” report sets out actions for the NICE compliance regime, so readers are advised to view this report.

The data presented here is insufficient to determine if an organisation is or is not compliant with NICE guidance.

The Innovation Scorecard should not be used for performance management. It is intended to identify where variation in the adoption of TAs may exist between healthcare organisations and for these organisations to understand, be challenged and explain any variation. This is based on the assumption that reduced variation will result in improved quality of care.

It is important to be aware that observed use of a medicine or technology may differ for a range of reasons and should not be assumed to definitely indicate either ‘under’ or ‘over’ prescribing or implementation. It is recommended that any exploration of the variation uses a structure such as the pyramid of investigation for special cause variation^[1] (see diagram below).

More likely explanations are listed towards the bottom of the pyramid, and so it is suggested that these are investigated first. The data presented in this publication requires careful interpretation and should be used in conjunction with other indicators and information from other sources that together form an holistic view of use of medicines and medical technologies. For example, a medicine may not be the only treatment for a particular condition recommended in NICE guidance, or otherwise available in the NHS. Medicines are generally recommended as options for treatments. Other options may include non-appraised medicines or other appraised medicines. Therefore variation in the use of individual medicines would be expected.

Assessment of compliance cannot be made using currently available data.

Indicators of variation may be more appropriate for sub ICB locations than for Trusts, as medicine use in primary care is more likely to be for the management of common long term conditions. Some appraised medicines are used wholly or mainly in tertiary care hospitals; others are more likely to be used in less specialist hospitals. A medicine which is used for a very rare condition (tens of doses per sub ICB location or Trust per quarter) would normally be expected to show wider variation between organisations.

Variation in the use of medicines or medical technologies between NHS organisations may be due to a number of valid factors including:

• Natural variation in populations, both in demographic profile and disease prevalence
• Variation in presentation to the NHS by the relevant populations
• Variation in choice of preferred treatment option at the local level
• Variation in the use of alternative products or procedures
• Differences in the extent to which local utilisation information is available
• Differences in services provided between organisations, for example differences in the extent to which a service is provided by primary or secondary care
• Difference in levels of informed patient dissent to intervention

A detailed examination of the reasons for variation for individual technologies is beyond the scope of this report.

The publications show use of medicines over time. Data on the Innovation Scorecard is presented by calendar quarter which covers up to 5 years of data at the National (England) level and up to 2 years of data at lower NHS Organisation levels. 

It is not appropriate to draw conclusions from direct comparisons of the data in each of the scorecards, as many factors may be responsible for any apparent variation. These include: 

Clinical and demographic issues:

• differences in populations served, local prevalence of disease, services provided and changes in clinical practice.

Data issues:

• The NHS reorganisation has led to changes in the individual hospitals included in some Hospital Trusts, and there have been changes in the general practices included in some ICBs.
• Data which has been standardised by Finished Consultant Episode (FCE) days of hospital care (introduced in March 2014 publication) cannot be compared with data standardised by FCE bed days in previous publications.

Data in the Scorecard are presented in several ways – based on available data at specific NHS organisation levels.

Data for medicine utilisation is reported at National (England), NHS Region level, ICB, sub ICB locations and Trust organisation level where appropriate. 

The web platform tool allows comparison of medicine use at sub ICB location level and medicines purchased at NHS Trust level. Additionally, maps presented on the tool show use based on quintile proportions. Data for all organisation levels are provided in the CSV files.

Data is provided at NHS Hospital Trust level. The majority of trusts are made up of a number of local hospitals.

No, we are legally obliged to avoid disclosure of potential identification of individual patients by implementing suppression handling where required.

NHS organisations differ widely in the populations they serve and so data which does not take this into account can be misleading.

For ICB, Region and national data the number of resident patients has been used to take account of the size of population served. 

For hospital Trusts data the number of Finished Consultant Episode (FCE) days of hospital care for the time period under consideration (taken from the Hospital Episode Statistics data) has been used to standardise the data. The values vary significantly, with more specialist hospitals, for example, Moorfields Eye Hospital NHS Foundation Trust reporting fewer than 40,000 days of hospital care per year, whereas larger Trusts such as Bart’s Health NHS Trust report over 800,000 days of hospital care per year.

Trust level data should not be compared with the national, Region or ICB data.

This scorecard covers some highly specialised medicines and technologies so differences in use across organisations are to be expected.

Medicines presented as grouped medicines are also standardised either by resident populations, FCE days of hospital care or by national disease registers i.e. QOF. 

Data on the number of patients diagnosed with a particular condition, eligible for a specific intervention, or receiving an intervention may be recorded locally but are not reported centrally by the NHS; therefore the eligible population for any TA can be problematic to estimate. To develop estimates of the eligible population, information is required to refine population numbers to the particular circumstances where the medicine is recommended by NICE; from overall disease prevalence to the proportion of patients within a particular stage of a disease and then to the particular indication recommended by NICE. In some cases further details are also required, for example, the proportion of patients likely to discontinue treatment or choose alternative treatments.

The NHS currently does not have any robust data reporting or collection process which gathers data which can be used to measure compliance with NICE TAs.

The data sets used in this publication were developed for other specific purposes and are shown here to provide an indication of the implementation of NICE appraised technologies. Therefore users of the Scorecard should be familiar with the limitations of the data when used for this purpose.

Patients can receive medicines from the NHS by a variety of routes. The most common is to receive a prescription from their general practitioner or other community based prescriber, which is then dispensed by a community pharmacy. Hospitals also administer and supply medicines direct to patients.

However there are a number of other ways patients can receive medicines which may limit the coverage of centrally available data, and result in under-reporting of utilisation. Some of the main areas are described below:

• Medicines are supplied directly to a patient without a prescription being issued or the supply being recorded in a system which supports central collection of data. For example:
  • Some medicines (e.g. for smoking cessation or contraception) are supplied directly to patients from specific clinics, and this use does not appear in any central data collection.
  • Some mental health services or clinics purchase medicines and supply or administer them directly to patients without using a prescription or order in such a way that relevant information is not routinely captured in national data sets.
  • Some urgent supplies of medicines are supplied directly to patients and this use does not appear in any central data collection, for example, supplies provided by ‘out of hours’ services.
• There are some known gaps in the central data collections, including:
  • Homecare: Supplies of medicines are delivered directly to the patient’s home, and may not be routinely recorded on hospital pharmacy systems. This service is generally contracted to a commercial supplier by hospital Trusts. The homecare market is a significant proportion of total medicines use in secondary care.
  • Some medicines are provided by an aseptic unit where the medicine has been individually prepared for administration to a particular patient. This data may not routinely appear in the hospital pharmacy systems.
  • Outsourced dispensing: Hospitals are increasingly using local commercial suppliers to dispense medicines to patients attending out-patient clinics and these supplies may not be routinely included in hospital pharmacy systems.
  • Hospitals can contract with the private sector to provide specific care or administer specific medicines, which are not recorded in the data from the commissioning hospital. This is most commonly seen with particular medicines, such as ranibizumab, where specific monitoring is not required.

Some medicines are included in multiple TAs; and some medicines are licensed for other indications than those covered by TAs. The available datasets do not allow identification of utilisation by indication, therefore it is not possible to report use for a particular condition.

In January 2016, medicine groupings were introduced to the Innovation Scorecard. The medicine grouping methodologies have been developed by analysts and pharmacists at NHS England, ABPI, OHE, NICE and OLS. All methods have been reviewed and agreed by the technical working group for the Innovation Scorecard. Comments and feedback on this section of the publication would be appreciated. Please email any feedback to [email protected].

These groupings have been developed to show combined use of medicines that are options for treatment of a specific condition, and where a TA covers more than one medicine for the same indication or two or more TAs cover the same specific condition. It is more informative to compare uptake of combined options for treatment than only showing uptake of the individual medicines. These groups only contain those medicines with a TA. Other options for treatment may be available but are not reported in the Innovation Scorecard.

Data for the medicine groupings is not comparable to the data published in previous Innovation Scorecard publications.

Actual Daily Dose (ADD) is a new prescribing measure developed for and introduced to the Innovation Scorecard in January 2016. They have been developed where current prescribing measures, such as DDDs, are not available or representative of prescribing practices in England.

• Unlike DDDs that use a value for a drug (chemical substance), ADDs use a unique value for each presentation based on the actual dose likely to be taken.
• ADDs assign a unique value for each presentation of a drug based on units (tablets, capsules, patches etc.) and the recommended frequency of daily use (e.g. one a day, three times a day).
• The general assumption being that each dose is equal to a unit of each preparation.
• The advantage of ADD is that a measure can be created that matches the prescribing patterns when a DDD has not been calculated; was created for a different indication (e.g. a DDD for DOACs is based on utilisation in secondary care, but different doses of medicines are used in primary care that are not covered by the DDD); the dose is variable depending on age or a dose and hence presentation is specific to an indication.
• Therefore, different presentations of the same chemical are likely to have different ADD values whereas a single DDD value for a chemical substance is usually applied to all presentations of the substance.
• For this publication, the ADDs have been created using each presentation strength of the medicines linked with licenced dose regimens and NICE guidance.

This report includes medicines where it has been possible to estimate the number of patients predicted to be treated with the medicines and comparing that to the observed use. The report is produced and written by NICE. This work is complex due to difficulties in deriving estimates of the number of patients expected to be treated and obtaining appropriate utilisation data for specific medicines and indications. Medicines may have multiple indications (some NICE appraised and some not) and there may be a number of medicines with the same indication. A TA generally recommends a medicine as a treatment option for consideration alongside other options. There are known gaps in medicine utilisation data (such as supplies made directly to patients via the homecare route or by outsourced dispensing).

For January 2016, the estimate report was reformatted to separate the estimates and measures of uptake from the more detailed methodology used to calculate the estimates. The methodology and reference data lists are now provided in a separate annex in the second half of the estimates report. Feedback on these changes made to the structure of the report and the changes to the methodology and the introduction of ranges is welcome. Any comments can be sent to [email protected]

The Government Statistical Service (GSS) Methodological Committee recommended that estimates should include a range to provide an indication of the level of uncertainty in the data. For this publication all estimates now include a range:

Full details of the calculations are available on Appendix 1: Estimate Calculations within the estimate report.

This report was originally based on the ‘Use of NICE Appraised Medicines in the NHS in England 2012’ report published in January 2014. This publication is a result of collaborative work between NHS Digital and NICE, with contributions from the pharmaceutical industry, DH and the NHS. The report was commissioned by DH, to meet their commitments in the Pharmaceutical Price Regulation Scheme agreements (PPRS 2009 and PPRS 2014). The report can be found at: https://webarchive.nationalarchives.gov.uk/20180328130852tf_/http:/content.digital.nhs.uk/catalogue/PUB13413/

Questions regarding policy and strategy behind the Innovation Scorecard should be sent to: [email protected]

Please quote ‘Innovation Scorecard’ in correspondence.

We welcome comments and suggestions to improve this publication. Please use the associated feedback form.

Alternatively, please send them to our contact centre at [email protected], quoting ‘Innovation Scorecard’ in the correspondence.

Please send them to our contact centre at [email protected], quoting ‘Innovation Scorecard’ in the correspondence.