skip to primary navigation skip to content

Improving Improvement

A toolkit for Engineering Better Care

 

Collect the Evidence

The evidence describes the information in support of solution(s) within a system and the means used to measure the performance of the system.

Contents

 

Introduction

The need to Collect the Evidence is the fourth of the seven strands in the improvement model. It underlines the importance of collecting the evidence that the system has been improved, following understanding the system, defining the problem and developing the solution. As a result, it is expected that such a definition will be developed in the intermediate stages of the improvement process and revised, as appropriate, as the process develops.

Purpose

Collecting the evidence has particular importance to systems improvement: defining the measures required to evaluate the performance of the system of interest; undertaking a range of evaluation activities to evaluate effectiveness, patient safety and patient experience; and synthesising the information to evidence the extent of the improvement to the system.

Activities

The process of collecting the evidence may include a wide range of activities including, but not limited to: Define Measures, Agree Quality Targets, Review Effectiveness, Review Safety, Review Experience and Synthesise Evidence.

Tools

The practice of collecting the evidence may draw on a wide range of tools including, but not limited to: Exclusion Audit, Expert Review, User Trials, Root Cause Analysis, Failure Mode and Effects Analysis, Fault Tree Analysis, Hazard and Operability Analysis, Structured What-if Technique and Risk Matrix.

Getting Started

Collecting the evidence is key to the successful evaluation of an improvement project. It is particularly important to collect the evidence starting early in the improvement process and to update the description as necessary during the remainder of the process.

The Understand, Design, Deliver and Sustain stages of an improvement process were previously described within the Introduction section, where each stage will likely comprise a preliminary activity, followed by a number of stage activities.

For the Collect the Evidence strand of the Understand phase, the preliminary activity is entitled Identify the Big Wins. For the Design, Delivery and Sustain stages, the evidence should be further developed into a full account of the effectiveness (or otherwise) of the implemented solution(s).

Identify the Big Wins

This preliminary activity encourages people to identify the big wins and justify the resources required to make the system changes proposed and deliver a description of the most likely outcome(s) from delivering the proposed solution(s) for improving the system.

Worksheets: Improvement Questions, Improvement Stakeholders, Improvement Strands Canvas, Improvement Plan

Top tips:

  • Quantify how many patients could benefit from the changes
  • Identify the potential benefits, costs and risks for the stakeholders
  • Consider the alignment of risks and benefits for each stakeholder

Following on from this preliminary activity, the remaining stage activities are: Define Measures, Agree Quality Targets, Review Effectiveness, Review Safety, Review Experiences and Synthesise Evidence. Each of these are now described in turn, together with the tools that may be used to support them.

Activities

Define Measures

The translation of the core themes for improvement into the definition of appropriate and robust measures of success

Purpose: to define performance measures to evidence the successful delivery of a measurably better system

Inputs: Define Requirements, Agree Quality Targets, Review Effectiveness, Review Safety, Review Experience

Tools: tbd

Outputs: a clear definition of the performance measures required to evidence success

Top tips:

  • Establish a clear rationale for the criteria for success
  • Identify performance measures necessary to demonstrate success
  • Align evaluation activities to the measures and criteria

Agree Quality Targets

The translation of the core themes for improvement into the definition of appropriate and robust quality targets to ensure success

Purpose: to define service quality targets to ensure the successful delivery of a measurably better system

Inputs: Define Requirements, Define Measures, Review Effectiveness, Review Safety, Review Experience

Tools: Literature Review, Soft Systems Method, Causal Loop Diagram, Entity Relationship Diagram, Swimlane Diagram, Dependency Structure Matrix

Outputs: a clear definition of the quality targets required to ensure success

Top tips:

  • Define appropriate targets for clinical and cost effectiveness
  • Define appropriate targets for patient safety
  • Define appropriate targets for patient experience

Review Effectiveness

The systematic review of new concepts, identifying their strengths and weaknesses in meeting the prioritised stakeholder needs

Purpose: to identify clinical and cost effectiveness risks in the adoption of new concepts by their intended users

Inputs: Define Requirements, Filter Ideas, Develop Concepts, Make Models, Define Measures, Agree Quality Targets, Review Experience, Review Safety

Tools: Exclusion Audit, Expert Review, User Trials

Outputs: a list of the strengths and weaknesses of the new concepts in meeting the stakeholder needs

Top tips:

  • Ensure concepts meet the demands in the requirements
  • Check if concepts meet the wishes in the requirements
  • Note requirements that relate to different stakeholders

Review Safety

The systematic review of the safety of the system, identifying hazards and the subsequent risks they pose to users

Purpose: to identify safety concerns in the adoption of new concepts by their intended users

Inputs: Define Requirements, Filter Ideas, Develop Concepts, Make Models, Define Measures, Agree Quality Targets, Review Experience, Review Effectiveness

Tools: Root Cause Analysis, Failure Mode and Effects Analysis, Fault Tree Analysis, Hazards and Operability Analysis, Structured What-if Technique, Risk Matrix

Outputs: a list of the hazards present within the proposed system and the safety risk they pose to users

Top tips:

  • List the hazards presented to users along the care journey
  • Estimate the likelihood of harm posed by these hazards
  • Identify the likely extent of harm posed by these hazards

Review Experience

The systematic review of the demands made by the system on patients and other stakeholders, and their corresponding capability to respond

Purpose: to identify potential barriers to the adoption of new concepts by their intended users

Inputs: Define Requirements, Filter Ideas, Develop Concepts, Make Models, Define Measures, Agree Quality Targets, Review Safety, Review Effectiveness

Tools: Exclusion Audit, Expert Review, User Trials

Worksheets: Improvement Design Wall

Outputs: a list of accessibility issues within the proposed system and the challenge they pose to users

Top tips:

  • Understand the demands made of users along the care journey
  • Estimate the users’ capability to respond to these demands
  • Identify where specific demands exceed users’ capabilities

Synthesise Evidence

The drawing together and summary of a body of evidence that describes the evaluation and selection of concepts for improvement

Purpose: to provide evidence of evaluation of new concepts against the system requirements

Inputs: Define Requirements, Consider Pre-existing Solutions, Stimulate Ideas, Filter Ideas, Develop Concepts, Make Models, Define Measures, Agree Quality Targets, Review Experience, Review Safety, Review Effectiveness

Tools: tbd

Outputs: a clear summary of evidence relating to the evaluation of the new concepts

Top tips:

  • Draw together, summarise and communicate all of the evidence
  • Do this in a systematic way to support the review criteria
  • Use the evidence to support the choice of solution concept

Tools

Feedback

We would welcome your feedback on this page:

Your name:


Your email:


Your comments:


Please leave this field blank (it's a spam trap):
Submit feedback

Privacy policy. If your feedback comments warrant follow-up communication, we will send you an email using the details you have provided. Feedback comments are anonymized and then stored on our file server

Read more about how we use your personal data. Any e-mails that are sent or received are stored on our mail server for up to 24 months.