WIREDGOV

This report explores and summarises the requirements from and options for a National Mental Health Experience of Services (MHES) survey.

The MHES would gather regular data on service user experiences and help inform standards measurement and service improvement, policy development and NHS reform.”

Executive Summary

Introduction

The Scottish Government commissioned The Lines Between (TLB) to conduct research to inform the development of a potential mental health experiences of services (MHES) survey. Key objectives for this work include proposing options for the MHES survey design and delivery and exploring possible survey content.

This report presents an analysis of data collected through qualitative interviews with 26 key stakeholders, two group events, an online survey with 105 responses and an evidence review of 34 existing mental health and other health and social care experience surveys from Scotland, the UK and abroad.

Aims, purpose, aspirations and need for a MHES survey

All stakeholders agreed that the key uses of data collected from a MHES survey would be to better understand experiences of mental health services in Scotland, improve services, and learn from good practice.

• Existing data was identified as either not being accessed or is insufficient to understand service user experiences.
• By ensuring data collected by a MHES survey is relevant, helpful and can be used to meet its aims, the survey could become valued as a way to identify and make positive change to services.

Survey scope and sampling

Stakeholders noted that an exemplary MHES survey should have a primary aim of understanding service user experiences and that it should be be comprehensive, capturing a wide range of experiences and voices, span all service users rather than a sample, and include those who are on waiting lists or those who have unsuccessfully attempted to access mental health services.

• Some advocated for the survey population to include carers, family members and friends of people with mental health conditions.
• It was recognised that a large scope may be unfeasible and that resource implications might shape the scope and scale of the first MHES survey.

There was no consensus on which services to include in a MHES survey.

• Suggestions ranged from inpatient services only to all mental health services.
• When considering the practicalities of developing the first MHES, an achievable first stage could be an initial pilot survey with a narrow focus that produces a clearly defined data set, for example surveying inpatient services.
• This approach would, however, limit understanding of wider mental health services.

Approaches to sampling for an MHES survey will depend on which services and service users are included.

• Challenges accessing existing databases to identify participants were noted.
• Alternative options include sampling the population directly after engagement and by services handing out a survey or providing an online link or QR (Quick Response) code at the time of, or end of, engagement.

A MHEs survey using a larger sample would likely be more diverse and minimise unintentional identification of service users, depending on the level of reporting.

• The sample required will also depend on which services and service users are included in the survey.
• Options to maximise the sample size, such as combining multiple waves of data, could be considered.
• Sample stratification was suggested to ensure representation of people by area, age and gender.

Survey content

An optimal MHES survey is likely to be similar to other surveys and take between 5-30 minutes to complete.

• Shorter surveys are likely to be more appropriate for some service users and, more generally, produce higher response rates and be cheaper to conduct.
• However, the length must be balanced with the need to ensure that all relevant information and voices are captured.

There was broad agreement that the survey should include at least some, or ideally all, key aspects of using a service.

• This includes accessing services and support, waiting times, moving between different services and support, and overall experiences.
• A mix of closed and open questions is required to understand how effectively services are performing and how they could be improved.
• The survey could include a wide range of measures, but only relevant information must be captured to ensure the length remains manageable.
• There was also agreement that the survey questions should align with the Mental Health Core Standards.
• Aligning a MHES survey with other surveys was considered important and helpful, but it needs to be considered whether this could limit the effectiveness of a Scottish survey.

Key demographic information should be collected in the survey, but only if there is a clear purpose for its collection and use.

• Asking a range of demographic questions will add to the survey length and could be a barrier to participation for some.
• Consideration should be given to how demographic data from other sources can be used to ensure that only relevant information needs to be captured through an MHES survey.

There was no clear consensus about how long or what experiences a survey should ask service users to reflect on.

• The long-term nature of some respondents’ experiences of poor mental health and services means defining an appropriate period may be more challenging than other service user experience surveys.

MHES survey administration

Overall, stakeholders expressed a preference for the MHES survey to be conducted at a fixed point in time rather than through continuous data collection.

• An annual or bi-annual survey was commonly felt to be sufficient to monitor significant change and would be less resource-intensive.
• There was no clear consensus on how long the survey should be open, with the need for a sufficient response rate being a key consideration.
• However, practical considerations were also raised, such as avoiding running the survey over holiday periods.

An online survey will be the quickest and most cost-effective approach.

• However, other alternatives to increase accessibility for those who do not want to complete a survey online will need to be considered.
• This could include paper and postal surveys, and the extent of any online or telephone support to assist with completion and provide translation.

Various survey distribution approaches were suggested with no clear consensus.

• The distribution approach will depend on other aspects of the survey, such as the services being included, response rates required and accessibility of service user contact details.
• Concerns about distribution by practitioners were raised on the basis this could influence service users’ responses.

Incentives may support the response rate. However, they are cost-dependent and are not widely used in other smaller-scale health experience surveys.

Ethical considerations, accessibility, and data governance

Clear, upfront information should be provided to respondents that participation will not negatively impact their care and that they can opt-out at any time.

• Situations when confidentiality is not applied should be highlighted.
• Challenges with informed consent will exist for those who are unwell and for children and young people, if the latter are included within the sample.

The survey will need to be available in multiple formats and languages to support accessibility and inclusion of diverse voices.

• Routes to support the involvement of seldom-heard groups were suggested such as face-to-face interviews and working with community organisations, but these approaches would be resource-intensive.

Multiple actions can be taken to minimise unintended harms and maintain confidentiality e.g. sensitive information not being placed on communication about the survey and not reporting on small service user populations.

• Data protection, storage security, and de-identification of responses will need to be determined and provided on the survey privacy notice.
• The processes of gaining ethical approval and establishing data governance arrangements are likely to require significant staff time, especially if personal or health data is used or collected.

Data analysis, reporting and dissemination

Local or service-level data is likely to be most useful in driving improvements and meeting the needs of a wide range of interested stakeholders.

• The number of reporting variables in the data will need to be considered in advance to ensure robust sample sizes at a local or service level.
• Support should be provided to services to understand and use service-level data. This could be through using tools or systems to identify changes or in service delivery evident in the data and possible ways to respond.

Results should be available at least at a national level to meet the desire for results to be published and for transparency.

• Publication must align with risks raised within ethics - de-identification, anonymisation, and the ethical use of the data given the resourcing and effort required to gather it.
• Data must be used in the way stated in the information given to participants.
• There is a need to be clear with all parties about how the data will be used, i.e., how it will benefit service users, improve mental health services, support funding, address service gaps, and inform government policy and planning.

Stakeholders may need results in different formats, e.g. written reports, raw data and dashboards.

• The number of outputs required will impact costs but focusing on national and local reports would meet many data users’ needs.
• Publishing results at a local or service level raises risks of respondent identification and potential league tables of services, undermining the survey among service users.

Resource and budget considerations

The cost of a MHEs survey will depend on which of the many factors outlined in this report are chosen, such as the target audience, sample size, methodology, and level of analysis. It will be important to consider the staff time and resources needed to manage or analyse the survey and the wider resources required to support service improvements generated from the survey findings.

Conclusions and recommendations

Acknowledging the complexities described above, we recommend that the Scottish Government develop a small-scale survey focused on a few services which span a range of mental health settings. Specifically, we recommend developing and piloting this with services that were early adopters of the Mental Health Core Standards.

This pilot approach provides the opportunity to evaluate the survey methodology, implementation, value, and results. It will also provide a model for embedding a patient experience survey within services to drive improvement before a larger-scale survey is conducted.

The recomended approach for developing a pilot is outlined below:

Recommended Approach for Mental Health Experiences of Services Survey

Aim: To understand service user experiences of mental health services in Scotland, support service development, identify areas of good performance and improvement, and inform local planning.

Services to Include: Inpatient services and adult secondary community services within NHS Boards that were early adopters of the Mental Health Core Standards.

Participants: Adults with experience using these services and support.

Sampling Strategy: Sample drawn from service-collected details, potentially using existing data collection structures from early adopters.

Sample Size: Include all service users to ensure a diverse range of voices.

Survey Length: Should take no more than 15 minutes to complete, balancing depth and respondent concentration.

Survey Questions: Cover key themes within the time limit, prioritising:

• Accessing services and support
• Waiting times and their impact
• Transitions between services
• Overall experiences
• Involvement in care
• Out-patient and in-patient care
• Staff interactions
• Information about diagnosis

Include both qualitative and quantitative questions aligned with Core Standards. Pilot questions with individuals with lived experience.

Demographic Information: Collect essential demographics (e.g. age, disability, deprivation, location, ethnicity, working status, sex at birth, gender identity). Additional demographic questions should be optional.

Period to Reflect Over: Most recent experience or the past year.

Survey Frequency: Conduct annually.

Timing: Administer at the same time each year, avoiding holiday periods.

Survey Duration: Keep open for a minimum of 4 weeks.

Data Collection Method: Primarily online, with paper options for accessibility.

Incentives: No incentives provided.

Distribution Method: Through services (e.g. email or reception staff), not directly from practitioners.

Risk of Harm: Ensure sensitivity, appropriate language, clarity that participation does not affect care, and signposting to support.

Ethical Approval: Obtain ethics and privacy approvals.

Consent and Feedback: Secure informed consent, explain anonymisation and confidentiality limits, and outline procedures.

Data Privacy: Implement robust data management to prevent unintended disclosure.

Accessibility: Design for screen reader compatibility, use plain language, and provide translations or easy-read versions.

Reporting and Dissemination: Share data locally to drive service improvement; report broader findings nationally.

Levels of Analysis: Analyse by demographics, reach, service type, and potentially by service, NHS Board, HSCP, and national levels.

The recommended next stages of the process are now set out.

1. Secure resources for survey development

• Internal resources (staff time)
• Budget for commercial survey suppliers

2. Establish a working group to develop a pilot MHES survey

• Agree on broad approach (e.g. which services to include)
• Secure involvement of early adopters of Core Standards

3. Agree on specific elements of survey design

• Confirm whether to proceed with aspects of survey design where consensus emerged in the scoping review
• Decide how to proceed where there is no consensus
• Key issues include:
• Sample size requirements
• Use of existing databases and related ethical/data privacy considerations
• Survey frequency
• Accessibility mitigations

4. Develop a detailed specification for the survey

• Establish internal staff time required to manage the survey
• Seek quotes from experienced commercial survey suppliers
• Revisit step 3 as needed and revise design elements to balance requirements with available resources

5. Commission and begin survey setup

• Collaborate with commercial supplier to develop and set up the survey
• Develop and pilot survey questions
• Engage service users with lived experience

6. Survey launch

Click here for the full press release