qualitative methods in research on healthcare quality

Qualitative Methods in Health Care Research

Affiliations.

• 1 School of Nursing and Midwifery, Royal College of Surgeons Ireland - Bahrain (RCSI Bahrain), Al Sayh Muharraq Governorate, Bahrain.
• 2 Department of Mental Health Nursing, Manipal College of Nursing Manipal, Manipal Academy of Higher Education, Manipal, Karnataka, India.
• 3 Department of OBG Nursing, Manipal College of Nursing Manipal, Manipal Academy of Higher Education, Manipal, Karnataka, India.
• 4 School of Nursing, MGH Institute of Health Professions, Boston, USA.
• 5 Department of Child Health Nursing, Manipal College of Nursing Manipal, Manipal Academy of Higher Education, Manipal, Karnataka, India.
• PMID: 34084317
• PMCID: PMC8106287
• DOI: 10.4103/ijpvm.IJPVM_321_19

Healthcare research is a systematic inquiry intended to generate robust evidence about important issues in the fields of medicine and healthcare. Qualitative research has ample possibilities within the arena of healthcare research. This article aims to inform healthcare professionals regarding qualitative research, its significance, and applicability in the field of healthcare. A wide variety of phenomena that cannot be explained using the quantitative approach can be explored and conveyed using a qualitative method. The major types of qualitative research designs are narrative research, phenomenological research, grounded theory research, ethnographic research, historical research, and case study research. The greatest strength of the qualitative research approach lies in the richness and depth of the healthcare exploration and description it makes. In health research, these methods are considered as the most humanistic and person-centered way of discovering and uncovering thoughts and actions of human beings.

Keywords: Ethnography; grounded theory; qualitative research; research design.

Copyright: © 2021 International Journal of Preventive Medicine.

Publication types

• - Google Chrome

Intended for healthcare professionals

• Access provided by Google Indexer
• My email alerts
• BMA member login
• Username * Password * Forgot your log in details? Need to activate BMA Member Log In Log in via OpenAthens Log in via your institution

Search form

• Advanced search
• Search responses
• Search blogs
• Methodological...

Methodological standards for qualitative and mixed methods patient centered outcomes research

• Related content
• Peer review
• Bridget Gaglio , senior program officer 1 ,
• Michelle Henton , program manager 1 ,
• Amanda Barbeau , program associate 1 ,
• Emily Evans , research health science specialist 2 ,
• David Hickam , director of clinical effectiveness and decision sciences 1 ,
• Robin Newhouse , dean 3 ,
• Susan Zickmund , research health scientist and professor 4 5
• 1 Patient-Centered Outcomes Research Institute, 1828 L Street, Suite 900, Washington, DC, 20036, USA
• 2 Veterans Health Administration, United States Department of Veterans Affairs, Washington, DC, USA
• 3 Indiana University School of Nursing, Indianapolis, IN, USA
• 4 United States Department of Veterans Affairs, Salt Lake City, UT, USA
• 5 University of Utah School of Medicine, Salt Lake City, UT, USA
• Correspondence to: B Gaglio bgaglio{at}pcori.org
• Accepted 20 October 2020

The Patient-Centered Outcomes Research Institute’s (PCORI) methodology standards for qualitative methods and mixed methods research help ensure that research studies are designed and conducted to generate the evidence needed to answer patients’ and clinicians’ questions about which methods work best, for whom, and under what circumstances. This set of standards focuses on factors pertinent to patient centered outcomes research, but it is also useful for providing guidance for other types of clinical research. The standards can be used to develop and evaluate proposals, conduct the research, and interpret findings. The standards were developed following a systematic process: survey the range of key methodological issues and potential standards, narrow inclusion to standards deemed most important, draft preliminary standards, solicit feedback from a content expert panel and the broader public, and use this feedback to develop final standards for review and adoption by PCORI’s board of governors. This article provides an example on how to apply the standards in the preparation of a research proposal.

Rigorous methodologies are critical for ensuring the trustworthiness of research results. This paper will describe the process for synthesizing the current literature providing guidance on the use of qualitative and mixed methods in health research; and the process for development of methodology standards for qualitative and mixed methods used in patient centered outcomes research. Patient centered outcomes research is comparative clinical effectiveness research that aims to evaluate the clinical outcomes resulting from alternative clinical or care delivery approaches for fulfilling specific health and healthcare needs. By focusing on outcomes that are meaningful to patients, studies on patient centered outcomes research strengthen the evidence base and inform the health and healthcare decisions made by patients, clinicians, and other stakeholders.

The methods used in patient centered outcomes research are diverse and often include qualitative methodologies. Broadly, qualitative research is a method of inquiry used to generate and analyze open ended textual data to enhance the understanding of a phenomenon by identifying underlying reasons, opinions, and motivations for behavior. Many different methodologies can be used in qualitative research, each with its own set of frameworks and procedures. 1 This multitude of qualitative approaches allows investigators to select and synergize methods with the specific needs associated with the aims of the study.

Qualitative methods can also be used to supplement and understand quantitative results; the integration of these approaches for scientific inquiry and evaluation is known as mixed methods. 2 This type of approach is determined a priori, because the research question drives the choice of methods, and draws on the strengths of both quantitative and qualitative approaches to resolve complex and contemporary issues in health services. This strategy is achieved by integrating qualitative and quantitative approaches at the design, methods, interpretation, and reporting levels of research. 3 Table 1 lists definitions of qualitative methods, mixed methods, and patient centered outcomes research. The methodology standards described here are intended to improve the rigor and transparency of investigations that include qualitative and mixed methods. The standards apply to designing projects, conducting the studies, and reporting the results. Owing to its focus on patient centered outcomes research, this article is not intended to be a comprehensive summary of the difficulties encountered in the conduct of qualitative and mixed methods research.

Terms and definitions used in the development of the Patient-Centered Outcomes Research Institute’s (PCORI) qualitative and mixed methods research methodology standards

• View inline

Summary points

Many publications provide guidance on how to use qualitative and mixed methods in health research

The methodological standards reported here and adopted by Patient-Centered Outcomes Research Institute (PCORI) synthesize and refine various recommendations to improve the design, conduct, and reporting of patient centered, comparative, clinical effectiveness research

PCORI has developed and adopted standards that provide guidance on key areas where research applications and research reports have been deficient in the plans for and use of qualitative and mixed methods in conducting patient centered outcomes research

The standards provide guidance to health researchers to ensure that studies of this research are designed and conducted to generate valid evidence needed to analyze patients’ and clinicians’ questions about what works best, for whom, and under what circumstances

Established by the United States Congress in 2010 13 and reauthorized in 2019, 14 the Patient-Centered Outcomes Research Institute (PCORI) funds scientifically rigorous comparative effectiveness research, previously defined as patient centered outcomes research, to improve the quality and relevance of evidence that patients, care givers, clinicians, payers, and policy makers need to make informed healthcare decisions. Such decisions might include choices about which prevention strategies, diagnostic methods, and treatment options are most appropriate based on personal preferences and unique patient characteristics.

PCORI’s focus on patient centeredness and stakeholder engagement in research has generated increased interest in and use of methodologies of qualitative and mixed methods research within comparative effectiveness research studies. Qualitative data have a central role in understanding the human experience. As with any research, the potential for these studies to generate high integrity, evidence based information depends on the quality of the methods and approaches that were used. PCORI’s authorizing legislation places a unique emphasis on ensuring scientific rigor, including the creation of a methodology committee that develops and approves methodology standards to guide PCORI funded research. 13 The methodology committee consists of 15 individuals who were appointed by the Comptroller General of the US and the directors of the Agency for Healthcare Research and Quality and the National Institutes of Health. The members of the committee are medical and public health professionals with expertise in study design and methodology for comparative effectiveness research or patient centered outcomes research ( https://www.pcori.org/about-us/governance/methodology-committee ).

The methodology committee began developing its initial group of methodology standards in 2012 (with adoption by the PCORI’s board of governors that year). Since then, the committee has revised and expanded the standards based on identified methodological issues and input from stakeholders. Before the adoption of the qualitative and mixed methods research standards, the PCORI methodology standards consisted of 56 individual standards in 13 categories. 15 The first five categories of the standards are crosscutting and relevant to most studies on patient centered outcomes research, while the other eight categories are applicable depending on a study’s purpose and design. 15

Departures from good research practices are partially responsible for weaknesses in the quality and subsequent relevance of research. The PCORI methodology standards provide guidance that helps to ensure that studies on patient centered outcomes research are designed and conducted to generate the evidence needed to answer patients’ and clinicians’ questions about what works best, for whom, and under what circumstances. These standards do not represent a complete, comprehensive set of all requirements for high quality patient centered outcomes research; rather, they cover topics that are likely to contribute to improvements in quality and value. Specifically, the standards focus on selected methodological issues that have substantial deficiencies or inconsistencies regarding how available methods are applied in practice. These methodological issues might include a lack of rigor or inappropriate use of approaches for conducting patient centered outcomes research. As a research funder, PCORI uses the standards in the scientific review of applications, monitoring of funded research projects, and evaluation of final reports of research findings.

Use of qualitative methods has become more prevalent over time. Based on a PubMed search in June 2020 (search terms “qualitative methods” and “mixed methods”), the publication of qualitative and mixed methods studies has grown steadily from 1980 to 2019. From 1980 to 1989, 63 qualitative and 110 mixed methods papers were identified. Between 1990 to 1999, the number of qualitative and mixed methods papers was 420 and 58, respectively; by 2010 to 2019, these numbers increased to 5481 and 17 031, respectively. The prominent increase in publications in recent years could be associated with more sophisticated indexing methods in PubMed as well as the recognition that both qualitative and mixed methods research are important approaches to scientific inquiry within the health sciences. These approaches allow investigators to obtain a more detailed perspective and to incorporate patients’ motivations, beliefs, and values.

Although the use of qualitative and mixed methods research has increased, consensus regarding definitions and application of the methods remain elusive, reflecting wide disciplinary variation. 16 17 Many investigators and organizations have attempted to resolve these differences by proposing guidelines and checklists that help define essential components. 12 16 18 19 20 21 22 23 24 25 26 27 28 29 For example, Treloar et al 20 offer direction for qualitative researchers in designing and publishing research by providing a 10 point checklist for assessing the quality of qualitative research in clinical epidemiological studies. Tong et al 22 provide a 32 item checklist to help investigators report important aspects of the research process for interviews and focus groups such as the study team, study methods, context of the study, findings, analysis, and interpretations.

The goal of the PCORI Methodology Standards on Qualitative and Mixed Methods is to provide authoritative guidance on the use of these methodologies in comparative effectiveness research and patient centered outcomes research. The purpose of these types of research is to improve the clinical evidence base and, particularly, to help end users understand how the evidence provided by individual research studies can be applied to particular clinical circumstances. Use of qualitative and mixed methods can achieve this goal but can also introduce specific issues that need to be captured in PCORI’s methodological guidance. The previously published guidelines generally have a broader focus and different points of emphasis.

This article describes the process for synthesizing the current literature providing guidance on the use of qualitative and mixed methods in health research; and developing methodology standards for qualitative and mixed methods used in patient centered outcomes research. We then provide an example showing how to apply the standards in the design of a patient centered outcomes research application.

Methodology standards development process

Literature review and synthesis.

The purpose of the literature review was to identify published journal articles that defined criteria for rigorous qualitative and mixed methods research in health research. With the guidance of PCORI’s medical librarian, we designed and executed searches in PubMed, and did four different keyword searches for both qualitative and mixed methods (eight searches in total; supplemental table 1). We aimed to identify articles that provided methodological guidance rather than studies that simply used the methods.

We encountered two major challenges. First, qualitative and mixed methods research has a broad set of perspectives. 30 31 Second, some medical subject headings (MeSH terms) in our queries were not introduced until recently (eg, “qualitative methods” introduced in 2003, “comparative effectiveness” introduced in 2010), which required us to search for articles by identifying a specific qualitative method (eg, interviews, focus groups) to capture the literature before 2003 ( table 1 ). These challenges could have led to missed publications. To refine and narrow our search results, we applied the following inclusion criteria:

Articles on health services or clinical research, published in English, and published between 1 January 1990 and 14 April 2017

Articles that proposed or discussed a guideline, standard, framework, or set of principles for conducting rigorous qualitative and mixed methods research

Articles that described or discussed the design, methods for, or reporting of qualitative and mixed methods research.

The search queries identified 1933 articles (1070 on qualitative methods and 863 on mixed methods). The initial citation lists were reviewed, and 204 duplicates were removed. Three authors (BG, MH, and AB) manually reviewed the 1729 remaining article abstracts. Titles and abstracts were independently evaluated by each of the three reviewers using the inclusion criteria. Disagreements were adjudicated by an in-person meeting to determine which articles to include. This initial round of review yielded 212 references, for which the full articles were obtained. The full articles were reviewed using the same inclusion and exclusion criteria as the abstracts. Most of these articles were studies that had used a qualitative or mixed methods approach but were only reporting on the results of the completed research. Therefore, these articles were not able to inform the development of standards for conducting qualitative and mixed methods research and they were excluded, resulting in the final inclusion of 56 articles (supplemental table 2). Following the original search, the literature was scanned for new articles providing guidance on qualitative and mixed methods, resulting in four articles being added to the final set of literature. These articles come from psychology and health psychology specialties and seek to provide not only minimal standards in relation to qualitative and mixed methods research but also standards for best practice that apply across a wide range of fields. 32 33 34 35

Initial set of methodology standards

Using an abstraction form that outlined criteria for qualitative and mixed methods manuscripts and research proposals, we abstracted the articles to identify key themes, recommendations, and guidance under each criterion. Additional information was noted when considered relevant. A comprehensive document was created to include the abstractions and notes for all articles. This document outlined the themes in the literature related to methodological guidance. We began with the broadest set of themes organized into 11 major domains: the theoretical approach, research topics, participants, data collection, analysis and interpretation, data management, validity and reliability, presentation of results, context of research, impact of the researchers (that is, reflexivity), and mixed methods. As our goal was to distill the themes into broad standards that did not overlap with pre-existing PCORI methodology standards, we initially condensed the themes into six qualitative and three mixed methods standards. Following discussion among members of the working group, some standards were combined and two were dropped because of substantial overlap with each other or with previously developed PCORI methodology standards.

The key themes identified from the abstracted information were used as the foundation for the first draft of the new methodology standards. We then further discussed the themes as a team and removed redundancies, refined the labeling of themes, and removed themes deemed extraneous through a team based adjudication process. The draft standards were presented to PCORI’s methodology committee to solicit feedback. Revisions were made on the basis of this feedback.

Expert panel one day workshop

A one day expert panel workshop was held in Washington, DC, on 18 January 2018. Ten individuals regarded as international leaders in qualitative and mixed methods were invited to attend—including those who had created standards previously or had a substantial number of peer reviewed publications reporting qualitative and mixed methods in health research; had many years’ experience as primary researchers; and had served as editors of major textbooks and journals. The panel was selected on the basis of their influence and experience in these methodologies as well as their broad representation from various fields of study. The representation of expertise spanned the fields of healthcare, anthropology, and the social sciences (supplemental table 3).

Before the meeting, we emailed the panel members the draft set of qualitative and mixed methods standards, PCORI’s methodology standards document, and the background document describing how the draft standards had been developed. At the meeting, the experts provided extensive feedback, including their recommendations regarding what needs to be done well when using these methodological approaches. The panel emphasized that when conducting mixed methods research, this approach should be selected a priori, based on the research question, and that integration of the mixed approaches is critical at all levels of the research process (from inception to data analysis). The panel emphasized that when conducting qualitative research, flexibility and reflexive iteration should be maintained throughout the process—that is, the sampling, data collection, and data analysis. The main theme from the meeting was that the draft standards were not comprehensive enough to provide guidance for studies on patient centered outcomes research or comparative effectiveness research that involved qualitative and mixed methods. After the conclusion of the workshop, feedback and recommendations were synthesized, and the draft standards were reworked in the spring of 2018 ( fig 1 ). This work resulted in a new set of four qualitative methods standards and three mixed methods standards representing the unique features of each methodology that were not already included in the methodology standards previously adopted by PCORI.

Process of development and adoption of the Patient-Centered Outcomes Research Institute’s (PCORI) methodology standards on qualitative and mixed methods research

• Download figure
• Open in new tab
• Download powerpoint

Continued refinement and approval of methodology standards

In late spring 2018, the revised draft methodology standards were presented to PCORI’s methodology committee first by sharing a draft of the standards and then via oral presentation. Feedback from the methodology committee centered around eliminating redundancy in the standards proposed (both across the draft standards and in relation to the previously adopted categories of standards) and making the standards more actionable. The areas where the draft standards overlapped with the current standards were those for formulating research questions, for patient centeredness, and for data integrity and rigorous analyses. Each draft standard was reviewed and assessed by the methodology committee members and the staff workgroup to confirm its unique contribution to PCORI’s methodology standards. After this exercise, each remaining standard was reworded to be primarily action guiding (rather than explanatory). This version of proposed standards was approved by the methodology committee to be sent to PCORI’s board of governors for a vote to approve for public comment. The board of governors approved the standards to be posted for public comment.

The public comment period hosted on PCORI’s website ( https://www.pcori.org/engagement/engage-us/provide-input/past-opportunities-provide-input ) was held from 24 July 2018 to 21 September 2018. Thirty nine comments were received from nine different stakeholders—seven health researchers, one training institution, and one professional organization. Based on the public comments, minor wording changes were made to most of the draft standards. The final version of the standards underwent review by both the methodology committee and PCORI’s board of governors. The board voted to adopt the final version of the standards on 26 February 2019 ( table 2 ).

Patient-Centered Outcomes Research Institute’s (PCORI) methodology standards for qualitative methods and mixed methods

Application of methodology standards in research design

The standards can be used across the research continuum, from research design and application development, conduct of the research, and reporting of research findings. We provide an example for researchers on how these standards can be used in the preparation of a research application ( table 3 ).

Guidance for researchers on how to use Patient-Centered Outcomes Research Institute’s (PCORI) methodology standards for qualitative and mixed methods research in application preparation

QM-1: State the qualitative approach to research inquiry, design, and conduct

Many research proposals on patient centered outcomes research or comparative effectiveness research propose the use of qualitative methods but lack adequate description of and justification for the qualitative approach that will be used. Often the rationale for using qualitative methods is not tied back to the applicable literature and the identified evidence gap, missing the opportunity to link the importance of the approach in capturing the human experience or patient voice in the research aims. The approach to inquiry should be explicitly stated along with the rationale and a description of how it ties to the research question(s). The research proposal should clearly define how the qualitative approach will be operationalized and supports the choice of methods for participant recruitment, data collection, and analysis. Moreover, procedures for data collection should be stated, as well as the types of data to be collected, when data will be collected (that is, one point in time v longitudinal), data management, codebook development, intercoder reliability process, data analysis, and procedures for ensuring full confidentiality.

QM-2: Select and justify appropriate qualitative methods and sampling strategy

While the number of participants who will be recruited for focus groups or in-depth interviews is usually described, the actual sampling strategy is often not stated. The description of the sampling strategy should state how it aligns with the qualitative approach, how it relates to the research question(s), and the variation in sampling that might occur over the course of the study. Furthermore, most research proposals state that data will be collected until thematic saturation is reached, but how this will be determined is omitted. As such, this standard outlines the information essential for understanding who is participating in the study and aims to reduce the likelihood of making unsupported statements, emphasizing transparency in the criteria used to determine the stopping point for recruitment and data collection.

QM-3: Link the qualitative data analysis, interpretations, and conclusions to the study question

Qualitative analysis transforms data into information that can be used by the relevant stakeholder. It is a process of reviewing, synthesizing, and interpreting data to describe and explain the phenomena being studied. The interpretive process occurs at many points in the research process. It begins with making sense of what is heard and observed during data gathering, and then builds understanding of the meaning of the data through data analysis. This is followed by development of a description of the findings that makes sense of the data, in which the researcher’s interpretation of the findings is embedded. Many research proposals state that the data will be coded, but it is unclear by whom, their qualifications, or the process. Very little, if any, description is provided as to how conclusions will be drawn and how they will be related to the original data, and this standard highlights the need for detailed information on the analytical and interpretive processes for qualitative data and its relationship to the overall study.

QM-4: Establish trustworthiness and credibility of qualitative research

The qualitative research design should incorporate elements demonstrating validity and reliability, which are also known by terms such as trustworthiness and credibility. Studies with qualitative components can use several approaches to help ensure the validity and reliability of their findings, including audit trail, reflexivity, negative or deviant case analysis, triangulation, or member checking (see table 1 for definitions).

MM-1: Specify how mixed methods are integrated across design, data sources, and/or data collection phases

This standard requires investigators to declare and support their intent to conduct a mixed methods approach a priori in order to avoid a haphazard approach to the design and resulting data. Use of mixed methods can enhance the study design, by using the strengths of both quantitative and qualitative research as investigators are afforded the use of multiple data collection tools rather than being restricted to one approach. Mixed methods research designs have three key factors: integration of data, relative timing, and implications of linkages for methods in each component. Additionally, the standards for mixed methods, quantitative, and qualitative methodologies must be met in the design, implementation, and reporting stages. This is different from a multimethod research design in which two or more forms of data (qualitative, quantitative, or both) are used to resolve different aspects of the research question independently and are not integrated.

MM-2: Select and justify appropriate mixed methods sampling strategy

Mixed methods research aims to contribute insights and knowledge beyond that obtained from quantitative or qualitative methods only, which should be reflected in the sampling strategies as well as in the design of the study and the research plan. Qualitative and quantitative components can occur simultaneously or sequentially, and researchers must select and justify the most appropriate mixed method sampling strategy and demonstrate that the desired number and type of participants can be achieved with respect to the available time, cost, research team skillset, and resources. Those sampling strategies that are unique to mixed methods (eg, interdependent, independent, and combined) should focus on the depth and breadth of information across research components.

MM-3: Integrate data analysis, data interpretations, and conclusions

Qualitative and quantitative data often are analyzed in isolation, with little thought given to when these analyses should occur or how the analysis, interpretation, and conclusions integrate with one another. There are multiple approaches to integration in the analysis of qualitative and quantitative data (eg, merging, embedding, and connecting). As such, the approach to integration should determine the priority of the qualitative and quantitative components, as well as the temporality with which analysis will take place (eg, sequentially, or concurrently; iterative or otherwise). Either a priori or emergently, where appropriate, researchers should define these characteristics, identify the points of integration, and explain how integrated analyses will proceed with respect to the two components and the selected approach.

The choice between multiple options for prevention, diagnosis, and treatment of health conditions presents a considerable challenge to patients, clinicians, and policy makers as they seek to make informed decisions. Patient centered outcomes research focuses on the pragmatic comparison of two or more health interventions to determine what works best for which patients and populations in which settings. 5 The use of qualitative and mixed methods research can enable more robust capture and understanding of information from patients, caregivers, clinicians, and other stakeholders in research, thereby improving the strength, quality, and relevance of findings. 4

Despite extensive literature on qualitative and mixed methods research in general, the use of these methodologies in the context of patient centered outcomes research or comparative effectiveness research continues to grow and requires additional guidance. This guidance could facilitate the appropriate design, conduct, analysis, and reporting of these approaches. For example, the need for including multiple stakeholder perspectives, understanding how an intervention was implemented across multiple settings, or documenting the clinical context so decision makers can evaluate whether findings would be transferable to their respective settings pose unique challenges to the rigor and agility of qualitative and mixed methods approaches.

PCORI’s methodology standards for qualitative and mixed methods research represent an opportunity for further strengthening the design, conduct, and reporting of patient centered outcomes research or comparative effectiveness research by providing guidance that encompasses the broad range of methods that stem from various philosophical assumptions, disciplines, and procedures. These standards directly affect factors related to methodological integrity, accuracy, and clarity as identified by PCORI staff, methodology committee members, and merit reviewers in studies on patient centered outcomes research or comparative effectiveness research. The standards are presented at a level accessible to researchers new to qualitative and mixed methods research; however, they are not a substitute for appropriate expertise.

The challenges of ensuring rigorous methodology in the design and conduct of research are not unique to qualitative and mixed methods research, because the imperative to increase value and reduce waste in research design, conduct, and analysis is widely recognized. 36 Consistent with such efforts, PCORI recognizes the importance of continued methodological development and evaluation and is committed to listening to the research community and providing updated guidance based on methodological advances and research needs. 37

Acknowledgments

We thank the Patient-Centered Outcomes Research Institute’s (PCORI) methodology committee during this work (Naomi Aronson, Ethan Bach, Stephanie Chang, David Flum, Cynthia Girman, Steven Goodman (chairperson), Mark Helfand, Michael S Lauer, David O Meltzer, Brian S Mittman, Sally C Morton, Robin Newhouse (vice chairperson), Neil R Powe, and Adam Wilcox); and Frances K Barg, Benjamin F Crabtree, Deborah Cohen, Michael Fetters, Suzanne Heurtin-Roberts, Deborah K Padgett, Janice Morse, Lawrence A Palinkas, Vicki L Plano Clark, and Catherine Pope, for participating in the expert panel meeting and consultation.

Contributors: BG led the development of the methodology standards and wrote the first draft of the paper. MH, AB, SZ, EE, DH, and RN made a substantial contribution to all stages of developing the methodology standards. BG, SZ, MH, and AB drafted the methodology standards. DH, EE, and RN gave critical insights into PCORI’s methodology standards development processes and guidance. SZ served as qualitative methods consultant to the workgroup. BG provided project leadership and guidance. MH and AB facilitated the expert panel meeting. SZ is senior author. BG and SZ are the guarantors of this work and accept full responsibility for the finished article and controlled the decision to publish. The corresponding author attests that all listed authors meet authorship criteria and that no others meeting the criteria have been omitted.

Funding: No funding was used to support this work. All statements in this report, including its findings and conclusions, are solely those of the authors and do not necessarily represent the views of PCORI, its board of governors or methodology committee. The views expressed in this article are those of the authors and do not necessarily represent the views of the Department of Veterans Affairs.

Competing interests: All authors have completed the ICMJE uniform disclosure form at http://www.icmje.org/conflicts-of-interest/ and declare: no support from any organization for the submitted work; no financial relationships with any organizations that might have an interest in the submitted work in the previous three years; no other relationships or activities that could appear to have influenced the submitted work.

The lead author affirms that the manuscript is an honest, accurate, and transparent account of the work being reported; that no important aspects of the study have been omitted; and that any discrepancies from the work as planned have been explained.

Provenance and peer review: Not commissioned; externally peer reviewed.

Patient and public involvement: Patients and stakeholders were invited to comment on the draft standards during the public comment period held from 24 July 2018 to 21 September 2018. Comments were reviewed and revisions made accordingly. Development of the standards, including the methods, were presented at two PCORI board of governors’ meetings, which are open to the public, recorded, and posted on the PCORI website.

This is an Open Access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/ .

• Collins CS ,
• Stockton CM
• Creswell JW ,
• Plano Clark VL
• Fetters MD ,
• Creswell JW
• ↵ Patient-Centered Outcomes Research Institute. Patient-centered outcomes research. 2010-19. https://www.pcori.org/research-results/about-our-research/research-we-support .
• Institute of Medicine
• Crabtree BF ,
• Klassen AC ,
• Plano Clark VL ,
• Clegg Smith KC ,
• Office of Behavioral and Social Sciences Research
• ↵ Patient Protection and Affordable Care Act, Pub. L. No. 111-148 Stat. 119 (March 23, 2010).
• ↵ Further Consolidated Appropriations Act, 2020, Pub. L. No. 116-94 (20 December 2019).
• ↵ Patient-Centered Outcomes Research Institute (PCORI). PCORI methodology standards. 2011-19. https://www.pcori.org/research-results/about-our-research/research-methodology/pcori-methodology-standards .
• Molina-Azorin JF
• Chapple A ,
• Treloar C ,
• Champness S ,
• Simpson PL ,
• Higginbotham N
• Cesario S ,
• Santa-Donato A
• Sainsbury P ,
• Flemming K ,
• McInnes E ,
• Davidoff F ,
• Batalden P ,
• Stevens D ,
• Mooney SE ,
• SQUIRE development group
• Gagnon MP ,
• Griffiths F ,
• Johnson-Lafleur J
• ↵ National Cancer Institute. Qualitative methods in implementation science. 2018. https://cancercontrol.cancer.gov/sites/default/files/2020-09/nci-dccps-implementationscience-whitepaper.pdf
• O’Brien BC ,
• Harris IB ,
• Beckman TJ ,
• ↵ National Institute for Health and Clinical Excellence. The guidelines manual. Appendix H: Methodology checklist: qualitative studies. https://www.nice.org.uk/process/pmg6/resources/the-guidelines-manual-appendices-bi-2549703709/chapter/appendix-h-methodology-checklist-qualitative-studies .
• Crabtree BF
• Johnson RB ,
• Onwuegbuzie AJ ,
• American Psychological Association
• Levitt HM ,
• Bamberg M ,
• Josselson R ,
• Suárez-Orozco C
• Motulsky SL ,
• Morrow SL ,
• Ponterotto JG
• Bishop FL ,
• Horwood J ,
• Chilcot J ,
• Ioannidis JPA ,
• Greenland S ,
• Hlatky MA ,
• ↵ Patient-Centered Outcomes Research Institute. The PCORI methodology report. 2019. https://www.pcori.org/sites/default/files/PCORI-Methodology-Report.pdf .

• Open access
• Published: 24 April 2024

Feasibility of a quality-improvement program based on routinely collected health outcomes in Dutch primary care physical therapist practice: a mixed-methods study

• LSF Smeekens 1 ,
• AC Verburg 1 ,
• MJM Maas 1 , 2 ,
• R van Heerde 1 ,
• A van Kerkhof 3 &
• PJ van der Wees 1  

BMC Health Services Research volume  24 , Article number:  509 ( 2024 ) Cite this article

Metrics details

This study evaluates the feasibility of a nine-month advanced quality-improvement program aimed at enhancing the quality of care provided by primary care physical therapists in the Netherlands. The evaluation is based on routinely collected health outcomes of patients with nonspecific low back pain, assessing three feasibility domains: (1) appropriateness, feasibility, and acceptability for quality-improvement purposes; (2) impact on clinical performance; and (3) impact on learning and behavioral change.

A mixed-methods quality-improvement study using a concurrent triangulation design was conducted in primary care physical therapist practice. Feedback reports on the processes and outcomes of care, peer assessment, and self-assessment were used in a Plan-Do-Study-Act cycle based on self-selected goals. The program’s appropriateness, feasibility, and acceptability, as well as the impact on clinical performance, were evaluated using the Intervention Appropriate Measure, Feasibility Intervention Measure, Acceptability Intervention Measure (for these three measure, possible scores range from 4 to 20), and with a self-assessment of clinical performance (scored 0–10), respectively. The impact on learning and behavioral change was evaluated qualitatively with a directed content analysis.

Ten physical therapists from two practices participated in this study. They rated the program with a mean of 16.5 (SD 1.9) for appropriateness, 17.1 (SD 2.2) for feasibility, and 16.4 (SD 1.5) for acceptability. Participants gave their development in clinical performance a mean score of 6.7 (SD 1.8). Participants became aware of the potential value of using outcome data and gained insight into their own routines and motivations. They changed their data collection routines, implemented data in their routine practice, and explored the impact on their clinical behavior.

Conclusions

This explorative study demonstrated that a quality-improvement program, using health outcomes from a national registry, is judged to be feasible.

Impact statement

This study provides preliminary evidence on how physical therapists may use health outcomes to improve their quality, which can be further used in initiatives to improve outcome-based care in primary physical therapy.

Peer Review reports

High-quality health care is defined as care that is safe, timely, equitable, effective, efficient, and patient centered [ 1 ]. Against a background of rapidly increasing healthcare costs, service restrictions, and differences in quality, there is an increasing need for initiatives to improve quality of care [ 2 ]. This has led the Royal Dutch Society for Physical Therapy (KNGF) to initiate the ‘Quality in Motion’ program, which aims to improve the effectiveness and patient centeredness of care in physical therapist practice by providing therapists with feedback on health outcomes [ 3 ]. Outcome measures include patient-reported outcomes (PROs), which are used to assess aspects of a patient’s health status coming directly from the patient. Patient-reported outcome measures (PROMs) are questionnaires or single-item scales used to assess PROs [ 4 ], and can be used to support quality improvement [ 3 ]; however, there is a clear lack of understanding about how physical therapists can best utilize feedback about PROs to improve quality of care [ 5 , 6 , 7 ].

Nonspecific low back pain (NSLBP) is one of the most common health conditions in primary physical therapist practice [ 8 , 9 ]. Based on health outcomes from a clinical registry and consensus among stakeholders (i.e., physical therapists, researchers, patients, and health insurers), Verburg et al. [ 4 ] developed a core set of PRO-based quality indicators for patients with NSLBP in primary physical therapist practice. The set was found to be useful for quality-improvement initiatives, and stakeholders reported that it added value for routine practice [ 3 , 4 ]. These outcomes can be aggregated across patients in clinical registries, providing data for managing clinical quality, benchmarking and public reporting across organizations, and in clinical research; however, their aggregated use for quality improvement was found to be suboptimal [ 10 , 11 , 12 ]. An earlier study found that electronic health record (EHR) compatibility and therapist knowledge of the PROMs are the two key barriers to wider PROM use [ 13 ], with similar issues reported in other professions [ 14 , 15 ].

Feedback interventions, particularly when provided by a colleague both verbally and in writing [ 16 ], have shown promise in improving physical therapist practice [ 17 , 18 ]. Correspondingly, feedback reporting on processes and outcomes of care has been identified as an effective intervention that can support the exchange of best practices and mutual learning [ 16 , 18 , 19 ]. Additionally, involving peers as feedback providers in peer assessment creates meaningful learning experiences and is associated with behavioral change and measurable performance improvement in healthcare professionals [ 20 , 21 , 22 ]. Maas et al. [ 23 ] showed that peer assessment using video recordings of client communication and clinical records is an effective feedback intervention method in enhancing commitment to change and improving the clinical performance of physical therapists. Furthermore, feedback interventions seem to be more effective in changing clinical behavior when including clear targets and an action plan [ 16 ]. Accordingly, the Plan-Do-Study-Act facilitates systematic testing of changes in real-world settings, allowing for rapid learning and adaptation. This approach has been effectively utilized in various healthcare studies to enhance clinical outcomes and process efficiencies [ 24 ]; however, most physical therapists are not familiar with such quality-improvement interventions based on health outcomes [ 25 ].

The aim of this study is therefore to evaluate the feasibility of an advanced quality-improvement program for physical therapists in primary care. The evaluation involves feedback, peer assessment, and self-assessment in a rapid improvement Plan-Do-Study-Act cycle, using the routinely collected health outcome data of patients with NSLBP.

Study design and setting

The program feasibility was evaluated through an explorative quality-improvement study using a mixed-methods approach in a concurrent triangulation design [ 26 ]. The following program feasibility domains were addressed [ 27 ]: (1) appropriateness, feasibility, and acceptability for quality-improvement; (2) impact on clinical performance; and (3) impact on learning and behavioral change. We used the Standards for QUality Improvement Reporting Excellence (SQUIRE) Guidelines [ 28 ]. The evaluation was conducted between January and October 2022. We tested our program in a convenience sample of Dutch primary care physical therapists organized in a regional network of communities of practice (the Cooperation of Physical Therapists Nijmegen; CFN).

Participants

All physical therapy practices within the CFN network ( n  = 30) were approached to recruit therapists for the study. Invitations were extended via a digital newsletter, which included the goals of the study and contact details of the first author (LS). Physical therapists willing to participate received detailed study information by email and were screened for eligibility using the inclusion criteria below. Participation was voluntary. All participants provided written informed consent.

Inclusion criteria

Licensed Dutch physical therapists were eligible to participate in this study if they provided primary care to patients with NSLBP aged 18 years or older [ 3 , 4 ]. They also had to evaluate selected outcomes as part of a standard clinical routine in patients with NSLBP using the following PROMs (associated domain): Numeric Pain Rating Scale (NPRS) (pain intensity), Patient Specific Functioning Scale (PSFS) (physical activity), Quebec Pain Disability Scale (QBPDS) (physical functioning), Global Perceived Effect (GPE-DV) (perceived treatment effect), and STarT Back Screening Tool (SBST) (profile grouping based on risk of poor outcome) [ 3 , 4 ]. Physical therapists collected outcomes using these PROMs, which were directly recorded into their EHRs. These data were transferred to the national data registry of the Royal Dutch Society for Physical Therapy (KNGF). Additionally, to facilitate meaningful participation in the quality-improvement program, particularly during peer assessment sessions and outcome discussions, it was essential for participants to have contributed sufficient data to the national clinical registry from January 2021 to November 2021 (a minimum requirement of five patients with a closed treatment episode). An episode was considered closed when the physical therapist closed the episode in the EHR, or if six weeks had passed after the last visit. Informed consent for delivering data to the national clinical registry was obtained from every patient. This approach ensured that participants could engage with actual data reflective of their clinical practices rather than hypothetical scenarios, fostering deeper learning and reflection on professional conduct and patient care. The requirement for therapists to have already been actively collecting and submitting data as part of their clinical routine underlines the study’s aim to engage therapists who were not only familiar with the use of PROMs, but who also had sufficient data to enable a meaningful analysis and discussion within the context of the quality-improvement program.

The quality-improvement program content

The nine-month program consisted of a rapid improvement cycle comprising multiple consecutive steps and quality-improvement interventions. In step 1, participants were offered the opportunity to complete an e-learning module on using data in clinical practice [ 29 ]. In step 2, personal data exports were extracted from the national clinical registry. Participants received feedback reports on the processes and outcomes of their care in step 3 [ 30 , 31 , 32 ], then attended peer assessment meetings in step 4 [ 18 , 23 , 33 ], In step 4, the therapists drafted a rapid improvement Plan-Do-Study-Act cycle and individual quality-improvement goals [ 6 , 34 , 35 ], and in step 5, they performed a self-assessment of their clinical performance [ 36 ]. See Additional File 1 for further details of the program. The process and outcome indicators of the PROMs for patients with NSLBP were used in the program (see Additional File 2 ) [ 3 , 4 ].

Evaluation of program feasibility and outcome measures

The program’s perceived appropriateness, acceptability, and feasibility for quality-improvement purposes were evaluated using the Dutch versions of the Intervention Appropriate Measure (IAM), the Feasibility Intervention Measure (FIM), and the Acceptability Intervention Measure (AIM), respectively [ 37 ], which have been demonstrated to be valid and reliable tools [ 37 ]. Each measure consists of four items scored on a five-point Likert scale, with higher scores indicating better appropriateness, acceptability, and feasibility, respectively (scoring range: 4–20 for each tool). The impact on clinical performance was evaluated using self-assessment checklists [ 36 ] (steps 5 and 7 of the quality-improvement program), while the impact on learning and behavioral change was qualitatively determined during the peer assessment (steps 4 and 6). We used a parallel approach in collecting the quantitative and qualitative data, giving equal weight to both methods.

Data collection

Participants were invited by email to attend the peer assessment meetings. A script (see Additional Files 5 and 6 ) for each meeting was designed by the research team, addressing different quality-improvement interventions. A participatory evaluation strategy was used, allowing an assessment of the impact of the program on learning and behavioral change during the actual implementation [ 38 ]. The peer assessment meetings lasted 100–120 min and were conducted face-to-face by an external coach (RvH) using open-ended questions, which facilitated group discussion and knowledge development. A safe environment was encouraged within each peer group [ 20 , 22 ]. The peer assessment meetings were audio-taped, video-recorded, and subsequently transcribed verbatim. Written informed consent was obtained from all participants. The identities of the participants were considered confidential; therefore, the transcripts of the meetings were processed anonymously. Participants were asked to complete a self-assessment checklist halfway through the program, at the end, and six months after via email. Likewise, participants completed the IAM, FIM, and AIM at the end of the study, following the second peer assessment meeting.

Data analysis

Quantitative analysis.

The mean scores and standard deviations (SDs) of the IAM, FIM, and IAM were calculated. For the quality-improvement program to be considered appropriate, feasible, and acceptable [ 37 ], a minimum mean score of 15 out of 20, averaged over all participants, was required for each measure. The mean scores and SDs were calculated separately for the self-assessment checklists at three timepoints. For the quality-improvement program to be considered to impact the development of clinical performance, a minimum mean score of 5 out of 10 was required [ 36 ], averaged over all competed self-assessment checklists. Our comparative analysis focused on the mean scores and differences in process and outcome indicators between two periods: the pre-improvement period (the 12 months before the start of the study) and the quality-improvement period (the nine months after the study began). The latter period integrates data from both the initial and subsequent phases of the quality-improvement program, reflecting insights consolidated from the two feedback reports received by the participants during the program (Fig.  1 ). Our analysis focused exclusively on complete case episodes with both baseline and endpoint measurements to ensure the integrity and applicability of the data for participation in the quality-improvement program. All quantitative data were analyzed using SPSS Statistics, version 26 (IBM, Armonk, New York, USA).

The structure of the quality-improvement (QI) program QI = quality improvement; PDSA = Plan-Do-Study-Act

Qualitative analysis

Transcripts of the peer assessment meetings were read in detail, and a directed content analysis was used to study them [ 39 , 40 ]. A codebook was developed in advance, informed by the research questions. Text fragments were labeled according to these a priori codes, which were further refined during the coding process. Meaningful text fragments that could not be labeled were coded inductively. The transcript analysis was supported by ATLAS.ti version 8.4 [ 41 ]. Two researchers (LS and AvK) independently coded the transcripts, discussed the codes to reach consensus, and created the codebook, allocating codes into categories based on their similarities [ 42 ]. A researcher (MM) with ample experience in peer assessment and qualitative research guided this process. Kirkpatrick’s model, which was designed to evaluate the impact of an educational program, was used to allocate the identified categories to four domains: reaction, learning, behavior, and results (see Additional File 7 ) [ 43 ]. Preliminary findings after both peer assessment meetings and the final codes, categories, and the allocation of categories to the domains were discussed by the research team (LS, MM, RvH, AV, and PvdW) in several meetings. A member checking procedure was conducted by sending a summary with preliminary results to all participants after the first meeting to increase the credibility of the results. To optimize the transferability of the results, we aimed to saturate the information by recruiting at least three peer groups.

In total, 10 physical therapists from two different practices participated in the program. Two mixed-practice peer groups were formed, each consisting of five participants. The participants’ characteristics are outlined in Table  1 .

Quantitative results

Table  2 provides an overview of the appropriateness, feasibility, and acceptability of the program, as well as the perceived development in clinical performance. All predefined criteria regarding the minimum score on the IAM, FIM, AIM, and the self-assessment checklists were met.

The mean process and outcome indicator scores for the three data periods are compared in Table  3 . All process indicators improved substantially during and after the quality-improvement cycle, with mean improvements ranging from 9 to 26%.

Qualitative results

We conducted four peer assessment meetings, two for each peer group. After analyzing the qualitative data, the codes were classified into eight major categories. These categories were allocated to the four domains of Kirkpatrick’s model of evaluation (see Table  4 ). Quotes are numbered and labeled by peer group (see Table  5 ).

Domsain: reaction

Program appreciation; suggestions for program improvement.

Participating in a quality-improvement program based on routinely collected health outcomes was novel for most participants. In general, the therapists considered the program’s content meaningful, pleasant, acceptable, and accessible (Q1-G2), and proposed several advancements to increase future program experiences and satisfaction (Q2-G1)(Q3-G1).

Domain: learning

Awareness and insight.

Most participants became more aware of the existing data and the possibilities for analyzing and comparing them. They developed an understanding of the clinical relevance of the data presented, and identified possible explanatory factors by interpreting and clarifying the data (Q4-G1). Participants also gained insight into how to appropriately design data collection, the importance of proper data collection methods (Q5-G1), and potential areas for implementing data in routine practice.

Participants became more aware of data collection throughout the quality-improvement cycle, but acknowledged the lack of a standardized, valid, and reliable data collection method (Q6-G1). Before the quality-improvement program, most participants did not routinely use data to guide and improve their practice, despite dedicating considerable effort to its collection (Q7-G1).

The evolving knowledge gained from the quality-improvement cycle led participants to realize that routinely implementing data can enhance their clinical practice, and more importantly can significantly benefit patients (Q8-G2). Some participants openly argued that using data will not improve the quality of their physical therapy. They challenged the perceived value of the data in comparison with their own expertise and discussed the required time investment in relation to the perceived returns.

Motivational change

Collecting and using data with the objective of improving quality of care for the patient was not a common mindset among participants. Instead, data collection was performed to meet obligatory external requirements and was not considered a priority. However, as the quality-improvement cycle continued, most participants reported a shift to more intrinsically motivated efforts for collecting data (Q9-G2).

Domain: Behavior

Intentions for behavioral change.

Participants were encouraged to reflect on their own clinical behavior and reported feeling motivated to change their routine practice. All participants planned to improve their process indicators and data collection routines, particularly by allowing patients to complete their own questionnaires. Some participants proposed integrating the data into their practice and investigating its impact on their clinical behavior (Q10-G2).

Demonstrated behavioral change

All participants revised the extent and approach of their data collection. Most participants successfully applied some form of data use in routine practice, such as to evaluate treatment progress, to guide treatment and decision-making processes, as input for taking patient histories, for patient empowerment, for goal setting with the patient, and to complement or contradict their own assumptions (Q11-G2). Although they changed their data collection routines, two participants admitted they still rarely used data to support their clinical behavior (Q12-G1).

Barriers to and facilitators of behavioral change

Participants identified several barriers and facilitators that hindered or helped them to achieve their intended behavioral changes (Q13-G2)(Q14-G2)(Q15-G1) (see Table  6 ). These factors impacted the quantity of data collected, influenced the data collection protocols used, and shaped efforts to integrate data into routine practice.

Domain: results

Goal attainment.

The majority of participants set goals related to processes and collection routines. Seven of the 10 participants accomplished their personal targets regarding improving process indicators (Q16-G2). All participants achieved their objectives around changing data collection routines. One participant openly debated the benefit of goal attainment on the added value and quality of care for the patient (Q17-G1).

This study explored the feasibility of a quality-improvement program designed to enhance the quality of primary care physical therapists. The program uses health outcomes from a national registry and incorporates feedback, peer assessment, and self-assessment in a Plan-Do-Study-Act cycle. We found that the participants considered the program an appropriate, feasible, and acceptable intervention for quality-improvement purposes, and found it beneficial for improving their clinical performance. All participants improved the completeness of the data they collected. They also gained insights into the potential value of using outcome data in clinical practice, as well as in examining their routines and motivation. Participants recognized the importance of handling data, revised their data collection methods, began to implement data use into their routine practice, and observed the impact on their clinical behavior. They acknowledged the added value of using data when formulating clear treatment targets, monitoring treatment processes, motivating patients, and, on an aggregated level, improving the quality of care. While most participants reacted positively to the program and acknowledged its added value, they faced significant challenges, such as the complexity of integrating systematic data collection into daily practice, external pressures to meet specific outcome benchmarks, and the need for more knowledge and skills in data interpretation and application. These factors sometimes hindered the full realization of the program’s benefits and highlighted areas for improvement that should be addressed to improve the program before wider implementation.

Comparison to similar studies

This study builds upon previous research that highlighted the potential value of outcome data in quality-improvement initiatives [ 4 , 24 ]. When evaluating the potential value of feedback, peer assessment, self-assessment, and Plan-Do-Study-Act cycles in physical therapist care, most previous studies did not use aggregated real-world data from clinical registries. Maas et al. [ 23 ] and Steenbruggen et al. [ 36 ] incorporated feedback, peer assessment, and self-assessment in comprehensive quality-improvement programs aimed at the professional development of physical therapists, using client records, video recordings of client communication, and the tracer methodology, respectively. Both programs were found to be feasible and led to improvements in clinical performance [ 23 , 36 ]. The results of the present study support and extend previous findings of these quality-improvement strategies in physical therapist practice.

During the initial peer assessment meeting, the participants gained new knowledge and became more conscious of their own behavior. These findings are consistent with previous research indicating that peer assessment promotes learning, increases self-awareness [ 22 , 44 ], and builds self-concept [ 45 , 46 ]. Additionally, participants developed a critical perspective regarding their daily routines and expressed a desire to change their behavior. A similar enhanced commitment to change was reported by physical therapists who underwent cycles of peer assessment and self-assessment [ 23 ]. These findings are in line with theories of health behavior, which suggest that all behavioral change begins with recognizing one’s own behavior [ 47 ], and with the intention to change [ 48 ].

Another important finding was the observed shift in motivation for collecting data. Prior to the quality-improvement program, data were often collected in a non-validated manner, driven by external factors such as health insurers, and were not used to improve patient care. This is consistent with previous findings that the use of feedback in quality improvement is hindered by a perceived political motive for public reporting rather than improved patient care [ 7 ], by financial incentives from health insurers [ 49 ], and by a lack of experience and skills [ 7 , 50 ]. Instead of collecting data to meet an external goal, most participants moving along the quality-improvement cycle reported a shift to a more intrinsic motivation. This could be attributed to participants giving new meaning to collecting and handling data in their daily practice, and establishing their own personal values. These findings are consistent with Ryan and Deci’s self-determination theory, which states that the basis for intrinsic motivation and behavior is formed by people finding a rationale within themselves [ 51 ]. Indeed, participants in the current study emphasized the importance of having clear self-directed motives for data collection as a key driver of behavioral change. Consistent with this, healthcare providers previously reported being more likely to take steps for quality improvement in response to the feedback of aggregated PROMs if they perceived these data to be credible and beneficial for improving patient care [ 19 ]. Throughout the quality-improvement cycle, learning and understanding of data management continued to be developed through experience and reflection, in line with Dewey’s experiential learning theory [ 52 ].

All participants made self-initiated behavioral changes during the program, which was believed to be supported by the application of knowledge gained by following the Plan-Do-Study-Act cycle [ 24 ]. Setting specific targets and making an action plan may increase the effectiveness of feedback and facilitate behavioral changes [ 18 ]. In the present study, feedback was provided by a colleague, more than once, both verbally and in writing to further increase its effectiveness. The participants were largely successful in changing their data-collection procedures; however, there is still room for improvement in the use of data in routine daily practice. Previous studies have shown that clinicians find PROMs useful for supporting the therapeutic process [ 19 ]; however, it took more time or effort to develop these application skills than was available within the timespan of the program. This assumption is supported by the feedback intervention theory, which assumes that the effectiveness of feedback is lower when the ‘task novelty’ and ‘task complexity’ are higher [ 53 ]. Indeed, participants mentioned a lack of knowledge and skills regarding data application as important barriers to its use. Feeling competent is very important for accomplishing behavioral change, according to the self-determination theory [ 51 ]. Correspondingly, previous research indicated that healthcare providers need more support and guidance on how to structurally implement data into their daily practice [ 19 ].

Strengths and limitations

In this explorative study, an innovative theory- and evidence-based quality-improvement program was developed and implemented in daily physical therapy practice. Integrating multiple proven quality-improvement interventions, combined and informed by outcome data, clearly contributed to the inventive character of this program. Using a participatory strategy for the evaluation of program feasibility during the implementation enhanced the evaluation relevance, as well as providing valuable information regarding the program’s beneficial features and suggestions for improvements from the direct perspectives of the intended end-users. Using both qualitative and quantitative data in a concurrent triangulation design also contributed to the rigor of this study.

This study has several limitations. First, although we intended to include three peer groups for data saturation, only two were ultimately recruited. This could have impacted the validity and transferability of the results. Despite this, the two peer groups provided us with rich data that were deemed sufficient for program evaluation and feasibility study purposes [ 54 ]. Second, the peer groups were comprised of physical therapists selected based on the amount of data they collected. As all participants needed to meet external requirements regarding data collection, they could be seen as early adopters. The voluntary participation and external motivation of the participants may have influenced the results and may limit generalizability to other physical therapists. Third, indicative of its exploratory nature, the study’s sample size was limited, but was deemed sufficient to address our research questions. Additionally, the gender distribution among participants, with nine out of 10 being male, does not reflect the typical gender distribution in primary care physiotherapy in the Netherlands. This discrepancy was unintentional, emerging from the recruitment process, but could nevertheless constitute a selection bias, and underscores the need for caution when generalizing findings across diverse physiotherapy contexts. Lastly, although the coach promoted a safe environment during the group meetings, they were not anonymous, and participants may have felt unable to talk openly. Alongside the fact that the assessments could not be blinded, this may have introduced social desirability bias.

Implications for research and practice

Our findings can be used by national physical therapist bodies and other stakeholders in the field to develop initiatives for improving outcome-based care. This program is well suited for use in primary physical therapy care as it integrates with the peer assessment methodology commonly used in many practices. Such integration minimizes the opportunity costs usually associated with new initiatives by leveraging existing peer-learning and feedback structures, making it a feasible and cost-effective strategy for quality improvement [ 55 ]. Additionally, recommendations for advancing the national clinical data registry may further improve the usability for end-users and future researchers, who may wish to study whether the findings are also generalizable to other primary care physical therapist practices. In this study, feedback reporting appeared to support the establishment of quality-improvement goals, and future research could investigate the value of these strategies in evaluating results and changing clinical practices. The sustainability of the observed participant’s behavioral changes and their translation of their revised data-collection routines into quality improvements in care require further consideration. Future studies could improve the program’s feasibility by directly addressing the identified facilitators. Additionally, the program’s impact on patient outcomes should be explored in a full-scale study with long-term follow up.

This explorative study demonstrated that a quality-improvement program incorporating feedback, peer assessment, and self-assessment in a Plan-Do-Study-Act cycle, and using health outcomes from a national registry, was deemed feasible for quality improvement. The implementation of the program led to knowledge development, perceived improvements in clinical performance, and a change in the behavior of the physical therapists regarding data handling in their routine practice.

Data availability

The datasets used and/or analyzed during the current study are available from the corresponding author upon reasonable request.

Abbreviations

Acceptability Intervention Measure

Cooperation of Physical Therapists Nijmegen

Electronic health system

Feasibility Intervention Measure

Global Perceived Effect

Intervention Appropriate Measure

Royal Dutch Society for Physical Therapy

• Nonspecific low back pain

Numeric Pain Rating Scale

Patient-reported outcome measures

Patient-reported outcomes

Patient Specific Functioning Scale

Quebec Pain Disability Scale

STarT Back Screening Tool

Standard deviation

QUality Improvement Reporting Excellence

Spath P, Kelly DL. Applying quality management in healthcare: a systems approach. Health Administration Press Chicago; 2017.

Porter ME. Value-based Health care Delivery. 2008;248:503–9.

Google Scholar  

Meerhoff GA, Verburg AC, Schapendonk RM, Cruijsberg J, Nijhuis-van der Sanden MW, van Dulmen SA et al. Reliability, validity and discriminability of patient reported outcomes for non-specific low back pain in a nationwide physical therapy registry: a retrospective observational cohort study 2021;16:e0251892.

Verburg AC, van Dulmen SA, Kiers H, Nijhuis-van der Sanden MW, van der Wees PJ. Patient-reported outcome-based quality indicators in Dutch primary care physical therapy for patients with nonspecific low back Pain. A Cohort Study; 2021.

Swinkels RA, Meerhoff GM, Custers JW, van Peppen RP, Beurskens AJ, Wittink H. Using outcome measures in daily practice: development and evaluation of an implementation strategy for physiotherapists in. Neth. 2015;67:357–64.

van Dulmen S, van der Wees P. Eindrapport toepassing van minimale dataset en kwaliteitsindicatoren voor lage rugklachten in de fysiotherapie n.d.

Greenhalgh J, Dalkin S, Gibbons E, Wright J, Valderas JM, Meads D et al. How do aggregated patient-reported outcome measures data stimulate health care improvement? A realist synthesis. 2018;23(1):57–65.

Koes BW, Van Tulder M, Lin C-WC, Macedo LG, McAuley J, Maher C. An updated overview of clinical guidelines for the management of non-specific low back pain in primary care. 2010;19(12):2075–94.

Koninklijk Nederlands Genootschap voor Fysiotherapie (KNGF). / vereniging van Oefentherapeuten Cesar en Mensendieck (VvOCM). KNGF-richtlijn Lage rugpijn en lumbosacraal radiculair syndroom. Amersfoort/Utrecht: KNGF/VvoCM; 2021. n.d.

Copeland JM, Taylor WJ, Dean SG. Factors influencing the use of outcome measures for patients with low back pain: a survey of New. Z Phys Therapists. 2008;88(12):1492–505.

Jette DU, Halbert J, Iverson C, Miceli E, Shah P. Use of standardized outcome measures. Phys Therapist Practice: Perceptions Appl. 2009;89(2):125–35.

van Dulmen SA, van der Wees PJ, Staal JB, Braspenning J, Nijhuis-van der Sanden MW. Patient reported outcome measures (PROMs) for goalsetting and outcome measurement in primary care physiotherapy. Explorative Field Study. 2017;103:66–72.

Meerhoff GA, van Dulmen SA, Cruijsberg JK, Nijhuis-van der Sanden MW, Van der Wees PJ. Which factors influence the use of patient-reported outcome measures in Dutch physiotherapy practice? A cross-sectional study. 2020;72(1):63–70.

Irizarry T, DeVito Dabbs A, Curran CR. Patient portals and patient engagement: a state of the science review. 2015;17(6):e148.

Sorondo B, Allen A, Bayleran J, Doore S, Fathima S, Sabbagh I et al. Using a patient portal to transmit patient reported health information into the electronic record: workflow implications and user experience. 2016;4(3).

Ivers N, Jamtvedt G, Flottorp S, Young JM, Odgaard-Jensen J, French SD et al. Audit and feedback: effects on professional practice and healthcare outcomes. 2012.

van der Wees P, van der Vleuten C. Marjo Maas Ria Nijhuis–van der Sanden Geert Rutten Yvonne Heerkens n.d.:187.

Gibbons C, Porter I, Gonçalves-Bradley DC, Stoilov S, Ricci-Cabello I, Tsangaris E et al. Routine provision of feedback from patient-reported outcome measurements to healthcare providers and patients in clinical practice 2021.

Greenhalgh J, Dalkin S, Gooding K, Gibbons E, Wright J, Meads D et al. Functionality and feedback: a realist synthesis of the collation, interpretation and utilisation of patient-reported outcome measures data to improve patient care. 2017;5:1–280.

Maas MJ, Nijhuis-van der Sanden MW, Driehuis F, Heerkens YF, van der Vleuten CP, van der Wees PJ. Feasibility of peer assessment and clinical audit to self-regulate the quality of physiotherapy services: a mixed methods study 2017;7:e013726.

Maas MJ, van der Wees PJ, Braam C, Koetsenruijter J, Heerkens YF, van der Vleuten CP et al. An innovative peer assessment approach to enhance guideline adherence in physical therapy: single-masked, cluster-randomized controlled trial. 2015;95(4):600–12.

Phillips Fox D. Peer review of health care professionals: a systematic review of the literature. Melbourne. 2009. https://www.safetyandquality.gov.au/wp-content/uploads/2012/01/25738-LitReview.pdf (accessed June 2022).

Maas MJ, Driehuis F, Meerhoff GA, Heerkens YF, van der Vleuten CP, Nijhuis-van der Sanden MW et al. Impact of self-and peer assessment on the clinical performance of physiotherapists in primary care: a cohort study 2018;70:393–401.

Kampstra NA, Zipfel N, van der Nat PB, Westert GP, van der Wees PJ, Groenewoud AS. Health outcomes measurement and organizational readiness support quality improvement: a. Syst Rev. 2018;18:1–14.

McDonald PL, Foley TJ, Verheij R, Braithwaite J, Rubin J, Harwood K et al. Data to knowledge to improvement: creating the learning health system. 2024;384.

Creswell JW, Plano Clark VL, Gutmann ML, Hanson WE. An expanded typology for classifying mixed methods research into designs 2003:209–40.

Bowen DJ, Kreuter M, Spring B, Cofta-Woerpel L, Linnan L, Weiner D, et al. How we Des Feasibility Stud. 2009;36:452–7.

Ogrinc G, Armstrong GE, Dolansky MA, Singh MK, Davies L. SQUIRE-EDU (standards for QUality Improvement Reporting Excellence in Education): publication guidelines for educational improvement. 2019;94(10):1461.

https://www.npi.nl/cursussen-en-opleidingen/e-learning-en-toetsen/13025-e-learning-hoe-gebruik-ik-data-in-de-praktijk . n.d.

Krägeloh CU, Czuba KJ, Billington DR, Kersten P, Siegert RJ. Using feedback from patient-reported outcome measures in mental health services: a scoping study and typology. 2015;66:224–41.

Hildon Z, Neuburger J, Allwood D, van der Meulen J, Black N. Clinicians’ and patients’ views of metrics of change derived from patient reported outcome measures (PROMs) for comparing providers’. Perform Surg. 2012;12:1–13.

Hildon Z, Allwood D, Black N. Making data more meaningful: patients’ views of the format and content of quality indicators comparing health. care Providers. 2012;88:298–304.

Pronovost PJ, Hudson DW. Improving healthcare quality through organisational peer-to-peer assessment: lessons from the nuclear power industry. 2012;21:872–5.

Wakefield J, Herbert CP, Maclure M, Dormuth C, Wright JM, Legare J, et al. Commit Change Statements can Predict Actual Change Pract. 2003;23:81–92.

Rehring SMA, Steiner JF, Reifler LM, Glenn KA, Daley MF. Commitment to change statements and actual practice change after a Continuing Medical. Educ Intervention. 2021;41:145–52.

Steenbruggen RA, van Heusden-Scholtalbers LA, Hoogeboom TJ, Maas M, Brand P, van der Wees P. Impact and feasibility of a tailor-made patient communication quality improvement programme for hospital-based physiotherapists: a mixed-methods study. 2021;10:e001286.

Weiner BJ, Lewis CC, Stanick C, Powell BJ, Dorsey CN, Clary AS, et al. Psychometric Assess Three New Developed Implement Outcome Measures. 2017;12:1–12.

Cousins JB, Whitmore E. Framing participatory evaluation 1998;1998:5–23.

van Staa A, de Vries K. Directed content analysis: een meer deductieve dan inductieve aanpak bij kwalitatieve analyse 2014;19.

Kiger ME, Varpio L. Thematic analysis of qualitative data. AMEE Guide 131. 2020;42:846–54.

Friese S. Qualitative data analysis with ATLAS. Ti. Sage; 2019.

O’Connor C, Joffe H. Intercoder reliability in qualitative research: debates and practical guidelines 2020;19:1609406919899220.

Kirkpatrick D. Great Ideas Revisit. 1996;50:54–60.

van Dulmen SA, Maas M, Staal JB, Rutten G, Kiers H, Nijhuis-van der Sanden M, et al. Effectiveness of peer assessment for implementing a Dutch physical therapy low back pain guideline. Cluster Randomized Controlled Trial. 2014;94:1396–409.

Epstein RM, Siegel DJ, Silberman J. Self-monitoring in clinical practice: a challenge for medical educators 2008;28:5–13.

Daves DA, Mazmanian PE, Fordis M, Van Harrision R, Thorpe KE, Perrier L. Accuracy of physician self-assessment compared with observed measures of competence 2006;296:1094–102.

Prochaska JO, Velicer WF. Transtheoretical Model Health Behav Change. 1997;12:38–48.

CAS   Google Scholar  

Ajzen I. The theory of planned behavior 1991;50:179–211.

Scholte M, Neeleman-van der Steen CW, van der Wees PJ, Nijhuis-van der Sanden MW, Braspenning J. The reasons behind the (non) use of feedback reports for quality improvement in physical therapy: a mixed-method study. 2016;11(8):e0161056.

Alvarado N, McVey L, Elshehaly M, Greenhalgh J, Dowding D, Ruddle R, et al. Analysis of a web-based dashboard to support the Use of National Audit Data in Quality Improvement. Realist Evaluation. 2021;23(11):e28854.

Ryan RM, Deci EL. Self-determination theory 2017.

Dewey J, Authentic IEL. Experiential learning 1938.

Kluger AN, DeNisi A. The effects of feedback interventions on performance: a historical review, a meta-analysis, and a preliminary feedback intervention theory. 1996;119:254.

Anderson R. New MRC guidance on evaluating complex interventions. 2008;337.

Maas MJ, van Poelgeest E, Weymans C, PJvd W. Module 3 feedback op proces en uitkomsten. KNGF; 201.

Download references

Acknowledgements

The authors express their gratitude to the participating physical therapists from the regional network of the Cooperation of Physical Therapists Nijmegen (CFN).

This research received no specific grant from any funding agency in the public, commercial, or not-for-profit sectors.

Author information

Authors and affiliations.

Scientific Institute for Quality of Healthcare, Radboud university medical center, Kapittelweg 54, 6525 EP, Nijmegen, The Netherlands

LSF Smeekens, AC Verburg, MJM Maas, R van Heerde & PJ van der Wees

HAN University of Applied Sciences, Nijmegen, The Netherlands

Leidsche Rijn Julius Gezondheidscentra, Utrecht, The Netherlands

A van Kerkhof

You can also search for this author in PubMed   Google Scholar

Contributions

Concept/idea/research design: LS, AV and PvdW Data analysis: LS, AvK and MM Project management: LS and AV Writing of the manuscript: LS and AV Peer assessment coach: RvH Approval of final manuscript: LS, MM, RvH, AV and PvdW.

Corresponding author

Correspondence to LSF Smeekens .

Ethics declarations

Ethics approval and consent to participate.

The study protocol was approved by the Medical Ethical Committee of Radboud university medical center (registration #2021–13329). This study complies with the Declaration of Helsinki. All participants provided written informed consent.

Consent for publication

Not applicable.

Competing interests

The authors declare that they have no competing interests.

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary Material 1

Supplementary material 2, supplementary material 3, supplementary material 4, supplementary material 5, supplementary material 6, rights and permissions.

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ . The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Cite this article.

Smeekens, L., Verburg, A., Maas, M. et al. Feasibility of a quality-improvement program based on routinely collected health outcomes in Dutch primary care physical therapist practice: a mixed-methods study. BMC Health Serv Res 24 , 509 (2024). https://doi.org/10.1186/s12913-024-10958-5

Download citation

Received : 27 July 2023

Accepted : 05 April 2024

Published : 24 April 2024

DOI : https://doi.org/10.1186/s12913-024-10958-5

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Health outcomes
• Physical therapy
• Quality improvement

BMC Health Services Research

ISSN: 1472-6963

• General enquiries: [email protected]

An official website of the Department of Health & Human Services

• Search All AHRQ Sites
• Email Updates

1. Use quotes to search for an exact match of a phrase.

2. Put a minus sign just before words you don't want.

3. Enter any important keywords in any order to find entries where all these terms appear.

• The PSNet Collection
• All Content
• Perspectives
• Current Weekly Issue
• Past Weekly Issues
• Curated Libraries
• Clinical Areas
• Patient Safety 101
• The Fundamentals
• Training and Education
• Continuing Education
• WebM&M: Case Studies
• Training Catalog
• Submit a Case
• Improvement Resources
• Innovations
• Submit an Innovation
• About PSNet
• Editorial Team
• Technical Expert Panel

Why do nurses miss nursing care? A qualitative meta-synthesis.

Peng M, Saito S, Mo W, et al. Why do nurses miss nursing care? A qualitative meta‐synthesis. Jpn J Nurs Sci. 2024;21(2):e12578. doi:10.1111/jjns.12578.

Missed nursing care is an indicator of poor quality. This review synthesizes what causes missed nursing care. The causes were grouped into three main themes: intrinsic resources, system structure, and social environment. The review also presents nurses' strategies to overcome challenges such as incorporating informal teaching into every patient interaction.

Chemotherapy regimen checks performed by pharmacists contribute to safe administration of chemotherapy. December 2, 2015

Medication errors in community pharmacies: the need for commitment, transparency, and research. February 20, 2019

Team structure and adverse events in home health care. June 6, 2007

Prevention of medical accidents caused by defective surgical instruments. April 11, 2012

Effect of lawsuits on professional well-being and medical error rates among orthopaedic surgeons. August 30, 2023

Perioperative handoff enhancement opportunities through technology and artificial intelligence: a narrative review. June 14, 2023

An evaluation of the effects of human factors and ergonomics on health care and patient safety practices: a systematic review. July 22, 2015

The mental health trigger tool: development and testing of a specialized trigger tool for mental health settings. July 10, 2019

Improved pain resolution in hospitalized patients through targeting of pain mismanagement as medical error. April 8, 2009

Intraoperative patient information handover between anesthesia providers. November 5, 2014

A multicenter, multidisciplinary, high-alert medication collaborative to improve patient safety: the Singapore experience. May 8, 2013

Status of patient safety culture in community pharmacy settings: a systematic review. August 23, 2023

Toward safer health care: a review strategy of FDA medical device adverse event database to identify and categorize health information technology related events. November 28, 2018

Unmasking bias in artificial intelligence: a systematic review of bias detection and mitigation strategies in electronic health record-based models. April 17, 2024

Transmission of community- and hospital-acquired SARS-CoV-2 in hospital settings in the UK: a cohort study. November 10, 2021

Postoperative adverse events inconsistently improved by the World Health Organization surgical safety checklist: a systematic literature review of 25 studies. May 25, 2016

Differences in hospitals' workplace violence incident reporting practices: a mixed methods study. May 18, 2022

Implementing the clinical occurrence reporting and learning system: a double-loop learning incident reporting system in long-term care. March 24, 2021

Prevalence and factors associated with patient-requested corrections to the medical record through use of a patient portal: findings from a national survey. March 9, 2022

Do drug interaction alerts between a chemotherapy order-entry system and an electronic medical record affect clinician behavior? July 17, 2013

Leveraging patient safety research: efforts made fifteen years since To Err Is Human. September 11, 2019

Lessons from walking the medical distancing tightrope. July 22, 2020

Anatomy of a cyberattack: part 4: quality assurance and error reduction, billing and compliance, transition to uptime. April 27, 2022

Stakeholder perceptions of smart infusion pumps and drug library updates: a multisite, interdisciplinary study. September 25, 2019

Engaging the front line: tapping into hospital-wide quality and safety initiatives. March 14, 2018

Improving patient safety in clinical oncology: applying lessons from Normal Accident Theory. October 21, 2015

Activation of a medical emergency team using an electronic medical recording–based screening system. June 18, 2014

End-to-end lung cancer screening with three-dimensional deep learning on low-dose chest computed tomography. June 5, 2019

Reduction in hospital-wide clinical laboratory specimen identification errors following process interventions: a 10-year retrospective observational study. October 26, 2016

Registration errors among patients receiving blood transfusions: a national analysis from 2008 to 2017. December 16, 2020

The effect of a clinical pharmacist-led training programme on intravenous medication errors: a controlled before and after study. April 2, 2014

Wrong-patient blood transfusion error: leveraging technology to overcome human error in intraoperative blood component administration. January 9, 2019

Effectiveness of a radiofrequency detection system as an adjunct to manual counting protocols for tracking surgical sponges: a prospective trial of 2,285 patients. October 10, 2012

Scoping review of the second victim syndrome among surgeons: understanding the impact, responses, and support systems. March 27, 2024

International evaluation of an AI system for breast cancer screening. January 29, 2020

Effect of genetic diagnosis on patients with previously undiagnosed disease. November 7, 2018

Association of diagnostic stewardship for blood cultures in critically ill children with culture rates, antibiotic use, and patient outcomes: results of the Bright STAR Collaborative. May 18, 2022

Evaluation and accurate diagnoses of pediatric diseases using artificial intelligence. February 27, 2019

Perspective

WebM&M Cases

Electronic Health Record Modernization Caused Pharmacy-Related Patient Safety Issues Nationally and at the VA Central Ohio Healthcare System in Columbus. April 10, 2024

Examining the relationship between nurse fatigue, alertness, and medication errors. April 3, 2024

The association between nurse staffing and quality of care in emergency departments: a systematic review. March 20, 2024

"Good care is slow enough to be able to pay attention": primary care time scarcity and patient safety. March 6, 2024

Medication safety gaps in English pediatric inpatient units: an exploration using work domain analysis. March 6, 2024

Physician and nurse well-being, patient safety and recommendations for interventions: cross-sectional survey in hospitals in six European countries. February 28, 2024

A qualitative study of systems-level factors that affect rural obstetric nurses' work during clinical emergencies. February 21, 2024

Longer work experience and age associated with safety attitudes in operating room nurses: an online cross-sectional study. February 7, 2024

The impact of nursing practice environments on patient safety culture in primary health care: a scoping review. January 31, 2024

Clinical deterioration as a nurse sensitive indicator in the out-of-hospital context: a scoping review. January 24, 2024

Characteristics and patient impact of boarding in the pediatric emergency department, 2018-2022. January 24, 2024

The impact of rationing nursing care on patient safety: a systematic review. January 24, 2024

Patient safety in nursing homes from an ecological perspective: an integrated review. January 17, 2024

Addressing nursing shortages and patient safety using Maslow's hierarchy of needs. January 10, 2024

Nurses' experience with presenteeism and the potential consequences on patient safety: a qualitative study among nurses at out-of-hours emergency primary care facilities. January 10, 2024

Voices from frontline nurses on care quality and patient safety during COVID-19: an application of the Donabedian Model. November 1, 2023

Toxic leadership and its relationship with outcomes on the nursing workforce and patient safety: a systematic review. October 25, 2023

Fragmented: A Doctor's Quest to Piece Together American Health Care. September 27, 2023

Wake-up call: night shifts adversely affect nurse health and retention, patient and public safety, and costs. September 20, 2023

Clinicians' insights on emergency department boarding: an explanatory mixed methods study evaluating patient care and clinician well-being. August 23, 2023

Healthcare-associated infections in adult intensive care units: a multisource study examining nurses' safety attitudes, quality of care, missed care, and nurse staffing. August 16, 2023

Missed nursing care in emergency departments: a scoping review. August 16, 2023

Unpacking the complexity of COVID-19 fatalities: adverse events as contributing factors--a single-center, retrospective analysis of the first two years of the pandemic. August 16, 2023

Associations between self-reported healthcare disruption due to COVID-19 and avoidable hospital admission: evidence from seven linked longitudinal studies for England. August 9, 2023

Healthcare workers' experiences of patient safety in the intensive care unit during the COVID-19 pandemic: a multicentre qualitative study. July 12, 2023

Approaches to improving patient safety in integrated care: a scoping review. June 14, 2023

Connect With Us

Sign up for Email Updates

To sign up for updates or to access your subscriber preferences, please enter your email address below.

Agency for Healthcare Research and Quality

5600 Fishers Lane Rockville, MD 20857 Telephone: (301) 427-1364

• Accessibility
• Disclaimers
• Electronic Policies
• HHS Digital Strategy
• HHS Nondiscrimination Notice
• Inspector General
• Plain Writing Act
• Privacy Policy
• Viewers & Players
• U.S. Department of Health & Human Services
• The White House
• Don't have an account? Sign up to PSNet

Submit Your Innovations

Please select your preferred way to submit an innovation.

Continue as a Guest

Track and save your innovation

in My Innovations

Edit your innovation as a draft

Continue Logged In

Please select your preferred way to submit an innovation. Note that even if you have an account, you can still choose to submit an innovation as a guest.

Continue logged in

New users to the psnet site.

Access to quizzes and start earning

CME, CEU, or Trainee Certification.

Get email alerts when new content

matching your topics of interest

in My Innovations.

Log in using your username and password

• Search More Search for this keyword Advanced search
• Latest content
• Current issue
• BMJ Journals More You are viewing from: Google Indexer

You are here

• Online First
• Locum doctor working and quality and safety: a qualitative study in English primary and secondary care
• Article Text
• Article info
• Citation Tools
• Rapid Responses
• Article metrics

• http://orcid.org/0000-0002-8129-8376 Jane Ferguson 1 ,
• http://orcid.org/0000-0001-9325-3362 Gemma Stringer 2 ,
• http://orcid.org/0000-0002-0696-480X Kieran Walshe 2 ,
• http://orcid.org/0000-0002-2972-7911 Thomas Allen 3 , 4 ,
• http://orcid.org/0000-0003-1621-8648 Christos Grigoroglou 3 ,
• http://orcid.org/0000-0002-2958-915X Darren M Ashcroft 5 ,
• http://orcid.org/0000-0001-6450-5815 Evangelos Kontopantelis 6 , 7
• 1 Health Services Management Centre, School of Social Policy , University of Birmingham , Birmingham , UK
• 2 Alliance Manchester Business School , University of Manchester , Manchester , UK
• 3 Manchester Centre for Health Economics, Division of Population Health, Health Services Research and Primary Care , University of Manchester , Manchester , UK
• 4 Danish Centre for Health Economics , University of Southern Denmark , Odense , Denmark
• 5 NIHR Greater Manchester Patient Safety Research Collaboration (PSRC), Division of Pharmacy and Optometry, Faculty of Biology Medicine and Health , University of Manchester , Manchester , UK
• 6 Division of Informatics, Imaging and Data Sciences , University of Manchester , Manchester , UK
• 7 NIHR School for Primary Care Research, Centre for Primary Care, Division of Population Health, Health Services Research and Primary Care , University of Manchester , Manchester , UK
• Correspondence to Dr Jane Ferguson, Health Services Management Centre, University of Birmingham, Birmingham, UK; j.ferguson.1{at}bham.ac.uk

Background The use of temporary doctors, known as locums, has been common practice for managing staffing shortages and maintaining service delivery internationally. However, there has been little empirical research on the implications of locum working for quality and safety. This study aimed to investigate the implications of locum working for quality and safety.

Methods Qualitative semi-structured interviews and focus groups were conducted with 130 participants, including locums, patients, permanently employed doctors, nurses and other healthcare professionals with governance and recruitment responsibilities for locums across primary and secondary healthcare organisations in the English NHS. Data were collected between March 2021 and April 2022. Data were analysed using reflexive thematic analysis and abductive analysis.

Results Participants described the implications of locum working for quality and safety across five themes: (1) ‘familiarity’ with an organisation and its patients and staff was essential to delivering safe care; (2) ‘balance and stability’ of services reliant on locums were seen as at risk of destabilisation and lacking leadership for quality improvement; (3) ‘discrimination and exclusion’ experienced by locums had negative implications for morale, retention and patient outcomes; (4) ‘defensive practice’ by locums as a result of perceptions of increased vulnerability and decreased support; (5) clinical governance arrangements, which often did not adequately cover locum doctors.

Conclusion Locum working and how locums were integrated into organisations posed some significant challenges and opportunities for patient safety and quality of care. Organisations should take stock of how they work with the locum workforce to improve not only quality and safety but also locum experience and retention.

• Health services research
• Patient safety
• Qualitative research
• Quality improvement

Data availability statement

No data are available.

This is an open access article distributed in accordance with the Creative Commons Attribution 4.0 Unported (CC BY 4.0) license, which permits others to copy, redistribute, remix, transform and build upon this work for any purpose, provided the original work is properly cited, a link to the licence is given, and indication of whether changes were made. See:  https://creativecommons.org/licenses/by/4.0/ .

https://doi.org/10.1136/bmjqs-2023-016699

Statistics from Altmetric.com

Request permissions.

If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.

WHAT IS ALREADY KNOWN ON THIS TOPIC

Despite longstanding policy concerns about the implications of locum working for quality and safety, there has been little empirical research. Understanding how organisations engage, support and work with locums and how locum doctors integrate and interact with the complex and changing systems in which they work is essential if quality and safety are to be improved.

WHAT THIS STUDY ADDS

This qualitative study examines the perspectives of locums, patients and people who work with locums to identify the implications of temporary medical working for quality and safety.

HOW THIS STUDY MIGHT AFFECT RESEARCH, PRACTICE OR POLICY

Organisations should examine how they engage, support and work with locums. Organisations and locums need to reflect on whether their practices support a collective approach to patient safety and quality of care.

Introduction

Temporary doctors, often known as locums, are a vital resource that enable healthcare organisations to deliver care by flexing capacity and covering staffing gaps. In the United Kingdom, all doctors, other than those in their first year of training after qualifying, can work as a locum. Locum work can vary from very short-term (a single shift) to longer-term assignments (weeks, months or even sometimes years). Locums find work through various platforms, including locum agencies, online job platforms, professional networks or word of mouth. Locum agencies typically have some governance responsibilities (such as compliance with regulations and licensing requirements), but the extent of these responsibilities varies and the NHS in England has no oversight over how recruitment agencies operate. Despite concerns among policymakers, healthcare providers, professional associations and professional regulators about the implications of locum working for quality and safety and cost, 1–3 there is limited robust empirical research to evidence or support those concerns.

The workforce retention crisis is a significant challenge in healthcare internationally 4–6 and persistent understaffing poses a serious risk to patient safety. 7 8 In the UK, high doctor turnover has been linked to poorer service and health outcomes 9 and has led NHS trusts and general practices (GPs) to be ‘overly reliant’ 3 on temporary staff to fill rota gaps. 10 11 Expenditure on temporary staff in the NHS in England increased from £3.45 billion to £5.2 billion between 2021 and 2022. 3 12 The NHS Long Term Workforce Plan aims to reduce reliance on temporary staff and make substantive employment the most cost-effective and attractive option. 3 However, with the vacancy rate in the NHS projected to increase, 13 locums are likely to continue to be essential to maintaining service provision, especially in shortage specialities such as psychiatry. 14

An obvious implication of locum working is a reduced likelihood of organisational and team integration, 15 familiarity and a shared understanding of ‘the way things are done around here’. 16 Locums are likely to be less familiar with teams and other contextual factors relevant to providing safe and effective care 17 and more likely to be situated on the periphery of organisational structures, teams and governance systems 1 18 Teamwork represents a powerful process to improve patient care, 19 20 and trust, shared understanding, communication and collaboration have been associated with better patient outcomes. 21 22 The ability of healthcare teams to develop and maintain team situational awareness, or a shared perception, comprehension and subsequent projection of what is going on in complex and changing clinical environments, has been described as crucial for patient safety. 23 24 Through participation and working together, 25 teams gain an understanding of the roles, skills and competencies of others to demonstrate ‘collective competence’, 26 27 which is critical for healthcare delivery, 28 29 and existing research on locums suggests a need for better integration into teams to improve quality and safety. 30 31

Context matters for patient safety and quality improvement, 32 33 yet the limited evidence 17 relating to locums practice is largely ‘acontextual’ and tends to ignore the role of the organisation in the integration of temporary staff, focusing instead on the potential risks locums present as individual clinicians, 17 30 which is perhaps unsurprising given the liminal space locums occupy. In the UK, responsibility for the quality and safety of healthcare services is shared primarily between organisations and the individual professionals working within them. 34 Organisations are responsible for creating systems and environments that promote and protect clinical governance and enable all doctors to meet their professional obligations, while doctors are expected to participate in the systems and processes put in place by regulators and organisations to protect and improve patient care. 35 However, NHS trusts and primary care organisations procure the services of locum doctors without assuming the responsibilities normally associated with an employer–employee relationship 30 and locums often struggle to participate in teams and governance systems that were designed for doctors working in conventional employment relationships. 18 36

There is longstanding debate about the role of individual accountability in patient safety and how responsibility is distributed between organisations and individuals. 37 A systems approach reasons that adverse events are likely to occur as a result of system failures rather than individual failures, 38 and patients are protected from mistakes by well designed systems and environments that promote safety cultures. 39 But locums are often positioned at the periphery of these systems, 30 and doctors who are new to and also peripheral to organisations, and organisations who are inexperienced with and unsupportive of locums are unlikely to be able to perform optimally. 40

The aim of this research was to provide evidence on how locum working arrangements impact quality and safety and the implications of locum working for patients, locums and health service organisations in primary and secondary care in the English NHS. Locum doctors are an essential and growing part of the healthcare workforce 1 who have been largely ignored in healthcare workforce research. This research addresses a gap in the empirical evidence base on how locum doctor working arrangements affect quality and safety, and provides, for the first time, an in-depth exploration that includes perspectives from patients, locums and the people they work with.

Study design and setting

A qualitative semi-structured interview and focus group study was conducted with locums, people working with locums, and patients with experience of being treated by locums. Participants were purposively sampled through 11 organisations, including NHS trusts, primary care practices, statutory NHS bodies and locum agencies. Locum doctor participants were recruited through these organisations, locum recruitment agencies and networks. We used purposive, snowball and convenience sampling, drawing on intelligence from stakeholders, including our project advisory group, to identify and recruit organisations and participants. Patient participants were recruited through patient and contributor forums. The forum involved active partnership between patients and researchers in the research process to develop research which is relevant and useful to patient and public needs. Participant demographics were monitored to ensure representation across a broad range of roles in primary and secondary care and to increase diversity in terms of gender and ethnicity (see table 1 ).

• View inline

Characteristics of study participants

Data collection

Three semi-structured interview and focus group guides were developed for use with locums, people working with locums and patients with experience of being treated by locums (as shown in online supplemental files 1-3 ). Our previous review of the literature relating to quality and safety and locum work 17 informed the schedules as well as the initial coding and thematic development. Schedules were also refined and informed by our patient and public involvement (PPI) forum and our project advisory group. Each schedule was intended to explore locum doctor working arrangements with a particular focus on understanding how locum doctor working may affect the safety and quality of care and what strategies or systems organisations and individuals used to assure or improve quality and safety. The topic guides for locums and people working with locums also covered governance and support, the impact of the COVID pandemic and policies and initiatives used to support locums.

Supplemental material

Interviews and focus groups were transcribed verbatim by a professional transcription company and organised into codes and themes using the software package NVivo. 41 Reflexive thematic analysis (RTA) 42 was used and involved familiarisation with the data by reading and re-reading the transcripts and field notes; coding the dataset and collating all relevant data extracts; generating initial themes by examining the codes and collated data to identify significant broader patterns of meaning across the dataset; reviewing themes by questioning whether themes answered the research question and told a convincing story of the data and combining, splitting and discarding themes as necessary; defining and naming themes by developing a detailed analysis of each theme; and finally the analytical write up which positioned the analysis in relation to existing literature. 43 RTA acknowledges the active role of researchers in knowledge production and the researcher’s subjectivity as the analytic resource. 42 RTA recognises interpretive variability between researchers based on differences in their knowledge and skills, theoretical assumptions and differences in how they responded to the dataset is acknowledged and expected. 42 The research team worked reflexively discussing their personal biases and their potential impact on the research at regular meetings throughout the data collection and analysis period. Our PPI forum were also involved in data collection and analysis, and offered a form of triangulation to enhance rigour, challenge and alternative interpretations of the findings. 44 Analysis adopted a constructionist epistemology, in that while we acknowledged the importance of recurrence in generating themes, meaning and meaningfulness were the central criteria in the coding process. 42

After themes were developed, an abductive approach was taken to position findings against a background of existing theory and knowledge. 17 30 This provided a way of constructing empirically based theorisations without confining theory to predefined concepts. 45 This approach integrated inductive data-driven coding with deductive theory-driven interpretation; aiming to find a middle ground between inductive and deductive methods and the most logical solution and useful explanation for phenomena. 45

We conducted 130 interviews with 88 participants who worked in healthcare and 42 patients took part in focus groups and one-to-one interviews. Participants included locums, permanently employed doctors; nurses and other health professionals; medical directors/clinical leaders; responsible officers (ROs are accountable for local clinical governance processes and focus on the performance of doctors) and appraisers; leads for medical staffing and clinical governance and practice managers (see table 2 ). Three experienced qualitative researchers (JF, GS and KW) and two members of the PPI forum (MM and MS) carried out five focus groups with 30 patients, and JF and GS carried out 12 one-to-one interviews. Data were collected between March 2021 and April 2022 during the COVID pandemic using video conferencing software (n=126) or over the phone (n=4) at a time convenient to participants. Interviews and focus groups ranged in length from 23 to 171 min, with the average interview being 59 min.

Healthcare organisations and participant roles

Thematic framework

Our findings are presented under five broad and interrelated themes that examine how locum work relates to and impacts quality and safety: ‘familiarity’ with an organisation and its patients and staff; ‘balance and stability’ in services with lots of locums; ‘discrimination and exclusion’ towards locums and their effects; ‘defensive practice’ by locums; and the positioning of locums outside clinical governance arrangements.

Familiarity: knowing who, where and how

Locums described often working in unfamiliar environments, sometimes with minimal induction and varying levels of support. Unfamiliarity, lack of access to or other restrictions on computer systems, policies, procedures and buildings meant that locums were not always able to do their job safely, productively or effectively.

That’s probably the biggest sort of safety aspect that sticks in my mind, is that it is unbelievably frustrating to have to learn a whole new set of patients from day to day … when I was signed up to four different hospitals, plus the locum agencies, I very quickly realised that not only is it the fact that you don’t know the patients from day to day, if you’re chopping and changing site the whole time, then store cupboards are laid out differently, ways of contacting relevant staff members are different, you’ve got to recognise what code to put in to bleep someone that’s different at every single site. (Interview 23, locum, secondary care)

Locum working sometimes created extra work for permanent staff who were responsible for inducting, training and supervising locums. The amount of additional workload was dependent on contextual factors, such as the experience of the locum, organisational support and length of placement, access to systems and what terms and conditions locums or organisations had negotiated. Locum reliance on permanent staff meant that care could be delayed, partially completed or not completed at all, which sometimes caused resentment.

Some of the things that we don’t … like, for example, procedures of limited clinical value that we don’t refer in for, they won’t know about those in our areas … So they’ll do referrals that we then will get pulled on. They’ll maybe prescribe medications that are not first line medications within our own formulary. So we see quite a bit of that, you know, there’s quite a lot of tidying up to be done afterwards or work. They generate that. So whilst we meet the patient numbers, they create a lot of work for the rest of the team. (Interview 3, practice manager, primary care)

Locums mitigated risks related to working in unfamiliar environments by avoiding organisations considered chaotic or unsafe, working below their grade to avoid having responsibility in unfamiliar organisations where they may not be supported or included in the team or working in a limited number of organisations to increase familiarity.

Most locums take jobs, locum work below their grade. So a person who’s at a registrar level would take a locum work as an SHO (senior house officer), because they don't know the trust that well. (Interview 55, locum, secondary care)

However, lack of familiarity and discontinuity could at times be beneficial for patients and organisations as fresh perspectives offered by locums led to different routes of treatment or management, and could alter organisational cultures or practices.

So that [locum] doctor, through that line of questioning and not having any sort of prior history … ordered the right tests and didn’t feel constrained in that practice about what tests that they could order. And someone subsequently … because when you get referred to hospital, the consultant said that that doctor was very much on the ball. And, of course, that’s a change to lifelong medication. And literally within a month of the medication kicking in, it transformed my life. (Focus group A, patient 1)

Balance and stability

The balance between locum and permanent staff had implications for quality and safety, organisational leadership, long-term planning and governance. Locums were often employed to deliver immediate services and consequently were less likely to be involved in team and organisational development. Locums recognised that having ‘an NHS run by locums’ was detrimental to quality and safety, and some avoided organisations that were locum dependent for this reason. Well functioning established teams were regarded as better able to incorporate a small number of locums without being significantly impacted.

Locum work, my view on it is they’re there to fill a gap. They shouldn’t be relied upon to deliver a service Monday to Friday, day in, day out, week in, week out. And unfortunately my trust see it as that, though, that’s my worry that they feel they’re not just plugging a gap, they’re almost as a workforce … (Interview 84, lead GP, primary and secondary care)

Departments that were disproportionately locum dependent were often perceived to lack clinical leadership and direction. An absence of consistent medical leadership meant that quality improvement was slower or less likely to happen, and trusting relationships between staff were harder to establish.

If you get a department that is disproportionately locum dependent, then it stagnates, it doesn't progress. Things like implementation of new NICE guidance, for example, that sort of thing tends not to happen or happen less well, less quickly. (Interview 30, responsible officer, secondary care)

Discrimination and exclusion

Most locums described negative behaviours and attitudes from staff and some patients, which impacted their involvement, inclusion and experiences in organisations. Negative attitudes and behaviours towards locums could affect turnover, locum well-being, team dynamics and potentially patient safety. Perceived disparities between pay, workload, competence and organisational and team commitment between locums and permanent staff could be sources of resentment and influenced how locums were treated and viewed. This compromised staff communication and reduced the sharing of important patient information.

I guess like any temporary post really, you struggle to invest in them, don't necessarily see them as being part of the team. Not very positive about them, particularly junior staff, particularly in the acute trusts. We'd have locums refusing to come back because of the treatment of the midwives. (Interview 86, clinical lead, secondary care)

Negative perceptions of competency and safety meant that locums were often stigmatised, marginalised and excluded. The identity of locum intersected and overlapped with other identities and was described as ‘layering up’ with ethnicity and gender to further exacerbate discrimination.

Oh, doctors coming over from Germany. There was one locum … that administered a dose of something and the patient died, and then there’s this whole layer of extra negativity attached to locum doctors in general because of what one doctor did, and that doctor happened to be someone from a different ethnicity … As a UK born and qualified doctor I can see that those overseas get it but I can also see that I have experienced that as well. So yeah, it can layer up with the whole locum thing. (Interview 59, locum GP, primary care)

A sense of othering and being seen as less was particularly evident during the COVID pandemic when resources were limited. Some locums described how they were not afforded the same protections as permanent members of staff and were sometimes expected to take on riskier work.

I’ve worked in another practice where, because they live on locums and they live on ad hoc locums, you’re a piece of dirt under the shoe. You don’t get gloves, you didn’t have aprons, you didn’t have a face visor, you didn’t have safety specs, you have to ask for a mask. Not only are you not treated as a service provider, you’re not treated as a colleague, someone with knowledge. (Interview 44, locum GP)

Defensive practice by locums

Locums recognised that they were likely to be scapegoated if things went wrong, and some locums described being more likely to practice defensively. Defensive practice has been defined as deviation from standard practice to avoid litigation, complaints or criticism. 46 Participants reported instances of defensive practice which involved providing services (eg, tests, referrals) or avoiding high-risk decisions, usually to reduce the risk of adverse outcomes such as patient complaints or potential termination of contract at short notice. Locums described practicing defensively because they were attempting to practice as safely as possible in complex unfamiliar environments where they were professionally isolated and perceived negatively. Permanent members of staff could perceive that locums practiced defensively because they lacked confidence in their abilities. The diversion of resources away from more clinically relevant activities placed additional burden on teams, who were already facing significant workload challenges.

Being risk averse and practising defensive medicine usually means more tests, more referrals, whereas holding risk tends to be disadvantageous for you as a locum because what’s the benefit to you of not doing that. You’re benefiting the system by rationing resource, the patient won’t thank you. (Interview 35, locum GP)

Locums described avoiding making decisions when risks to employment or medical licenses were perceived as high. Locums felt they were more vulnerable to criticisms of their clinical competence and disempowered to make decisions. Others felt that some locums were simply avoiding work and evaded responsibility for patients by pushing work onto others or into the future.

You don’t interfere, very simple. Over time locums have learned that if you interfere, if you participate in the team, you participate in patient care, [and this] is when you get into trouble … Well most of the locums that I know will just say, okay, there’s already somebody else who’s made a decision, it’s not my job to make a decision, I just follow through. If things go wrong, call the senior person and be done with it, that’s the end of my role. Actually doing something to protect a patient is not important for a locum because the risk is too high. (Interview 55, locum, secondary care)

Locums fall outside clinical governance arrangements

Governance practices in relation to locums varied widely and were not generally regarded as being as robust in comparison to permanently employed doctors. Responsibility for involving locum doctors in performance feedback, supervision, educational opportunities, appraisal and quality improvement was unclear. While some organisations included locums in their governance activities, others regarded locum work as transactional; where the locum was there to provide a finite service and the organisation assumed no responsibilities for their performance, development or oversight. There were concerns that governance structures were modelled on and designed for permanently employed doctors and did not work for locums. When deficits in performance were undetected or unaddressed, doctor performance and patient safety could be jeopardised.

I think it’s a remote world. It’s like a cloud, you know, it’s like the cloud. We talk about the cloud when it comes to storing information. And I think locum world is a bit like that … And I don’t know the doctors anywhere like as much as I did when I was an RO in the NHS, I knew them all personally. If I used to have a problem, I used to get them in my office there and then, chat it all through, sort it. Can’t do that in locum world, it might take me four days to get hold of the doctor, some of them won’t respond immediately … They don’t know me and I don’t know them. (Interview 51, responsible officer, locum agency)

The absence of typical recruitment processes (involving meeting a doctor, carrying out an interview and following up on references) meant that healthcare organisations were reliant on partial information from locum agencies, which made it difficult to determine competency, scope of practice and suitability for a role. However, staff shortages and a requirement to meet safe staffing ratios meant that organisational leaders had little recourse of action if they were unsure about a doctor’s capability, which caused anxiety and frustration. This suggests that the provision of healthcare superseded ensuring safety standards and necessitated accepting one of two objectionable alternatives; accepting gaps in staffing that may jeopardise patient safety or accepting unknown doctors; each of which may compromise patient safety.

If a locum turns up and I have serious doubts about their ability to do the job to the required standard, I don’t have any recourse … And therefore I’m in a position where either I accept this locum or I don’t. There’s not much in the way of middle ground. Not accepting them is a really unpalatable choice because if I say look, I’m sorry, I don’t think you’re up to this, I think you should go home, that leaves me with a gap. (Interview 30, consultant and responsible officer, secondary care)

Similar governance and information sharing problems were described by locum agencies and NHS organisations; both described difficulties in gathering and sharing feedback. When concerns were raised, participants were often uncertain as to what happened to the information they provided and whether it was shared or acted on. Locums often did not get to hear about concerns raised about them, meaning learning opportunities were missed.

It would give you more confidence if you heard back. And sometimes I'll pick up the phone and you try to do the best you can to make sure this information gets passed on. But I just have this nagging doubt that I'm not always convinced it does. (Interview 30, responsible officer, secondary care)

There was also a perception from some locum agency responsible officers that while most locum doctors were excellent, there were some locums who were isolated and in need of organisational and professional support.

You have to accept that whilst within the agency world, 80 per cent of the doctors we place are excellent, and have no problems, and do a great job, perhaps 20 per cent are those that have shaken down to that 20 per cent in the agency world, because they’ve not succeeded in the NHS, they’ve not got a substantive place, they are lost souls. And they are less able to cope with the vicissitudes of busy clinical life and professional life within a large organisation such as the NHS. (Interview 47, responsible officer, locum agency)

Our findings provide some profound and concerning insights for patient safety and quality of care. The ways in which locums were recruited, inducted, deployed and integrated, and supported by organisations undoubtedly affected quality and safety. Our findings indicate that regardless of their level of experience, it was unlikely that locum doctors would be able to function optimally in unfamiliar environments; and organisations who had poor supportive infrastructure and governance mechanisms for locums were less likely to deliver high-quality safe services.

Locums were often regarded as organisational outsiders—positioned at the periphery of the team and the organisation. The implications of transience and peripheral participation were weaker relationships with organisations, teams, peers and patients, leading some to suggest locum working is better suited to experienced doctors. 47 Consistent with previous research, 48 frequent variation in process, systems and equipment, combined with disruption in relationships and a lack of mutual awareness of team skills and competencies, decreased collective competence, placed additional burden on the wider healthcare team and reduced patient safety. As others have found in research on safe staffing and nursing, 49 temporary staff are not effective substitutes for staff who regularly work in the organisation. Safe medical staffing is not just achieved by filling rota gaps, but also team composition and doctors’ familiarity with the team and organisation must be taken into account. Regulatory agencies should consider locum usage in their inspections and perhaps be particularly concerned when organisations have ‘services run on locums’.

Our research found, as others have, 18 that organisations and doctors sometimes struggled to meet their governance obligations and that governance activities differed based on contractual status and organisational policies and norms, with systems being less robust for locums. This research has highlighted that much still needs to be done to develop governance systems that promote and protect the interests of patients and create an environment which supports locum doctors in meeting their professional obligations.

More positively, locum doctors are a potentially valuable source of information about safety concerns, faulty systems or poor conduct. 50 Locums move between organisations, have broad systems knowledge and are perhaps better placed to identify some quality and safety issues than permanent doctors. However, findings indicate opportunities for shared learning were often missed. Locums recognised their precarity and vulnerability when offering second opinions, sharing improvement ideas or voicing safety concerns; meaning opinions were not always offered and concerns were not always raised. Failure to voice concerns is a persistent problem in healthcare, 51 and locums may be even less inclined to offer potentially valuable information about safety concerns because of their perceptions of unsupportive organisational climates.

Our findings shed light on how temporary doctors fit into the enduring debate 37 around how responsibility between organisational systems and individual professionals is distributed. Locums appear to represent a subsection of the medical profession for whom the wider paradigm shift from a focus on individual blame to a systems approach 52 appeared not to have been made. Locums were often not regarded as a part of the organisation, and therefore the system, and not afforded the same protections as permanent staff when things went wrong. Blaming locums when things go wrong and punishing or sanctioning individuals who make errors in contexts that were not designed to incorporate temporary workers may divert attention from understanding inadequately designed, poorly functioning systems, or indeed the individual practice of other doctors. While we should take into account systemic factors that impede locums from performing safely, we should expect high standards of healthcare professionals, be cognisant of individual agency and recognise the distinction between blaming someone and holding them responsible. 53

Strengths and limitations

This large qualitative study explores locum working and quality and safety in an under-researched, yet growing area of the medical workforce. However, sites were all based in England, which means caution should be taken when extrapolating findings. Similar research in other countries and contexts to understand more about locum doctor working and quality and safety is therefore important. It is possible that our sample may have been skewed towards locums, healthcare professionals and patients who had more negative perceptions and experiences, although accounts resonate with previous research 30 and patient perspectives were generally positive. Our data were collected during the COVID pandemic, which may have affected findings as there was a reduction in locum working during that time 10 11 ; it also meant we were unable to carry out observations, which would have strengthened our findings and mitigated some of the inherent limitations of interviews, such as recall bias. We used both one-to-one interviews and focus groups in data collection. Although flexibility in data collection meant that participants had the option to take part in an interview or a focus group, these methods are used for different reasons and produce different data. There may have been differences in what participants disclosed depending on the method

Our findings show that the way in which doctors who worked on a temporary basis were integrated into organisations posed some significant challenges and opportunities for patient safety and quality of care, and that both organisations and locums had a part to play in improvement. Doctors working as locums are a heterogeneous group with differing backgrounds, experiences, skills and capabilities that likely reflect the variability seen in the wider population of doctors. Locums are working in the same pressured and imperfect systems as other health workers; it is vital that systemic problems are not mistaken for problems about individuals and important to recognise that a locum is not a type of doctor but a way of working. Our findings are a call to action for organisations to take stock of how they engage, support and work with locums, and asks both locums and organisations to reflect on whether their practices support a collective approach to patient safety and quality of care.

Ethics statements

Patient consent for publication.

Not applicable.

Ethics approval

This study involves human participants and was approved by the Health Research Authority North West—Haydock Research Ethics Committee 20/NW/0386. Participants gave informed consent to participate in the study before taking part.

• General Medical Council
• NHS England
• World Health Organization
• Health Social Care Committee
• Harvey PR ,
• Trudgill NJ
• Bower P , et al
• Grigoroglou C ,
• Kontopantelis E , et al
• Shembavnekar N ,
• Bazeer N , et al
• Glatter R ,
• Papadakos P ,
• Ferguson J ,
• Schmutz JB ,
• DiazGranados D ,
• Dietz AS , et al
• Sutradhar R ,
• Jerath A , et al
• Oeppen RS , et al
• Pollack AH ,
• Mishra SR ,
• Apodaca C , et al
• McLaney E ,
• Morassaei S ,
• Hughes L , et al
• Firth-Cozens J
• Tazzyman A ,
• Walshe K , et al
• Pouwels KB ,
• van Hecke O , et al
• Aveling E-L ,
• Dixon-Woods M
• Blumenthal DM ,
• Olenski AR ,
• Tsugawa Y , et al
• QSR International Pty Ltd
• Stewart D ,
• Tierney T , et al
• Timmermans S
• Baungaard N ,
• Skovvang PL ,
• Assing Hvidt E , et al
• Lewis L , et al
• Liberati EG ,
• Tarrant C ,
• Willars J , et al
• Zaranko B ,
• Sanford NJ ,
• Kelly E , et al
• Martin GP ,
• Dixon-Woods M ,
• Aveling EL ,
• Campbell A , et al
• Karsh B-T ,
• Holden RJ ,
• Alper SJ , et al

Supplementary materials

Supplementary data.

This web only file has been produced by the BMJ Publishing Group from an electronic file supplied by the author(s) and has not been edited for content.

• Data supplement 1
• Data supplement 2
• Data supplement 3
• Data supplement 4

Twitter @janefergo, @@kieran_walshe

Contributors JF, KW, DA, TA and EK conceived the study. Recruitment was led by JF and supported by GS. JF, GS and KW conducted the interviews, reviewed and analysed the transcripts, and JF wrote the first version of the manuscript. Two members of the patient and public involvement (PPI) forum also assisted with focus groups. JF conducted data analysis with input from KW and GS, the PPI forum, and review by all authors. JF and KW were involved in initial critical review and revision of the manuscript, followed by all authors. All authors read and approved the final version of the manuscript. JF is the guarantor.

Funding This study was funded by the National Institute for Health and Care Research (NIHR) Health and Social Care Delivery Research programme (NIHR128349), and the NIHR Greater Manchester Patient Safety Research Collaboration (PSRC). The views expressed are those of the authors and not necessarily those of the NIHR or the Department of Health and Social Care.

Competing interests None declared.

Provenance and peer review Not commissioned; externally peer reviewed.

Supplemental material This content has been supplied by the author(s). It has not been vetted by BMJ Publishing Group Limited (BMJ) and may not have been peer-reviewed. Any opinions or recommendations discussed are solely those of the author(s) and are not endorsed by BMJ. BMJ disclaims all liability and responsibility arising from any reliance placed on the content. Where the content includes any translated material, BMJ does not warrant the accuracy and reliability of the translations (including but not limited to local regulations, clinical guidelines, terminology, drug names and drug dosages), and is not responsible for any error and/or omissions arising from translation and adaptation or otherwise.

Linked Articles

• Editorial Locums: threat or opportunity Richard Lilford BMJ Quality & Safety 2024; - Published Online First: 16 Apr 2024. doi: 10.1136/bmjqs-2023-016951

Read the full text or download the PDF:

• Open access
• Published: 24 April 2024

Disaster literacy in disaster emergency response: a national qualitative study among nurses

• Di Zhang 1 ,
• Li-Yan Zhang 1 , 2 ,
• Ke Zhang 3 ,
• Han Zhang 4 ,
• Huan-fang Zhang 5 &
• Kai Zhao 6  

BMC Nursing volume  23 , Article number:  267 ( 2024 ) Cite this article

Metrics details

As the largest group of healthcare professionals, nurses play an indispensable and crucial role in disaster response. The enhancement of nurses’ disaster literacy is imperative for effective disaster emergency management. However, there is currently a lack of knowledge regarding nurses’ disaster literacy. This study represents the first attempt to explore the key components and characteristics of disaster literacy among nurses.

A qualitative descriptive design was employed, and the reporting followed the Consolidated Criteria for Reporting Qualitative Research (COREQ) guidelines (refer to File S1). The purposive sampling method was utilized. Thirty-one rescue nurses from 31 medical institutions across 25 provinces and regions in China were recruited to participate in the study. The respondents were requested to share their experiences and insights regarding disaster rescue operations. Inductive content analysis was employed for data examination.

The results indicated that rescue nurses universally recognized that there was a pressing need to enhance the level of disaster literacy among nurses. The disaster literacy of nurses encompasses nine dimensions: physical and mental quality, disaster rescue general knowledge, professional and technical competence, professional ethics, teamwork, emotional ability, information literacy, leadership, and knowledge transformation.

Conclusions

To ensure national sustainability, government departments, healthcare organizations, and hospital administrators can accurately evaluate the disaster literacy of individual clinical nurses, groups, and the workforce as a whole through nine dimensions, which also can provide evidence to support the development of precision strategies to strengthen the disaster literacy of nurses.

Peer Review reports

Disaster prevention has long been a priority of the international healthcare system [ 1 ]. Effective disaster risk management is intricately intertwined with the attainment of the Global Sustainable Development Goals [ 2 ]. In 2020, for instance, the unforeseen proliferation of COVID-19 impeded global endeavors to accomplish the Sustainable Development Goals worldwide [ 3 ]. Nevertheless, this pandemic has also engendered an unparalleled surge in media and public attention towards nursing [ 4 ], along with a global discourse regarding the pivotal role played by nursing professionals in national disaster response and public safety.

Nurses are the largest group of healthcare professionals globally and serve as the primary responders during disaster rescue operations [ 5 ]. Enhancing disaster literacy within this extensive group can significantly enhance treatment efficiency, an aspect that is often overlooked [ 6 ]. Disasters frequently occur unexpectedly, resulting in chaotic and intricate environments at rescue sites, which starkly contrasts with structured clinical settings comprising fixed clinical departments and medical staff. Consequently, experienced or trained nurses are indispensable for managing the complexities associated with such situations. The demanding treatment environment characterized by inadequate resources, scarcity of supplies and medications, urgent tasks, and psychological stress places heightened demands on nurses engaged in disaster rescue efforts. According to the State of the World’s Nursing Report 2020, advancing disaster nursing disciplines will play a pivotal role in shaping the future of global nursing [ 7 ]. Consequently, adapting traditional first aid structures, nursing skills, and theoretical frameworks to suit disaster scenarios has proven challenging.

Despite this, not every nurse is adequately prepared to confront these situations. Studies have confirmed that nurses’ knowledge, skills, and abilities in disaster emergency response are insufficient to deal with such circumstances [ 8 – 9 ]. Furthermore, it should be noted that the concepts of knowledge, skill, ability, and literacy are distinct and cannot be equated. In the context of disaster rescue operations, saving lives is a crucial ability, life first is the value of faith, and dedication is the necessary character. The disaster rescue scene serves as a litmus test for nurses’ emergency response capabilities, however, possessing the necessary ability does not necessarily imply having literacy in this domain. According to the Organisation for Economic Co-operation and Development (OECD), literacy encompasses not only knowledge and skills but also the capacity to utilize psychosocial resources effectively in complex situations [ 10 ]. Literacy possesses multiple dimensions, extending beyond the mere acquisition of knowledge or skills, with a greater emphasis on their practical application for problem-solving during critical incidents. Consequently, disaster literacy aims at enhancing nurses’ disaster nursing abilities by integrating and promoting their knowledge and skills, processes and methods, emotional attitudes, as well as values [ 11 ]. The development of disaster literacy proves advantageous for nurses when confronted with diverse emergencies that arise from local conditions within a disaster scenario.

However, there is currently a lack of global studies examining the conceptual connotation of disaster literacy in professional groups, specifically focusing on nurses. Furthermore, existing research indicates that nurses may be inadequately prepared to effectively respond to disasters [ 12 – 13 ]. Therefore, it is imperative to discuss the significance of disaster literacy among nurses and elucidate the characteristics that nurses should possess.

The qualitative and descriptive design was used in this study to determine the disaster literacy characteristics that nurses should have from the perspective of rescue nurses. The present study employed a qualitative and descriptive design to ascertain the disaster literacy characteristics that nurses should possess from the perspective of rescue nurses. Utilizing a qualitative descriptive design is considered the most effective approach for directly gathering information from survey participants [ 14 ]. Qualitative descriptive research aims to provide comprehensive accounts of experiences in authentic settings using straightforward language, thereby enabling detailed descriptions of otherwise unknown occurrences [ 15 ]. Consequently, employing a qualitative descriptive study design ensures that data analysis remains faithful to participants’ descriptions and facilitates transparency in researchers’ judgments.

To ensure the interviews remain focused on the topic, this study developed an interview outline based on the STAR tool [ 16 ]. The STAR method is a commonly employed technique by interviewees to effectively respond to behavioral interview questions. STAR stands for Situation, Task, Action, and Result. Through utilizing the STAR method, interviewees articulate their job-related scenarios eloquently, describe their responses comprehensively, and provide detailed outcomes when addressing behavior-based inquiries. Thus, the utilization of the STAR method guarantees that interviewees are capable of delivering concise and persuasive answers. The formal interview outline is finalized after thorough review and modification by experts in the qualitative research field within the project team as presented in Table  1 .

Setting and participants

The purposive sampling method was employed to recruit participants. The selection criteria for participants are as follows: (1) Licensed registered nurses from medical and healthcare institutions nationwide; (2) Participation in at least two out of the four major disaster types, including natural disasters (earthquakes, floods, snowstorms, etc.), accident disaster (mining accidents, traffic accidents, accidents at public facilities and equipment, etc.), public health events (SARS, Avian Influenza, New Crown Pneumonia Pandemic, etc.), and social security events (mass incidents, terrorist attacks, emergencies affecting market stability, particularly significant foreign-related incidents, etc.); (3) Willingness to be interviewed. Recruitment and selection encompass China’s seven administrative geographic regions. The exclusion criteria are as follows: (1) Involvement in fewer than two disaster rescue events; (2) Incomplete provision of personal and disaster relief information during formal interviews leading to unanalyzable data; (3) Insufficient interview time available for the participant.

Data collection

The recruitment notice was disseminated to all provincial disaster nursing professional committees through the Disaster Nursing Professional Committee of the Chinese Nursing Association in this study. Rescue nurses who met the inclusion criteria were contacted via email or phone as stated in the recruitment notice, and they provided us with their personal contact information. Before conducting pre-interviews and formal interviews, we communicated with each individual by phone or email and sent them an informed consent form, which they signed and returned to our research team. The qualitative descriptive design was employed to collect individual interview data from July to September 2021. Due to the influence of epidemic prevention and control measures during that period, telephone interviews were conducted for this study. Before the formal interviews, two participants were selected by the interview team for pre-interviews to identify and address any issues that may arise during the interviews, ensuring their effectiveness. The formal interviews commenced by adhering to the interview outline and requirements. The researchers meticulously recorded each interview, capturing nuances such as tone of voice, intonation, pauses, etc., and appropriately labeling them. Each interview was transcribed verbatim upon initial recording. The duration of these interviews ranged from 30 to 77 min with an average of 47 min.

Data analysis

• Open access
• Published: 23 April 2024

The Role of Chief Medical officers in making Public Health Overview Documents in Norwegian Municipalities. A qualitative Study

• Dag-Helge Rønnevik 1 ,
• Betty J. Pettersen 1 , 2 ,
• Anders Grimsmo 1 &
• Aslak Steinsbekk 1  

BMC Public Health volume  24 , Article number:  1132 ( 2024 ) Cite this article

Metrics details

To investigate how Chief Medical Officers experience their role in the municipalities´ work with making the public health overview documents, demanded by the Norwegian Public Health Act from 2012.

A qualitative study with semi-structured focus group interviews with 21 Chief Medical Officers from 20 different municipalities in Norway. The interviews were conducted in 2017. The data were analyzed thematically.

The Chief Medical Officers were mainly positive to participating in making public health overview documents. They took on roles as leaders of the work, medical advisors, data collectors towards local GPs and listening post to other sectors. Organizational factors like too small positions and a lack of tradition to involve the CMO in public health work were experienced as barriers to their involvement. The collaboration with the public health coordinators was said to be rewarding, and the intersectoral process involved employees from other sectors in a new way in public health. Although there were some positive experiences, several CMOs considered the use and impact of the public health overview document as limited.

There was a large variation in the amount and the type of involvement the Chief Medical Officers had in making the public health overview documents in Norwegian municipalities. More research is needed to understand if this has any consequences for the quality of public health work in the municipalities and whether it is a sign of a changing role of the Chief Medical Officers.

Peer Review reports

The role of Chief Medical Officers (CMOs), also termed public health physicians, medical officer of health and district medical officer, has developed over time [ 1 , 2 , 3 ]. From 1984 it became mandatory for Norwegian municipalities to have one or more CMO as medical advisor [ 4 ].

Two thirds of the 356 Norwegian municipalities have only one CMO, and about half of all CMOs have combined positions as general practitioners (GPs), often with the CMO position as a small add on, typical 20% of a man-year [ 5 ]. The position as a CMO may cover public health, infection control, environmental health care, primary health care management, quality assurance and planning processes, often in a hybrid role as advisors on different managerial levels [ 6 ]. During the COVID-19 pandemic, CMOs in Norway as well as in other countries were heavily involved in infection control and vaccination programs and they were publicly exposed through communication with the general population, management, and politicians [ 7 , 8 ].

The Public health Act (PHA) introduced in 2012 [ 9 ], stated that Norwegian municipalities must involve all sectors in promoting public health, and have a public health overview document. The overview document shall identify the public health challenges in the municipality, including assessment of causal factors and impact on population health. It should be designed to be used as a base for the municipality’s planning and policy development, also in accordance with the Planning and Building Act [ 10 ], representing a ‘Health in All Policies’ (HiAP) as proposed by the WHO [ 11 , 12 ]. The medical expertise represented by the CMOs is stated in the PHA as a prerequisite for this work. One or more CMOs must be employed as medical advisors for the municipality to take care of community medical advice in the municipality’s public health work, including epidemiological analyses.

Previous studies have found that following the introduction of the Public health act in Norway, municipalities have introduced new working methods and professionals and implemented organizational changes on both micro and macro levels [ 13 , 14 ]. This includes appointment of public health coordinators [ 15 ] and a change towards increased understanding and adoption of the new, comprehensive public health policy [ 16 ]. Municipalities that developed health overviews reported more often to prioritising fair distribution of social and economic resources among social groups in political decision-making [ 17 ]. However, a countrywide supervision in 2015 showed that many of the municipalities had not decided how they wanted the CMOs to be involved in the work [ 18 ]. Also internationally there are tensions related to the role of CMOs [ 19 ], but no study focusing on the CMOs role in making overview documents has been identified.

The aim of this study was to investigate how Chief Medical Officers experience their role in the municipalities work with making the public health overview documents demanded by the Norwegian Public Health Act from 2012.

Materials and methods

Study design.

This was a qualitative study with semi-structured focus group interviews with Chief Medical Officers, conducted in 2017. Focus groups were chosen as they foster discussions among the participants and are suitable to explore phenomena that concern common experiences, attitudes or views in a field where people interact [ 20 ]. The Consolidated criteria for Reporting Qualitative research checklist (COREQ) [ 21 ] was consulted for reporting the study.

Participants and recruitment

The goal was to include CMOs from rural and urban areas in Norway, with variation in age, experience in public health, gender and size of the municipality they worked in. Eligible CMOs were recruited from the following three meetings. A yearly course arranged by the Chief County Medical Officer in a county in Central Norway, a compulsory course in the specialist program in community medicine held in Oslo and a workshop on new ways of organizing and managing GP practices in the western part of Norway. The identified participants were invited by e-mail which included information about the study.

Data collection

The data was collected in semi-structured focus group interviews, which lasted about 90 min and were audiotaped and transcribed verbatim. The participants were given verbal information about the study before the interviews started. The interview guide (Fig.  1 ) was developed based on relevant public documents related to public health. It was discussed with a group of 20 stakeholders interested in this topic, as well as discussions among the researchers. The main questions concerned experiences with public health work in general, and in particular the public health overview document.

Interview guide

Data were analyzed thematically, inspired by Braun and Clarke [ 22 ], in an iterative process. First, meanings and patterns were identified and discussed. Then meaning units related to the aim were coded and organized in themes. Seven initial main themes were identified (challenges related to the CMO-role in general, “yes the CMO is important”, intersectoral public health, challenges related to the CMO-role in public health, the role of GPs in public health, organizational conditions, public health vs. health care services). This process was repeated in an iterative process involving all authors and ended with three main themes (types of involvement, reasons for being involved or not, and the impact of the overview document).

A total of 21 CMOs participated in the three focus group interviews (Table  1 ). Twelve were specialists in community medicine/public health, and 11 worked fulltime as CMOs. Half of them were over 45 years old. The intended variation in geography and municipality size was reached. Exact data on years of experience from community medicine and public health were not systematically collected, but it varied from a few years to more than 30 years.

Types of involvement

Most of the informants expressed it as natural for them to participate in making the public health overview document, and that they had made a substantial contribution. A few had been leaders in the process, while others took a role as medical advisors. The main contributions were collecting, selecting and evaluating data, and making recommendations in the written report. Still almost none participated in all parts of the process, and some did not take part at all.

I have given advice regarding structure, scope and working method. I wasn’t involved in the actual writing, but I know it was an extensive struggle to get data from the various sectors.

It was said that having combined positions as GP made it easier to contribute to the collection of and ensuring the quality of the data from the health care sector. This included collecting data from local GPs including helping the GPs in extracting relevant data from the electronic health records. One senior CMO rejected using local data for this purpose because he did not find it trustworthy and would rather rely on national data. Some had interviewed GPs to get their input to the overview document.

It was very rewarding [collecting data from GPs]. We had a series of meetings and the GPs received course credits, and it was just as good to discuss this as discussing other professional matters. So GPs are not hopeless in relation to thinking about public health work, I would say!

The CMOs said that their role in the process was based on their knowledge about the municipality and the health care sector, and their relationships to other stakeholders in public health. One said that the CMO is a listening post to other sectors. Some emphasized their epidemiological training as important, giving them the ability to see relationships between data, do health impact assessments and understand the burden of disease in the local community. Also having skills in presenting knowledge, training in prioritizing and giving advice were mentioned as contributions.

It is the competency in community medicine, to prioritize among many suggested proposals. An employee from the planning department may have a completely different understanding than the CMO. In a sense, it is in a group process the community medicine competency is relevant.

Reasons for being involved or not

The variation in what degree the CMOs had been involved in the process of making the public health overview documents, was mainly said to be due to organizational factors. Too small positions, a lack of tradition to involve the CMO in public health work, municipal directors with limited interests in public health, and limited access to other expertise locally were mentioned as barriers. One said he had deliberately chosen to let the public health coordinator take care of the process, so he could prioritize other tasks. Two CMOs in combined positions said it was hard to combine this kind of planning process with clinical work as GPs due to time constraint and difficulty in finding suitable meeting times. One talked about being involved as a medical alibi to legitimize the work, while another senior doctor felt being on the outside and left behind:

From one perspective, it is good that the work involves more people, but my role has become more obscure. In some way, I have been somewhat degraded […]. The structure of public health work is no longer such that I am an explicit part of the leadership, which I think I should have been, even if it shall be distributed. And not just someone they ask when it suits them.

Those who expressed being most involved described the process as rewarding. The work with the overview document was said to breathe new life into public health work, and giving the CMOs knowledge they would have missed without this process. The collaboration with the public health coordinators was especially mentioned as a valuable experience. One emphasized the importance of having office space in the same building as the public health coordinator. The intersectoral process was said to give good discussions with employees from other sectors, and through this work they saw that they gave politicians a good basis for making important decisions for the health of the population. Also, being organized at a cross-sectoral level with access to other stakeholders involved in public health, with enough time to interact with them, was seen as an advantage.

It is important to be there and remind them of this [the public health perspective] when the decisions are made, and it is time-consuming, because then you have to take part in the planning processes and be involved early enough.

Impact of the health overview

There were different experiences as to the result of the process and how the overview document had been used after completion. One CMO said that she and her CMO colleague had created a web-based version of the overview document on the municipality’s website, with navigation features to make it more accessible. Several informants had witnessed how the process of making the overview document made employees in other sectors connect to public health in a new way. One had been included in intersectoral working groups where they used the overview document as a foundation for further municipal planning.

When the planning program is approved by the politicians, we see many signs of the idea behind. We can now see the results from the work done by the public health coordinator and the CMO, having worked to collect this data that we have collected in a health overview, and that it will be used in the future.

Still, several informants said that they did not see much use or impact of the public health overview document. Reasons given were that the politicians did not care, the plan was neglected and put in a drawer and the recommendations in the overview document were not prioritized in the municipality`s budget. The perspectives of the politicians were also commented. One CMO referred to politicians becoming very focused on measures concerning somatic health, asking “do we have more or less cancer than our neighbors?”. Others commented that they had a hard time making the politicians realize that effects of interventions in public health take time.

But as it turned out, as soon as there was a cost connected to it, it went down to zero and nothing was granted for follow up. The plan was adopted, but interventions in the plan are completely dead. I haven´t bothered to get involved further since it got so negative from the administration and the politicians.

The Chief Medical Officers said they were involved in various roles and to varying degrees in making public health overview documents. This was related to the size of their position as CMOs and different traditions in the municipalities to involve CMOs. How the municipalities used the overview document afterwards also varied.

Study strength and limitations

Although no data were recorded about those invited who did not volunteer to participate, the characteristic of the participants indicates that the sought after variation was achieved. There was good variation in experiences, e.g. positive and negative attitudes towards public health work. Furthermore, the main topics were identifiable in all the interviews, which indicates that more interviews would not be likely to change the findings. Some of the informants were known by the authors’ networks and some knew each other. However, in the transcripts it was not found that the participants tried to please others, rather there were lively discussions about different experiences and viewpoints. This also shows that the use of focus groups was a good choice. All the authors were involved in the inductive analysis which ensured that they contributed with different perspectives and increased the likelihood of identifying the variations in the data. The data were collected before the Covid-19 pandemic, and it may be that interviews conducted after the pandemic would have given some different experiences. However, we assume that this would mainly concern infectious disease control, and that the experiences related to public health work would be similar to the situation in 2017.

Health in all policy

The intention behind the public health overview documents and part of the public health act in Norway, is to formalize the implementation of health in all policies (HiAP) [ 13 , 23 ]. None of the participants expressed any objection to this approach. On the contrary, the CMOs mainly expressed positive attitudes to the process of making public health overviews regardless of their own involvement. This is in accordance with an evaluation report that found that making overview documents contributed to a strengthened anchoring of public health in the municipal planning system and an increase in the use of resources for the organization and planning of public health work [ 24 ].

Some of the CMOs expressed an appreciation of and motivation for cross-sectoral work, but overall, there were few who talked about having a leading role in working with the overview. This is in line with previous evaluation of actions that is termed ‘intersectoral action for health’ (ISA), which concerns how the health sector works with other governmental and non-state sectors to improve health and well-being [ 25 ]. A Danish document analysis showed that the ideas that ISA builds on, has proven difficult to implement, and that the guidelines for municipal ISA were vague and constructed with buzzwords [ 26 ].

Furthermore, Health in all policy in Norway is nationally decided upon, but not fully recognized locally [ 27 ]. There are reports pointing out that Norwegian municipalities are at different levels regarding building competence necessary to achieve the objectives of the Public Health Act [ 28 ]. A report from 2015 concluded that the work required more resources, time, expertise and capacity than the municipalities had at their disposal [ 29 ]. It was stated that especially small municipalities did not have enough resources to meet the expectations from the governments. This is a likely reason for some of the experiences of the CMOs about the role of both the planning and use of the overview document.

The role of the CMOs

There was a large variation of how involved the CMOs were in making of the public health overview documents, from hardly taking part, to having a central role. There were also comments about the changing role of the CMOs in the municipalities over time. One possible explanation is the increased focus on implementing inter-sectoral working groups and increasing numbers of public health coordinators. Such activities are used to coordinate and implement public health policies and measures in the municipalities, and are considered important when building capacity to reduce health inequities [ 30 ].

A major contribution from the CMOs that participated in the work, was the contact with GPs, both through helping to collect data from electronic health record and to interview GPs. This could in part be explained by the opportunities that come with CMOs working in combined positions [ 31 ], i.e. as both CMOs and GPs. Although CMOs have different attitudes towards their responsibility to go beyond clinical medicine to help even out social inequality in health [ 32 ], this activity points to a wish among the CMOs to use knowledge from the clinical work in primary care as a contribution to the population health perspective.

Chief Medical Officers have experienced large variation in their involvement in making public health overview documents in Norwegian municipalities. More research is needed to understand if this has any consequences for the quality of public health work in the municipalities and whether it is a sign of a changing role of the Chief Medical Officers.

Data availability

The datasets generated and analysed during the current study are not publicly available due to data regulations but are available from the corresponding author on reasonable request.

Larsen Ø, Olsen BO. The shaping of a profession: Physicians in Norway, past and present. Oslo: Science History; 1996.

Google Scholar  

Larsen Ø. Legerollen–verktøy som må voktes [The physician role - tools to be guarded]. Michael. 2022;19:215–24.

Pettersen BJ. Local public health physicians in Norway from 1994 to 2002: workload, work content, and interaction: a story of everyday life in primary health care. Tromsø: University of Tromsø; 2008.

Lov om kommunale. Helse- og omsorgstjenester m.m. (helse- og omsorgstjenesteloven) [Health and Care services Act]. LOV-2011-06-24-30, https://lovdata.no/dokument/NL/lov/2011-06-24-30 (2012).

NOU. 2022: 5. Myndighetenes håndtering av koronapandemien– del 2 — Rapport fra Koronakommisjonen [The governments handling of the corona pandemic - part 2 - Report from the Corona Commission], https://www.regjeringen.no/no/dokumenter/nou-2022-5/id2910055/ (2022).

Fossberg BC, Frich JC. Kommuneoverlegers opplevelse Av Egen rolle [How Chief Medical officers experience their role]. Tidsskrift Norske Legeforening. 2022;142:121–6. https://doi.org/10.4045/tidsskr.21.0589 .

Article   Google Scholar  

Hungnes T, Vik E, Veddeng O. Kommunelegens rolle under koronapandemien–en kvalitativ studie [The role of chief medical doctors during the corona pandemic– a qualitative study]. Tidsskr nor Laegeforen. 2022;142:1391–5.

MacAulay M, Macintyre AK, Yashadhana A, et al. Under the spotlight: understanding the role of the Chief Medical Officer in a pandemic. J Epidemiol Community Health. 2022;76:100–4. https://doi.org/10.1136/jech-2021-216850 .

Article   PubMed   Google Scholar  

Lov om folkehelsearbeid (Folkehelseloven) [The Public Health Act]. LOV-2011-06-24-29, https://lovdata.no/dokument/NL/lov/2011-06-24-29?q=folkehelseloven (2012).

Lov om planlegging og. Byggesaksbehandling (Plan- Og bygningsloven) [The Planning and Building Act]. LOV-2008-06-27-71, https://lovdata.no/dokument/NL/lov/2008-06-27-71?q=plan-%20og%20bygning (2008).

World Health Organization. The Helsinki Statement on Health in All Policies, https://www.who.int/publications/i/item/9789241506908 (2013).

World Health Organization. Norway: Health system review, https://apps.who.int/iris/handle/10665/331786 (2020).

Hofstad H. The ambition of Health in all policies in Norway: the role of political leadership and bureaucratic change. Health Policy. 2016;120:567–75. https://doi.org/10.1016/j.healthpol.2016.03.001 .

Helgesen M, Hofstad H. Regionalt og lokalt folkehelsearbeid. Ressurser, organisering og koordinering. En baselineundersøkelse [Regional and local public health work. Resources, organization and coordination. A baseline survey], https://oda.oslomet.no/oda-xmlui/bitstream/handle/20.500.12199/5507 /2012-13.pdf?sequence=1 (2012).

Hagen SHM, Torp S. Health in all policies: a cross-sectional study of the public health coordinators’ role in Norwegian municipalities. Scand J Public Health. 2015;43:597–605. https://doi.org/10.1177/1403494815585614 .

Fosse E, Helgesen MK, Hagen S, et al. Addressing the social determinants of health at the local level: opportunities and challenges. Scand J Public Health. 2018;46:47–52. https://doi.org/10.1177/1403494817743896 .

Article   CAS   PubMed   Google Scholar  

Hagen S, Øvergård KI, Helgesen M, et al. Health Promotion at Local Level in Norway: the Use of Public Health Coordinators and Health Overviews to promote fair distribution among Social groups. Int J Health Policy Manage. 2018;7:807–17. https://doi.org/10.15171/ijhpm.2018.22 .

Norwegian Board of Health Supervision. Med blikk for betre folkehelse [Improving public health]. Report of the Norwegian Board of Health Supervision 4/2015, https://www.helsetilsynet.no/publikasjoner/rapport-fra-helsetilsynet/2015/med-blikk-for-betre-folkehelse-oppsummering-av-landsomfattande-tilsyn-i-2014/ (2015).

Cassola A, Fafard P, Nagi R, et al. Tensions and opportunities in the roles of senior public health officials in Canada: a qualitative study. Health Policy. 2022;126:988–95. https://doi.org/10.1016/j.healthpol.2022.07.009 .

Article   PubMed   PubMed Central   Google Scholar  

Morgan DL, Krueger RA. When to use focus groups and why. In: Morgan DL, editor. Successful focus groups: advancing the state of the art. Newsbury Park: Sage; 1993. pp. 3–19.

Chapter   Google Scholar  

Tong A, Sainsbury P, Craig J. Consolidated criteria for reporting qualitative research (COREQ): a 32-item checklist for interviews and focus groups. Int J Qual Health Care. 2007;19:349–57. https://doi.org/10.1093/intqhc/mzm042 .

Braun V, Clarke V. Using thematic analysis in psychology. Qual Res Psychol. 2006;3. https://doi.org/10.1191/1478088706qp063oa .

Nilsen HM, Feiring E. Local public health projectification in practice: a qualitative study of facilitators and barriers to a public health plan implementation. Scand J Public Health. 2023;51:874–81. https://doi.org/10.1177/14034948221080402 .

Løset GK, Veenstra M. Forebyggende helsearbeid i kommunene. Prosessevaluering av Samhandlingsreformen. [Process evaluation of the coordination reform]. NOVA Rapport 6/15, https://oda.oslomet.no/oda-xmlui/handle/20.500.12199/5088 (2015).

Rantala R, Bortz M, Armada F. Intersectoral action: local governments promoting health. Health Promot Int. 2014;29:92–102. https://doi.org/10.1093/heapro/dau047 .

Holt DH, Waldorff SB, Tjørnhøj-Thomsen T, et al. Ambiguous expectations for intersectoral action for health: a document analysis of the Danish case. Crit Public Health. 2018;28:35–47. https://doi.org/10.1080/09581596.2017.1288286 .

Amdam R. Health in all policies in Norwegian municipalities—dilemmas and lack of legitimacy and capacity. Int J Health Plann Mgmt: 1–15. https://doi.org/10.1002/hpm.3664 .

Synnevåg E, Amdam R, Fosse E. Legitimising Inter-sectoral Public Health policies: a challenge for Professional identities? Int J Integr Care. 2019;19:1–10. https://doi.org/10.5334/ijic.4641 .

Hofstad H, Lid S, Schou A et al. Lokal oversikt over folkehelsen– muligheter og utfordringer [Local public health overview - possibilities and challenges], http://www.hioa.no/extension/hioa/design/hioa/images/nibr/files/filer/2015-13.pdf (2015).

Helgesen M, Fosse E, Hagen S. Capacity to reduce inequities in health in Norwegian municipalities. Scand J Public Health. 2017;45(18suppl):77–82. https://doi.org/10.1177/1403494817709412 .

Thorlby R. Reclaiming a population health perspective. Future challenges for primary care, https://www.nuffieldtrust.org.uk/files/2017-01/reclaiming-a-population-health-perspective-web-final.pdf (2013).

Bringedal B, Bærøe K. Bør leger bidra til å utjevne sosial ulikhet i helse? [Should doctors help even out social inequality in health?]. Tidsskr nor Legeforen. 2010;130:1024–7. https://doi.org/10.4045/tidsskr.09.0648 .

Download references

Acknowledgements

We especially would like to thank the Chief Medical Officers taking part in the interviews.

This project has been made possible by funding from the Dam Foundation in partnership with The Norwegian Health Association. Additional funding has been given by Norwegian Centre of Rural Medicine, The Liaison Committee for education, research and innovation in Central Norway and the research fund of The Medical Association of Rogaland.

Open access funding provided by Norwegian University of Science and Technology

Author information

Authors and affiliations.

ISM, NTNU, Trondheim, Norway

Dag-Helge Rønnevik, Betty J. Pettersen, Anders Grimsmo & Aslak Steinsbekk

Municipality of Trondheim, Trondheim, Norway

Betty J. Pettersen

You can also search for this author in PubMed   Google Scholar

Contributions

DHR, BJP and AG led the conceptualization of the study, conducted the key informant interviews, and wrote the first draft of the article. AS provided oversight for the investigation, methodological support, and substantive editorial feedback. All authors read and approved the final manuscript.

Corresponding author

Correspondence to Dag-Helge Rønnevik .

Ethics declarations

Ethics approval and consent to participate.

The study was approved by The Norwegian Centre for Research Data (project 52922) who ensure that the projects´ processing of data is done in accordance with the national data protection legislation. The study was carried out in accordance with the Declaration of Helsinki. Informed consent was obtained from all participants.

Consent for publication

Not applicable.

Competing interests

The authors declare no competing interests.

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ . The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Cite this article.

Rønnevik, DH., Pettersen, B.J., Grimsmo, A. et al. The Role of Chief Medical officers in making Public Health Overview Documents in Norwegian Municipalities. A qualitative Study. BMC Public Health 24 , 1132 (2024). https://doi.org/10.1186/s12889-024-18608-5

Download citation

Received : 20 September 2023

Accepted : 15 April 2024

Published : 23 April 2024

DOI : https://doi.org/10.1186/s12889-024-18608-5

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Chief Medical Officer
• Public health
• Community medicine
• Epidemiology
• Preventive medicine
• General practice

BMC Public Health

ISSN: 1471-2458

• Submission enquiries: [email protected]
• General enquiries: [email protected]

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings

Preview improvements coming to the PMC website in October 2024. Learn More or Try it out now .

• Advanced Search
• Journal List
• Sultan Qaboos Univ Med J
• v.8(1); 2008 Mar

Qualitative Research and its Uses in Health Care

Although relatively uncommon in health care research, qualitative research is now receiving recognition and is increasingly used in health care research with social and cultural dimensions. Unlike quantitative research, which is deductive and tends to analyze phenomena in terms of trends and frequencies, qualitative research seeks to determine the meaning of a phenomenon through description. It aims to develop concepts that aid in the understanding of natural phenomena with emphasis on the meaning, experiences and views of the participants. Differences among qualitative researchers exist on matters of ontology, epistemology, data collection methods and methods of evaluation. The aim of this article is not to act as a practical guide on how to conduct qualitative research, but is an attempt to give an introduction to qualitative research methods and their use in health-related research.

Q ualitative research is defined as an umbrella term covering an array of interpretative techniques which seek to describe, decode, translate and otherwise come to terms with the meaning, not the frequency, of certain more or less naturally occurring phenomena in the social world. 1 , 2

As a method of inquiry, it was first used by sociologists and anthropologists in the early twentieth century, although it existed much earlier than that in its non-structural form. Researchers studied cultures and groups in their own and foreign settings and told stories of their experience long before then. In the 1920s and 1930s, social anthropologists and sociologists implemented a more focused approach compared to the old unsystematic and journalistic style used in those days. Since the 1960s, qualitative research has experienced a steady growth starting with the development of grounded theory and new publications in ethnography. 3 , 4 The number of books, articles and papers related to qualitative research has increased tremendously during the past 20 years and more researchers, including health-related professionals, have moved to a more qualitative paradigm adapting and modifying these approaches to the study needs of their own areas. 4

Since qualitative research does not aim to enumerate, it is sometimes viewed as the exact opposite to quantitative methods and the two methods are frequently presented as antagonists. Quantitative research is based on structure and uses experiments and surveys as methods. In addition, it is deductive in nature and uses statistical sampling methods. In contrast, qualitative research is described as an action research using observation and interview methods. It is inductive in nature and depends on the purposeful selection of participants. Whereas quantitative researchers use reliability as a way of verification, qualitative researchers use validity. Recently, there has been a growing recognition that the quantitative-qualitative debate and distinction is unnecessary and that it would be more fruitful for the relation between the two methods to be complementary and overlapping rather than exclusive. 6 In fact, neither qualitative nor quantitative research is superior to the other; there are weaknesses and strengths in each method. Qualitative and quantitative research methods can indeed be seen as complementary and both are necessary to provide an understanding of a phenomenon.

Several researchers have argued that the research questions and the phenomenon under investigation should determine the methodology to used. 7 , 8 , 9 The crucial question, therefore, is not “what is the best research method?” but “what is the best research method for answering this question most effectively and efficiently?” 10 So, while qualitative and quantitative research may well investigate similar topics, each will address a different type of question. Holman sums up this position: “true understanding in medicine cannot be achieved without adding qualitative methods to the research arsenal”. 11

Traditional quantitative methods, such as randomised controlled trials, are the appropriate means of testing, for example, the effect of an intervention or a treatment, while a qualitative exploration of beliefs and understandings is needed to find out why the results of research are often not implemented in clinical practice. The aim of qualitative research is to develop concepts that can help us understand social phenomena in natural settings, giving emphasis on the meanings, experiences and views of the participants. 10

Qualitative research methods are the most suitable for this approach because of their emphasis on people’s lived experience. They are considered to be well suited for locating the meanings that people place on the events, processes, and structures of their lives and their perceptions, presuppositions and assumptions. 5 In his advice to graduate students, Patton lists a number of conditions that are suitable for a qualitative study. These include: questions about people’s experiences; inquiry into the meanings people make of their experiences; studying a person in the context of her or his social/interpersonal environment and research where it is difficult to develop a standardised instrument due to the lack of knowledge on the phenomenon. 12

ARGUMENTS FOR AND AGAINST QUALITATIVE METHODS

There are major differences of opinions among qualitative researchers on matters of ontology and epistemology as well as the methods to be used and criteria of evaluation. There are also disagreements about the nature, purpose, status and practice of its methods. A large number of authors take a predominantly method-based approach; authors such as Miles and Huberman 5 and Patton 12 put emphasis on data collection techniques. Another approach is to classify qualitative research according to research traditions, i.e. whether phenomenological, grounded theory or ethnography, amongst others. Authors such as Creswel 9 and Denzin and Lincolin 3 prefer this approach, which has the advantage of being based on systematisation of knowledge providing a sense of order and orientation. On the other hand, it has the disadvantage of oversimplification, ignoring the issues of the research question and conceptual frameworks used and the way these issues can shape the research process and the findings. 13 In addition, some researchers have decided to classify qualitative research according to the research question or the method of analysis.

Although the majority of qualitative researchers stress that qualitative research is inductive in nature, in contrast to quantitative research which is deductive, there are qualitative researchers who argue that both can be used for different purposes and at different times, and that qualitative research can be done in a deductive way where prior assumptions are tested on new cases. 14 Retroduction, which is defined as the movement backward and forward between theory and data or the combination of deduction and induction, is said to be a characteristic of qualitative research. 11 The degree of deduction or induction and which one follows the other depends on the research question. 12 .

Guba and Lincoln 15 used the term “emergent design” to describe a qualitative study design that emerges as the study progresses in response to the researcher’s early observations. There are also qualitative studies in health care that base their research questions on the results of prior literature of quantitative studies on the subject. 16 , 17 While the research questions in these studies were relatively precise, the method used was flexible.

Miles and Huberman 5 state that no study conforms exactly to a standard methodology and that the researcher bends the methodology to the peculiarities of the study. According to Creswell, 9 traditions of inquiry used “need not be pure, and one might mix procedures from several”. Mixing approaches and procedures is encouraged by some authors and regarded as a creative approach to qualitative research. 12

Studies using qualitative research in health care have been criticised for the misguided separation of method from theory and of technique from the conceptual underpinnings. 18 However, qualitative health researchers respond by stating that the choice of method and how it is used can perfectly well be matched to what is being studied rather than to the methodological leanings of the researcher. 10 It has been suggested that incorporating qualitative research method experts into health research teams enriches research and ensures that the right methodology is used for answering the right questions. Finally, using qualitative methods in health-related research has resulted in more insight into health professionals’ perceptions of lay participation in care and identification of barriers to changing healthcare practice.

QUALITATIVE RESEARCH APPROACHES

Grounded theory, phenomenology and ethnography are three approaches used in qualitative research. Grounded theory approach is a commonly used qualitative method in the social sciences to inductively generate or discover a theory out of the data. 9 Phenomenology and ethnography are more commonly used qualitative approaches in health care and will be highlighted below.

PHENOMENOLOGY

One of the major strengths of qualitative research is its emphasis upon understanding the phenomenon of interest holistically and in its context. The term phenomenology is popular and widely embraced, but its meaning has become confusing and faint. Different researchers refer to phenomenology differently. It can refer to an inquiry paradigm, an interpretive theory, a philosophy, an analytical perspective, a major qualitative research tradition or a research method framework. 12

In spite of the differences, all of these approaches share the focus of phenomenology, which is exploring how human beings make sense of experience and the meaning they give to these experiences. Phenomenology is being used in the social and human sciences including sociology, education, psychology, nursing and health sciences. 9

Phenomenologists are interested in how people put together the phenomena they experience in such a way as to make sense of the world and develop a worldview. They assume commonality in human experience and focus on meaning-making as the essence of human experience. The essence is the core meaning mutually understood through a phenomenon commonly experienced. 12 The phenomenon under study may be emotions, relationships, a programme, an organisation or a culture. 12 Bracketing is one of the central ideas in phenomenology. It means that the researcher has to set aside all of his prejudgments and his previous experience about the phenomena and approach the field with an open mind, imagination and intuition. Although important, bracketing is often said to be a difficult task. 9

Patton 12 describes the difference between conducting a phenomenological study and using a phenomenological perspective to a study. He argues that, “one can employ a general phenomenological perspective to elucidate the importance of using methods that capture people’s experience of the world without conducting a phenomenological study that focuses on the essence of shared experience”.

ETHNOGRAPHY

In ethnography, the researcher studies the structure and function of a group of people. An example of a structure or configuration is the kinship, while the function refers to patterns of relationships affecting and regulating behaviour. 9 The aim of ethnographic studies is to give a holistic picture of the social group studied, attempting to describe aspects of the cultural and social system of that particular group. These aspects could be the group history, religion, economy, politics or environment.

Data collection methods vary in ethnographic studies with observation and interviews being the most popular methods. Although some authors expressed concern that anthropological methods may be misused or applied superficially by the medical profession, 19 others, however, expressed the need for acknowledging and incorporating ethnographic methods in health care research. 20 Savage 19 explains that today the term ethnography can be applied to any small-scale social research carried out in everyday settings and uses several methods evolving in design and focusing on an individual’s meanings and explanations. In health care, ethnography has been used in topics related to health beliefs and practices, allowing these issues to be viewed in the context in which they occur and therefore helping broaden the understanding of behaviours related to health and illness. 20

In addition, there has been an increase in the amount of cross-cultural research and a change in the form of this research. Recently, more short-term ethnographic and cross-cultural studies have been undertaken, for example, studies sponsored by international development agencies or student exchange projects. 12

QUALITATIVE RESEARCH INSTRUMENTS

Qualitative research instruments used for data collection include interviews, observations, and analysis of documents. Interviews are the most common techniques used to gather research information. There are three types of interviews: structured, semi-structured and unstructured, described in some books as structured, informed and guided, respectively. 13

The semi-structured interview is more commonly used in health care-related qualitative research. Such an interview is characteristically based on a flexible topic guide that provides a loose structure of open-ended questions to explore experiences and attitudes. It has the advantage of great flexibility, enabling the researcher to enter new areas and produce richer data. In addition, it helps the researcher to develop a rapport with the informants. Semi-structured interviews elicit people’s own views and descriptions and have the benefit of uncovering issues or concerns that have not been anticipated by the researcher. 6 They are commonly used when the aim is to gain information on the perspectives, understandings and meanings constructed by people regarding the events and experiences of their lives. However, this type of interviewing is claimed to reduce the researcher’s control over the interview situation and take a longer time to conduct and analyse, in addition to the difficulties of the analysis process. 21

In health care, interviews are the appropriate tool to be used if the research is concerned with interpersonal aspects of care or if the available evidence is limited. 6

Another qualitative research instrument that can be used in health care research is the focus group. Focus group interviews have the advantage of being more time efficient as more people can be interviewed for the same amount of time. They also provide a richer source of data. On the other hand, focus group interviews tend to document the ‘public’ rather than the ‘private’ views of the individuals. In addition some people do not interview well in-group situations. 13

THE USE OF THE INTERVIEW GUIDE

Some qualitative researchers are reluctant to plan a design of their study in advance of the data collection. They argue that the phenomenon studied must first be discovered and they describe their design as emergent. However, increasingly more qualitative researchers appear ready to define a research question and develop an interview guide prior to starting the data collection. 11 Patton defines an interview guide as a series of topics or broad interview questions which the researcher is free to explore and probe with the interviewee. 12 The advantage of an interview guide is that it helps the interviewer pursue the same basic lines of inquiry with each person interviewed and manage the interviews in a more systematic and comprehensive way. The findings of earlier work are increasingly being used as a facilitator for further research. However, concepts drawn from earlier work are supposed to be held lightly and to be subject to reformulation or rejection by the researcher especially as the study goes on and the research progresses. 11 The extent to which such a flexibility of design is important will vary depending on the topic and the aim of the study.

Sampling strategies in qualitative research are largely determined by the purpose of the study. Statistical representativeness is not considered as a prime requirement in qualitative research and is not normally sought. Furthermore, qualitative data collection is more time consuming and expensive, which makes the use of a probability sample impractical. 6 The aim of sampling in qualitative research is to identify specific groups of people who hold characteristics or live in circumstances relevant to the phenomena being studied. In this way, identified informants are expected to enable enriched exploration of attitudes and aspects of behaviour relevant to the research. 10 Two types of samples used in qualitative research, maximum variation and homogenous samples, are explained below.

According to Patton, 12 maximum variation sampling is where the researcher attempts to study a phenomenon by seeking out settings or persons that represent the greatest differences in that phenomenon. A maximum variation sample documents diverse variations and identifies important common patterns by representing diverse cases to develop fully multiple perspectives about the cases. 9 This means that the participants are sampled based on particular predetermined criteria in order to cover a range of constituencies, such as different age, cultural background or class. 6

Similarly, the researcher can select the setting of the data collection on the basis that it is sufficiently similar to other settings in which generalisation is sought. This way the researcher is demonstrating the possibility that the setting studied is representative of the population studied. 2

In homogeneous sampling, the researcher chooses a small homogenous sample with the purpose of describing some particular subgroups in depth. 12

The sample size in qualitative research is not determined by fixed rules, but by factors such as the depth and duration of the interview and what is feasible for a single interviewer. 22 Although it is theoretically possible to carry out qualitative research on large samples, qualitative researchers find themselves obliged by time and resource limits to trade breadth for depth. 11 The sample size for interview studies is usually much smaller than those of a quantitative research, usually not exceeding 50 participants, although this can vary with the research question asked. 6 Patton 12 states, “there are no rules for sample size in qualitative inquiry”. In other words, sample size depends on the aim of the study and what is possible, given the time and resources available.

ANALYSING QUALITATIVE RESEARCH

Although there are plenty of guidelines for analysing qualitative research, applying these guidelines requires judgment and creativity because each qualitative study is unique. 12 A researcher might also be confused by the different terms used by qualitative researchers when describing analysis. Analysis might be described as interpretation, making sense of data, or transforming data. Analysis is sometimes presented to indicate different procedures based on language, theory or what is described as interpretive/descriptive analysis. 11 However, overlap can take place between these different methods and a researcher might decide to use a method of analysis that is based on language, such as symbolic interactionist, while using grounded theory to develop a theory at the same time. Most of the analytical approaches to qualitative research in health care are ‘generic’ and are not labelled within one of the specific traditions of qualitative research. A common approach in most of these studies is general and inductive in nature, but does not comply with the very systematic and rigorous inductive approach of grounded theory. In addition, it has been described that many researchers use a simple two-level analysis scheme followed by a more specific level. This means that the researcher can initiate the analysis based on the conceptual framework used in order to produce more inductive data, and the coding moves from the descriptive to the more interpretative and inferential codes. 5

COMPUTER USE IN QUALITATIVE DATA ANALYSIS

Qualitative research studies typically produce very large amount of data that needs to be managed efficiently. Computer packages can improve the efficiency of data management. 11 Computer programs provide a way of storing and retrieving material. They are therefore useful in locating cases, statements, phrases or even words, thereby replacing the tedious and time-consuming process of “cutting and pasting” and “colour coding”. The use of computer packages, however, is claimed to distance the analyst from the data, 11 and may take the place of a close and careful analysis. Using a computer programme can lead to quantitative analysis instead of qualitative, for example, counting occurrences, giving more weight to more frequent events, and ignoring isolated incidences. 23 In addition, computer programmes are said to fix and label categories during the analysis process and the researcher may be reluctant to change these categories. 9 Furthermore, the researcher is required to learn the computer programme, which may add to the time and effort he or she will need to spend on the research project. The researcher also has to be aware of the limitations of computer programmes. While computer packages can help with the intensive process of analysis and the management of large data sets, they are not a substitute for “immersion” in the data, and thorough knowledge that can enable the researcher to make comparisons, identify patterns and develop interpretations. 6 There are different packages available: Ethnograph, Atlas and NUD.IST are the mostly used. QSR NVivo is a new product developed by the makers of NUD.IST and is user-friendlier, more suitable for individual research projects and more visually attractive than previous packages.

THE ROLE OF THE RESEARCHER IN QUALITATIVE RESEARCH

Patton 12 states that, “the human element in qualitative research is both its strength and weakness”. It is considered a point of strength because it allows human insight and experience to develop new understandings of the world, and a point of weakness because it depends heavily on the researcher’s skills, creativity, training and intellect. Qualitative methods depend on both critical and creative thinking and the balance between the two in conducting the study and interpreting its results. The role of the researcher is approached differently according to the type of research tradition used. A phenomenologist researcher is required to bracket his/her own assumptions when collecting data. On the other hand, in an ethnographic study, the effect of the researcher on the interview and the interaction is seen as inevitable and important in shaping the results of the study.

Although practising clinicians routinely interview patients during their clinical work, interviewing is a well-established technique in sociology and related disciplines. One of the differences between clinical and research-aimed interviews lies in their different purposes. The usual aim of the clinical interview is to fit the patient’s problem into the appropriate medical category for diagnosis and management. On the other hand, the aim of a qualitative research interview is to discover the interviewee’s own meaning and avoid prior assumptions and preset categories. 22 Having said that, there are general skills in clinical interviewing that can be useful, such as listening and observation. A good level of self-awareness is necessary in the researcher in order to reduce possible biases.

Researchers in qualitative research need to consider how they are perceived by interviewees and the effect of features related to the researcher, such as class, race, and sex on the interview. This question is more important if the interviewee knows that the interviewer is a doctor. It has been described that a patient, or someone who is likely to be become one, may give what he or she thinks is a desirable response, thinking that the doctor will be pleased.

RIGOUR IN QUALITATIVE RESEARCH

Health field research is generally quantitative and based on biomedical traditions and experimental methods. In this field, qualitative research is criticized for being subject to researcher bias and for lacking reproducibility and generalisability. 10 Researchers presenting their qualitative work in health-related research are partly responsible for this view. Many qualitative researchers neglect the importance of giving an adequate description of their theoretical concepts and methods used in their research. A systematic research method is also essential when conducting qualitative research. Rigour in qualitative research includes procedures taken at different stages of the research process including during data collection and analysis. Several procedures have been described to increase rigour in qualitative research. For example, triangulation is commonly used as a way of validating of data.

TRIANGULATION

In triangulation, the researcher uses multiple methods, sources, researchers or theories to provide evidence that strengthens his or her study. Triangulation provides different ways of looking at the same phenomenon and adds credibility and confidence in the conclusions drawn from the study. There are two main types of triangulation, triangulation of sources and analyst triangulation. Patton 12 defines triangulation of sources as “checking out the consistency of different data sources within the same method”. When using this type of triangulation, the researcher compares the perspectives of people from different points of view. For example, studies in programme evaluation might compare the views of staff, clients or funding bodies. 12 Studies in health care have used this method of verification to study the accounts of doctors, patients, and managers in order to identify similarities and differences in views. An example is the study of patients’ versus doctors’ agendas in general practice. 23 Both similarities and differences from different sources, when given reasonable explanation, could contribute significantly to the credibility of the findings. Triangulation with multiple analysts can also be used as a method of verification. It is defined as “having two or more persons independently analyse the same qualitative data and compare their findings”. 12

QUALITATIVE RESEARCH IN HEALTH CARE MANAGEMENT

Recently, there has been a greater acceptance of the qualitative approach, even as a stand-alone method, in health care research. Institutions that control funding for medical research have developed ethical guidelines for assessing qualitative studies which indicates formal acceptance of this form of research within an area previously dominated by quantitative methods. 13 More qualitative research articles are published in health-related journals, in addition to a new qualitative research journal (Qualitative Health Research).

Quality of health care is one of the areas where qualitative methods can be used. The concept of quality in health care is multidimensional and multifaceted and some of the questions asked related to the quality of care or services may not be acquiescent to quantitative methods. 6 Qualitative research offers a variety of methods to be used for identifying what is really important to both patients and carers. It can also be used to identify and detect obstacles to change and the reasons why improvement does not occur. 6 It is therefore an essential component of health services research because it enables us to reach areas not amenable to quantitative research, for example, lay and professional health beliefs. In addition, qualitative description can be a prerequisite of quantitative research, particularly in areas that have received little previous investigation.

Qualitative research is widely used to study issues related to doctor-patient interaction especially in general practice. Studies concerning patients’ versus doctors’ agendas in general practice and general practitioners perceptions of effective health care are examples. 23 , 24 Recently there have been more studies concentrating on patients’ own perceptions and views regarding their health and health care services, for example a study on women’s views on the impact of operative delivery in the second stage of labour. 25 Another example is the study on middle-aged person’s experience of living with severe heart failure. 26

Another area where qualitative research is being used in health care is to identify obstacles and barriers to practice change by exploring the reasons behind certain behaviours. A good example of this is the study of patients’ decisions about whether or not to take anti-hypertensive drugs. 27 There are similar studies on issues such as the use of antibiotics in general practice and patient compliance.

In addition to issues related to the patients’ perception, some qualitative studies concentrated on factors fostering the doctor’s motivation and the effect of doctors’ social life and culture, in addition to issues related to the doctor’s own health. Examples are the study by Dumelow et al. 28 on the relation between a career and family life for English hospital consultants. Another example is a study aimed at exploring general practitioners’ perceptions of the effects of their profession and training on their attitudes to illness in themselves and colleagues. 29

Qualitative work can help in identifying cultural and social factors that affect health care positively or negatively. Such information can be helpful in improving service delivery. 6 Studies on patients from ethnic minorities have identified administrative and language barriers that affected health care and shed light on some of the beliefs and behaviours of these patients that might have affected help seeking and compliance. Good examples are the studies by Bush et al. 30 exploring the influences on smoking in Bangladeshi and Pakistani adults in the UK, and the multi-centre cross-cultural postnatal depression study. 31

There has been an increasing interest and use of qualitative research methods in primary health care and general practice articles. Britten 32 states that “the nature of general practice is such that a variety of research methods are needed to explore all its intricacies” He adds that qualitative methods can enrich research in general practice by opening up areas not amenable to quantitative methods, topics such as patient satisfaction, doctor-patient interaction, in addition to identifying and explaining attitudes, beliefs and behaviour. In addition, qualitative research has been used in the assessment of new technology methods used in health care. For example, the studies on the implementation of the National Health System information technology programme in the UK. 11 , 33

Qualitative research methods are receiving an increasing recognition in health care related research. The use of qualitative research in health care enables researchers to answer questions that may not be easily answered by quantitative methods. Moreover, it seeks to understand the phenomenon under study in the context of the culture or the setting in which it has been studied, therefore, aiding in the development of new research instruments, such as questionnaires that are more culturally acceptable. However, because health care related research has, for decades, been based on quantitative methods, the introduction of a new method requires researchers in health care who attempt to use it, to have a thorough understanding of its theoretical basis, methodology and evaluation techniques.

Acknowledgments

I would like to thank Sultan Qaboos University, Oman, for granting me a scholarship to do my PhD at the University of Nottingham, UK. I am also grateful to my PhD supervisor, Dr Margaret Oates, for her guidance and support.