sources of empirical literature review

Want to create or adapt books like this? Learn more about how Pressbooks supports open publishing practices.

Module 2 Chapter 3: What is Empirical Literature & Where can it be Found?

In Module 1, you read about the problem of pseudoscience. Here, we revisit the issue in addressing how to locate and assess scientific or empirical literature . In this chapter you will read about:

• distinguishing between what IS and IS NOT empirical literature
• how and where to locate empirical literature for understanding diverse populations, social work problems, and social phenomena.

Probably the most important take-home lesson from this chapter is that one source is not sufficient to being well-informed on a topic. It is important to locate multiple sources of information and to critically appraise the points of convergence and divergence in the information acquired from different sources. This is especially true in emerging and poorly understood topics, as well as in answering complex questions.

What Is Empirical Literature

Social workers often need to locate valid, reliable information concerning the dimensions of a population group or subgroup, a social work problem, or social phenomenon. They might also seek information about the way specific problems or resources are distributed among the populations encountered in professional practice. Or, social workers might be interested in finding out about the way that certain people experience an event or phenomenon. Empirical literature resources may provide answers to many of these types of social work questions. In addition, resources containing data regarding social indicators may also prove helpful. Social indicators are the “facts and figures” statistics that describe the social, economic, and psychological factors that have an impact on the well-being of a community or other population group.The United Nations (UN) and the World Health Organization (WHO) are examples of organizations that monitor social indicators at a global level: dimensions of population trends (size, composition, growth/loss), health status (physical, mental, behavioral, life expectancy, maternal and infant mortality, fertility/child-bearing, and diseases like HIV/AIDS), housing and quality of sanitation (water supply, waste disposal), education and literacy, and work/income/unemployment/economics, for example.

Three characteristics stand out in empirical literature compared to other types of information available on a topic of interest: systematic observation and methodology, objectivity, and transparency/replicability/reproducibility. Let’s look a little more closely at these three features.

Systematic Observation and Methodology. The hallmark of empiricism is “repeated or reinforced observation of the facts or phenomena” (Holosko, 2006, p. 6). In empirical literature, established research methodologies and procedures are systematically applied to answer the questions of interest.

Objectivity. Gathering “facts,” whatever they may be, drives the search for empirical evidence (Holosko, 2006). Authors of empirical literature are expected to report the facts as observed, whether or not these facts support the investigators’ original hypotheses. Research integrity demands that the information be provided in an objective manner, reducing sources of investigator bias to the greatest possible extent.

Transparency and Replicability/Reproducibility.   Empirical literature is reported in such a manner that other investigators understand precisely what was done and what was found in a particular research study—to the extent that they could replicate the study to determine whether the findings are reproduced when repeated. The outcomes of an original and replication study may differ, but a reader could easily interpret the methods and procedures leading to each study’s findings.

What is NOT Empirical Literature

By now, it is probably obvious to you that literature based on “evidence” that is not developed in a systematic, objective, transparent manner is not empirical literature. On one hand, non-empirical types of professional literature may have great significance to social workers. For example, social work scholars may produce articles that are clearly identified as describing a new intervention or program without evaluative evidence, critiquing a policy or practice, or offering a tentative, untested theory about a phenomenon. These resources are useful in educating ourselves about possible issues or concerns. But, even if they are informed by evidence, they are not empirical literature. Here is a list of several sources of information that do not meet the standard of being called empirical literature:

• your course instructor’s lectures
• political statements
• advertisements
• newspapers & magazines (journalism)
• television news reports & analyses (journalism)
• many websites, Facebook postings, Twitter tweets, and blog postings
• the introductory literature review in an empirical article

You may be surprised to see the last two included in this list. Like the other sources of information listed, these sources also might lead you to look for evidence. But, they are not themselves sources of evidence. They may summarize existing evidence, but in the process of summarizing (like your instructor’s lectures), information is transformed, modified, reduced, condensed, and otherwise manipulated in such a manner that you may not see the entire, objective story. These are called secondary sources, as opposed to the original, primary source of evidence. In relying solely on secondary sources, you sacrifice your own critical appraisal and thinking about the original work—you are “buying” someone else’s interpretation and opinion about the original work, rather than developing your own interpretation and opinion. What if they got it wrong? How would you know if you did not examine the primary source for yourself? Consider the following as an example of “getting it wrong” being perpetuated.

Example: Bullying and School Shootings . One result of the heavily publicized April 1999 school shooting incident at Columbine High School (Colorado), was a heavy emphasis placed on bullying as a causal factor in these incidents (Mears, Moon, & Thielo, 2017), “creating a powerful master narrative about school shootings” (Raitanen, Sandberg, & Oksanen, 2017, p. 3). Naturally, with an identified cause, a great deal of effort was devoted to anti-bullying campaigns and interventions for enhancing resilience among youth who experience bullying.  However important these strategies might be for promoting positive mental health, preventing poor mental health, and possibly preventing suicide among school-aged children and youth, it is a mistaken belief that this can prevent school shootings (Mears, Moon, & Thielo, 2017). Many times the accounts of the perpetrators having been bullied come from potentially inaccurate third-party accounts, rather than the perpetrators themselves; bullying was not involved in all instances of school shooting; a perpetrator’s perception of being bullied/persecuted are not necessarily accurate; many who experience severe bullying do not perpetrate these incidents; bullies are the least targeted shooting victims; perpetrators of the shooting incidents were often bullying others; and, bullying is only one of many important factors associated with perpetrating such an incident (Ioannou, Hammond, & Simpson, 2015; Mears, Moon, & Thielo, 2017; Newman &Fox, 2009; Raitanen, Sandberg, & Oksanen, 2017). While mass media reports deliver bullying as a means of explaining the inexplicable, the reality is not so simple: “The connection between bullying and school shootings is elusive” (Langman, 2014), and “the relationship between bullying and school shooting is, at best, tenuous” (Mears, Moon, & Thielo, 2017, p. 940). The point is, when a narrative becomes this publicly accepted, it is difficult to sort out truth and reality without going back to original sources of information and evidence.

What May or May Not Be Empirical Literature: Literature Reviews

Investigators typically engage in a review of existing literature as they develop their own research studies. The review informs them about where knowledge gaps exist, methods previously employed by other scholars, limitations of prior work, and previous scholars’ recommendations for directing future research. These reviews may appear as a published article, without new study data being reported (see Fields, Anderson, & Dabelko-Schoeny, 2014 for example). Or, the literature review may appear in the introduction to their own empirical study report. These literature reviews are not considered to be empirical evidence sources themselves, although they may be based on empirical evidence sources. One reason is that the authors of a literature review may or may not have engaged in a systematic search process, identifying a full, rich, multi-sided pool of evidence reports.

There is, however, a type of review that applies systematic methods and is, therefore, considered to be more strongly rooted in evidence: the systematic review .

Systematic review of literature. A systematic reviewis a type of literature report where established methods have been systematically applied, objectively, in locating and synthesizing a body of literature. The systematic review report is characterized by a great deal of transparency about the methods used and the decisions made in the review process, and are replicable. Thus, it meets the criteria for empirical literature: systematic observation and methodology, objectivity, and transparency/reproducibility. We will work a great deal more with systematic reviews in the second course, SWK 3402, since they are important tools for understanding interventions. They are somewhat less common, but not unheard of, in helping us understand diverse populations, social work problems, and social phenomena.

Locating Empirical Evidence

Social workers have available a wide array of tools and resources for locating empirical evidence in the literature. These can be organized into four general categories.

Journal Articles. A number of professional journals publish articles where investigators report on the results of their empirical studies. However, it is important to know how to distinguish between empirical and non-empirical manuscripts in these journals. A key indicator, though not the only one, involves a peer review process . Many professional journals require that manuscripts undergo a process of peer review before they are accepted for publication. This means that the authors’ work is shared with scholars who provide feedback to the journal editor as to the quality of the submitted manuscript. The editor then makes a decision based on the reviewers’ feedback:

• Accept as is
• Accept with minor revisions
• Request that a revision be resubmitted (no assurance of acceptance)

When a “revise and resubmit” decision is made, the piece will go back through the review process to determine if it is now acceptable for publication and that all of the reviewers’ concerns have been adequately addressed. Editors may also reject a manuscript because it is a poor fit for the journal, based on its mission and audience, rather than sending it for review consideration.

Indicators of journal relevance. Various journals are not equally relevant to every type of question being asked of the literature. Journals may overlap to a great extent in terms of the topics they might cover; in other words, a topic might appear in multiple different journals, depending on how the topic was being addressed. For example, articles that might help answer a question about the relationship between community poverty and violence exposure might appear in several different journals, some with a focus on poverty, others with a focus on violence, and still others on community development or public health. Journal titles are sometimes a good starting point but may not give a broad enough picture of what they cover in their contents.

In focusing a literature search, it also helps to review a journal’s mission and target audience. For example, at least four different journals focus specifically on poverty:

• Journal of Children & Poverty
• Journal of Poverty
• Journal of Poverty and Social Justice
• Poverty & Public Policy

Let’s look at an example using the Journal of Poverty and Social Justice . Information about this journal is located on the journal’s webpage: http://policy.bristoluniversitypress.co.uk/journals/journal-of-poverty-and-social-justice . In the section headed “About the Journal” you can see that it is an internationally focused research journal, and that it addresses social justice issues in addition to poverty alone. The research articles are peer-reviewed (there appear to be non-empirical discussions published, as well). These descriptions about a journal are almost always available, sometimes listed as “scope” or “mission.” These descriptions also indicate the sponsorship of the journal—sponsorship may be institutional (a particular university or agency, such as Smith College Studies in Social Work ), a professional organization, such as the Council on Social Work Education (CSWE) or the National Association of Social Work (NASW), or a publishing company (e.g., Taylor & Frances, Wiley, or Sage).

Indicators of journal caliber.  Despite engaging in a peer review process, not all journals are equally rigorous. Some journals have very high rejection rates, meaning that many submitted manuscripts are rejected; others have fairly high acceptance rates, meaning that relatively few manuscripts are rejected. This is not necessarily the best indicator of quality, however, since newer journals may not be sufficiently familiar to authors with high quality manuscripts and some journals are very specific in terms of what they publish. Another index that is sometimes used is the journal’s impact factor . Impact factor is a quantitative number indicative of how often articles published in the journal are cited in the reference list of other journal articles—the statistic is calculated as the number of times on average each article published in a particular year were cited divided by the number of articles published (the number that could be cited). For example, the impact factor for the Journal of Poverty and Social Justice in our list above was 0.70 in 2017, and for the Journal of Poverty was 0.30. These are relatively low figures compared to a journal like the New England Journal of Medicine with an impact factor of 59.56! This means that articles published in that journal were, on average, cited more than 59 times in the next year or two.

Impact factors are not necessarily the best indicator of caliber, however, since many strong journals are geared toward practitioners rather than scholars, so they are less likely to be cited by other scholars but may have a large impact on a large readership. This may be the case for a journal like the one titled Social Work, the official journal of the National Association of Social Workers. It is distributed free to all members: over 120,000 practitioners, educators, and students of social work world-wide. The journal has a recent impact factor of.790. The journals with social work relevant content have impact factors in the range of 1.0 to 3.0 according to Scimago Journal & Country Rank (SJR), particularly when they are interdisciplinary journals (for example, Child Development , Journal of Marriage and Family , Child Abuse and Neglect , Child Maltreatmen t, Social Service Review , and British Journal of Social Work ). Once upon a time, a reader could locate different indexes comparing the “quality” of social work-related journals. However, the concept of “quality” is difficult to systematically define. These indexes have mostly been replaced by impact ratings, which are not necessarily the best, most robust indicators on which to rely in assessing journal quality. For example, new journals addressing cutting edge topics have not been around long enough to have been evaluated using this particular tool, and it takes a few years for articles to begin to be cited in other, later publications.

Beware of pseudo-, illegitimate, misleading, deceptive, and suspicious journals . Another side effect of living in the Age of Information is that almost anyone can circulate almost anything and call it whatever they wish. This goes for “journal” publications, as well. With the advent of open-access publishing in recent years (electronic resources available without subscription), we have seen an explosion of what are called predatory or junk journals . These are publications calling themselves journals, often with titles very similar to legitimate publications and often with fake editorial boards. These “publications” lack the integrity of legitimate journals. This caution is reminiscent of the discussions earlier in the course about pseudoscience and “snake oil” sales. The predatory nature of many apparent information dissemination outlets has to do with how scientists and scholars may be fooled into submitting their work, often paying to have their work peer-reviewed and published. There exists a “thriving black-market economy of publishing scams,” and at least two “journal blacklists” exist to help identify and avoid these scam journals (Anderson, 2017).

This issue is important to information consumers, because it creates a challenge in terms of identifying legitimate sources and publications. The challenge is particularly important to address when information from on-line, open-access journals is being considered. Open-access is not necessarily a poor choice—legitimate scientists may pay sizeable fees to legitimate publishers to make their work freely available and accessible as open-access resources. On-line access is also not necessarily a poor choice—legitimate publishers often make articles available on-line to provide timely access to the content, especially when publishing the article in hard copy will be delayed by months or even a year or more. On the other hand, stating that a journal engages in a peer-review process is no guarantee of quality—this claim may or may not be truthful. Pseudo- and junk journals may engage in some quality control practices, but may lack attention to important quality control processes, such as managing conflict of interest, reviewing content for objectivity or quality of the research conducted, or otherwise failing to adhere to industry standards (Laine & Winker, 2017).

One resource designed to assist with the process of deciphering legitimacy is the Directory of Open Access Journals (DOAJ). The DOAJ is not a comprehensive listing of all possible legitimate open-access journals, and does not guarantee quality, but it does help identify legitimate sources of information that are openly accessible and meet basic legitimacy criteria. It also is about open-access journals, not the many journals published in hard copy.

An additional caution: Search for article corrections. Despite all of the careful manuscript review and editing, sometimes an error appears in a published article. Most journals have a practice of publishing corrections in future issues. When you locate an article, it is helpful to also search for updates. Here is an example where data presented in an article’s original tables were erroneous, and a correction appeared in a later issue.

• Marchant, A., Hawton, K., Stewart A., Montgomery, P., Singaravelu, V., Lloyd, K., Purdy, N., Daine, K., & John, A. (2017). A systematic review of the relationship between internet use, self-harm and suicidal behaviour in young people: The good, the bad and the unknown. PLoS One, 12(8): e0181722. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5558917/
• Marchant, A., Hawton, K., Stewart A., Montgomery, P., Singaravelu, V., Lloyd, K., Purdy, N., Daine, K., & John, A. (2018).Correction—A systematic review of the relationship between internet use, self-harm and suicidal behaviour in young people: The good, the bad and the unknown. PLoS One, 13(3): e0193937.  http://journals.plos.org/plosone/article?id=10.1371/journal.pone.0193937

Search Tools. In this age of information, it is all too easy to find items—the problem lies in sifting, sorting, and managing the vast numbers of items that can be found. For example, a simple Google® search for the topic “community poverty and violence” resulted in about 15,600,000 results! As a means of simplifying the process of searching for journal articles on a specific topic, a variety of helpful tools have emerged. One type of search tool has previously applied a filtering process for you: abstracting and indexing databases . These resources provide the user with the results of a search to which records have already passed through one or more filters. For example, PsycINFO is managed by the American Psychological Association and is devoted to peer-reviewed literature in behavioral science. It contains almost 4.5 million records and is growing every month. However, it may not be available to users who are not affiliated with a university library. Conducting a basic search for our topic of “community poverty and violence” in PsychINFO returned 1,119 articles. Still a large number, but far more manageable. Additional filters can be applied, such as limiting the range in publication dates, selecting only peer reviewed items, limiting the language of the published piece (English only, for example), and specified types of documents (either chapters, dissertations, or journal articles only, for example). Adding the filters for English, peer-reviewed journal articles published between 2010 and 2017 resulted in 346 documents being identified.

Just as was the case with journals, not all abstracting and indexing databases are equivalent. There may be overlap between them, but none is guaranteed to identify all relevant pieces of literature. Here are some examples to consider, depending on the nature of the questions asked of the literature:

• Academic Search Complete—multidisciplinary index of 9,300 peer-reviewed journals
• AgeLine—multidisciplinary index of aging-related content for over 600 journals
• Campbell Collaboration—systematic reviews in education, crime and justice, social welfare, international development
• Google Scholar—broad search tool for scholarly literature across many disciplines
• MEDLINE/ PubMed—National Library of medicine, access to over 15 million citations
• Oxford Bibliographies—annotated bibliographies, each is discipline specific (e.g., psychology, childhood studies, criminology, social work, sociology)
• PsycINFO/PsycLIT—international literature on material relevant to psychology and related disciplines
• SocINDEX—publications in sociology
• Social Sciences Abstracts—multiple disciplines
• Social Work Abstracts—many areas of social work are covered
• Web of Science—a “meta” search tool that searches other search tools, multiple disciplines

Placing our search for information about “community violence and poverty” into the Social Work Abstracts tool with no additional filters resulted in a manageable 54-item list. Finally, abstracting and indexing databases are another way to determine journal legitimacy: if a journal is indexed in a one of these systems, it is likely a legitimate journal. However, the converse is not necessarily true: if a journal is not indexed does not mean it is an illegitimate or pseudo-journal.

Government Sources. A great deal of information is gathered, analyzed, and disseminated by various governmental branches at the international, national, state, regional, county, and city level. Searching websites that end in.gov is one way to identify this type of information, often presented in articles, news briefs, and statistical reports. These government sources gather information in two ways: they fund external investigations through grants and contracts and they conduct research internally, through their own investigators. Here are some examples to consider, depending on the nature of the topic for which information is sought:

• Agency for Healthcare Research and Quality (AHRQ) at https://www.ahrq.gov/
• Bureau of Justice Statistics (BJS) at https://www.bjs.gov/
• Census Bureau at https://www.census.gov
• Morbidity and Mortality Weekly Report of the CDC (MMWR-CDC) at https://www.cdc.gov/mmwr/index.html
• Child Welfare Information Gateway at https://www.childwelfare.gov
• Children’s Bureau/Administration for Children & Families at https://www.acf.hhs.gov
• Forum on Child and Family Statistics at https://www.childstats.gov
• National Institutes of Health (NIH) at https://www.nih.gov , including (not limited to):
• National Institute on Aging (NIA at https://www.nia.nih.gov
• National Institute on Alcohol Abuse and Alcoholism (NIAAA) at https://www.niaaa.nih.gov
• National Institute of Child Health and Human Development (NICHD) at https://www.nichd.nih.gov
• National Institute on Drug Abuse (NIDA) at https://www.nida.nih.gov
• National Institute of Environmental Health Sciences at https://www.niehs.nih.gov
• National Institute of Mental Health (NIMH) at https://www.nimh.nih.gov
• National Institute on Minority Health and Health Disparities at https://www.nimhd.nih.gov
• National Institute of Justice (NIJ) at https://www.nij.gov
• Substance Abuse and Mental Health Services Administration (SAMHSA) at https://www.samhsa.gov/
• United States Agency for International Development at https://usaid.gov

Each state and many counties or cities have similar data sources and analysis reports available, such as Ohio Department of Health at https://www.odh.ohio.gov/healthstats/dataandstats.aspx and Franklin County at https://statisticalatlas.com/county/Ohio/Franklin-County/Overview . Data are available from international/global resources (e.g., United Nations and World Health Organization), as well.

Other Sources. The Health and Medicine Division (HMD) of the National Academies—previously the Institute of Medicine (IOM)—is a nonprofit institution that aims to provide government and private sector policy and other decision makers with objective analysis and advice for making informed health decisions. For example, in 2018 they produced reports on topics in substance use and mental health concerning the intersection of opioid use disorder and infectious disease,  the legal implications of emerging neurotechnologies, and a global agenda concerning the identification and prevention of violence (see http://www.nationalacademies.org/hmd/Global/Topics/Substance-Abuse-Mental-Health.aspx ). The exciting aspect of this resource is that it addresses many topics that are current concerns because they are hoping to help inform emerging policy. The caution to consider with this resource is the evidence is often still emerging, as well.

Numerous “think tank” organizations exist, each with a specific mission. For example, the Rand Corporation is a nonprofit organization offering research and analysis to address global issues since 1948. The institution’s mission is to help improve policy and decision making “to help individuals, families, and communities throughout the world be safer and more secure, healthier and more prosperous,” addressing issues of energy, education, health care, justice, the environment, international affairs, and national security (https://www.rand.org/about/history.html). And, for example, the Robert Woods Johnson Foundation is a philanthropic organization supporting research and research dissemination concerning health issues facing the United States. The foundation works to build a culture of health across systems of care (not only medical care) and communities (https://www.rwjf.org).

While many of these have a great deal of helpful evidence to share, they also may have a strong political bias. Objectivity is often lacking in what information these organizations provide: they provide evidence to support certain points of view. That is their purpose—to provide ideas on specific problems, many of which have a political component. Think tanks “are constantly researching solutions to a variety of the world’s problems, and arguing, advocating, and lobbying for policy changes at local, state, and federal levels” (quoted from https://thebestschools.org/features/most-influential-think-tanks/ ). Helpful information about what this one source identified as the 50 most influential U.S. think tanks includes identifying each think tank’s political orientation. For example, The Heritage Foundation is identified as conservative, whereas Human Rights Watch is identified as liberal.

While not the same as think tanks, many mission-driven organizations also sponsor or report on research, as well. For example, the National Association for Children of Alcoholics (NACOA) in the United States is a registered nonprofit organization. Its mission, along with other partnering organizations, private-sector groups, and federal agencies, is to promote policy and program development in research, prevention and treatment to provide information to, for, and about children of alcoholics (of all ages). Based on this mission, the organization supports knowledge development and information gathering on the topic and disseminates information that serves the needs of this population. While this is a worthwhile mission, there is no guarantee that the information meets the criteria for evidence with which we have been working. Evidence reported by think tank and mission-driven sources must be utilized with a great deal of caution and critical analysis!

In many instances an empirical report has not appeared in the published literature, but in the form of a technical or final report to the agency or program providing the funding for the research that was conducted. One such example is presented by a team of investigators funded by the National Institute of Justice to evaluate a program for training professionals to collect strong forensic evidence in instances of sexual assault (Patterson, Resko, Pierce-Weeks, & Campbell, 2014): https://www.ncjrs.gov/pdffiles1/nij/grants/247081.pdf . Investigators may serve in the capacity of consultant to agencies, programs, or institutions, and provide empirical evidence to inform activities and planning. One such example is presented by Maguire-Jack (2014) as a report to a state’s child maltreatment prevention board: https://preventionboard.wi.gov/Documents/InvestmentInPreventionPrograming_Final.pdf .

When Direct Answers to Questions Cannot Be Found. Sometimes social workers are interested in finding answers to complex questions or questions related to an emerging, not-yet-understood topic. This does not mean giving up on empirical literature. Instead, it requires a bit of creativity in approaching the literature. A Venn diagram might help explain this process. Consider a scenario where a social worker wishes to locate literature to answer a question concerning issues of intersectionality. Intersectionality is a social justice term applied to situations where multiple categorizations or classifications come together to create overlapping, interconnected, or multiplied disadvantage. For example, women with a substance use disorder and who have been incarcerated face a triple threat in terms of successful treatment for a substance use disorder: intersectionality exists between being a woman, having a substance use disorder, and having been in jail or prison. After searching the literature, little or no empirical evidence might have been located on this specific triple-threat topic. Instead, the social worker will need to seek literature on each of the threats individually, and possibly will find literature on pairs of topics (see Figure 3-1). There exists some literature about women’s outcomes for treatment of a substance use disorder (a), some literature about women during and following incarceration (b), and some literature about substance use disorders and incarceration (c). Despite not having a direct line on the center of the intersecting spheres of literature (d), the social worker can develop at least a partial picture based on the overlapping literatures.

Figure 3-1. Venn diagram of intersecting literature sets.

Take a moment to complete the following activity. For each statement about empirical literature, decide if it is true or false.

Social Work 3401 Coursebook Copyright © by Dr. Audrey Begun is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License , except where otherwise noted.

Share This Book

Have a language expert improve your writing

Run a free plagiarism check in 10 minutes, generate accurate citations for free.

• Knowledge Base

Methodology

• How to Write a Literature Review | Guide, Examples, & Templates

How to Write a Literature Review | Guide, Examples, & Templates

Published on January 2, 2023 by Shona McCombes . Revised on September 11, 2023.

What is a literature review? A literature review is a survey of scholarly sources on a specific topic. It provides an overview of current knowledge, allowing you to identify relevant theories, methods, and gaps in the existing research that you can later apply to your paper, thesis, or dissertation topic .

There are five key steps to writing a literature review:

• Search for relevant literature
• Evaluate sources
• Identify themes, debates, and gaps
• Outline the structure
• Write your literature review

A good literature review doesn’t just summarize sources—it analyzes, synthesizes , and critically evaluates to give a clear picture of the state of knowledge on the subject.

Instantly correct all language mistakes in your text

Upload your document to correct all your mistakes in minutes

Table of contents

What is the purpose of a literature review, examples of literature reviews, step 1 – search for relevant literature, step 2 – evaluate and select sources, step 3 – identify themes, debates, and gaps, step 4 – outline your literature review’s structure, step 5 – write your literature review, free lecture slides, other interesting articles, frequently asked questions, introduction.

• Quick Run-through
• Step 1 & 2

When you write a thesis , dissertation , or research paper , you will likely have to conduct a literature review to situate your research within existing knowledge. The literature review gives you a chance to:

• Demonstrate your familiarity with the topic and its scholarly context
• Develop a theoretical framework and methodology for your research
• Position your work in relation to other researchers and theorists
• Show how your research addresses a gap or contributes to a debate
• Evaluate the current state of research and demonstrate your knowledge of the scholarly debates around your topic.

Writing literature reviews is a particularly important skill if you want to apply for graduate school or pursue a career in research. We’ve written a step-by-step guide that you can follow below.

Here's why students love Scribbr's proofreading services

Discover proofreading & editing

Writing literature reviews can be quite challenging! A good starting point could be to look at some examples, depending on what kind of literature review you’d like to write.

• Example literature review #1: “Why Do People Migrate? A Review of the Theoretical Literature” ( Theoretical literature review about the development of economic migration theory from the 1950s to today.)
• Example literature review #2: “Literature review as a research methodology: An overview and guidelines” ( Methodological literature review about interdisciplinary knowledge acquisition and production.)
• Example literature review #3: “The Use of Technology in English Language Learning: A Literature Review” ( Thematic literature review about the effects of technology on language acquisition.)
• Example literature review #4: “Learners’ Listening Comprehension Difficulties in English Language Learning: A Literature Review” ( Chronological literature review about how the concept of listening skills has changed over time.)

You can also check out our templates with literature review examples and sample outlines at the links below.

Download Word doc Download Google doc

Before you begin searching for literature, you need a clearly defined topic .

If you are writing the literature review section of a dissertation or research paper, you will search for literature related to your research problem and questions .

Make a list of keywords

Start by creating a list of keywords related to your research question. Include each of the key concepts or variables you’re interested in, and list any synonyms and related terms. You can add to this list as you discover new keywords in the process of your literature search.

• Social media, Facebook, Instagram, Twitter, Snapchat, TikTok
• Body image, self-perception, self-esteem, mental health
• Generation Z, teenagers, adolescents, youth

Search for relevant sources

Use your keywords to begin searching for sources. Some useful databases to search for journals and articles include:

• Your university’s library catalogue
• Google Scholar
• Project Muse (humanities and social sciences)
• Medline (life sciences and biomedicine)
• EconLit (economics)
• Inspec (physics, engineering and computer science)

You can also use boolean operators to help narrow down your search.

Make sure to read the abstract to find out whether an article is relevant to your question. When you find a useful book or article, you can check the bibliography to find other relevant sources.

You likely won’t be able to read absolutely everything that has been written on your topic, so it will be necessary to evaluate which sources are most relevant to your research question.

For each publication, ask yourself:

• What question or problem is the author addressing?
• What are the key concepts and how are they defined?
• What are the key theories, models, and methods?
• Does the research use established frameworks or take an innovative approach?
• What are the results and conclusions of the study?
• How does the publication relate to other literature in the field? Does it confirm, add to, or challenge established knowledge?
• What are the strengths and weaknesses of the research?

Make sure the sources you use are credible , and make sure you read any landmark studies and major theories in your field of research.

You can use our template to summarize and evaluate sources you’re thinking about using. Click on either button below to download.

Take notes and cite your sources

As you read, you should also begin the writing process. Take notes that you can later incorporate into the text of your literature review.

It is important to keep track of your sources with citations to avoid plagiarism . It can be helpful to make an annotated bibliography , where you compile full citation information and write a paragraph of summary and analysis for each source. This helps you remember what you read and saves time later in the process.

To begin organizing your literature review’s argument and structure, be sure you understand the connections and relationships between the sources you’ve read. Based on your reading and notes, you can look for:

• Trends and patterns (in theory, method or results): do certain approaches become more or less popular over time?
• Themes: what questions or concepts recur across the literature?
• Debates, conflicts and contradictions: where do sources disagree?
• Pivotal publications: are there any influential theories or studies that changed the direction of the field?
• Gaps: what is missing from the literature? Are there weaknesses that need to be addressed?

This step will help you work out the structure of your literature review and (if applicable) show how your own research will contribute to existing knowledge.

• Most research has focused on young women.
• There is an increasing interest in the visual aspects of social media.
• But there is still a lack of robust research on highly visual platforms like Instagram and Snapchat—this is a gap that you could address in your own research.

There are various approaches to organizing the body of a literature review. Depending on the length of your literature review, you can combine several of these strategies (for example, your overall structure might be thematic, but each theme is discussed chronologically).

Chronological

The simplest approach is to trace the development of the topic over time. However, if you choose this strategy, be careful to avoid simply listing and summarizing sources in order.

Try to analyze patterns, turning points and key debates that have shaped the direction of the field. Give your interpretation of how and why certain developments occurred.

If you have found some recurring central themes, you can organize your literature review into subsections that address different aspects of the topic.

For example, if you are reviewing literature about inequalities in migrant health outcomes, key themes might include healthcare policy, language barriers, cultural attitudes, legal status, and economic access.

Methodological

If you draw your sources from different disciplines or fields that use a variety of research methods , you might want to compare the results and conclusions that emerge from different approaches. For example:

• Look at what results have emerged in qualitative versus quantitative research
• Discuss how the topic has been approached by empirical versus theoretical scholarship
• Divide the literature into sociological, historical, and cultural sources

Theoretical

A literature review is often the foundation for a theoretical framework . You can use it to discuss various theories, models, and definitions of key concepts.

You might argue for the relevance of a specific theoretical approach, or combine various theoretical concepts to create a framework for your research.

Like any other academic text , your literature review should have an introduction , a main body, and a conclusion . What you include in each depends on the objective of your literature review.

The introduction should clearly establish the focus and purpose of the literature review.

Depending on the length of your literature review, you might want to divide the body into subsections. You can use a subheading for each theme, time period, or methodological approach.

As you write, you can follow these tips:

• Summarize and synthesize: give an overview of the main points of each source and combine them into a coherent whole
• Analyze and interpret: don’t just paraphrase other researchers — add your own interpretations where possible, discussing the significance of findings in relation to the literature as a whole
• Critically evaluate: mention the strengths and weaknesses of your sources
• Write in well-structured paragraphs: use transition words and topic sentences to draw connections, comparisons and contrasts

In the conclusion, you should summarize the key findings you have taken from the literature and emphasize their significance.

When you’ve finished writing and revising your literature review, don’t forget to proofread thoroughly before submitting. Not a language expert? Check out Scribbr’s professional proofreading services !

This article has been adapted into lecture slides that you can use to teach your students about writing a literature review.

Scribbr slides are free to use, customize, and distribute for educational purposes.

Open Google Slides Download PowerPoint

If you want to know more about the research process , methodology , research bias , or statistics , make sure to check out some of our other articles with explanations and examples.

• Sampling methods
• Simple random sampling
• Stratified sampling
• Cluster sampling
• Likert scales
• Reproducibility

 Statistics

• Null hypothesis
• Statistical power
• Probability distribution
• Effect size
• Poisson distribution

Research bias

• Optimism bias
• Cognitive bias
• Implicit bias
• Hawthorne effect
• Anchoring bias
• Explicit bias

A literature review is a survey of scholarly sources (such as books, journal articles, and theses) related to a specific topic or research question .

It is often written as part of a thesis, dissertation , or research paper , in order to situate your work in relation to existing knowledge.

There are several reasons to conduct a literature review at the beginning of a research project:

• To familiarize yourself with the current state of knowledge on your topic
• To ensure that you’re not just repeating what others have already done
• To identify gaps in knowledge and unresolved problems that your research can address
• To develop your theoretical framework and methodology
• To provide an overview of the key findings and debates on the topic

Writing the literature review shows your reader how your work relates to existing research and what new insights it will contribute.

The literature review usually comes near the beginning of your thesis or dissertation . After the introduction , it grounds your research in a scholarly field and leads directly to your theoretical framework or methodology .

A literature review is a survey of credible sources on a topic, often used in dissertations , theses, and research papers . Literature reviews give an overview of knowledge on a subject, helping you identify relevant theories and methods, as well as gaps in existing research. Literature reviews are set up similarly to other  academic texts , with an introduction , a main body, and a conclusion .

An  annotated bibliography is a list of  source references that has a short description (called an annotation ) for each of the sources. It is often assigned as part of the research process for a  paper .  

Cite this Scribbr article

If you want to cite this source, you can copy and paste the citation or click the “Cite this Scribbr article” button to automatically add the citation to our free Citation Generator.

McCombes, S. (2023, September 11). How to Write a Literature Review | Guide, Examples, & Templates. Scribbr. Retrieved July 5, 2024, from https://www.scribbr.com/dissertation/literature-review/

Is this article helpful?

Shona McCombes

Other students also liked, what is a theoretical framework | guide to organizing, what is a research methodology | steps & tips, how to write a research proposal | examples & templates, what is your plagiarism score.

Literature Reviews: Types of Literature

• Library Basics
• 1. Choose Your Topic
• How to Find Books
• Types of Clinical Study Designs

Types of Literature

• 3. Search the Literature
• 4. Read & Analyze the Literature
• 5. Write the Review
• Keeping Track of Information
• Style Guides
• Books, Tutorials & Examples

Different types of publications have different characteristics.

Primary Literature Primary sources means original studies, based on direct observation, use of statistical records, interviews, or experimental methods, of actual practices or the actual impact of practices or policies. They are authored by researchers, contains original research data, and are usually published in a peer-reviewed journal. Primary literature may also include conference papers, pre-prints, or preliminary reports. Also called empirical research .

Secondary Literature Secondary literature consists of interpretations and evaluations that are derived from or refer to the primary source literature. Examples include review articles (such as meta-analysis and systematic reviews) and reference works. Professionals within each discipline take the primary literature and synthesize, generalize, and integrate new research.

Tertiary Literature Tertiary literature consists of a distillation and collection of primary and secondary sources such as textbooks, encyclopedia articles, and guidebooks or handbooks. The purpose of tertiary literature is to provide an overview of key research findings and an introduction to principles and practices within the discipline.

Adapted from the Information Services Department of the Library of the Health Sciences-Chicago , University of Illinois at Chicago.

Types of Scientific Publications

These examples and descriptions of publication types will give you an idea of how to use various works and why you would want to write a particular kind of paper.

• Scholarly article aka empirical article
• Review article
• Conference paper

Scholarly (aka empirical) article -- example

Empirical studies use data derived from observation or experiment. Original research papers (also called primary research articles) that describe empirical studies and their results are published in academic journals.  Articles that report empirical research contain different sections which relate to the steps of the scientific method.

      Abstract - The abstract provides a very brief summary of the research.

     Introduction - The introduction sets the research in a context, which provides a review of related research and develops the hypotheses for the research.

     Method - The method section describes how the research was conducted.

     Results - The results section describes the outcomes of the study.

     Discussion - The discussion section contains the interpretations and implications of the study.

     References - A references section lists the articles, books, and other material cited in the report.

Review article -- example

A review article summarizes a particular field of study and places the recent research in context. It provides an overview and is an excellent introduction to a subject area. The references used in a review article are helpful as they lead to more in-depth research.

Many databases have limits or filters to search for review articles. You can also search by keywords like review article, survey, overview, summary, etc.

Conference proceedings, abstracts and reports -- example

Conference proceedings, abstracts and reports are not usually peer-reviewed.  A conference article is similar to a scholarly article insofar as it is academic. Conference articles are published much more quickly than scholarly articles. You can find conference papers in many of the same places as scholarly articles.

How Do You Identify Empirical Articles?

To identify an article based on empirical research, look for the following characteristics:

     The article is published in a peer-reviewed journal .

     The article includes charts, graphs, or statistical analysis .

     The article is substantial in size , likely to be more than 5 pages long.

     The article contains the following parts (the exact terms may vary): abstract, introduction, method, results, discussion, references .

• << Previous: Types of Clinical Study Designs
• Next: 3. Search the Literature >>
• Last Updated: Dec 29, 2023 11:41 AM
• URL: https://research.library.gsu.edu/litrev

• Types of Sources
• Evaluating Sources
• Use LibKey Nomad to Search Anywhere!

Quick Links

• How to Find...
• Reference and Research Support Services
• Interlibrary Loan (ILL)
• Regional Centers, Online Programs, and Dual Enrollment
• Library Policies
• Paul Meek Library Assessment Initiative

Peer-Reviewed

Peer-Reviewed articles are sources that have been reviewed by experts in a specific field and determined to meet the expectations of that field for publication in specific scholarly journals. Not all journals use this process. And some peer-reviewed journals have sections that include non peer-reviewed articles. Resources on the open web, such as blogs, magazines, news sites, and general webpages are not the same as peer-reviewed articles. 

In the peer-review process , scholars review manuscripts submitted for publication. The process is made as anonymous as possible. Manuscripts are reviewed and critiqued in the following areas:  Journal needs/appropriateness, methodology (quality of research), citations/evidence/lit review, relevance to the field, etc. After review, manuscripts are either accepted, rejected, or returned to the author for revision and resubmission.  This process can be involved and take many months (sometimes longer).  

While this process generally guarantees accurate and relevant resources, you still need to use some critical thinking skills when searching the library databases for peer-reviewed articles. Keep in mind that in some fields, progress and advancements in understanding can happen rapidly (within a few years), so some resources can be out of date (think about medical fields). And also be aware that not everything in a peer-reviewed journal is necessarily peer-reviewed. Pay attention to the article details and the journal in which the article is published. 

Empirical Sources

Your professor wants you to utilize empirical sources in your work. Empirical articles refer to sources that are reporting on original research. You may also hear these sources referred to as primary sources.  Empirical sources will contain several sections, including a methods or methodology section. The methodology will tell you how the authors conducted their study or experiment, including how they selected participants, how they determined or selected survey instruments, and other processes.  

Empirical sources can also include literature review, results, discussion, and suggestions for future research sections, in addition to the methods section.  You can limit to empirical studies in the advanced search features of many psychological databases, including PsycArticles. 

• PsycARTICLES (ProQuest) This link opens in a new window Articles from all of the publications of the American Psychological Association.

You may also discover and use secondary sources , like review articles - these articles will generally evaluate empirical sources.  Some examples of review articles include literature reviews, systematic reviews, analysis papers, synthesis papers, etc. Unlike an empirical source, you won't see an experiment or survey tool described in the secondary sources. Review articles offer interpretation and commentary on the empirical source and its methodology (experiment/survey).  This is the main way you can differentiate an empirical study from a review article. 

• << Previous: PSYC 301
• Next: Evaluating Sources >>
• Last Updated: Sep 8, 2023 3:44 PM
• URL: https://libguides.utm.edu/PSYC301

Want to create or adapt books like this? Learn more about how Pressbooks supports open publishing practices.

Chapter 2: What is a Literature Review?

Learning objectives.

At the conclusion of this chapter, you will be able to:

• Recognize how information is created and how it evolves over time.
• Identify how the information cycle impacts the reliability of the information.
• Select information sources appropriate to information need.

2.1 Overview of information

Because a literature review is a summary and analysis of the relevant publications on a topic, we first have to understand what is meant by ‘the literature’.  In this case, ‘the literature’ is a collection of all of the relevant written sources on a topic.  It will include both theoretical and empirical works.  Both types provide scope and depth to a literature review.

2.1.1 Disciplines of knowledge

When drawing boundaries around an idea, topic, or subject area, it helps to think about how and where the information for the field is produced. For this, you need to identify the disciplines of knowledge production in a subject area.

Information does not exist in the environment like some kind of raw material. It is produced by individuals working within a particular field of knowledge who use specific methods for generating new information. Disciplines are knowledge-producing and -disseminating systems which consume, produce and disseminate knowledge. Looking through a  course catalog of a post-secondary educational institution gives clues to the structure of a discipline structure. Fields such as political science, biology, history and mathematics are unique disciplines, as are education and nursing, with their own logic for how and where new knowledge is introduced and made accessible.

You will need to become comfortable with identifying the disciplines that might contribute information to any search strategy. When you do this, you will also learn how to decode the way how people talk about a topic within a discipline. This will be useful to you when you begin a  review of the literature in your area of study.

For example, think about the disciplines that might contribute information to a the topic such as  the role of sports in society. Try to anticipate the type of perspective each discipline might have on the topic. Consider the following types of questions as you examine what different disciplines might contribute:

• What is important about the topic to the people in that discipline?
• What is most likely to be the focus of their study about the topic?
• What perspective would they be likely to have on the topic?

In this example, we identify two disciplines that have something to say about the role of sports in society: allied health and education. What would each of these disciplines raise as key questions or issues related to that topic?

2.1.1.1 Nursing

• how sports affect individuals’ health and well-being
• assessing and treating sports injuries
• physical conditioning for athletes

2.1.1.2 Education

• how schools privilege or punish student athletes
• how young people are socialized into the ideal of team cooperation
• differences between boys’ and girls’ participation in organized sports

We see that a single topic can be approached from many different perspectives depending on how the disciplinary boundaries are drawn and how the topic is framed. This step of the research process requires you to make some decisions early on to focus the topic on a manageable and appropriate scope for the rest of the strategy. ( Hansen & Paul, 2015 ).

‘The literature’ consists of the published works that document a scholarly conversation in a field of study. You will find, in ‘the literature,’ documents that explain the background of your topic so the reader knows where you found loose ends in the established research of the field and what led you to your own project.  Although your own literature review will focus on primary, peer-reviewed resources, it will begin by first grounding yourself in background subject information generally found in secondary and tertiary sources such as books and encyclopedias.  Once you have that essential overview, you delve into the seminal literature of the field. As a result, while your literature review may consist of research articles tightly focused on your topic with secondary and tertiary sources used more sparingly, all three types of information (primary, secondary, tertiary) are critical to your research.

2.1.2 Definitions

• Theoretical – discusses a theory, conceptual model or framework for understanding a problem.
• Empirical – applies theory to a behavior or event and reports derived data to findings.
• Seminal – “A classic work of research literature that is more than 5 years old and is marked by its uniqueness and contribution to professional knowledge.” ( Houser, 4th ed., 2018, p. 112 ).
• Practical – “…accounts of how things are done” ( Wallace & Wray, 3rd ed., 2016, p. 20 ). Action research, in Education, refers to a wide variety of methods used to develop practical solutions. ( Great Schools Partnership, 2017 ).
• Policy – generally produced by policy-makers, such as government agencies.
• Primary – published results of original research studies .
• Secondary – interpret, discuss, summarize original sources
• Tertiary – synthesize or distill primary and secondary sources.  Examples include: encyclopedias, directories, dictionaries, handbooks, guides, classification, chronology, and other fact books.
• Grey literature – research and information released by non-commercial publishers, such as government agencies, policy organizations, and think-tanks.

‘The literature’ is published in books, journal articles, conference proceedings, theses and dissertations.  It can also be found in newspapers, encyclopedias, textbooks, as well as websites and reports written by government agencies and professional organizations. While these formats may contain what we define as ‘the literature’, not all of it will be appropriate for inclusion in your own literature review.

These sources are found through different tools that we will discuss later in this section. Although a discovery tool, such as a database or catalog, may link you to the ‘the literature’ not every tool is appropriate to every literature review.  No single source will have all of the information resources you should consult.  A comprehensive literature review should include searches in the following:

• Multiple subject and article databases
• Library and other book catalogs
• Grey literature sources

2.2 Information Cycle

To get a better idea of how the literature in a discipline develops, it’s useful to see how the information publication lifecycle works.  These distinct stages show how information is created, reviewed, and distributed over time.

The following chart can be used to guide you in searching literature existing at various stages of the scholarly communication process (freely accessible sources are linked, subscription or subscribed sources are listed but not linked):

2.3 Information Types

To continue our discussion of information sources, there are two ways published information in the field can be categorized:

• Articles by the type of periodical in which an article it is published, for example, magazine, trade, or scholarly publications .
• Where the material is located in the information cycle, as in primary, secondary, or tertiary information sources .

2.3.1 Popular, Trade, or Scholarly publications

2.3.1.1 types of periodicals.

Journals, trade publications, and magazines are all periodicals, and articles from these publications they can all look similar article by article when you are searching in the databases. It is good to review the differences and think about when to use information from each type of periodical.

2.3.1.2 Magazines

A magazine is a collection of articles and images about diverse topics of popular interest and current events.

Features of magazines:

• articles are usually written by journalists
• articles are written for the average adult
• articles tend to be short
• articles rarely provides a list of reference sources at the end of the article
• lots of color images and advertisements
• the decision about what goes into the magazine is made by an editor or publisher
• magazines can have broad appeal, like Time and Newsweek , or a narrow focus, like Sports Illustrated and Mother Earth News .

Popular magazines like Psychology Today , Sports Illustrated , and Rolling Stone can be good sources for articles on recent events or pop-culture topics, while Harpers , Scientific American , and The New Republic will offer more in-depth articles on a wider range of subjects. These articles are geared towards readers who, although not experts, are knowledgeable about the issues presented.

2.3.1.3 Trade Publications

Trade publications or trade journals are periodicals directed to members of a specific profession. They often have information about industry trends and practical information for people working in the field.

Features of trade publications:

• Authors are specialists in their fields
• Focused on members of a specific industry or profession
• No peer review process
• Include photographs, illustrations, charts, and graphs, often in color
• Technical vocabulary

Trade publications are geared towards professionals in a discipline. They report news and trends in a field, but not original research. They may provide product or service reviews, job listings, and advertisements.

2.3.1.4 Scholarly, Academic, and Scientific Publications

Scholarly, academic, and scientific publications are a collections of articles written by scholars in an academic or professional field. Most journals are peer-reviewed or refereed, which means a panel of scholars reviews articles to decide if they should be accepted into a specific publication. Journal articles are the main source of information for researchers and for literature reviews.

Features of journals:

• written by scholars and subject experts
• author’ credentials and institution will be identified
• written for other scholars
• dedicated to a specific discipline that it covers in depth
• often report on original or innovative research
• long articles, often 5-15 pages or more
• articles almost always include a list of sources at the end (Works Cited, References, Sources, or Bibliography) that point back to where the information was derived
• no or very few advertisements
• published by organizations or associations to advance their specialized body of knowledge

Scholarly journals provider articles of interest to experts or researchers in a discipline. An editorial board of respected scholars (peers) reviews all articles submitted to a journal. They decide if the article provides a noteworthy contribution to the field and should be published. There are typically few  little or no advertisements. Articles published in scholarly journals will include a list of references.

2.3.1.5 A word about open access journals

Increasingly, scholars are publishing findings and original research in open access journals .   Open access journals are scholarly and peer-reviewed and open access publishers provide unrestricted access and unrestricted use.  Open access is a means of disseminating scholarly research that breaks from the traditional subscription model of academic publishing. It is free of charge to readers and because it is online, it is available at anytime, anywhere in the world, to anyone with access to the internet.  The Directory of Open Access Journals ( DOAJ ) indexes and provides access to high-quality, peer-reviewed scholarly articles.

In summary, newspapers and other popular press publications are useful for getting general topic ideas. Trade publications are useful for practical application in a profession and may also be a good source of keywords for future searching. Scholarly journals are the conversation of the scholars who are doing research in a specific discipline and publishing their research findings.

2.3.1.6 Primary, Secondary, and Tertiary Sources

Primary sources of information are those types of information that come first. Some examples of primary sources are:

• original research, like data from an experiment with plankton.
• diaries, journals, photographs
• data from the census bureau or a survey you have done
• original documents, like the constitution or a birth certificate
• newspapers are primary sources when they report current events or current opinion
• speeches, interviews, email, letters
• religious books
• personal memoirs and autobiographies
• pottery or weavings

There are different types of primary sources for different disciplines.  In the discipline of history, for example, a diary or transcript of a speech is a primary source.  In education and nursing, primary sources will generally be original research, including data sets.

Secondary sources are written about primary sources to interpret or analyze them. They are a step or more removed from the primary event or item. Some examples of secondary sources are:

• commentaries on speeches
• critiques of plays, journalism, or books
• a journal article that talks about a primary source such as an interpretation of Steinbeck’s The Grapes of Wrath, or the flower symbolism of Monet’s water garden paintings
• textbooks (can also be considered tertiary)
• biographies
• encyclopedias

Tertiary sources are further removed from the original material and are a distillation and collection of primary and secondary sources. Some examples are:

• bibliography of critical works about an author
• textbooks (also considered secondary)

A comparison of information sources across disciplines:

2.4 Information Sources

In this section, we discuss how to find not only information, but the sources of information in your discipline or topic area.  As we see in the graphic and chart above, the information you need for your literature review will be located in multiple places.  How and where research and publication occurs drives how and where the information is located, which in turn determines how you will discover and retrieve it.  When we talk about information sources for a literature review in education or nursing, we generally mean these five areas: the internet, reference material and other books, empirical or evidence-based articles in scholarly, peer-reviewed journals, conference proceedings and papers, dissertations and theses, and grey literature.

The World Wide Web can be an excellent place to satisfy some initial research needs.

• It is a good resource for background information and for finding keywords for searching in the library catalog and databases.
• It is a good tool for locating professional organizations and searching for information and the names of experts in a given discipline.
• Google Scholar is a useful discovery tool for citations, especially if you are trying to get the lay of the land surrounding your topic or if you are having a problem with keywords in the databases. You can find some information to refine your search terms. It is NOT acceptable to depend on Google Scholar for finding articles because of the spotty coverage and lack of adequate search features.

2.4.2 Books and Reference Sources

Reference materials and books are available in both print and electronic formats. They provide gateway knowledge to a subject area and are useful at the beginning of the research process to:

• Get an overview of the topic, learn the scope, key definitions, significant figures who are involved, and important timelines
• Discover the foundations of a topic
• Learn essential definitions, vocabulary terms, and keywords you can use in your literature searching strategy

2.4.3 Scholarly Articles in Journals

Another major category of information sources is scholarly information produced by subject experts working in academic institutions, research centers and scholarly organizations. Scholars and researchers generate information that advances our knowledge and understanding of the world. The research they do creates new opportunities for inventions, practical applications, and new approaches to solving problems or understanding issues.

Academics, researchers and students at universities make their contributions to scholarly knowledge available in many forms:

• masters’ theses
• doctoral dissertations
• conference papers
• journal articles and books
• individual scholars’ web pages
• web pages developed by the researcher’s’ home institution (Hansen & Paul, 2015).

Scholars and researchers introduce their discoveries to the world in a formal system of information dissemination that has developed over centuries. Because scholarly research undergoes a process of “peer review” before being published (meaning that other experts review the work and pass judgment about whether it is worthy of publication), the information you find from scholarly sources meets preset standards for accuracy, credibility and validity in that field.

Likewise, scholarly journal articles are generally considered to be among the most reliable sources of information because they have gone through a peer-review process.

2.4.5 Conference Papers & Proceedings

Conferences are a major source of  emerging research where researchers present papers on their current research and obtain feedback from the audience.  The papers presented in the conference are then usually published in a volume called a conference proceeding.  Conference proceedings highlight current discussion in a discipline and can lead you to scholars who are interested in specific research areas.

A word about conference papers: several factors contribute to making these documents difficult to find.  It may be months before a paper is published as a journal article, or it may never be published.  Publishers and professional associations are inconsistent in how they publish proceedings.  For example, the papers from an annual conference may be published as individual, stand-alone titles, which may be indexed in a library catalog, or the conference proceedings may be treated more like a periodical or serial and, therefore, indexed in a journal database.

It is not unusual that papers delivered at professional conferences are not published in print or electronic form, although an abstract may be available.  In these cases, the full paper may only be available from the author or authors.

The most important thing to remember is that if you have any difficulty finding a conference proceeding or paper, ask a librarian for assistance.

2.4.6 Dissertations and Theses

Dissertations and theses can be rich sources of information and have extensive reference lists to scan for resources. They are considered gray literature, so are not “peer reviewed”. The accuracy and validity of the paper itself may depend on the school that awarded the doctoral or master’s  degree to the author.

2.5 Conclusion

In thinking about ‘the literature’ of your discipline, you are beginning the first step in writing your own literature review.  By understanding what the literature in your field is, as well as how and when it is generated, you begin to know what is available and where to look for it.

We briefly discussed seven types of (sometimes overlapping) information:

• information found on the web
• information found in reference books and monographs
• information found in scholarly journals
• information found in conference proceedings and papers
• information found in dissertations and theses
• information found in magazines and trade journals
• information that is primary, secondary, or tertiary.

By conceptualizing or scoping how and where the literature of your discipline or topic area is generated, you have started on your way to writing your own literature review.

Finally, remember:

“All information sources are not created equal. Sources can vary greatly in terms of how carefully they are researched, written, edited, and reviewed for accuracy. Common sense will help you identify obviously questionable sources, such as tabloids that feature tales of alien abductions, or personal websites with glaring typos. Sometimes, however, a source’s reliability—or lack of it—is not so obvious…You will consider criteria such as the type of source, its intended purpose and audience, the author’s (or authors’) qualifications, the publication’s reputation, any indications of bias or hidden agendas, how current the source is, and the overall quality of the writing, thinking, and design.”  ( Writing for Success, 2015, p. 448 ).

We will cover how to evaluate sources in more detail in Chapter 5.

For each of these information needs, indicate what resources would be the best fit to answer your question. There may be more than one source so don’t feel like you have to limit yourself to only one. See Answer Key for the correct response.

• You are to write a brief paper on a theory that you only vaguely understand. You need some basic information. Where would you look?
• If you heard something on the radio about a recent research involving an herbal intervention for weight loss where could you find the actual study?
• You are going to be doing an internship in a group home for young men. You have heard that one issue that comes up for them is anger. Where would you look for practical interventions to help you manage this problem if it came up?
• You have the opportunity to work on a research project through a grant proposal. You need to justify the research question and show that there is an interest and a need for this research. What resources would you cite in your application?
• You have been assigned a project to find primary sources about classroom discipline used in early 20th-century schools. What primary sources could you use and where would you find them?
• You have an idea for a great thesis but you are afraid that it has been done before. Since you would like to do something original, where could you find out if someone else has done the project?
• There was a post on Facebook that welfare recipients in Arizona were recently tested for drug use with only three in 140,000 having positive results. Where can I find out if this number is accurate?

Test Yourself

Question 1  match the type of periodical to its content.

Trade publication Scholarly journal Magazine

• Contains articles about a variety of topics of popular interest; also contains advertising.
• Has information about industry trends and practical information for professionals in a field.
• Contains articles written by scholars in an academic field and reviewed by experts in that field.

Question 2: Given what you know about information types and sources, put the following information sources in order from the least accurate and reliable  to the most accurate and reliable. (1 least accurate/4 most accurate)

• Books and encyclopedias
• News broadcasts and social media directly following an event.
• Analysis of an event in the news media or popular magazine weeks after an event.
• Articles written by scholars and published in a journal.

Question 3: What is information called that is either a diary, a speech, original research, data, artwork, or a religious book.

Question 4: to find the best information in the databases you need to use keywords that are used by the scholars. where do you find out what keywords to try.

• From websites
• In journal articles
• All of the above

Question 5: Which of the following is NOT true about scholarly journals?

• They contain the conversation of the scholars on a particular subject.
• They are of interest to the general public.
• The articles are followed by an extensive reference list.
• They contain reports of original research.

Image Attribution

Literature Reviews for Education and Nursing Graduate Students Copyright © by Linda Frederiksen is licensed under a Creative Commons Attribution 4.0 International License , except where otherwise noted.

Share This Book

Literature review sources

Sources for literature review can be divided into three categories as illustrated in table below. In your dissertation you will need to use all three categories of literature review sources:

Sources for literature review and examples

Generally, your literature review should integrate a wide range of sources such as:

• Books . Textbooks remain as the most important source to find models and theories related to the research area. Research the most respected authorities in your selected research area and find the latest editions of books authored by them. For example, in the area of marketing the most notable authors include Philip Kotler, Seth Godin, Malcolm Gladwell, Emanuel Rosen and others.
• Magazines . Industry-specific magazines are usually rich in scholarly articles and they can be effective source to learn about the latest trends and developments in the research area. Reading industry magazines can be the most enjoyable part of the literature review, assuming that your selected research area represents an area of your personal and professional interests, which should be the case anyways.
• Newspapers can be referred to as the main source of up-to-date news about the latest events related to the research area. However, the proportion of the use of newspapers in literature review is recommended to be less compared to alternative sources of secondary data such as books and magazines. This is due to the fact that newspaper articles mainly lack depth of analyses and discussions.
• Online articles . You can find online versions of all of the above sources. However, note that the levels of reliability of online articles can be highly compromised depending on the source due to the high levels of ease with which articles can be published online. Opinions offered in a wide range of online discussion blogs cannot be usually used in literature review. Similarly, dissertation assessors are not keen to appreciate references to a wide range of blogs, unless articles in these blogs are authored by respected authorities in the research area.

Your secondary data sources may comprise certain amount of grey literature as well. The term grey literature refers to type of literature produced by government, academics, business and industry in print and electronic formats, which is not controlled by commercial publishers. It is called ‘grey’ because the status of the information in grey literature is not certain. In other words, any publication that has not been peer reviewed for publication is grey literature.

The necessity to use grey literature arises when there is no enough peer reviewed publications are available for the subject of your study.

John Dudovskiy

Library DIY: Finding Books and Articles

• Academic Books
• Children's Books
• Print Books on the Shelf
• Empirical/Primary Articles
• Significant Articles
• Paying for Articles?
• The Library Doesn't Have What I Need
• WISE Link Problem

What is an empirical source or primary article?

In many of your classes, especially in the sciences, your professor may specify that you should use primary sources or empirical articles. It is important to understand what that means before you begin your search.

Sources come in three types: Primary, secondary, and tertiary.

• A primary source, in the sciences (sometimes called an empirical source), is the first writing of new research. It reports on a single experiment and is written by those who did the research. This is different than in History, where a primary source is a source from the time period you are studying, or English where a primary source may refer to a piece of literature you are analyzing.
• Secondary sources synthesize and compare primary sources. Secondary sources give an overview on ‘recent research’ on a topic. If you find a secondary source, you can use it to find primary sources, because it will probably cite a whole lot of primary sources.
• Tertiary sources are written for the public or people without in-depth knowledge of the topic. Most of the time, if we want to know something in our life, we use tertiary sources.

Peer-reviewed journals publish many things that are not primary sources, including:

• Literature reviews
• Meta-Analyses/Review articles (These are studies that arrive at conclusions based on research from many other studies.)
• Chapters in books
• Encyclopedia articles
• Speeches and interviews

Identifying Primary Sources Using the Parts of an Article

Identifying a primary or empirical article takes practice.  You have to carefully review each of the parts of the article.

Abstract: The abstract of an article is a short summary of the research. A primary source will have an abstract that includes a hypothesis and an active statement of research that the author(s) performed. Pay special attention to the way the abstract describes the type of research that was conducted.

Here is an abstract of a primary source . One of the big clues is the inclusion of participants in the study and the description of how the experiment was carried out.

Here is an abstract of a secondary source . The authors include an active statement of research, but the process they describe includes searching other people’s research and comparing multiple studies.

Methods: A Primary source should have a methods section. Secondary sources occasionally have a methods section, so be sure to read carefully to understand whether they did original research (lab work, clinical trial, interviews, surveys, analyzing historical records) or if they found and analyzed the research of others (the authors might discuss conducting a literature review or searching different databases).

Here is the methods section of a primary source , with excerpts from “participants” and “measures” sections. (Other primary articles might look different, but will include similar information)

Here is the methods section of a secondary article . Not all secondary articles will have a methods section, but if they do, it should outline the search process and qualities of studies that would indicate their inclusion in the research.

Results: A primary source should also have a results section, where the authors present the raw data that they collected. Some secondary sources will have a section labeled results that summarize their analysis of the primary sources they reviewed. Remember, if it is a primary source, there should be data collected from the study.

Here is an excerpt from the results section of a primary source.  It reports data and trends in data that were gathered by the author(s).

Here is an excerpt from the results section of a secondary source . It reports the summary of data and data trends found in other studies, not directly conducted by the author(s).

The results section of a primary source will usually include tables, charts, and graphs that help to make the data more understandable. Some secondary sources also contain graphs or tables that explain the process they used to select which primary articles to review. Be sure that you understand what the table, chart, or graph is trying to show.

Here is a data table from a primary source . This table shows the data gathered from different treatment groups in the study.

Here is a data table from a secondary source . This table shows the details of the articles they included in their study.

Discussion: In the discussion section, a primary source will analyze and explain the results to draw preliminary conclusions and discuss how their findings compare to existing research.

Here is an excerpt from the discussion section of a primary source (case study). It discusses the evidence and outcome of a specific study conducted by the author(s).

Here is an excerpt from the discussion section of a secondary source . Some keywords include "exploratory examination" of other studies and "systemic review".

If you aren't sure if an article you've found is a primary source - ask a librarian!  

Your professor is the final judge of whether an article is appropriate for a particular assignment, so be sure to discuss your source selection with them. 

• << Previous: Peer-Reviewed Articles
• Next: Significant Articles >>
• Last Updated: Jun 29, 2023 2:15 PM
• URL: https://libguides.westminster.edu/c.php?g=826484

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings

Preview improvements coming to the PMC website in October 2024. Learn More or Try it out now .

• Advanced Search
• Journal List
• CBE Life Sci Educ
• v.21(3); Fall 2022

Literature Reviews, Theoretical Frameworks, and Conceptual Frameworks: An Introduction for New Biology Education Researchers

Julie a. luft.

† Department of Mathematics, Social Studies, and Science Education, Mary Frances Early College of Education, University of Georgia, Athens, GA 30602-7124

Sophia Jeong

‡ Department of Teaching & Learning, College of Education & Human Ecology, Ohio State University, Columbus, OH 43210

Robert Idsardi

§ Department of Biology, Eastern Washington University, Cheney, WA 99004

Grant Gardner

∥ Department of Biology, Middle Tennessee State University, Murfreesboro, TN 37132

Associated Data

To frame their work, biology education researchers need to consider the role of literature reviews, theoretical frameworks, and conceptual frameworks as critical elements of the research and writing process. However, these elements can be confusing for scholars new to education research. This Research Methods article is designed to provide an overview of each of these elements and delineate the purpose of each in the educational research process. We describe what biology education researchers should consider as they conduct literature reviews, identify theoretical frameworks, and construct conceptual frameworks. Clarifying these different components of educational research studies can be helpful to new biology education researchers and the biology education research community at large in situating their work in the broader scholarly literature.

INTRODUCTION

Discipline-based education research (DBER) involves the purposeful and situated study of teaching and learning in specific disciplinary areas ( Singer et al. , 2012 ). Studies in DBER are guided by research questions that reflect disciplines’ priorities and worldviews. Researchers can use quantitative data, qualitative data, or both to answer these research questions through a variety of methodological traditions. Across all methodologies, there are different methods associated with planning and conducting educational research studies that include the use of surveys, interviews, observations, artifacts, or instruments. Ensuring the coherence of these elements to the discipline’s perspective also involves situating the work in the broader scholarly literature. The tools for doing this include literature reviews, theoretical frameworks, and conceptual frameworks. However, the purpose and function of each of these elements is often confusing to new education researchers. The goal of this article is to introduce new biology education researchers to these three important elements important in DBER scholarship and the broader educational literature.

The first element we discuss is a review of research (literature reviews), which highlights the need for a specific research question, study problem, or topic of investigation. Literature reviews situate the relevance of the study within a topic and a field. The process may seem familiar to science researchers entering DBER fields, but new researchers may still struggle in conducting the review. Booth et al. (2016b) highlight some of the challenges novice education researchers face when conducting a review of literature. They point out that novice researchers struggle in deciding how to focus the review, determining the scope of articles needed in the review, and knowing how to be critical of the articles in the review. Overcoming these challenges (and others) can help novice researchers construct a sound literature review that can inform the design of the study and help ensure the work makes a contribution to the field.

The second and third highlighted elements are theoretical and conceptual frameworks. These guide biology education research (BER) studies, and may be less familiar to science researchers. These elements are important in shaping the construction of new knowledge. Theoretical frameworks offer a way to explain and interpret the studied phenomenon, while conceptual frameworks clarify assumptions about the studied phenomenon. Despite the importance of these constructs in educational research, biology educational researchers have noted the limited use of theoretical or conceptual frameworks in published work ( DeHaan, 2011 ; Dirks, 2011 ; Lo et al. , 2019 ). In reviewing articles published in CBE—Life Sciences Education ( LSE ) between 2015 and 2019, we found that fewer than 25% of the research articles had a theoretical or conceptual framework (see the Supplemental Information), and at times there was an inconsistent use of theoretical and conceptual frameworks. Clearly, these frameworks are challenging for published biology education researchers, which suggests the importance of providing some initial guidance to new biology education researchers.

Fortunately, educational researchers have increased their explicit use of these frameworks over time, and this is influencing educational research in science, technology, engineering, and mathematics (STEM) fields. For instance, a quick search for theoretical or conceptual frameworks in the abstracts of articles in Educational Research Complete (a common database for educational research) in STEM fields demonstrates a dramatic change over the last 20 years: from only 778 articles published between 2000 and 2010 to 5703 articles published between 2010 and 2020, a more than sevenfold increase. Greater recognition of the importance of these frameworks is contributing to DBER authors being more explicit about such frameworks in their studies.

Collectively, literature reviews, theoretical frameworks, and conceptual frameworks work to guide methodological decisions and the elucidation of important findings. Each offers a different perspective on the problem of study and is an essential element in all forms of educational research. As new researchers seek to learn about these elements, they will find different resources, a variety of perspectives, and many suggestions about the construction and use of these elements. The wide range of available information can overwhelm the new researcher who just wants to learn the distinction between these elements or how to craft them adequately.

Our goal in writing this paper is not to offer specific advice about how to write these sections in scholarly work. Instead, we wanted to introduce these elements to those who are new to BER and who are interested in better distinguishing one from the other. In this paper, we share the purpose of each element in BER scholarship, along with important points on its construction. We also provide references for additional resources that may be beneficial to better understanding each element. Table 1 summarizes the key distinctions among these elements.

Comparison of literature reviews, theoretical frameworks, and conceptual reviews

This article is written for the new biology education researcher who is just learning about these different elements or for scientists looking to become more involved in BER. It is a result of our own work as science education and biology education researchers, whether as graduate students and postdoctoral scholars or newly hired and established faculty members. This is the article we wish had been available as we started to learn about these elements or discussed them with new educational researchers in biology.

LITERATURE REVIEWS

Purpose of a literature review.

A literature review is foundational to any research study in education or science. In education, a well-conceptualized and well-executed review provides a summary of the research that has already been done on a specific topic and identifies questions that remain to be answered, thus illustrating the current research project’s potential contribution to the field and the reasoning behind the methodological approach selected for the study ( Maxwell, 2012 ). BER is an evolving disciplinary area that is redefining areas of conceptual emphasis as well as orientations toward teaching and learning (e.g., Labov et al. , 2010 ; American Association for the Advancement of Science, 2011 ; Nehm, 2019 ). As a result, building comprehensive, critical, purposeful, and concise literature reviews can be a challenge for new biology education researchers.

Building Literature Reviews

There are different ways to approach and construct a literature review. Booth et al. (2016a) provide an overview that includes, for example, scoping reviews, which are focused only on notable studies and use a basic method of analysis, and integrative reviews, which are the result of exhaustive literature searches across different genres. Underlying each of these different review processes are attention to the s earch process, a ppraisa l of articles, s ynthesis of the literature, and a nalysis: SALSA ( Booth et al. , 2016a ). This useful acronym can help the researcher focus on the process while building a specific type of review.

However, new educational researchers often have questions about literature reviews that are foundational to SALSA or other approaches. Common questions concern determining which literature pertains to the topic of study or the role of the literature review in the design of the study. This section addresses such questions broadly while providing general guidance for writing a narrative literature review that evaluates the most pertinent studies.

The literature review process should begin before the research is conducted. As Boote and Beile (2005 , p. 3) suggested, researchers should be “scholars before researchers.” They point out that having a good working knowledge of the proposed topic helps illuminate avenues of study. Some subject areas have a deep body of work to read and reflect upon, providing a strong foundation for developing the research question(s). For instance, the teaching and learning of evolution is an area of long-standing interest in the BER community, generating many studies (e.g., Perry et al. , 2008 ; Barnes and Brownell, 2016 ) and reviews of research (e.g., Sickel and Friedrichsen, 2013 ; Ziadie and Andrews, 2018 ). Emerging areas of BER include the affective domain, issues of transfer, and metacognition ( Singer et al. , 2012 ). Many studies in these areas are transdisciplinary and not always specific to biology education (e.g., Rodrigo-Peiris et al. , 2018 ; Kolpikova et al. , 2019 ). These newer areas may require reading outside BER; fortunately, summaries of some of these topics can be found in the Current Insights section of the LSE website.

In focusing on a specific problem within a broader research strand, a new researcher will likely need to examine research outside BER. Depending upon the area of study, the expanded reading list might involve a mix of BER, DBER, and educational research studies. Determining the scope of the reading is not always straightforward. A simple way to focus one’s reading is to create a “summary phrase” or “research nugget,” which is a very brief descriptive statement about the study. It should focus on the essence of the study, for example, “first-year nonmajor students’ understanding of evolution,” “metacognitive prompts to enhance learning during biochemistry,” or “instructors’ inquiry-based instructional practices after professional development programming.” This type of phrase should help a new researcher identify two or more areas to review that pertain to the study. Focusing on recent research in the last 5 years is a good first step. Additional studies can be identified by reading relevant works referenced in those articles. It is also important to read seminal studies that are more than 5 years old. Reading a range of studies should give the researcher the necessary command of the subject in order to suggest a research question.

Given that the research question(s) arise from the literature review, the review should also substantiate the selected methodological approach. The review and research question(s) guide the researcher in determining how to collect and analyze data. Often the methodological approach used in a study is selected to contribute knowledge that expands upon what has been published previously about the topic (see Institute of Education Sciences and National Science Foundation, 2013 ). An emerging topic of study may need an exploratory approach that allows for a description of the phenomenon and development of a potential theory. This could, but not necessarily, require a methodological approach that uses interviews, observations, surveys, or other instruments. An extensively studied topic may call for the additional understanding of specific factors or variables; this type of study would be well suited to a verification or a causal research design. These could entail a methodological approach that uses valid and reliable instruments, observations, or interviews to determine an effect in the studied event. In either of these examples, the researcher(s) may use a qualitative, quantitative, or mixed methods methodological approach.

Even with a good research question, there is still more reading to be done. The complexity and focus of the research question dictates the depth and breadth of the literature to be examined. Questions that connect multiple topics can require broad literature reviews. For instance, a study that explores the impact of a biology faculty learning community on the inquiry instruction of faculty could have the following review areas: learning communities among biology faculty, inquiry instruction among biology faculty, and inquiry instruction among biology faculty as a result of professional learning. Biology education researchers need to consider whether their literature review requires studies from different disciplines within or outside DBER. For the example given, it would be fruitful to look at research focused on learning communities with faculty in STEM fields or in general education fields that result in instructional change. It is important not to be too narrow or too broad when reading. When the conclusions of articles start to sound similar or no new insights are gained, the researcher likely has a good foundation for a literature review. This level of reading should allow the researcher to demonstrate a mastery in understanding the researched topic, explain the suitability of the proposed research approach, and point to the need for the refined research question(s).

The literature review should include the researcher’s evaluation and critique of the selected studies. A researcher may have a large collection of studies, but not all of the studies will follow standards important in the reporting of empirical work in the social sciences. The American Educational Research Association ( Duran et al. , 2006 ), for example, offers a general discussion about standards for such work: an adequate review of research informing the study, the existence of sound and appropriate data collection and analysis methods, and appropriate conclusions that do not overstep or underexplore the analyzed data. The Institute of Education Sciences and National Science Foundation (2013) also offer Common Guidelines for Education Research and Development that can be used to evaluate collected studies.

Because not all journals adhere to such standards, it is important that a researcher review each study to determine the quality of published research, per the guidelines suggested earlier. In some instances, the research may be fatally flawed. Examples of such flaws include data that do not pertain to the question, a lack of discussion about the data collection, poorly constructed instruments, or an inadequate analysis. These types of errors result in studies that are incomplete, error-laden, or inaccurate and should be excluded from the review. Most studies have limitations, and the author(s) often make them explicit. For instance, there may be an instructor effect, recognized bias in the analysis, or issues with the sample population. Limitations are usually addressed by the research team in some way to ensure a sound and acceptable research process. Occasionally, the limitations associated with the study can be significant and not addressed adequately, which leaves a consequential decision in the hands of the researcher. Providing critiques of studies in the literature review process gives the reader confidence that the researcher has carefully examined relevant work in preparation for the study and, ultimately, the manuscript.

A solid literature review clearly anchors the proposed study in the field and connects the research question(s), the methodological approach, and the discussion. Reviewing extant research leads to research questions that will contribute to what is known in the field. By summarizing what is known, the literature review points to what needs to be known, which in turn guides decisions about methodology. Finally, notable findings of the new study are discussed in reference to those described in the literature review.

Within published BER studies, literature reviews can be placed in different locations in an article. When included in the introductory section of the study, the first few paragraphs of the manuscript set the stage, with the literature review following the opening paragraphs. Cooper et al. (2019) illustrate this approach in their study of course-based undergraduate research experiences (CUREs). An introduction discussing the potential of CURES is followed by an analysis of the existing literature relevant to the design of CUREs that allows for novel student discoveries. Within this review, the authors point out contradictory findings among research on novel student discoveries. This clarifies the need for their study, which is described and highlighted through specific research aims.

A literature reviews can also make up a separate section in a paper. For example, the introduction to Todd et al. (2019) illustrates the need for their research topic by highlighting the potential of learning progressions (LPs) and suggesting that LPs may help mitigate learning loss in genetics. At the end of the introduction, the authors state their specific research questions. The review of literature following this opening section comprises two subsections. One focuses on learning loss in general and examines a variety of studies and meta-analyses from the disciplines of medical education, mathematics, and reading. The second section focuses specifically on LPs in genetics and highlights student learning in the midst of LPs. These separate reviews provide insights into the stated research question.

Suggestions and Advice

A well-conceptualized, comprehensive, and critical literature review reveals the understanding of the topic that the researcher brings to the study. Literature reviews should not be so big that there is no clear area of focus; nor should they be so narrow that no real research question arises. The task for a researcher is to craft an efficient literature review that offers a critical analysis of published work, articulates the need for the study, guides the methodological approach to the topic of study, and provides an adequate foundation for the discussion of the findings.

In our own writing of literature reviews, there are often many drafts. An early draft may seem well suited to the study because the need for and approach to the study are well described. However, as the results of the study are analyzed and findings begin to emerge, the existing literature review may be inadequate and need revision. The need for an expanded discussion about the research area can result in the inclusion of new studies that support the explanation of a potential finding. The literature review may also prove to be too broad. Refocusing on a specific area allows for more contemplation of a finding.

It should be noted that there are different types of literature reviews, and many books and articles have been written about the different ways to embark on these types of reviews. Among these different resources, the following may be helpful in considering how to refine the review process for scholarly journals:

• Booth, A., Sutton, A., & Papaioannou, D. (2016a). Systemic approaches to a successful literature review (2nd ed.). Los Angeles, CA: Sage. This book addresses different types of literature reviews and offers important suggestions pertaining to defining the scope of the literature review and assessing extant studies.
• Booth, W. C., Colomb, G. G., Williams, J. M., Bizup, J., & Fitzgerald, W. T. (2016b). The craft of research (4th ed.). Chicago: University of Chicago Press. This book can help the novice consider how to make the case for an area of study. While this book is not specifically about literature reviews, it offers suggestions about making the case for your study.
• Galvan, J. L., & Galvan, M. C. (2017). Writing literature reviews: A guide for students of the social and behavioral sciences (7th ed.). Routledge. This book offers guidance on writing different types of literature reviews. For the novice researcher, there are useful suggestions for creating coherent literature reviews.

THEORETICAL FRAMEWORKS

Purpose of theoretical frameworks.

As new education researchers may be less familiar with theoretical frameworks than with literature reviews, this discussion begins with an analogy. Envision a biologist, chemist, and physicist examining together the dramatic effect of a fog tsunami over the ocean. A biologist gazing at this phenomenon may be concerned with the effect of fog on various species. A chemist may be interested in the chemical composition of the fog as water vapor condenses around bits of salt. A physicist may be focused on the refraction of light to make fog appear to be “sitting” above the ocean. While observing the same “objective event,” the scientists are operating under different theoretical frameworks that provide a particular perspective or “lens” for the interpretation of the phenomenon. Each of these scientists brings specialized knowledge, experiences, and values to this phenomenon, and these influence the interpretation of the phenomenon. The scientists’ theoretical frameworks influence how they design and carry out their studies and interpret their data.

Within an educational study, a theoretical framework helps to explain a phenomenon through a particular lens and challenges and extends existing knowledge within the limitations of that lens. Theoretical frameworks are explicitly stated by an educational researcher in the paper’s framework, theory, or relevant literature section. The framework shapes the types of questions asked, guides the method by which data are collected and analyzed, and informs the discussion of the results of the study. It also reveals the researcher’s subjectivities, for example, values, social experience, and viewpoint ( Allen, 2017 ). It is essential that a novice researcher learn to explicitly state a theoretical framework, because all research questions are being asked from the researcher’s implicit or explicit assumptions of a phenomenon of interest ( Schwandt, 2000 ).

Selecting Theoretical Frameworks

Theoretical frameworks are one of the most contemplated elements in our work in educational research. In this section, we share three important considerations for new scholars selecting a theoretical framework.

The first step in identifying a theoretical framework involves reflecting on the phenomenon within the study and the assumptions aligned with the phenomenon. The phenomenon involves the studied event. There are many possibilities, for example, student learning, instructional approach, or group organization. A researcher holds assumptions about how the phenomenon will be effected, influenced, changed, or portrayed. It is ultimately the researcher’s assumption(s) about the phenomenon that aligns with a theoretical framework. An example can help illustrate how a researcher’s reflection on the phenomenon and acknowledgment of assumptions can result in the identification of a theoretical framework.

In our example, a biology education researcher may be interested in exploring how students’ learning of difficult biological concepts can be supported by the interactions of group members. The phenomenon of interest is the interactions among the peers, and the researcher assumes that more knowledgeable students are important in supporting the learning of the group. As a result, the researcher may draw on Vygotsky’s (1978) sociocultural theory of learning and development that is focused on the phenomenon of student learning in a social setting. This theory posits the critical nature of interactions among students and between students and teachers in the process of building knowledge. A researcher drawing upon this framework holds the assumption that learning is a dynamic social process involving questions and explanations among students in the classroom and that more knowledgeable peers play an important part in the process of building conceptual knowledge.

It is important to state at this point that there are many different theoretical frameworks. Some frameworks focus on learning and knowing, while other theoretical frameworks focus on equity, empowerment, or discourse. Some frameworks are well articulated, and others are still being refined. For a new researcher, it can be challenging to find a theoretical framework. Two of the best ways to look for theoretical frameworks is through published works that highlight different frameworks.

When a theoretical framework is selected, it should clearly connect to all parts of the study. The framework should augment the study by adding a perspective that provides greater insights into the phenomenon. It should clearly align with the studies described in the literature review. For instance, a framework focused on learning would correspond to research that reported different learning outcomes for similar studies. The methods for data collection and analysis should also correspond to the framework. For instance, a study about instructional interventions could use a theoretical framework concerned with learning and could collect data about the effect of the intervention on what is learned. When the data are analyzed, the theoretical framework should provide added meaning to the findings, and the findings should align with the theoretical framework.

A study by Jensen and Lawson (2011) provides an example of how a theoretical framework connects different parts of the study. They compared undergraduate biology students in heterogeneous and homogeneous groups over the course of a semester. Jensen and Lawson (2011) assumed that learning involved collaboration and more knowledgeable peers, which made Vygotsky’s (1978) theory a good fit for their study. They predicted that students in heterogeneous groups would experience greater improvement in their reasoning abilities and science achievements with much of the learning guided by the more knowledgeable peers.

In the enactment of the study, they collected data about the instruction in traditional and inquiry-oriented classes, while the students worked in homogeneous or heterogeneous groups. To determine the effect of working in groups, the authors also measured students’ reasoning abilities and achievement. Each data-collection and analysis decision connected to understanding the influence of collaborative work.

Their findings highlighted aspects of Vygotsky’s (1978) theory of learning. One finding, for instance, posited that inquiry instruction, as a whole, resulted in reasoning and achievement gains. This links to Vygotsky (1978) , because inquiry instruction involves interactions among group members. A more nuanced finding was that group composition had a conditional effect. Heterogeneous groups performed better with more traditional and didactic instruction, regardless of the reasoning ability of the group members. Homogeneous groups worked better during interaction-rich activities for students with low reasoning ability. The authors attributed the variation to the different types of helping behaviors of students. High-performing students provided the answers, while students with low reasoning ability had to work collectively through the material. In terms of Vygotsky (1978) , this finding provided new insights into the learning context in which productive interactions can occur for students.

Another consideration in the selection and use of a theoretical framework pertains to its orientation to the study. This can result in the theoretical framework prioritizing individuals, institutions, and/or policies ( Anfara and Mertz, 2014 ). Frameworks that connect to individuals, for instance, could contribute to understanding their actions, learning, or knowledge. Institutional frameworks, on the other hand, offer insights into how institutions, organizations, or groups can influence individuals or materials. Policy theories provide ways to understand how national or local policies can dictate an emphasis on outcomes or instructional design. These different types of frameworks highlight different aspects in an educational setting, which influences the design of the study and the collection of data. In addition, these different frameworks offer a way to make sense of the data. Aligning the data collection and analysis with the framework ensures that a study is coherent and can contribute to the field.

New understandings emerge when different theoretical frameworks are used. For instance, Ebert-May et al. (2015) prioritized the individual level within conceptual change theory (see Posner et al. , 1982 ). In this theory, an individual’s knowledge changes when it no longer fits the phenomenon. Ebert-May et al. (2015) designed a professional development program challenging biology postdoctoral scholars’ existing conceptions of teaching. The authors reported that the biology postdoctoral scholars’ teaching practices became more student-centered as they were challenged to explain their instructional decision making. According to the theory, the biology postdoctoral scholars’ dissatisfaction in their descriptions of teaching and learning initiated change in their knowledge and instruction. These results reveal how conceptual change theory can explain the learning of participants and guide the design of professional development programming.

The communities of practice (CoP) theoretical framework ( Lave, 1988 ; Wenger, 1998 ) prioritizes the institutional level , suggesting that learning occurs when individuals learn from and contribute to the communities in which they reside. Grounded in the assumption of community learning, the literature on CoP suggests that, as individuals interact regularly with the other members of their group, they learn about the rules, roles, and goals of the community ( Allee, 2000 ). A study conducted by Gehrke and Kezar (2017) used the CoP framework to understand organizational change by examining the involvement of individual faculty engaged in a cross-institutional CoP focused on changing the instructional practice of faculty at each institution. In the CoP, faculty members were involved in enhancing instructional materials within their department, which aligned with an overarching goal of instituting instruction that embraced active learning. Not surprisingly, Gehrke and Kezar (2017) revealed that faculty who perceived the community culture as important in their work cultivated institutional change. Furthermore, they found that institutional change was sustained when key leaders served as mentors and provided support for faculty, and as faculty themselves developed into leaders. This study reveals the complexity of individual roles in a COP in order to support institutional instructional change.

It is important to explicitly state the theoretical framework used in a study, but elucidating a theoretical framework can be challenging for a new educational researcher. The literature review can help to identify an applicable theoretical framework. Focal areas of the review or central terms often connect to assumptions and assertions associated with the framework that pertain to the phenomenon of interest. Another way to identify a theoretical framework is self-reflection by the researcher on personal beliefs and understandings about the nature of knowledge the researcher brings to the study ( Lysaght, 2011 ). In stating one’s beliefs and understandings related to the study (e.g., students construct their knowledge, instructional materials support learning), an orientation becomes evident that will suggest a particular theoretical framework. Theoretical frameworks are not arbitrary , but purposefully selected.

With experience, a researcher may find expanded roles for theoretical frameworks. Researchers may revise an existing framework that has limited explanatory power, or they may decide there is a need to develop a new theoretical framework. These frameworks can emerge from a current study or the need to explain a phenomenon in a new way. Researchers may also find that multiple theoretical frameworks are necessary to frame and explore a problem, as different frameworks can provide different insights into a problem.

Finally, it is important to recognize that choosing “x” theoretical framework does not necessarily mean a researcher chooses “y” methodology and so on, nor is there a clear-cut, linear process in selecting a theoretical framework for one’s study. In part, the nonlinear process of identifying a theoretical framework is what makes understanding and using theoretical frameworks challenging. For the novice scholar, contemplating and understanding theoretical frameworks is essential. Fortunately, there are articles and books that can help:

• Creswell, J. W. (2018). Research design: Qualitative, quantitative, and mixed methods approaches (5th ed.). Los Angeles, CA: Sage. This book provides an overview of theoretical frameworks in general educational research.
• Ding, L. (2019). Theoretical perspectives of quantitative physics education research. Physical Review Physics Education Research , 15 (2), 020101-1–020101-13. This paper illustrates how a DBER field can use theoretical frameworks.
• Nehm, R. (2019). Biology education research: Building integrative frameworks for teaching and learning about living systems. Disciplinary and Interdisciplinary Science Education Research , 1 , ar15. https://doi.org/10.1186/s43031-019-0017-6 . This paper articulates the need for studies in BER to explicitly state theoretical frameworks and provides examples of potential studies.
• Patton, M. Q. (2015). Qualitative research & evaluation methods: Integrating theory and practice . Sage. This book also provides an overview of theoretical frameworks, but for both research and evaluation.

CONCEPTUAL FRAMEWORKS

Purpose of a conceptual framework.

A conceptual framework is a description of the way a researcher understands the factors and/or variables that are involved in the study and their relationships to one another. The purpose of a conceptual framework is to articulate the concepts under study using relevant literature ( Rocco and Plakhotnik, 2009 ) and to clarify the presumed relationships among those concepts ( Rocco and Plakhotnik, 2009 ; Anfara and Mertz, 2014 ). Conceptual frameworks are different from theoretical frameworks in both their breadth and grounding in established findings. Whereas a theoretical framework articulates the lens through which a researcher views the work, the conceptual framework is often more mechanistic and malleable.

Conceptual frameworks are broader, encompassing both established theories (i.e., theoretical frameworks) and the researchers’ own emergent ideas. Emergent ideas, for example, may be rooted in informal and/or unpublished observations from experience. These emergent ideas would not be considered a “theory” if they are not yet tested, supported by systematically collected evidence, and peer reviewed. However, they do still play an important role in the way researchers approach their studies. The conceptual framework allows authors to clearly describe their emergent ideas so that connections among ideas in the study and the significance of the study are apparent to readers.

Constructing Conceptual Frameworks

Including a conceptual framework in a research study is important, but researchers often opt to include either a conceptual or a theoretical framework. Either may be adequate, but both provide greater insight into the research approach. For instance, a research team plans to test a novel component of an existing theory. In their study, they describe the existing theoretical framework that informs their work and then present their own conceptual framework. Within this conceptual framework, specific topics portray emergent ideas that are related to the theory. Describing both frameworks allows readers to better understand the researchers’ assumptions, orientations, and understanding of concepts being investigated. For example, Connolly et al. (2018) included a conceptual framework that described how they applied a theoretical framework of social cognitive career theory (SCCT) to their study on teaching programs for doctoral students. In their conceptual framework, the authors described SCCT, explained how it applied to the investigation, and drew upon results from previous studies to justify the proposed connections between the theory and their emergent ideas.

In some cases, authors may be able to sufficiently describe their conceptualization of the phenomenon under study in an introduction alone, without a separate conceptual framework section. However, incomplete descriptions of how the researchers conceptualize the components of the study may limit the significance of the study by making the research less intelligible to readers. This is especially problematic when studying topics in which researchers use the same terms for different constructs or different terms for similar and overlapping constructs (e.g., inquiry, teacher beliefs, pedagogical content knowledge, or active learning). Authors must describe their conceptualization of a construct if the research is to be understandable and useful.

There are some key areas to consider regarding the inclusion of a conceptual framework in a study. To begin with, it is important to recognize that conceptual frameworks are constructed by the researchers conducting the study ( Rocco and Plakhotnik, 2009 ; Maxwell, 2012 ). This is different from theoretical frameworks that are often taken from established literature. Researchers should bring together ideas from the literature, but they may be influenced by their own experiences as a student and/or instructor, the shared experiences of others, or thought experiments as they construct a description, model, or representation of their understanding of the phenomenon under study. This is an exercise in intellectual organization and clarity that often considers what is learned, known, and experienced. The conceptual framework makes these constructs explicitly visible to readers, who may have different understandings of the phenomenon based on their prior knowledge and experience. There is no single method to go about this intellectual work.

Reeves et al. (2016) is an example of an article that proposed a conceptual framework about graduate teaching assistant professional development evaluation and research. The authors used existing literature to create a novel framework that filled a gap in current research and practice related to the training of graduate teaching assistants. This conceptual framework can guide the systematic collection of data by other researchers because the framework describes the relationships among various factors that influence teaching and learning. The Reeves et al. (2016) conceptual framework may be modified as additional data are collected and analyzed by other researchers. This is not uncommon, as conceptual frameworks can serve as catalysts for concerted research efforts that systematically explore a phenomenon (e.g., Reynolds et al. , 2012 ; Brownell and Kloser, 2015 ).

Sabel et al. (2017) used a conceptual framework in their exploration of how scaffolds, an external factor, interact with internal factors to support student learning. Their conceptual framework integrated principles from two theoretical frameworks, self-regulated learning and metacognition, to illustrate how the research team conceptualized students’ use of scaffolds in their learning ( Figure 1 ). Sabel et al. (2017) created this model using their interpretations of these two frameworks in the context of their teaching.

Conceptual framework from Sabel et al. (2017) .

A conceptual framework should describe the relationship among components of the investigation ( Anfara and Mertz, 2014 ). These relationships should guide the researcher’s methods of approaching the study ( Miles et al. , 2014 ) and inform both the data to be collected and how those data should be analyzed. Explicitly describing the connections among the ideas allows the researcher to justify the importance of the study and the rigor of the research design. Just as importantly, these frameworks help readers understand why certain components of a system were not explored in the study. This is a challenge in education research, which is rooted in complex environments with many variables that are difficult to control.

For example, Sabel et al. (2017) stated: “Scaffolds, such as enhanced answer keys and reflection questions, can help students and instructors bridge the external and internal factors and support learning” (p. 3). They connected the scaffolds in the study to the three dimensions of metacognition and the eventual transformation of existing ideas into new or revised ideas. Their framework provides a rationale for focusing on how students use two different scaffolds, and not on other factors that may influence a student’s success (self-efficacy, use of active learning, exam format, etc.).

In constructing conceptual frameworks, researchers should address needed areas of study and/or contradictions discovered in literature reviews. By attending to these areas, researchers can strengthen their arguments for the importance of a study. For instance, conceptual frameworks can address how the current study will fill gaps in the research, resolve contradictions in existing literature, or suggest a new area of study. While a literature review describes what is known and not known about the phenomenon, the conceptual framework leverages these gaps in describing the current study ( Maxwell, 2012 ). In the example of Sabel et al. (2017) , the authors indicated there was a gap in the literature regarding how scaffolds engage students in metacognition to promote learning in large classes. Their study helps fill that gap by describing how scaffolds can support students in the three dimensions of metacognition: intelligibility, plausibility, and wide applicability. In another example, Lane (2016) integrated research from science identity, the ethic of care, the sense of belonging, and an expertise model of student success to form a conceptual framework that addressed the critiques of other frameworks. In a more recent example, Sbeglia et al. (2021) illustrated how a conceptual framework influences the methodological choices and inferences in studies by educational researchers.

Sometimes researchers draw upon the conceptual frameworks of other researchers. When a researcher’s conceptual framework closely aligns with an existing framework, the discussion may be brief. For example, Ghee et al. (2016) referred to portions of SCCT as their conceptual framework to explain the significance of their work on students’ self-efficacy and career interests. Because the authors’ conceptualization of this phenomenon aligned with a previously described framework, they briefly mentioned the conceptual framework and provided additional citations that provided more detail for the readers.

Within both the BER and the broader DBER communities, conceptual frameworks have been used to describe different constructs. For example, some researchers have used the term “conceptual framework” to describe students’ conceptual understandings of a biological phenomenon. This is distinct from a researcher’s conceptual framework of the educational phenomenon under investigation, which may also need to be explicitly described in the article. Other studies have presented a research logic model or flowchart of the research design as a conceptual framework. These constructions can be quite valuable in helping readers understand the data-collection and analysis process. However, a model depicting the study design does not serve the same role as a conceptual framework. Researchers need to avoid conflating these constructs by differentiating the researchers’ conceptual framework that guides the study from the research design, when applicable.

Explicitly describing conceptual frameworks is essential in depicting the focus of the study. We have found that being explicit in a conceptual framework means using accepted terminology, referencing prior work, and clearly noting connections between terms. This description can also highlight gaps in the literature or suggest potential contributions to the field of study. A well-elucidated conceptual framework can suggest additional studies that may be warranted. This can also spur other researchers to consider how they would approach the examination of a phenomenon and could result in a revised conceptual framework.

It can be challenging to create conceptual frameworks, but they are important. Below are two resources that could be helpful in constructing and presenting conceptual frameworks in educational research:

• Maxwell, J. A. (2012). Qualitative research design: An interactive approach (3rd ed.). Los Angeles, CA: Sage. Chapter 3 in this book describes how to construct conceptual frameworks.
• Ravitch, S. M., & Riggan, M. (2016). Reason & rigor: How conceptual frameworks guide research . Los Angeles, CA: Sage. This book explains how conceptual frameworks guide the research questions, data collection, data analyses, and interpretation of results.

CONCLUDING THOUGHTS

Literature reviews, theoretical frameworks, and conceptual frameworks are all important in DBER and BER. Robust literature reviews reinforce the importance of a study. Theoretical frameworks connect the study to the base of knowledge in educational theory and specify the researcher’s assumptions. Conceptual frameworks allow researchers to explicitly describe their conceptualization of the relationships among the components of the phenomenon under study. Table 1 provides a general overview of these components in order to assist biology education researchers in thinking about these elements.

It is important to emphasize that these different elements are intertwined. When these elements are aligned and complement one another, the study is coherent, and the study findings contribute to knowledge in the field. When literature reviews, theoretical frameworks, and conceptual frameworks are disconnected from one another, the study suffers. The point of the study is lost, suggested findings are unsupported, or important conclusions are invisible to the researcher. In addition, this misalignment may be costly in terms of time and money.

Conducting a literature review, selecting a theoretical framework, and building a conceptual framework are some of the most difficult elements of a research study. It takes time to understand the relevant research, identify a theoretical framework that provides important insights into the study, and formulate a conceptual framework that organizes the finding. In the research process, there is often a constant back and forth among these elements as the study evolves. With an ongoing refinement of the review of literature, clarification of the theoretical framework, and articulation of a conceptual framework, a sound study can emerge that makes a contribution to the field. This is the goal of BER and education research.

Supplementary Material

• Allee, V. (2000). Knowledge networks and communities of learning . OD Practitioner , 32 ( 4 ), 4–13. [ Google Scholar ]
• Allen, M. (2017). The Sage encyclopedia of communication research methods (Vols. 1–4 ). Los Angeles, CA: Sage. 10.4135/9781483381411 [ CrossRef ] [ Google Scholar ]
• American Association for the Advancement of Science. (2011). Vision and change in undergraduate biology education: A call to action . Washington, DC. [ Google Scholar ]
• Anfara, V. A., Mertz, N. T. (2014). Setting the stage . In Anfara, V. A., Mertz, N. T. (eds.), Theoretical frameworks in qualitative research (pp. 1–22). Sage. [ Google Scholar ]
• Barnes, M. E., Brownell, S. E. (2016). Practices and perspectives of college instructors on addressing religious beliefs when teaching evolution . CBE—Life Sciences Education , 15 ( 2 ), ar18. https://doi.org/10.1187/cbe.15-11-0243 [ PMC free article ] [ PubMed ] [ Google Scholar ]
• Boote, D. N., Beile, P. (2005). Scholars before researchers: On the centrality of the dissertation literature review in research preparation . Educational Researcher , 34 ( 6 ), 3–15. 10.3102/0013189x034006003 [ CrossRef ] [ Google Scholar ]
• Booth, A., Sutton, A., Papaioannou, D. (2016a). Systemic approaches to a successful literature review (2nd ed.). Los Angeles, CA: Sage. [ Google Scholar ]
• Booth, W. C., Colomb, G. G., Williams, J. M., Bizup, J., Fitzgerald, W. T. (2016b). The craft of research (4th ed.). Chicago, IL: University of Chicago Press. [ Google Scholar ]
• Brownell, S. E., Kloser, M. J. (2015). Toward a conceptual framework for measuring the effectiveness of course-based undergraduate research experiences in undergraduate biology . Studies in Higher Education , 40 ( 3 ), 525–544. https://doi.org/10.1080/03075079.2015.1004234 [ Google Scholar ]
• Connolly, M. R., Lee, Y. G., Savoy, J. N. (2018). The effects of doctoral teaching development on early-career STEM scholars’ college teaching self-efficacy . CBE—Life Sciences Education , 17 ( 1 ), ar14. https://doi.org/10.1187/cbe.17-02-0039 [ PMC free article ] [ PubMed ] [ Google Scholar ]
• Cooper, K. M., Blattman, J. N., Hendrix, T., Brownell, S. E. (2019). The impact of broadly relevant novel discoveries on student project ownership in a traditional lab course turned CURE . CBE—Life Sciences Education , 18 ( 4 ), ar57. https://doi.org/10.1187/cbe.19-06-0113 [ PMC free article ] [ PubMed ] [ Google Scholar ]
• Creswell, J. W. (2018). Research design: Qualitative, quantitative, and mixed methods approaches (5th ed.). Los Angeles, CA: Sage. [ Google Scholar ]
• DeHaan, R. L. (2011). Education research in the biological sciences: A nine decade review (Paper commissioned by the NAS/NRC Committee on the Status, Contributions, and Future Directions of Discipline Based Education Research) . Washington, DC: National Academies Press. Retrieved May 20, 2022, from www7.nationalacademies.org/bose/DBER_Mee ting2_commissioned_papers_page.html [ Google Scholar ]
• Ding, L. (2019). Theoretical perspectives of quantitative physics education research . Physical Review Physics Education Research , 15 ( 2 ), 020101. [ Google Scholar ]
• Dirks, C. (2011). The current status and future direction of biology education research . Paper presented at: Second Committee Meeting on the Status, Contributions, and Future Directions of Discipline-Based Education Research, 18–19 October (Washington, DC). Retrieved May 20, 2022, from http://sites.nationalacademies.org/DBASSE/BOSE/DBASSE_071087 [ Google Scholar ]
• Duran, R. P., Eisenhart, M. A., Erickson, F. D., Grant, C. A., Green, J. L., Hedges, L. V., Schneider, B. L. (2006). Standards for reporting on empirical social science research in AERA publications: American Educational Research Association . Educational Researcher , 35 ( 6 ), 33–40. [ Google Scholar ]
• Ebert-May, D., Derting, T. L., Henkel, T. P., Middlemis Maher, J., Momsen, J. L., Arnold, B., Passmore, H. A. (2015). Breaking the cycle: Future faculty begin teaching with learner-centered strategies after professional development . CBE—Life Sciences Education , 14 ( 2 ), ar22. https://doi.org/10.1187/cbe.14-12-0222 [ PMC free article ] [ PubMed ] [ Google Scholar ]
• Galvan, J. L., Galvan, M. C. (2017). Writing literature reviews: A guide for students of the social and behavioral sciences (7th ed.). New York, NY: Routledge. https://doi.org/10.4324/9781315229386 [ Google Scholar ]
• Gehrke, S., Kezar, A. (2017). The roles of STEM faculty communities of practice in institutional and departmental reform in higher education . American Educational Research Journal , 54 ( 5 ), 803–833. https://doi.org/10.3102/0002831217706736 [ Google Scholar ]
• Ghee, M., Keels, M., Collins, D., Neal-Spence, C., Baker, E. (2016). Fine-tuning summer research programs to promote underrepresented students’ persistence in the STEM pathway . CBE—Life Sciences Education , 15 ( 3 ), ar28. https://doi.org/10.1187/cbe.16-01-0046 [ PMC free article ] [ PubMed ] [ Google Scholar ]
• Institute of Education Sciences & National Science Foundation. (2013). Common guidelines for education research and development . Retrieved May 20, 2022, from www.nsf.gov/pubs/2013/nsf13126/nsf13126.pdf
• Jensen, J. L., Lawson, A. (2011). Effects of collaborative group composition and inquiry instruction on reasoning gains and achievement in undergraduate biology . CBE—Life Sciences Education , 10 ( 1 ), 64–73. https://doi.org/10.1187/cbe.19-05-0098 [ PMC free article ] [ PubMed ] [ Google Scholar ]
• Kolpikova, E. P., Chen, D. C., Doherty, J. H. (2019). Does the format of preclass reading quizzes matter? An evaluation of traditional and gamified, adaptive preclass reading quizzes . CBE—Life Sciences Education , 18 ( 4 ), ar52. https://doi.org/10.1187/cbe.19-05-0098 [ PMC free article ] [ PubMed ] [ Google Scholar ]
• Labov, J. B., Reid, A. H., Yamamoto, K. R. (2010). Integrated biology and undergraduate science education: A new biology education for the twenty-first century? CBE—Life Sciences Education , 9 ( 1 ), 10–16. https://doi.org/10.1187/cbe.09-12-0092 [ PMC free article ] [ PubMed ] [ Google Scholar ]
• Lane, T. B. (2016). Beyond academic and social integration: Understanding the impact of a STEM enrichment program on the retention and degree attainment of underrepresented students . CBE—Life Sciences Education , 15 ( 3 ), ar39. https://doi.org/10.1187/cbe.16-01-0070 [ PMC free article ] [ PubMed ] [ Google Scholar ]
• Lave, J. (1988). Cognition in practice: Mind, mathematics and culture in everyday life . New York, NY: Cambridge University Press. [ Google Scholar ]
• Lo, S. M., Gardner, G. E., Reid, J., Napoleon-Fanis, V., Carroll, P., Smith, E., Sato, B. K. (2019). Prevailing questions and methodologies in biology education research: A longitudinal analysis of research in CBE — Life Sciences Education and at the Society for the Advancement of Biology Education Research . CBE—Life Sciences Education , 18 ( 1 ), ar9. https://doi.org/10.1187/cbe.18-08-0164 [ PMC free article ] [ PubMed ] [ Google Scholar ]
• Lysaght, Z. (2011). Epistemological and paradigmatic ecumenism in “Pasteur’s quadrant:” Tales from doctoral research . In Official Conference Proceedings of the Third Asian Conference on Education in Osaka, Japan . Retrieved May 20, 2022, from http://iafor.org/ace2011_offprint/ACE2011_offprint_0254.pdf
• Maxwell, J. A. (2012). Qualitative research design: An interactive approach (3rd ed.). Los Angeles, CA: Sage. [ Google Scholar ]
• Miles, M. B., Huberman, A. M., Saldaña, J. (2014). Qualitative data analysis (3rd ed.). Los Angeles, CA: Sage. [ Google Scholar ]
• Nehm, R. (2019). Biology education research: Building integrative frameworks for teaching and learning about living systems . Disciplinary and Interdisciplinary Science Education Research , 1 , ar15. https://doi.org/10.1186/s43031-019-0017-6 [ Google Scholar ]
• Patton, M. Q. (2015). Qualitative research & evaluation methods: Integrating theory and practice . Los Angeles, CA: Sage. [ Google Scholar ]
• Perry, J., Meir, E., Herron, J. C., Maruca, S., Stal, D. (2008). Evaluating two approaches to helping college students understand evolutionary trees through diagramming tasks . CBE—Life Sciences Education , 7 ( 2 ), 193–201. https://doi.org/10.1187/cbe.07-01-0007 [ PMC free article ] [ PubMed ] [ Google Scholar ]
• Posner, G. J., Strike, K. A., Hewson, P. W., Gertzog, W. A. (1982). Accommodation of a scientific conception: Toward a theory of conceptual change . Science Education , 66 ( 2 ), 211–227. [ Google Scholar ]
• Ravitch, S. M., Riggan, M. (2016). Reason & rigor: How conceptual frameworks guide research . Los Angeles, CA: Sage. [ Google Scholar ]
• Reeves, T. D., Marbach-Ad, G., Miller, K. R., Ridgway, J., Gardner, G. E., Schussler, E. E., Wischusen, E. W. (2016). A conceptual framework for graduate teaching assistant professional development evaluation and research . CBE—Life Sciences Education , 15 ( 2 ), es2. https://doi.org/10.1187/cbe.15-10-0225 [ PMC free article ] [ PubMed ] [ Google Scholar ]
• Reynolds, J. A., Thaiss, C., Katkin, W., Thompson, R. J. Jr. (2012). Writing-to-learn in undergraduate science education: A community-based, conceptually driven approach . CBE—Life Sciences Education , 11 ( 1 ), 17–25. https://doi.org/10.1187/cbe.11-08-0064 [ PMC free article ] [ PubMed ] [ Google Scholar ]
• Rocco, T. S., Plakhotnik, M. S. (2009). Literature reviews, conceptual frameworks, and theoretical frameworks: Terms, functions, and distinctions . Human Resource Development Review , 8 ( 1 ), 120–130. https://doi.org/10.1177/1534484309332617 [ Google Scholar ]
• Rodrigo-Peiris, T., Xiang, L., Cassone, V. M. (2018). A low-intensity, hybrid design between a “traditional” and a “course-based” research experience yields positive outcomes for science undergraduate freshmen and shows potential for large-scale application . CBE—Life Sciences Education , 17 ( 4 ), ar53. https://doi.org/10.1187/cbe.17-11-0248 [ PMC free article ] [ PubMed ] [ Google Scholar ]
• Sabel, J. L., Dauer, J. T., Forbes, C. T. (2017). Introductory biology students’ use of enhanced answer keys and reflection questions to engage in metacognition and enhance understanding . CBE—Life Sciences Education , 16 ( 3 ), ar40. https://doi.org/10.1187/cbe.16-10-0298 [ PMC free article ] [ PubMed ] [ Google Scholar ]
• Sbeglia, G. C., Goodridge, J. A., Gordon, L. H., Nehm, R. H. (2021). Are faculty changing? How reform frameworks, sampling intensities, and instrument measures impact inferences about student-centered teaching practices . CBE—Life Sciences Education , 20 ( 3 ), ar39. https://doi.org/10.1187/cbe.20-11-0259 [ PMC free article ] [ PubMed ] [ Google Scholar ]
• Schwandt, T. A. (2000). Three epistemological stances for qualitative inquiry: Interpretivism, hermeneutics, and social constructionism . In Denzin, N. K., Lincoln, Y. S. (Eds.), Handbook of qualitative research (2nd ed., pp. 189–213). Los Angeles, CA: Sage. [ Google Scholar ]
• Sickel, A. J., Friedrichsen, P. (2013). Examining the evolution education literature with a focus on teachers: Major findings, goals for teacher preparation, and directions for future research . Evolution: Education and Outreach , 6 ( 1 ), 23. https://doi.org/10.1186/1936-6434-6-23 [ Google Scholar ]
• Singer, S. R., Nielsen, N. R., Schweingruber, H. A. (2012). Discipline-based education research: Understanding and improving learning in undergraduate science and engineering . Washington, DC: National Academies Press. [ Google Scholar ]
• Todd, A., Romine, W. L., Correa-Menendez, J. (2019). Modeling the transition from a phenotypic to genotypic conceptualization of genetics in a university-level introductory biology context . Research in Science Education , 49 ( 2 ), 569–589. https://doi.org/10.1007/s11165-017-9626-2 [ Google Scholar ]
• Vygotsky, L. S. (1978). Mind in society: The development of higher psychological processes . Cambridge, MA: Harvard University Press. [ Google Scholar ]
• Wenger, E. (1998). Communities of practice: Learning as a social system . Systems Thinker , 9 ( 5 ), 2–3. [ Google Scholar ]
• Ziadie, M. A., Andrews, T. C. (2018). Moving evolution education forward: A systematic analysis of literature to identify gaps in collective knowledge for teaching . CBE—Life Sciences Education , 17 ( 1 ), ar11. https://doi.org/10.1187/cbe.17-08-0190 [ PMC free article ] [ PubMed ] [ Google Scholar ]

Summer Undergraduate Research Experience (SURE)

• Start Your Research Project

What is a Literature Review?

Scholarly article types.

• Find Background Information
• Find Articles
• Evaluate Sources
• Manage Citations and Sources
• SURE Research Scenarios

Scholarly Workflow

What is a literature review?

A literature review provides an overview of the scholarly literature (e.g. books, articles, dissertations, proceedings) relevant to an area of research or theory. The review typically will include a summary of the major questions in a area and critical evaluations of work that has already been done. Literature reviews are also helpful for their comprehensive bibliographies. This  webpage by the UC Santa Cruz Library  does a good job of explaining lit reviews.

Literature reviews typically include these components:

• An overview of the subject
• Organization of relevant publications into subtopics, theoretical areas, or key debates
• An analysis and discussion of how various works relate to one another the the relevant questions
• A discussion of unresolved questions or future directions
• Some will also include discussions of key data collection and analysis methodologies

The following resources are great places to start when compiling a comprehensive bibliography.

• Oxford Bibliographies
• Browse by Political Science, Economics or just run search. Note that you can limit by access that Emory has. In cases where we do not, check discoverE, and request needed chapters.
• Great resource for building initial literature reviews, identifying important figures in the literature, and engaging with major theories. Emory has a subscription to all the Handbooks.
• In-depth reviews of the literature in a discipline, published annually, with an emphasis on overviews and more recent approaches and theories.  Note that the HTML version is good for tracking citations, but does not have needed page numbers.

Scholarly journals may contain various types of scholarly literature. Knowing the different types of articles will help you find the best resources for your research needs.

• Research articles (aka empirical articles)  - primary sources of authors reporting their own studies.
• Theoretical articles  - concepts, frameworks, models, and perspectives.
• Literature reviews  - summary of literature on a given topic.
• Meta-analyses  - use statistical methods to summarize studies.
• Systematic reviews  - high-level summary of literature to answer focused--often clinical--question.
• Case studies and clinical trials  - describe real patient cases/treatments.
• Conference materials  - proceedings, notes, etc.
• Commentaries, book reviews, opinions, letters, etc.  - opinion and thought pieces.
• << Previous: Start Your Research Project
• Next: Find Background Information >>
• Last Updated: Jun 4, 2024 4:14 PM
• URL: https://guides.libraries.emory.edu/sure

• Getting Started
• Topic Overviews
• Other Databases
• Tests and Measures

What is an Empirical Study?

Literature review.

• Organize Your Sources
• NEW: LibKey Nomad

An empirical article reports the findings of a study conducted by the authors and uses data gathered from an experiment or observation. An empirical study is verifiable and "based on facts, systematic observation, or experiment, rather than theory or general philosophical principle" ( APA Databases Methodology Field Values ).  In other words, it tells the story of a research conducted, doing it in great detail. The study may utilize quantitative research methods to produce numerical data and seek to find a causal relationship between two or more variables. Conversely, it may use qualitative research methods, which involves collecting non-numerical data to analyze concepts, opinions, or experiences.

Key parts of an empirical article:

• Abstract  - Provides a brief overview of the research.
• Introduction  - The introduction provides a review of previous research on the topic and states the hypothesis. 
• Methods  - The methods area describes how the research was conducted, identifies the design of the study, the participants, and any measurements that were taken during the study.
• Results  - The results section describes the outcome of the study. 
• Discussion (or conclusion)  - The discussion section addresses the researchers' interpretations of their study and any future implications from their findings.
• References  - A list of works that were cited in the study.
• What is a Lit. Review?
• Purpose of a Lit. Review
• Limitations
• Non-Empirical Research
• Useful Links/Additional Info

A review of the published resources related to a specific issue, area of research, or specific theory. It provides a summary, description, and critical evaluation of each resource.

A literature review:

•  Synthesizes and places into context the research and scholarly literature relevant to the topic.
• Maps the different approaches to a given question and reveals patterns.
• Forms the foundation for subsequent research 
• Justifies the significance of the new investigation.
• Contains the most pertinent studies and points to important past and current research and practices.

A Lit. Review provides background and context; it shows how your research will contribute to the field. 

There are generally five parts to a literature review:

• Introduction
• Bibliography

A literature review should: 

• Provide a comprehensive and updated review of the literature
• Explain why this review has taken place
• Articulate a position or hypothesis
• Acknowledge and account for conflicting and corroborating points of view

 Add / Reorder  

A lit. review's purpose is to offer an overview of the significant works published on a topic. It can be written as an introduction to a study in order to:

• Demonstrate how a study fills a gap in research
• Compare a study with other research that's been done

It could be a separate work (a research article on its own) that:

• Organizes or describes a topic
• Describes variables within a particular issue/problem

Some limitations of a literature review include:

• It's a snapshot in time. Unlike other reviews, this one has beginning, a middle and an end. Future developments could make your work less relevant.
• It may be too focused. Some niche studies may miss the bigger picture.
• It can be difficult to be comprehensive. There is no way to ensure that all the literature on a topic was considered.
• It is easy to be biased if you stick to top tier journals. There may be other places where people are publishing exemplary research. Look to open access publications and conferences to reflect a more inclusive collection. Also, make sure to include opposing views (and not just supporting evidence).

Non-Empirical Research articles focus more on theories, methods and their implications for research. Non-Empirical Research can include comprehensive reviews and articles that focus on methodology. They rely on empirical research literature as well but does not need to be essentially data-driven.

Write a Literature Review (UCSC)

• Literature Review (Purdue)
• Overview: Lit Reviews (UNC)
• Review of Literature (UW-Madison)
• << Previous: Tests and Measures
• Next: Organize Your Sources >>
• Last Updated: Apr 9, 2024 10:58 AM
• URL: https://libguides.macalester.edu/psyc

• UTEP Library
• UTEP Library Research Guides

PSYC 3101: General Experimental Methods

Empirical articles vs. review articles.

• Use Library Resources from Anywhere Off-Campus
• Search Tips
• Article Anatomy
• Use RefWorks This link opens in a new window
• University Writing Center
• What is a Peer-Reviewed Article Anyway?

Empirical Articles

Review Articles

What is a Peer Reviewed Article?

Peer review is a process that many, but not all, journals use. Article manuscripts submitted to peer-reviewed journals are not automatically accepted and published.

In peer review, a panel of experts in the given field review the manuscript to determine aspects such as the quality of research, appropriateness for the journal, and relevance to the field. One of three decisions is made: accept, reject, or revise based on commentary from reviewers.

The process of peer review is thought to help ensure that high quality articles appear in journals.

Another term for peer-reviewed is  refereed . Peer-reviewed journals may also be called  scholarly.

Remember that magazines, Internet sources, and books are not the same as peer-reviewed journals.

In psychology, articles that report on original/new research studies may be referred to as primary sources or empirical .

• Common sections in a research/empirical article include introduction, literature review, methods/process, data, results, discussion, conclusion / suggestions for further study, and references.
• If the article is not divided into sections, it does not automatically mean it is not an empirical article.
• You cannot assume that an article is empirical just because it is divided into sections.
• In the methods section [which may be called something similar], or otherwise usually toward the beginning or middle of the article [if it does not have sections]: The authors will describe how they actively conducted new or original research -- such as an experiment or survey. Examples of what would likely be explained: How they identified participants, that they received Institutional Review Board (IRB) approval, control vs. experimental groups, and so forth about their research.

Articles that either interpret or analyze empirical articles are considered  review articles.  Such articles are often referred to as  secondary  sources  or  secondary research.

• An entire article that is purely a literature review [usually a review of select other articles considered to be the best support for a research question/topic]
• Systematic review
• Meta-analysis / meta-analyses
• Meta-synthesis / meta-syntheses
• If an article is a review article, it is likely [but not always] to have the words literature review, systematic review, integrative review, meta-analysis, meta-synthesis, or other mention of review in the title.
• The main indicator of a review article is if authors are just interpreting, analyzing, and/or comparing the results of empirical articles. So, in comparison to an empirical article, the authors of a review article do not describe an experiment or survey they conducted.
• If there is a methods section, it will usually describe how the authors searched for other articles [which databases they searched, what search terms they used] and decided the criteria for articles to include and exclude as part of their review. Again, they will not be describing how they conducted new or original research, such as an experiment or survey.
• In a literature review, for example, the authors' might point out what they believe to be the most pertinent/applicable research articles.
• << Previous: Use Library Resources from Anywhere Off-Campus
• Next: Databases >>
• Last Updated: Feb 20, 2024 9:46 AM
• URL: https://libguides.utep.edu/psyc3101

• Open access
• Published: 03 July 2024

Healthcare leaders navigating complexity: a scoping review of key trends in future roles and competencies

• Samantha Spanos   ORCID: orcid.org/0000-0003-3734-3907 1 ,
• Elle Leask   ORCID: orcid.org/0000-0003-1698-9151 1 ,
• Romika Patel   ORCID: orcid.org/0009-0000-6523-8798 1 ,
• Michael Datyner 1 ,
• Erwin Loh   ORCID: orcid.org/0000-0001-7157-0826 2 &
• Jeffrey Braithwaite   ORCID: orcid.org/0000-0003-0296-4957 1  

BMC Medical Education volume  24 , Article number:  720 ( 2024 ) Cite this article

31 Accesses

Metrics details

As healthcare systems rapidly become more complex, healthcare leaders are navigating expanding role scopes and increasingly varied tasks to ensure the provision of high-quality patient care. Despite a range of leadership theories, models, and training curricula to guide leadership development, the roles and competencies required by leaders in the context of emerging healthcare challenges (e.g., disruptive technologies, ageing populations, and burnt-out workforces) have not been sufficiently well conceptualized. This scoping review aimed to examine these roles and competencies through a deep dive into the contemporary academic and targeted gray literature on future trends in healthcare leadership roles and competencies.

Three electronic databases (Business Source Premier, Medline, and Embase) were searched from January 2018 to February 2023 for peer-reviewed literature on key future trends in leadership roles and competencies. Websites of reputable healthcare- and leadership-focused organizations were also searched. Data were analyzed using descriptive statistics and thematic analysis to explore both the range and depth of literature and the key concepts underlying leadership roles and competencies.

From an initial 348 articles identified in the literature and screened for relevance, 39 articles were included in data synthesis. Future leadership roles and competencies were related to four key themes: innovation and adaptation (e.g., flexibility and vision setting), collaboration and communication (e.g., relationship and trust building), self-development and self-awareness (e.g., experiential learning and self-examination), and consumer and community focus (e.g., public health messaging). In each of these areas, a broad range of strategies and approaches contributed to effective leadership under conditions of growing complexity, and a diverse array of contexts and situations for which these roles and competencies are applicable.

Conclusions

This research highlights the inherent interdependence of leadership requirements and health system complexity. Rather than as sets of roles and competencies, effective healthcare leadership might be better conceptualized as a set of broad goals to pursue that include fostering collaboration amongst stakeholders, building cultures of capacity, and continuously innovating for improved quality of care.

Peer Review reports

Healthcare leadership has grown in scope and importance in response to the increasing complexity of healthcare delivery [ 1 ]. Healthcare systems have become increasingly multifaceted, delivering a vast array of services across multiple levels, from preventative and primary care to acute, specialized care, and long-term care, to address the care needs of a changing population [ 1 ]. As populations age, chronic diseases rise, and the epidemiology and demographics of disease shift, new models of care rapidly emerge to address the ever-expanding spectrum of patient needs [ 2 ]. Advancements in technologies, tests and treatments and personalized medicine come with regulatory and ethical implications, and a growth in workforce specializations [ 3 , 4 ]. Healthcare leaders are navigating evermore complex webs of actors in the system – doctors, nurses, technicians, administrators, insurers, and patients – striving to balance priorities, foster collaboration, and provide strategic direction toward high-quality and safe patient care [ 5 ]. At the same time as running complex services, healthcare leaders need to continually assess, implement, and govern new technologies and services, adhere to the latest regulations and guidelines, operate within the confines of budgetary allocations, and meet growing consumer expectations for affordable and accessible care [ 6 , 7 ].

Competent healthcare leadership is widely considered to be critical for improving patient safety, system performance, and the effectiveness of healthcare teams [ 8 , 9 , 10 ]. Leadership has been identified as a key shaping influence on organizational culture [ 11 ], including workplace commitment to safety [ 12 ], and on preventing workforce burnout [ 13 , 14 ]. The increased need for multidisciplinary and integrated care models has shed growing light on the leadership roles of clinicians, including physicians, nurses, and allied health practitioners [ 15 , 16 , 17 ]. Individuals with both clinical and leadership expertise have been considered vital in complex healthcare landscapes because of their ability to balance administrative needs while prioritizing safety and high-quality care provision [ 18 , 19 , 20 , 21 , 22 ]. For example, physician leaders, through their deep understanding of clinical care and their credibility and influence, have been considered best able to devise strategies that improve patient care amidst stringent financial conditions [ 23 , 24 , 25 , 26 ]. Clinical leaders, particularly physician leaders, might also be of key importance for facilitating the success of collaborative care and care integration [ 27 ].

The formalization of healthcare leadership emerged as the importance of specialized healthcare leadership skills became increasingly needed, recognized and understood [ 1 , 28 , 29 ]. Leadership in healthcare has been conceptualized in several different ways, and a multitude of theories, frameworks, and models have been proposed to explain leadership roles and responsibilities [ 30 , 31 , 32 , 33 ]. For example, the CanMEDS Framework describes the Leader Role of physicians, which is comprised of key and enabling competencies, tasks, and abilities [ 34 , 35 ], and adaptations to this Framework emphasize the varying roles that leadership comprises and the competencies that fulfill them [ 36 ]. Although these frameworks present a good starting point for articulating leadership role scopes and their associated competencies, many fall short in explaining how leaders navigate complex, dynamic, multi-dimensional, and highly variable healthcare systems [ 37 ]. This is becoming increasingly recognized; CanMEDS is due to be updated in 2025 to incorporate competencies related to complexity [ 38 ]. Meanwhile, on the front lines, lack of role clarity and ambiguity about tasks and responsibilities presents a significant barrier for healthcare leaders [ 1 , 15 ]. In complex and unpredictable systems like healthcare, leaders spend substantial time ‘sense-making’, understanding, prioritizing and responding adaptively according to the needs of the situation [ 39 , 40 ]. The latest research on future healthcare trends tells us that increasing complexity associated with digital innovation, healthcare costs, regulatory compliance, sustainability concerns and equitable resource distribution will pose challenges to all actors in health systems [ 41 , 42 , 43 , 44 , 45 ]. In the face of these emerging challenges, it is vital to understand the range and type of roles and competencies that leaders will need to fulfil in the imminent future.

The aim of this scoping review is to examine the literature on the key trends in roles and competencies required for healthcare leaders in the future. We conceptualized ‘competencies’ as the attributes, skills, and abilities that comprise the fulfilment of varying leadership roles, as informed by the CanMEDS Framework [ 34 , 36 ]. Scoping review methodology was utilized to capture a broad range of literature types and identify key themes or groupings of future trends in leadership roles and competencies. Rather than focusing on answering specific questions (as per previous systematic reviews on leadership [ 46 , 47 ]) or developing theory (by utilizing a theoretical review approach to leadership literature [ 48 , 49 ]), we sought to map and identify patterns and trends within the leadership literature [ 50 ]. To investigate trends in leadership roles and competencies, we targeted emerging perspectives from key reputable thought leaders to supplement academic research [ 51 , 52 ].

The conduct and reporting of this review followed the Preferred Reporting Items for Systematic Reviews and Meta-Analyses Extension for Scoping Reviews (PRISMA-ScR) guidelines [ 53 ].

Search strategy

Comprehensive search strategies were developed, adapting search strategies utilized in a previous systematic review on physician leadership [ 26 ], and receiving input and expertise from two clinical librarians at Macquarie University (see supplementary file 1 for database search strategies). Medline, Embase, and Business Source Premier were searched from January 2018 to February 2023 to enable meaningful inferences to be made about future trends based on current perspectives. To capture key trends, patterns, shifts, and forecast changes to healthcare leadership, the Medline database search was limited to the ‘Trends’ subheading, “ used for the manner in which a subject changes, qualitatively or quantitatively, with time, whether past, present, or future. Includes “forecasting” & “futurology" ” (see supplementary file 1 ) [ 54 ]. For Embase and Business Source Premier, the ‘Trends’ subheading was not available, and instead key search terms were included to capture future trends, including “predict*”, “forecast*”, “shift*” and “transform*”. Efforts were made to locate texts that could not be retrieved, by searching Macquarie University’s digital library records and contacting authors to request the full text.

To complement the database searches, targeted searches of the Faculty of Medical Leadership and Management (FMLM; UK) website and The King’s Fund (UK) website were undertaken to identify emerging perspectives on the future roles and requirements of healthcare leaders. Targeted website searches can aid in uncovering unpublished yet relevant research identified by advocacy organizations or subject specialists, and research potentially missed by database searches [ 52 , 55 ]. Key search terms entered into the websites included ‘future healthcare’, ‘medical leader’, ‘clinical leader’, ‘medical manager’, ‘physician executive’, and ‘education and training’. We included articles that focused on leaders with a clinical background and leaders without a clinical background, to provide a comprehensive overview of leadership roles and requirements of reference to health systems [ 26 ].

Article selection

Database literature search.

References were uploaded into online data management software Rayyan [ 56 ], and duplicate records were identified and removed. Titles and abstracts of results were screened by three team members (SS, EL, RP) according to the inclusion and exclusion criteria (Table  1 ). Articles were included if they focused on future trends in the roles, competencies, attributes, or requirements of healthcare leaders, and if they reported on countries within the Organization for Economic Co-operation and Development (OECD). We limited our search to OECD countries to maximize the generalizability of findings within a developed context and enable meaningful trends to be identified. A subset of the articles was screened by all three team members to ensure that decisions were being made in a standardized manner. After this article subset was screened, the three team members discussed screening decisions, and disagreements were resolved by consensus or through discussion with JB [ 57 ]. During this process, two further exclusion criteria (#4 and #5, Table  1 ) were added to ensure that the screening process adhered to the aim of the current review. We excluded articles that focused on theories and definitions of leadership (e.g., for the purpose of developing educational or professional frameworks) without highlighting trends or changes in roles and competencies for future leadership. We also excluded articles that focused on healthcare interventions in which leaders may have been participants, but their roles or competencies were not the focus. Articles included at title and abstract screening were independently read in full and assessed for eligibility. Disagreements about inclusion were resolved through discussion, with JB available for arbitration if necessary. It was determined at this stage that if articles were conference abstracts in which the full presentation could not be accessed, the article of focus was sought and included in the analysis.

Targeted gray literature search

References were screened according to the inclusion and exclusion criteria (Table  1 ), except that articles only needed to report (rather than focus) on future leadership roles and requirements. This is because we wanted to ensure that our analysis broadly captured the most recent sources of information on healthcare leadership requirements, even if these sources did not focus exclusively on leadership.

Data charting process

Data from all records were appraised and charted simultaneously using a purpose-designed Excel data charting form designed by SS (and subsequently reviewed and endorsed by RP and EL). Multimedia records arising from targeted gray literature searches were listened to and transcribed by RP and checked by SS. Extracted data included article details (authors, year, country, text type), leadership focus (training or educational approaches, styles of leadership), and major and minor themes. Database literature were extracted first to identify and develop themes, and the targeted gray literature were extracted second to extend and embellish those themes.

Synthesis of results

Data from included articles were synthesized according to the Arksey and O’Malley framework for scoping reviews, selected for its detailed guidance on data collation, synthesis, and presentation [ 58 ]. The breadth, range, and type of data were analyzed using descriptive statistics, and underlying groups of leadership roles and competencies were analyzed using thematic analysis. First, the authorship team familiarized themselves with the articles to gain a broad overview of contexts in which leadership was discussed. An inductive approach was used to identify emerging themes of leadership roles and competencies in the database literature, where common concepts were identified, coded, and grouped together to form themes. Team discussion facilitated the final set of themes that were interpreted from the data. During this process, the extracted data were compared to the codes, groups, and resultant themes to examine the degree of consistency between the data and the interpreted findings. Where inconsistencies were identified, suggested changes (e.g., to code labels or groupings) were compared, and the most appropriate changes adopted. Targeted gray literature sources were deductively analyzed according to the identified themes.

Selection of sources of evidence

Figure  1 displays the process of identification and screening of included studies. Database searches yielded 160 records, from which 11 duplicates were removed. The remaining 149 database records were screened by title and abstract, after which a further 114 records were excluded. Of the remaining 35 that were assessed for eligibility, 22 were excluded, and 13 were included in the current review. Targeted gray literature searches yielded an additional 188 records, from which 146 were identified as duplicates and removed. The remaining 42 records were read in full and assessed for eligibility, from which a further 16 were excluded, and 26 were included in the current review. In total, 39 records were retained and synthesized.

PRISMA flowchart displaying the process of identification and selection of included articles

Characteristics of sources of evidence

The characteristics of the included records are displayed in Tables  2 and 3 . Of the database literature, most articles were published in the USA ( n  = 11), and the remaining two articles were published in Canada and Australia. Seven articles were empirical; three studies employed qualitative methods [ 59 , 60 , 61 ], three were quantitative [ 62 , 63 , 64 ], and one mixed methods [ 65 ]. Six articles were non-empirical; three were perspective pieces [ 66 , 67 , 68 ], and three were reports on training or organizational interventions [ 69 , 70 , 71 ]. Of the targeted literature, blog-type articles were most common ( n  = 11) [ 72 , 73 , 74 , 75 , 76 , 77 , 78 , 79 , 80 , 81 , 82 ], followed by news articles ( n  = 5) [ 83 , 84 , 85 , 86 , 87 ], reports ( n  = 4) [ 88 , 89 , 90 , 91 ], editorials ( n  = 2) [ 92 , 93 ], podcasts ( n  = 2) [ 94 , 95 ], and video and interview transcripts ( n  = 2) [ 96 , 97 ]. As targeted gray sources selected were The King’s Fund website and the FMLM website, the records from these websites were published in the UK.

Leadership roles and competencies

All 13 articles derived from the database searches focused on innovation and adaptation in future leadership. Two empirical articles reported on the ways in which clinical and non-clinical leaders innovated during the COVID-19 pandemic, rapidly designing new models of hospital care [ 61 ] and extending their roles to encompass the implementation of virtual leadership [ 64 ]. Qualitative investigations explored the importance of entrepreneurial leadership for implementing clinical genomics [ 59 ] and key leadership attributes for practice-level innovation and sustainability [ 60 ]. Four articles examined leadership training approaches that build physicians’ capacity to understand, adapt to, and manage change, overcome resistance, and think entrepreneurially [ 62 , 63 , 65 , 70 ]. Two reports described the necessity for healthcare leaders to be able to create a shared vision for an organization; one highlighted the importance of leaders being confident and “self-propelled to intervene” [ 69 ], and one emphasized physician leaders’ credibility as a catalyst for change management among healthcare providers [ 71 ]. The latter report also identified that visible and committed leadership that is sensitive to workplace cultures is critical for the success of change management activities [ 71 ]. Three perspective pieces discussed increasing opportunities for medical and other clinical leaders to create positive change in increasingly complex healthcare landscapes and fulfill the demands of the industry and public [ 66 , 67 , 68 ].

In the targeted gray literature, 19 of 26 records (73%) focused on innovative and adaptive leadership. Records primarily explored adaptive leadership behaviors during COVID-19, such as service redesign, introducing improved flexibility, learning mechanisms, and support platforms [ 73 , 76 , 77 , 97 ], and future innovation to manage climate change impacts [ 81 ], growing inequities [ 89 ], and emerging technologies [ 75 , 83 , 94 , 96 ]. Comfort with change, vision setting, and a desire to innovate were emphasized as key leadership attributes for future healthcare [ 82 , 83 , 88 , 96 ]. Records also explored how to best train and develop leaders for transforming health systems, including the National Health Service (NHS) [ 84 , 90 , 96 ]. New leadership training structures were proposed that foster innovation and adaptability in leaders [ 80 , 90 , 96 ] and encourage flexibility for cross-disciplinary learning.

Collaboration and communication  was a second theme that emerged across all 13 database articles. Three studies explored how collaborative leadership can foster innovation with regards to implementing genomics testing [ 59 ], creating new work models during COVID-19 [ 61 ] and developing new leadership styles via telecommunications [ 64 ]. Six articles focused on the importance of collaborating to build relationships across organisations [ 67 , 68 , 71 ] and within teams [ 65 , 69 , 70 ]. Three articles highlighted that effective communication contributes to organizational success, through fostering psychologically safe cultures [ 60 , 66 ] and generating the trust and rapport necessary for implementing technological innovations [ 71 ]. Two studies examined the impact of leadership training on physicians’ communication competencies [ 62 , 63 ].

In the targeted gray literature, 17 of 26 records (65%) focused on collaboration and communication. Records discussed specific initiatives to improve communication in clinical teams, such as staff surveys, daily huddles, and dedicated days for networking [ 75 , 77 , 80 , 95 ]. Cross-boundary collaboration and collective leadership (e.g., between clinicians and managers) [ 83 ] were advocated as a means to solve challenges [ 81 , 90 ], help build public trust [ 79 , 83 ], and improve quality of care [ 78 , 83 , 85 , 94 ]. Twelve records focused on the importance of team and leadership collaboration to create positive workplace cultures and improve staff wellbeing, through communication strategies such as openness and honesty [ 78 , 80 , 95 ], active listening and empathy [ 73 , 78 , 86 , 88 , 90 ], transparency [ 88 , 94 , 95 ], and inclusivity [ 85 , 94 ]. Three articles emphasized that encouraging staff autonomy, building trust, and demonstrating compassion facilitate better quality care than demanding and punitive leadership actions [ 73 , 74 , 88 ].

Nine of 13 database articles (69%) focused on a third theme, self-development and self-awareness in leadership. Four articles examined approaches to leadership development that incorporated self-development and self-awareness (e.g., personality testing) [ 63 , 65 , 69 , 70 ], with two articles describing these competencies as enablers for the development of other more advanced competencies (e.g., execution) [ 69 , 70 ]. Similar competencies explored included landscape awareness [ 60 ], self-organisation [ 60 ], emotional intelligence [ 64 ], and self-examination, the last of which was described as essential to gain skills beyond clinical roles [ 68 ], facilitate positive perceptions of others [ 66 ], and to remain relevant and effective in a changing healthcare environment [ 67 ]. One article also proposed strategies such as journaling, mindfulness, and feedback to encourage ongoing reflection on leadership decisions and biases [ 67 ].

In the targeted gray literature, seven of 26 records (27%) focused on self-development and self-awareness. Records examinedd the importance of continual personal leadership development, including mentoring and experiential learning, to facilitate understanding of one’s own skills [ 78 , 80 , 97 ]. Tools to facilitate self-reflection in physician leaders were advocated including the FMLM smartphone app [ 92 ] and leadership longitudinal assessments [ 91 ]. Self-care and resilience practices (e.g., meditation, social support) were also advocated for physician leaders as a means to manage “greater levels of stress and responsibility” [ 94 ].

Consumer engagement and advocacy  was a fourth theme and a focus of nine targeted gray literature records (35%). Records discussed patient and community engagement as essential for health system improvement, and examples included involving patients in health service design [ 74 , 77 ], creating channels of ongoing dialogue [ 79 , 83 ] and building stronger health system-community relationships [ 79 , 88 ]. Two records described the importance of public health messaging in improving health literacy [ 83 ] and countering misinformation [ 86 ], and two focused on the role of leaders in advocating for social justice and striving to improve equitable outcomes [ 75 , 93 ].

This scoping review identified 39 key resources that explored future trends in healthcare leadership roles and competencies. These records were derived from a combination of academic and targeted gray literature searches, juxtaposed and synthesized to build understanding of leadership to improve health systems into the future. Four themes of competencies emerged from the findings – innovation and adaptation, communication and collaboration, self-development and self-awareness, and consumer engagement and advocacy.

The competencies of healthcare leaders given the most attention in the literature over the last five years relate to innovation and adaptability . Both the academic and targeted gray literature focused on how leaders, clinical and non-clinical, demonstrated innovativeness and adapted to the demands of COVID-19, including rethinking and redesigning systems to support staff and patients [ 64 , 77 ]. The second focus of the literature on innovation and adaptability was geared toward the development of these capacities in leaders through education and training, as well as through opportunities for leaders to actualize their skills [ 70 , 90 ]. The literature indicated that as the complexity of healthcare is accelerating, leaders must both understand, and have opportunities to demonstrate, innovation amidst dynamic, variable, and demanding environments [ 59 , 60 , 71 ]. This aligns with prior research demonstrating that innovation uptake requires strong change management, and the ability to rapidly assess, understand, and apply innovative changes (e.g., medical technologies) [ 1 , 98 ]. While innovations might improve the system’s ability to deal with complex challenges in the long-term, their implementation can be challenged by a number of moving parts – including workforce changes, new rules and regulations, fluctuating resources and new patient groups – which leaders must consider and appropriately plan for [ 99 , 100 ]. Perhaps an even greater challenge for leaders to overcome when embracing innovation is the tendency for growing complexity to lock the organization into suboptimal conditions (i.e., inertia) [ 101 ]. Building awareness of the interacting components of complex systems and the flexibility required for adaptation and resilience should be a key focus of healthcare leadership education and training [ 102 ].

Competencies associated with communication and collaboration have also been a focus of the healthcare leadership literature. Academic literature dealt primarily with how collaborative structures and behaviors can help leaders innovate and build organizational cultures geared for success [ 59 , 61 , 71 ]. Targeted gray literature focused on how leaders can foster communication within teams, and the positive impacts of an open and accountable culture on staff wellbeing and productivity [ 73 , 74 ]. These findings echo research on resilient health systems emphasizing that ‘over-managing’ restricts the adaptive capacities needed by teams within dynamic healthcare environments [ 100 , 103 ]. The literature pointed to the need for leaders to strengthen communication and collaboration at varying levels – environmental, team, and organizational – to enable more efficient and better-quality healthcare delivery, and during this process they should endeavor to model the balance between autonomy and accountability [ 104 ]. Implementing regular touchpoints that engage multiple stakeholders, such as communities of practice, can help to create positive feedback loops that enable systems change [ 105 ], and overcome organizational barriers to collaboration and information sharing, such as weak relationships and inadequate communication [ 106 , 107 ].

Self-development and self-awareness  also emerged as an important aspect of leadership. Academic literature focused primarily on how these capacities are developed in leaders through structured education and training, including self-assessments and targeted educational modules [ 65 , 69 ]. Targeted gray literature discussed a range of activities outside of structured training (e.g., experiential learning) that can support leaders’ self-reflection and development, for physician leaders in particular to assess their performance and improve their leadership approaches [ 91 , 92 ]. These findings suggest that personal leadership development must go beyond formal curriculum requirements to incorporate everyday learning inputs [ 78 ], and align with other recent literature suggesting that self-regulation in leaders can be fostered through practicing self-discipline, boundary-setting, and managing disruptions, particularly in the digital age [ 108 , 109 ]. Practicing self-awareness can help leaders not only to sense-make in complex systems – to adapt to new situations and make appropriate trade-offs – but also to sense-give – to articulate and express the organization’s vision [ 40 ]. A minor theme, observed only in the targeted gray literature, was related to leaders’ roles and competencies in consumer engagement and advocacy . The importance of increasing consumer engagement in healthcare was emphasized, as well as the structures that are needed to facilitate these changes [ 79 ]. Working alongside consumers was highlighted as critical during times of changing care and need, such as during COVID-19 [ 77 , 86 ]. Although the involvement of consumers and the public in the co-production of care is increasing [ 110 ], there is limited academic literature focused on the roles of leaders in creating optimal environments for co-production. Consumer and community involvement in change efforts helps to improve care processes and outcomes [ 111 ], but leaders might face challenges understanding and operationalizing local engagement mechanisms [ 112 ]. Identifying the organizational and system levers that enable greater consumer involvement, and how leaders can advocate for these levers in their local context, is a fruitful area for future investigation.

The findings of the current review have implications for professional organizations that train healthcare leaders, such as the Australian College of Health Services Management (ACHSM) in Australia, and train clinicians to be leaders, including the UK’s FMLM. Creating a future-focused curriculum addressing the competencies related to the themes identified, in particular innovation and adaptability, is essential to prepare healthcare leaders for growing and changing scopes of responsibility. Such competencies are less amenable to formal theoretical teaching solely and require carefully crafted experiential learning programs in health settings, with supervision by experienced and effective healthcare leaders.

Strengths and limitations

A notable strength of this scoping review was the inclusion of a broad range of sources and perspectives on the future of healthcare leadership. We captured empirical studies, theoretical academic contributions (e.g., commentaries from healthcare leaders), and targeted grey literature, which is often a more useful source of information on emerging topics [ 52 ]. As a result, our findings identified key future trends in the roles and competencies of leaders, both clinical and non-clinical, across a wide range of contexts and situations. Another strength of this review was its specific focus on contemporary literature that examined future trends in leadership, to inform how leaders can prepare for upcoming challenges, rather than focusing on leadership that was effective in the past.

There are limitations to this review. Our search strategies may not have adequately captured other leadership trends applicable across contemporary healthcare settings or those faced by leaders and teams on the front lines of care [ 113 ]. Incorporating search terms related to specific settings, as well as complex systems concepts, may have enabled greater inferences to be made about how unique future challenges require new approaches to the development of healthcare leaders. To scope future-focused research and perspectives, database searches were narrowly restricted, and it is likely that key articles were missed. Targeted gray literature searches represent key thought leaders in healthcare and leadership, and while this enabled relevant information to be efficiently collected, undertaking highly focused searches may have introduced bias associated with geographical area (i.e., the UK) and particular stakeholder groups (e.g., policy-makers) [ 55 ]. Our choice to limit the current review to studies reporting in OECD countries further limited generalizability to other settings including in low-income and middle-income countries (LMICs) [ 1 ].

The roles and competencies of leaders are deeply enmeshed in, and reflective of, a complex and continuously transforming healthcare system. This research highlights the types of roles and competencies important for leaders facing a myriad of challenges, and the range of contexts and situations in which these types of roles and competencies can be applied. The ways in which roles and competencies manifest is highly contextual, dependent on both role responsibilities and the situational demands of healthcare environments.

Data availability

Data supporting these research findings are available upon reasonable request. Further inquiries can be directed to the corresponding author.

Abbreviations

Faculty of Medical Leadership and Management

National Health Service

Preferred Reporting Items for Systematic Reviews and Meta-Analyses Extension for Scoping Reviews

Royal Australasian College of Medical Administrators

United States of America

United Kingdom

Figueroa CA, Harrison R, Chauhan A, et al. Priorities and challenges for health leadership and workforce management globally: a rapid review. BMC Health Serv Res. 2019;19:1–11. https://doi.org/10.1186/s12913-019-4080-7

Article   Google Scholar  

Braithwaite J, Mannion R, Matsuyama Y, et al. The future of health systems to 2030: a roadmap for global progress and sustainability. Int J Qual Health Care. 2018;30(10):823–31. https://doi.org/10.1093/intqhc/mzy242

Anklam E, Bahl MI, Ball R, et al. Emerging technologies and their impact on regulatory science. Exp Biol Med. 2022;247(1):1–75. https://doi.org/10.1177/15353702211052280

Maeda A, Socha-Dietrich K. Skills for the future health workforce: preparing health professionals for people-centred care. OECD Health Working Papers. Paris: OECD Publishing,; 2021.

Reich MR, Javadi D, Ghaffar A. Introduction to the special issue on effective leadership for health systems. Health Syst Reform. 2016;2(3):171–5. https://doi.org/10.1080/23288604.2016.1223978

Shaw J, Brewer LC, Veinot T. Recommendations for health equity and virtual care arising from the COVID-19 pandemic: narrative review. JMIR Form Res. 2021;5(4):e23233. https://doi.org/10.2196/23233

Graham R, Woodhead T. Leadership for continuous improvement in healthcare during the time of COVID-19. Clin Radiol. 2021;76(1):67–72. https://doi.org/10.1016/j.crad.2020.08.008

Parr JM, Teo S, Koziol-McLain J. A quest for quality care: exploration of a model of leadership relationships, work engagement, and patient outcomes. J Adv Nurs. 2021;77(1):207–20. https://doi.org/10.1111/jan.14583

Ree E, Wiig S. Linking transformational leadership, patient safety culture and work engagement in home care services. Nurs Open. 2020;7(1):256–64. https://doi.org/10.1002/nop2.386

Wu X, Hayter M, Lee AJ, et al. Positive spiritual climate supports transformational leadership as means to reduce nursing burnout and intent to leave. J Nurs Manag. 2020;28(4):804–13. https://doi.org/10.1111/jonm.12994

West M, Armit K, Loewenthal L, et al. Leadership and leadership development in health care: the evidence base. London: The King’s Fund; 2015.

Google Scholar  

Churruca K, Ellis LA, Pomare C, et al. Dimensions of safety culture: a systematic review of quantitative, qualitative and mixed methods for assessing safety culture in hospitals. BMJ Open. 2021;11(7):e043982. https://doi.org/10.1136/bmjopen-2020-043982

Smallwood N, Bismark M, Willis K. Burn-out in the health workforce during the COVID-19 pandemic: opportunities for workplace and leadership approaches to improve well-being. BMJ Lead. 2023;leader–2022. https://doi.org/10.1136/leader-2022-000687

Kelly RJ, Hearld LR. Burnout and leadership style in behavioral health care: a literature review. J Behav Health Serv Res. 2020;47(4):581–600. https://doi.org/10.1007/s11414-019-09679-z

Hutchinson K, Ryder T, Coleman H, et al. Determining the role and responsibilities of the community epilepsy nurse in the management of epilepsy. J Clin Nurs. 2023;32(13–14):3730–45. https://doi.org/10.1111/jocn.16582

Mianda S, Voce A. Developing and evaluating clinical leadership interventions for frontline healthcare providers: a review of the literature. BMC Health Serv Res. 2018;18(1):1–15. https://doi.org/10.1186/s12913-018-3561-4

Veronesi G, Kirkpatrick I, Vallascas F. Clinicians on the board: what difference does it make? Soc Sci Med. 2013;77:147–55. https://doi.org/10.1016/j.socscimed.2012.11.019

Ferreira TDM, de Mesquita GR, de Melo GC, et al. The influence of nursing leadership styles on the outcomes of patients, professionals and institutions: an integrative review. J Nurs Manag. 2022;30(4):936–53. https://doi.org/10.1111/jonm.13592

Boamah SA. Emergence of informal clinical leadership as a catalyst for improving patient care quality and job satisfaction. J Adv Nurs. 2019;75(5):1000–9. https://doi.org/10.1111/jan.13895

Dickinson H, Ham C. Engaging doctors in leadership: review of the literature. Birmingham: University of Birmingham; 2008.

Hershkovich O, Gilad D, Zimlichman E, et al. Effective medical leadership in times of emergency: a perspective. Disaster Mil Med. 2016;2:1–5. https://doi.org/10.1186/s40696-016-0013-8

Stoller JK. Developing physician-leaders: a call to action. J Gen Intern Med. 2009;24:876–8. https://doi.org/10.1007/s11606-009-1007-8

Warren OJ, Carnall R. Medical leadership: why it’s important, what is required, and how we develop it. Postgrad Med J. 2011;87(1023):27–32. https://doi.org/10.1136/pgmj.2009.093807

Griffith R. NHS management inquiry: report to the secretary of state for social services. London: Department of Health and Social Security; 1983.

Braithwaite J. Changing how we think about healthcare improvement. BMJ. 2018;361. https://doi.org/10.1136/bmj.k2014

Clay-Williams R, Ludlow K, Testa L, et al. Medical leadership, a systematic narrative review: do hospitals and healthcare organisations perform better when led by doctors? BMJ Open. 2017;7(9):e014474. https://doi.org/10.1136/bmjopen-2016-014474

Nieuwboer MS, van der Sande R, van der Marck MA, et al. Clinical leadership and integrated primary care: a systematic literature review. Eur J Gen Pract. 2019;25(1):7–18. https://doi.org/10.1080/13814788.2018.1515907

Dwyer AJ. Roles, attributes and career paths of medical administrators in public hospitals: survey of victorian metropolitan directors of Medical services. Aust Health Rev. 2010;34(4):506–13. https://doi.org/10.1071/AH09750

Dwyer AJ. Medical managers in contemporary healthcare organisations: a consideration of the literature. Aust Health Rev. 2010;34(4):514–22. https://doi.org/10.1071/AH09736

Health Workforce Australia. Health LEADS Australia: the Australian health leadership framework. Adelaide: Health Workforce Australia; 2013.

NHS Institute for Innovation and Improvement and Academy of Medical Royal Colleges. Medical leadership competency framework: enhancing engagement in medical leadership. Coventry: NHS Institute for Innovation and Improvement; 2010.

National Centre for Healthcare Leadership. NCHL Health Leadership Competency Model 3.0. National Centre for Healthcare Leadership. 2022. https://www.nchl.org/research/ . Accessed 28 December, 2023.

NHS Leadership Academy. Healthcare Leadership Model: the nine dimensions of leadership behaviour. Leeds: NHS Leadership Academy; 2013.

Frank JR, Snell L, Sherbino J, et al. CanMEDS 2015 Physician Competency Framework. Royal College of Physicians and Surgeons of Canada; 2015.

Dath DCM-K, Anderson G, Burke A, Razack S, Lieff S, Moineau G, Chiu A, Ellison P. Leader. In: Frank JRSL, Sherbino J, editors. CanMEDS 2015 Physician Competency Framework. Royal College of Physicians and Surgeons of Canada; 2015.

Royal Australasian College of Medical Administrators. Medical Leadership and Management Curriculum Framework. Malvern, Victoria: Royal Australasian College of Medical Administrators; 2011.

Best S, Stark Z, Brown H, et al. The leadership behaviors needed to implement clinical genomics at scale: a qualitative study. Genet Med. 2020;22(8):1384–90. https://doi.org/10.1038/s41436-020-0818-1

Thoma B, Karwowska A, Samson L, et al. Emerging concepts in the CanMEDS physician competency framework. Can Med Educ J. 2023;14(1):4–12. https://doi.org/10.36834/cmej.75591

Ahern S, Loh E. Leadership during the COVID-19 pandemic: building and sustaining trust in times of uncertainty. BMJ Lead. 2020;leader–2020. https://doi.org/10.1136/leader-2020-000271

Braithwaite J, Churruca K, Hibbert P, et al. At the heart of getting things done in complex healthcare ecosystems: leadership, strategy, sensemaking and sensegiving. In: Chambers N, editor. Research Handbook on Leadership in Healthcare. Edward Elgar Publishing; 2023. pp. 775–92.

Braithwaite J, Fisher G. Beyond the aspirational: creating the future of health care in Australia. Int Med J. 2024;54(2):342–7. https://doi.org/10.1111/imj.16286

Cernega A, Nicolescu DN, Meleșcanu Imre M, et al. Volatility, uncertainty, complexity, and ambiguity (VUCA) in healthcare. Healthc. 2024;12(7):773. https://doi.org/10.3390/healthcare12070773

Kasula BY. Revolutionizing healthcare delivery: innovations and challenges in supply chain management for improved patient care. Trans Latest Trends Health Sect. 2023;15(15).

Niaz M, Nwagwu U. Managing healthcare product demand effectively in the post-covid-19 environment: navigating demand variability and forecasting complexities. Am J Econ Manag Bus. 2023;2(8):316–30. https://doi.org/10.58631/ajemb.v2i8.55

Thirumalai S, Devaraj S. Mitigating the curse of complexity: the role of focus and the implications for costs of care. J Oper Manag. 2024;70(1):157–79. https://doi.org/10.1002/joom.1278

Savage M, Savage C, Brommels M, et al. Medical leadership: boon or barrier to organisational performance? A thematic synthesis of the literature. BMJ Open. 2020;10(7):e035542. https://doi.org/10.1136/bmjopen-2019-035542

Ahti M, Taipale-Walsh L, Kuha S, et al. Health-care leaders’ experiences of the competencies required for crisis management during COVID-19: a systematic review of qualitative studies. Leadersh Health Serv. 2023. https://doi.org/10.1108/LHS-10-2022-0104

Krasikova DV, Green SG, LeBreton JM. Destructive leadership: a theoretical review, integration, and future research agenda. J Manag. 2013;39(5):1308–38. https://doi.org/10.1177/0149206312471388

Pellegrini MM, Ciampi F, Marzi G, et al. The relationship between knowledge management and leadership: mapping the field and providing future research avenues. J Knowl Manag. 2020;24(6):1445–92. https://doi.org/10.1108/JKM-01-2020-0034

Paré G, Trudel M-C, Jaana M, et al. Synthesizing information systems knowledge: a typology of literature reviews. Inf Manag. 2015;52(2):183–99. https://doi.org/10.1016/j.im.2014.08.008

Heath A, Levay P, Tuvey D. Literature searching methods or guidance and their application to public health topics: a narrative review. Health Info Libr J. 2022;39(1):6–21. https://doi.org/10.1111/hir.12414

Adams J, Hillier-Brown FC, Moore HJ, et al. Searching and synthesising ‘grey literature’and ‘grey information’in public health: critical reflections on three case studies. Syst Rev. 2016;5(1):1–11. https://doi.org/10.1186/s13643-016-0337-y

Tricco AC, Lillie E, Zarin W, et al. PRISMA extension for scoping reviews (PRISMA-ScR): checklist and explanation. Ann Intern Med. 2018;169(7):467–73. https://doi.org/10.7326/M18-0850

Wolters Kluwer Health I. Get to know the advantages of Ovid MEDLINE ® . 2024. https://tools.ovid.com/ovidtools/medline.html . Accessed.

Stansfield C, Dickson K, Bangpan M. Exploring issues in the conduct of website searching and other online sources for systematic reviews: how can we be systematic? Syst Rev. 2016;5:1–9. https://doi.org/10.1186/s13643-016-0371-9

Ouzzani M, Hammady H, Fedorowicz Z, et al. Rayyan—a web and mobile app for systematic reviews. Syst Rev. 2016;5:1–10. https://doi.org/10.1186/s13643-016-0384-4

Peters MD, Marnie C, Tricco AC, et al. Updated methodological guidance for the conduct of scoping reviews. JBI Evid Synth. 2020;18(10):2119–26. https://doi.org/10.11124/JBIES-20-00167

Arksey H, O’Malley L. Scoping studies: towards a methodological framework. Int J Soc Res Methodol. 2005;8(1):19–32. https://doi.org/10.1080/1364557032000119616

Best S, Long J, Stark Z, et al. Clinical leadership and genomics - what works? Twin Res Hum Genet. 2019;22(5):350. 43rd Annual Scientific Meeting of the Human Genetics Society of Australasia, HGSA 2019. Wellington New Zealand.

Crabtree BF, Howard J, Miller WL, et al. Leading innovative practice: leadership attributes in LEAP practices. Milbank Q. 2020;98(2):399–445. https://doi.org/10.1111/1468-0009.12456

Woiceshyn J, Huq JL, Kannappan S, et al. We need to work differently in a crisis: peer-professional leadership to redesign physicians’ work. BMJ Lead. 2022;6(2):98–103. https://doi.org/10.1136/leader-2021-000490

Appelbaum NP, Liang C, Whitney SE, et al. What business competencies are needed for the modern physician leader and when. Physician Leadersh J. 2022;9(2):39–45.

Hopkins J, Fassiotto M, Ku MC, et al. Designing a physician leadership development program based on effective models of physician education. Health Care Manage Rev. 2018;43(4):293–302. https://doi.org/10.1097/HMR.0000000000000146

Sandberg DS, Pennington CM, Lindquist MA. Virtual leadership: CEOs and C-Level executives of healthcare organizations in the United States reimagined new roles as virtual leaders. J Leadersh Stud. 2022;16(3):61–9. https://doi.org/10.1002/jls.21827

Satiani B, Dawson K, Mehta LS, et al. Lessons learned after implementing an academic faculty leadership program over seven years. Physician Leadersh J. 2022;9(2):30–8.

Angood PB. Transitions and transformations — our choice? Physician Leadersh J. 2022;9(5):6–8.

Brace R, James D. Adapting to changing roles. J Healthc Manag. 2020;35(3):46–7.

Wright C, Pister K. New thinking leads to better transitions. Physician Leadersh J. 2018;5(1):38–40.

Ramirez B, West DJ Jr, Ramirez CL. Healthcare and leadership competencies, training methods, and venues. J Leadersh Acc Ethics. 2021;18(4):93–9.

Wain MJ, Handel DA, Williford K, et al. Transitioning from physician to hospital leader: a competency-based model. Physician Leadersh J. 2020;7(2):32–3.

Borgwardt HL, Botz CT, Doppler JM, et al. Electronic health record implementation: the people side of change. Manage Healthc. 2019;4(1):7–20.

Anandaciva S. The biggest hospital in England. The King’s Fund . 2020. https://www.kingsfund.org.uk/blog/2020/09/biggest-hospital-england . Accessed 25 February, 2023.

Bailey S, West M. Learning from staff experiences of Covid-19: let the light come streaming in. The King’s Fund . 2020. https://www.kingsfund.org.uk/blog/2020/06/learning-staff-experiences-covid-19 . Accessed 26 February, 2023.

Bailey S, West M. Naming the issue: chronic excessive workload in the NHS. The King’s Fund . 2021. https://www.kingsfund.org.uk/blog/2021/06/naming-issue-chronic-excessive-workload-nhs . Accessed 26 February, 2023.

Bhopal A, Hood G. From disaster preparedness to digital disruption: A whistle-stop tour through public health. Faculty of Medical Leadership and Management. 2018. https://www.fmlm.ac.uk/news-opinion/from-disaster-preparedness-to-digital-disruption-a-whistle-stop-tour-through-public . Accessed 27 February, 2023.

Cheetham J. How Leaders in Healthcare 2020 gave an enlightened spotlight to junior doctor leadership. Faculty of Medical Leadership and Management. 2021. https://www.fmlm.ac.uk/news-opinion/how-leaders-in-healthcare-2020-gave-an-enlightened-spotlight-to-junior-doctor . Accessed 27 February, 2023.

Collins B. Leadership and innovation during Covid-19: lessons from the Cardiff and Vale Health System. The King’s Fund . 2021. https://www.kingsfund.org.uk/blog/2021/05/leadership-innovation-covid-19-cardiff-vale . Accessed 27 February, 2023.

Cottey L. The trainee and the team. Faculty of Medical Leadership and Management. 2018. https://www.fmlm.ac.uk/news-opinion/the-trainee-and-the-team . Accessed 28 February, 2023.

Dixon M. Engaging the public in designing the future for the NHS. The King’s Fund . 2018. https://www.kingsfund.org.uk/blog/2018/01/engaging-public-designing-future-nhs . Accessed 28 February, 2023.

Leveson D. Developing clinical leaders through innovation. The King’s Fund . 2021. https://www.kingsfund.org.uk/blog/2021/03/developing-clinical-leaders-through-innovation . Accessed 25 February, 2023.

Naylor C. Getting to grips with climate change: how can the NHS get to net zero? The King’s Fund . 2020. https://www.kingsfund.org.uk/blog/2020/01/climate-change-nhs-net-zero . Accessed 19 February, 2023.

Tweedie J. Leadership - so what is it anyway? February, 2018. https://www.fmlm.ac.uk/news-opinion/blog/leadership-so-what-is-it-anyway

Faculty of Medical Leadership and Management. FMLM response to HEE’s Call for Evidence on the Future Doctor. Faculty of Medical Leadership and Management. 2019. https://www.fmlm.ac.uk/news-opinion/fmlm-response-to-hees-call-for-evidence-on-the-future-doctor . Accessed 23 February, 2023.

Faculty of Medical Leadership and Management. Supporting and developing a leading resource for the future. Faculty of Medical Leadership and Management. 2021. https://www.fmlm.ac.uk/news-opinion/supporting-and-developing-a-leading-resource-for-the-future . Accessed 27 February, 2023.

Faculty of Medical Leadership and Management. Support, wellbeing and leadership. Faculty of Medical Leadership and Management. 2021. https://www.fmlm.ac.uk/news-opinion/support-wellbeing-and-leadership . Accessed 27 February, 2023.

Faculty of Medical Leadership and Management. Reflections from the International Healthcare Leadership Conference. 2022. Faculty of Medical Leadership and Management. 2022. https://www.fmlm.ac.uk/news-opinion/reflections-from-the-international-healthcare-leadership-conference-2022 . Accessed 27 February, 2023.

Lakhani M, Osuji N, Lees P et al. FMLM statement: racism and good leadership are incompatible. Faculty of Medical Leadership and Management. 2020. https://www.fmlm.ac.uk/news-opinion/fmlm-statement-racism-and-good-leadership-are-incompatible . Accessed 28 February, 2023.

Dougall D, Ross S. Transformational change in health and care: reports from the field. 2018. Accessed February, 2023. The King’s Fund.

Fogden R, Buck D, Franklin B et al. Poverty and the health and care system: the role of data and partnership in bringing change. The King’s Fund . 2022. https://www.kingsfund.org.uk/publications/poverty-health-care-system-data-partnership . Accessed 27 February, 2023.

Ham C, Murray R. The NHS 10-year plan: how should the extra funding be spent. The King’s Fund . 2018. https://www.kingsfund.org.uk/publications/nhs-10-year-plan . Accessed 28 February, 2023.

Swanwick T, McKimm J. The assessment of leadership development in the medical undergraduate curriculum: a consensus statement. Faculty of Medical Leadership and Management. 2020. https://www.fmlm.ac.uk/resources/the-assessment-of-leadership-development-in-the-medical-undergraduate-curriculum-a . Accessed 22 February, 2023.

Evans P. A radical upward shift to benefit members and fellows. Faculty of Medical Leadership and Management. 2020. https://www.fmlm.ac.uk/news-opinion/a-radical-upward-shift-to-benefit-members-and-fellows . Accessed 22 February, 2023.

Lakhani M. Black History Month and why it matters. Faculty of Medical Leadership and Management. 2021. https://www.fmlm.ac.uk/news-opinion/black-history-month-and-why-it-matters . Accessed 28 February, 2023.

Moralee S, Exworthy M, Rajasingam D et al. What next for medical leaders? Faculty of Medical Leadership and Management. 2018. https://www.fmlm.ac.uk/resources/what-next-for-medical-leaders . Accessed 25 February, 2023.

Warren S, Laverty A. Leading with compassion: supporting the health and wellbeing of NHS staff. The King’s Fund . 2022. https://www.kingsfund.org.uk/audio-video/podcast/leading-compassion-health-wellbeing-nhs-staff . Accessed 22 February, 2023.

Hancock M. Matt Hancock at The King’s Fund annual conference 2018. 2018. https://www.kingsfund.org.uk/audio-video/matt-hancock-kings-fund-annual-conference-2018 . Accessed.

Olsson-Brown A, Interview with Dr Anna Olsson-Brown - Chair of the Academy of Medical Royal Colleges Trainee Group. Faculty of Medical Leadership and Management. 2020. https://www.fmlm.ac.uk/news-opinion/interview-with-dr-anna-olsson-brown-chair-of-the-academy-of-medical-royal-colleges . Accessed 19 February, 2023.

Denicolai S, Previtali P. Innovation strategy and digital transformation execution in healthcare: the role of the general manager. Technovation. 2023;121:102555. https://doi.org/10.1016/j.technovation.2022.102555

Borges do Nascimento IJ, Abdulazeem H, Vasanthan LT, et al. Barriers and facilitators to utilizing digital health technologies by healthcare professionals. NPJ Digit Med. 2023;6(1):161. https://doi.org/10.1038/s41746-023-00899-4

Braithwaite J, Churruca K, Ellis LA, et al. Complexity science in healthcare: aspirations, approaches, applications, and accomplishments. A white paper. Sydney: Australian Institute of Health Innovation, Macquarie University; 2017.

Coiera E. Why system inertia makes health reform so difficult. BMJ. 2011;342 https://doi.org/10.1136/bmj.d3693

Clay-Williams R, Travaglia J, Hibbert P, et al. Clinical Governance Framework. A Report prepared for the Royal Australasian College of Medical Administrators (RACMA). Melbourne: RACMA; 2017.

Plsek PE, Wilson T. Complexity, leadership, and management in healthcare organisations. BMJ. 2001;323(7315):746–9. https://doi.org/10.1136/bmj.323.7315.746

van Diggele C, Burgess A, Roberts C, et al. Leadership in healthcare education. BMC Med Educ. 2020;20:1–6. https://doi.org/10.1186/s12909-020-02288-x

Golden RE, Klap R, Carney DV, et al. Promoting learning health system feedback loops: experience with a VA practice-based research network card study. Healthc. 2021;8:100484. https://doi.org/10.1016/j.hjdsi.2020.100484

Smith CL, Fisher G, Dharmayani PNA, et al. Progress with the Learning Health System 2.0: a rapid review of Learning Health Systems’ responses to pandemics and climate change. BMC Med. 2024;22(1):131. https://doi.org/10.1186/s12916-024-03345-8

Hunt RC, Struminger BB, Redd JT, et al. Virtual peer-to-peer learning to enhance and accelerate the health system response to COVID-19: the HHS ASPR Project ECHO COVID-19 Clinical rounds Initiative. Ann Emerg Med. 2021;78(2):223–8. https://doi.org/10.1016/j.annemergmed.2021.03.035

Mander R, Hellert U, Antoni CH. Self-leadership strategies for coping with flexibility requirements of digital work with a high degree of latitude for time, place and scope for action—a qualitative study. Gr Interakt Org. 2021;52:163–71. https://doi.org/10.1007/s11612-021-00560-2

Vay C, Steinherr V. Leadership in a digitalized and crisis-ridden world: towards a comprehensive overview of relevant competencies for leaders. University of Hawaiʻi at Mānoa; 2023.

Churruca K, Pomare C, Ellis LA, et al. Patient-reported outcome measures (PROMs): a review of generic and condition‐specific measures and a discussion of trends and issues. Health Expect. 2021;24(4):1015–24. https://doi.org/10.1111/hex.13254

Best A, Greenhalgh T, Lewis S, et al. Large-system transformation in health care: a realist review. Milbank Q. 2012;90(3):421–56. https://doi.org/10.1111/j.1468-0009.2012.00670.x

Bolden R, Kars-Unluoglu S, Jarvis C, et al. Paradoxes of multi-level leadership: insights from an integrated care system. J Change Manag. 2023;23(4):337–57. https://doi.org/10.1080/14697017.2023.2234388

Braithwaite J, Dammery G, Spanos S, et al. Learning health systems 2.0: future-proofing healthcare against pandemics and climate change. A White Paper. Sydney: Australian Institute of Health Innovation, Macquarie University; 2023.

Download references

Acknowledgements

The authors thank Mary Simons and Jeremy Cullis for their specialist guidance on database searches.

This work was funded in part by RACMA. RACMA contributed to the conceptualization and design of the research. JB is funded and supported by an NHMRC Leadership Investigator Award (1176620).

Author information

Authors and affiliations.

Centre for Healthcare Resilience and Implementation Science, Australian Institute of Health Innovation, Macquarie University, 75 Talavera Rd, North Ryde, Sydney, NSW, 2109, Australia

Samantha Spanos, Elle Leask, Romika Patel, Michael Datyner & Jeffrey Braithwaite

Royal Australasian College of Medical Administrators, Melbourne, VIC, Australia

You can also search for this author in PubMed   Google Scholar

Contributions

JB and SS led the research conception and design. SS, ELe, and RP executed the research, and acquired, and analyzed the data. JB, SS, and MD significantly contributed to the interpretation of the data. SS wrote the original manuscript draft which was critically reviewed by JB, ELe, RP, MD and ELo. SS led the manuscript revisions with significant contribution from JB, and input from ELe, RP, MD, and ELo. All authors agree on the journal to which the article will be submitted, have reviewed, and approved the manuscript, and agree to be personally accountable for its contents.

Corresponding author

Correspondence to Samantha Spanos .

Ethics declarations

Ethics approval and consent to participate.

Not applicable.

Consent for publication

Competing interests.

The authors declare no competing interests.

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary Material 1

Rights and permissions.

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ . The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Cite this article.

Spanos, S., Leask, E., Patel, R. et al. Healthcare leaders navigating complexity: a scoping review of key trends in future roles and competencies. BMC Med Educ 24 , 720 (2024). https://doi.org/10.1186/s12909-024-05689-4

Download citation

Received : 13 February 2024

Accepted : 21 June 2024

Published : 03 July 2024

DOI : https://doi.org/10.1186/s12909-024-05689-4

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Physician leadership
• Future health care
• Future healthcare
• Complex systems

BMC Medical Education

ISSN: 1472-6920

• Submission enquiries: [email protected]
• General enquiries: [email protected]

Framework for Health Care Services Delivery Under the Devolved System of Government in Kenya: An Empirical Review of Literature

• Conference paper
• First Online: 04 July 2024
• Cite this conference paper

• Emmanuel Awuor 13 &
• Kalyan Kumar Sahoo 14  

Part of the book series: Lecture Notes in Networks and Systems ((LNNS,volume 1))

Included in the following conference series:

• International Conference on Advanced Computing and Intelligent Engineering

This paper presents literature review seeking to establish a framework for management of health care services under the devolved system of government in Kenya. The key variables investigated included: health care systems structure; health care ecosystem; health care delivery operations; and management of health care services under the devolved system of government. The rationale for the study was based on inefficiencies reported, perennial shortage of staff, strikes and inequitable distribution of resources in public hospitals in Kenya. There are those who feel that management of health services should be taken back to the national level since its implementation was rushed and has made management of health service provision at the county level worse. There is still a belief that devolution has contributed to the significant inefficiencies and confusion of health service management in the county. The target population for the study were all the all-county referral hospitals in Kenya. Census sampling was adopted since the entire target population of eleven county referral hospitals and a population of 180 was used. The research philosophy adopted was a triangulation of both qualitative and quantitative methods. Hence, the research philosophy is a blend of phenomenological and positivist approach. A cross-sectional survey design was used as the research design for the study. Qualitative data analysis was done using content analysis with the help of computer-based softwares such as NVIVO. The quantitative part of the data analysis was done using descriptive statistics and multiple regression analysis using statistical methods such as multiple regression analysis and diagnostic tests such as heteroscedasticity and multicollinearity tests. The findings of the study should be able to influence policy on the management of health care services under the devolved system of government in Kenya, as well as form a body of theory in the study of health care as a complex adaptive system.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Institutional subscriptions

Amendah D (2015) How much is spent on Healthcare in Kenya and where does the money come from? www.aphrc.org : http://aphrc.org/how-much-is-spent-is-healthcare-in-kenya-and-where-does-the-money-come-from-lessons-from

Cilliers P (2002) Complexity and postmodernism—understanding complex systems. Sage Publications

Book   Google Scholar  

Cooper DR, Shindler PS (2014) Business research methods. McGraw-Hill, Irwin

Google Scholar  

MOH (2014) Kenya health sector referral and implementation guidelines—1st Edition. Ministry of Health, Division of Emergency and Disaster Risk Management. Ministry of Health, Nairobi. http://www.health.go.ke

McDermott T, Rouse W, Goodman S, Loper M (2013) Multi-level modelling of complex socio-technical systems. Proc Comput Sci 16 (2013) 1132–1141

Mulaki A, Lipsky A, Williamson T, Sullivan Z (2015) Political will for health devolution in kenya: insights from three countries. USAID/KENYA, Health Policy Project

Ojanji W (2016, March 17) Kenya’s health sector is starved of funds. http://AllAfrica.com : http://allafrica.com/stories/201603120266.html

Rouse WB (2007) Health care as a complex adaptive system: implications for design and management. In: Presentation at NAE Annual Meeting Technical Symposium

Rouse WB, Cortese DA (2010) Engineering the system of health delivery. IOS Press, Amsterdam

Rouse WB (2007) Complex engineered, organizational and natural systems—issues underlying the complexity of systems and fundamental research needed to address these issues. Syst Eng 10(3):260–271

Article   Google Scholar  

Sekaran U, Bougie R (2010) Research methods for business: A skill building approach. Wiley, Hoboken

Vision2030 (2007) Kenya Vision 2030—the most popular version, vol. 1. Ministry of Planning and National Development, Nairobi. www.nesc.go.ke

World Health Organisation (WHO) (2001) Report of the scientific peer group on health systems performance assessment. WHO, Geneva

World Health Organisation (WHO) (2005) International health regulations. JAI Press, Greenwich, CT

Download references

Author information

Authors and affiliations.

Management University of Africa, Nairobi, Kenya

Emmanuel Awuor

Department of Strategy and Marketing, DRIEMS University, Cuttack, India

Kalyan Kumar Sahoo

You can also search for this author in PubMed   Google Scholar

Corresponding author

Correspondence to Emmanuel Awuor .

Editor information

Editors and affiliations.

Department of Computer Science, Rama Devi Women's University, Bhubaneswar, Odisha, India

Bibudhendu Pati

Department of Computer Science, Rama Devi Women's University, Bhubaneswar, India

Chhabi Rani Panigrahi

Vice Chancellor for Research, University of California, Davis, Davis, CA, USA

Prasant Mohapatra

Dept of Computer Science and Information Engr. (CSIE),, Providence University, Taichung, Taiwan

Kuan-Ching Li

Rights and permissions

Reprints and permissions

Copyright information

© 2024 The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd.

About this paper

Cite this paper.

Awuor, E., Sahoo, K.K. (2024). Framework for Health Care Services Delivery Under the Devolved System of Government in Kenya: An Empirical Review of Literature. In: Pati, B., Panigrahi, C.R., Mohapatra, P., Li, KC. (eds) Proceedings of the 7th International Conference on Advance Computing and Intelligent Engineering. ICACIE 2022. Lecture Notes in Networks and Systems, vol 1. Springer, Singapore. https://doi.org/10.1007/978-981-99-5015-7_31

Download citation

DOI : https://doi.org/10.1007/978-981-99-5015-7_31

Published : 04 July 2024

Publisher Name : Springer, Singapore

Print ISBN : 978-981-99-5014-0

Online ISBN : 978-981-99-5015-7

eBook Packages : Engineering Engineering (R0)

Share this paper

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Publish with us

Policies and ethics

• Find a journal
• Track your research

Click through the PLOS taxonomy to find articles in your field.

For more information about PLOS Subject Areas, click here .

Loading metrics

Open Access

Peer-reviewed

Research Article

The impact of the Drug Marketing Authorization Holder system on sustainable improvement of innovation quality in the pharmaceutical manufacturing industry in China—An empirical study using synthetic control methods

Roles Conceptualization, Data curation, Formal analysis, Software, Writing – original draft

Affiliation School of Business Administration, Shenyang Pharmaceutical University, Shenyang, Liaoning, China

Roles Funding acquisition, Writing – review & editing

* E-mail: [email protected] (ZH); [email protected] (SW)

Affiliations School of Business Administration, Shenyang Pharmaceutical University, Shenyang, Liaoning, China, Research Institute of Drug Regulatory Science, Shenyang Pharmaceutical University, Shenyang, China

Roles Methodology, Supervision, Writing – review & editing

• Qiang Liu, 
• Su Wang, 

• Published: July 5, 2024
• https://doi.org/10.1371/journal.pone.0304056
• Reader Comments

In November 2015, China launched a pilot program for its Drug Marketing Authorization Holder (MAH) system, aiming to integrate production and research and development resources to spur drug innovation. While many scholars have examined the theoretical relationship between the MAH system and pharmaceutical manufacturing innovation, empirical studies have been scarce.

This study uses inter-provincial panel data on China’s pharmaceutical manufacturing industry from 2009 to 2019, along with synthetic control methods, to offer a robust analysis of the impact of the MAH paradigm on innovation quality in pilot provinces. We treat the 2015 MAH system implementation in ten provinces as a natural experiment and account for the mediating effects of R&D investments, allowing us to discern the underlying mechanisms.

Our findings demonstrate a significant positive effect of China’s MAH system on innovation quality in the pharmaceutical manufacturing industry. Specifically, the system is most effective in Jiangsu, Guangdong, and Shandong provinces. We also identify R&D investments as playing a mediating role in facilitating the favorable impact of the MAH system on innovation quality.

Our study highlights the crucial importance of the MAH system in promoting innovation quality in China’s pharmaceutical manufacturing industry, providing vital empirical support for its adoption and further enhancement. This research also offers practical insights for policymakers and R&D decision-makers in the pharmaceutical sector.

Citation: Liu Q, Wang S, Huang Z (2024) The impact of the Drug Marketing Authorization Holder system on sustainable improvement of innovation quality in the pharmaceutical manufacturing industry in China—An empirical study using synthetic control methods. PLoS ONE 19(7): e0304056. https://doi.org/10.1371/journal.pone.0304056

Editor: Yu Zhou, Inner Mongolia University, CHINA

Received: July 7, 2023; Accepted: May 6, 2024; Published: July 5, 2024

Copyright: © 2024 Liu et al. This is an open access article distributed under the terms of the Creative Commons Attribution License , which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.

Data Availability: All original data files are available from the database of National Bureau of Statistics of China ( http://www.stats.gov.cn/ ).

Funding: This work was supported by the Shenyang Social Science Federation Philosophy and Social Science Research Base Project (Project ID: SYSK2024-JD-24 to SW) and Liaoning Province Social Science Planning Fund Project (Project ID: L23BGL006 to SW).

Competing interests: The authors have declared that no competing interests exist.

1. Introduction

The Marketing Authorization Holder (MAH) system, a type of drug regulation, has been implemented for many years in developed regions such as Europe, the United States, and Japan. The concept of a MAH was first introduced in the European Economic Community (EEC) Council’s Act 65/65/EEC, which established provisions for drug marketing authorization in the European Union, thereby creating a widely used international MAH system [ 1 ]. In the early 20th century, the FDA established its drug registration system, but the federal Food, Drug, and Cosmetic Act (FD&C Act) does not use the term "marketing authorization holder." Instead, "applicant" and "applicant holder" are used to refer to those seeking authorization to market a drug, with "applicant holder" sharing similar obligations as the EU’s MAH [ 2 , 3 ]. Japan adopted the MAH system following a 2004 amendment to the Pharmaceutical Affairs Law, becoming the first Asian country to do so [ 4 ]. China has also formally implemented the MAH system in December 2019.

Currently, China’s economic development is undergoing a shift from high-speed growth to high-quality development [ 5 ]. As innovation remains the most significant driving force for economic growth, promoting high-quality innovation activities is essential for achieving high-quality economic development. The pharmaceutical manufacturing industry stands as a strategic pillar industry in ensuring both national security and citizens’ livelihood, making it crucial to achieve sustainable and healthy development. However, more relevant theoretical research is needed. Therefore, the research objective of this paper is to apply the synthetic control method, consider the pilot of the MAH system carried out in some provinces and cities in China from 2015 to 2019 as a quasi-natural experiment, empirically analyze the impact of the MAH system on the quality of innovation of China’s pharmaceutical manufacturing industry and the mechanism of its action, to make up for the deficiencies in the academic research, and to provide the enhancement of the quality of innovation of the pharmaceutical manufacturing industry through the implementation of the MAH system by promoting the Theoretical guidance.

The main objective and innovation of this paper is: Firstly, unlike previous studies, we provide the first empirical analysis exploring the links between the MAH system and the innovation quality of the pharmaceutical manufacturing industry. As such, this paper represents a significant step towards establishing an empirical foundation for the MAH system’s ability to improve the innovation quality in the industry. Secondly, our use of the synthetic control method is noteworthy for its data-driven weighting of the control group, which enables us to obtain a counterfactual control group that closely resembles the treatment group. As a result, this approach minimizes the impact of subjective biases and endogeneity, in contrast to methods such as DID or PSM-DID. Thirdly, alongside assessing the policy’s overall effect, we also examine in-depth the mechanism through which the MAH system can impact the quality of innovation in the pharmaceutical manufacturing industry. Overall, these findings provide valuable insights into the MAH system’s potential to enhance innovation quality in the industry, with wider implications for policymakers and practitioners.

To fulfil these objectives, the study is structured as follows. The first part introduces the basic information of the MAH system and reviews the related research literature. The second part introduces the theoretical foundation of this article and presents the research hypotheses. The third part describes the overall research design ideas, including an introduction to the empirical research methodology and the logic of model setting, data sources, and variable selection. Next, we conducted the empirical analysis, reported the results and conducted validity tests. The fourth part was the mediation effect test. The last part summarizes the conclusions and insights gained from this study and discusses the significance of the study, its limitations and future research directions.

2. Institutional background and literature review

Before 2016, China’s drug marketing license system implemented a unified management system that combined the "production license" and the "marketing license" for drugs, i.e., only enterprises holding a "drug production license" could apply for registration, production and sales of drugs. With the rapid development of the economy and the continuous improvement of the industrial system, China’s pharmaceutical manufacturing industry further upgraded the development of this drug production license and marketing license bundled regulatory system, at that time led to a lot of chaos in China’s pharmaceutical manufacturing industry, such as the lack of incentive for research and development of new drugs, a large number of drugs idle approval number, the responsibility for the quality of the drug is not clear and other issues, resulting in the pharmaceutical manufacturing industry as a whole redundant production capacity, the drug supervision departments review and approval resources are wasted, and it is also difficult to protect the legitimate rights and interests of citizens, legal persons and other organizations to participate in the research and development of new drugs at the legal level, which seriously restricts the high-quality development of China’s pharmaceutical manufacturing industry, especially the high-quality development of drug research, development and innovation [ 6 , 7 ].

To solve the above problems, in August 2015, the State Council formally put forward for the first time in the "Opinions on the Reform of the Review and Approval System for Drugs and Medical Devices" the requirement of "carrying out the pilot of the system of drug listing license holders", and explicitly proposed that "drug research and development institutions and scientific researchers are allowed to apply for registration of new drugs" [ 8 ]. and explicitly proposed to "allow drug research and development organizations and researchers to apply for registration of new drugs" [ 8 ]. In November of the same year, the Standing Committee of the National People’s Congress authorized the State Council to carry out the pilot system of drug listing permit holders in 10 provinces/municipalities directly under the central government, and the pilot period started from November 5, 2015 to November 4, 2018, with an implementation period of three years; the pilot period was extended for one year on October 26, 2018; and the new version of the Drug Administration Law of the People’s Republic of China was officially released and implemented on December 1, 2019 [ 9 ]. On December 1, 2019, the new version of the Drug Administration Law of the People’s Republic of China was formally released and implemented, clearly stating that "the holder of the marketing authorization for drugs shall, in accordance with the provisions of this law, assume responsibility for the non-clinical research, clinical trials, production and operation, post-market research, monitoring of adverse reactions, as well as the reporting and handling of drugs" [ 9 ]. At this point, the MAH system was formally established in China through the law.

The existing research literature on the MAH system mainly focuses on the comparative study of the MAH system among countries and the study of the implementation effect of the system. For example, Zhu Jiaxian et al. compared the differences between the marketing authorization holder systems implemented in the European Union, the United States, and Japan, analyzed the regulatory requirements of the MAH system in each country before and after marketing approval, and put forward reference suggestions for improving China’s MAH system [ 10 ]. Xie Jinping et al. sorted out the supporting regulatory systems in the whole life cycle management of drugs closely related to the MAH system analyzed the impact on the above regulatory systems before and after implementing the MAH system and put forward targeted suggestions for convergence [ 11 ]. Wu Xiaoyan and Huang Zhe analyzed the problems of the existing quality system of domestic commissioned manufacturers after implementing the MAH system. They provided suggestions for the improvement of the quality system of commissioned manufacturers [ 12 ]. Jia Jinsheng et al. focused on the evaluation index system of the quality of pharmacovigilance work carried out by the production body after the implementation of the MAH system and explored the design of an informatization platform for pharmacovigilance assessment of marketing license holders, which provides a reference for exploring the intelligence of pharmacovigilance assessment of the holders [ 13 ].

However, after literature research and screening, it is found that there needs to be more research on the impact of the implementation of the MAH system on the overall innovation quality of the pharmaceutical manufacturing industry.

3. Theoretical foundation and research hypothesis

Considering the implementation experience of countries such as Europe, the United States, and Japan, the core regulatory objective of the MAH system is to realize the separation of the production license from the marketing authorization. It means that MAHs—i.e. holders of marketing authorization documents for drugs, who may be drug manufacturers, R&D institutions, or researchers—are given more autonomy, and they can choose to produce drugs on their own or entrust other manufacturers to carry out production. Through a well-designed institutional framework and implementation pathway, the MAH system revolves around the two core dimensions of rights and responsibilities, aiming to help the regulatory authorities implement management more effectively, thereby effectively preventing risks to drug quality and safety while stimulating the initiative and flexibility of the market’s resource allocation, promoting technology transfer, and incentivizing research and development (R&D) innovation.

According to scholarly consensus, the implementation of the MAH system has the potential to optimize resource allocation within the pharmaceutical manufacturing industry. This would result in more efficient transformation of scientific research outcomes [ 14 – 16 ] and heightened innovation fervor among businesses. Given the high levels of capital investment, long innovation cycles, and frequent uncertainties associated with this field, the externalities of innovation activities have significant sway, particularly within pharmaceutical manufacturing.

In China, the pharmaceutical sector is of strategic importance, with close ties to the nation’s economy and welfare. Therefore, the externalities of innovation activities within the sector mostly arise from national policies. Taken as a whole, the introduction of the MAH system represents a promising development, pointing towards enhanced innovation capabilities and a more robust economic trajectory for China.

The new institutional economic theory presents a fresh approach to analyzing social and economic issues, particularly during the transition between old and new systems [ 17 ]. According to this theory, the aim of each system design is to constrain individual behavior that might otherwise maximize welfare or utility by establishing "game rules." At the same time, system design has profound efficiency impacts [ 18 ], fostering economic growth via institutional innovation even without increased input factors [ 19 ]. It should be noted that not all institutions have historically contributed to economic growth since some institutional arrangements were inefficient or even ineffective. This is because optimal resource allocation is closely tied to the incentive structure of economic society, which itself is determined by the formal system of the government’s supply. Therefore, for the government’s intervention in market operations to promote economic development, it must do so on the precondition of not impinging on the market’s basic regulatory role [ 19 ]. The implementation of the MAH system in China aims to transform the previous "drug production and marketing combined management" system into a novel one. The purpose of the MAH system’s implementation is to create more collaborations for the industry, encourage the involvement of more stakeholders in the development of new drugs, as well as internalize various external elements during the new drug research, development, and production process [ 20 ]. Additionally, from the standpoint of signal theory, the implementation of the MAH system itself will communicate a signal of the state’s intention to "encourage and innovate in the pharmaceutical manufacturing sector." This signal boosts corporate confidence and enhances their research, development, and innovation capabilities. Consequently, it promotes the entire industry’s high-quality innovation development.

The development of innovative drugs is a complex and multi-faceted process that relies heavily on advancements in basic science and significant investment in research and development. Increasing R&D investment can help companies acquire advanced technologies and high-end equipment, establishing a robust R&D infrastructure while recruiting top scientific talent to enhance innovation capabilities and overall competitiveness. However, the "aggregation effect" of capital allows enterprises to pool resources to tackle cutting-edge technical issues, leading to the production of quality results [ 21 ]. While the implementation of the MAH system seems only to relax restrictions on drug production and listing, it also benefits small and medium-sized research institutions, university labs, and other companies lacking large-scale production capacity by allowing them to retain ownership of their innovations after listing and avoiding "invisible" holding [ 22 ]. The MAH system also broadens the circulation channels for pharmaceutical industry knowledge assets, attracting private investment in innovative drug R&D and sharing the financial burden of new drug research and development with companies and institutions. This approach allows innovative companies and research institutions to concentrate on developing innovative drugs, dispersing financial risk and enhancing innovation potential [ 23 ]. Through policy guidance and market regulation, R&D companies and institutions can achieve relatively equivalent returns from their investments and early-stage risks, enhancing the "liquidity" of innovative drug technologies and stimulating the innovation potential of China’s pharmaceutical manufacturing industry, thus improving overall innovation quality.

Drawing on this information, this paper puts forth the following thesis:

• H1 :The implementation of the MAH system has the potential to enhance the quality of innovation within China’s pharmaceutical manufacturing industry.
• H2 :The adoption of the MAH system enhances the quality of innovation in China’s pharmaceutical manufacturing industry by bolstering the intensity of capital investment in research and development.

The theoretical model hypothesized in this paper is depicted in Fig 1 :

• PPT PowerPoint slide
• PNG larger image
• TIFF original image

https://doi.org/10.1371/journal.pone.0304056.g001

4. Materials and methods

4.1 research methods and modeling.

This paper adopts the Synthetic Control Method (SCM), which was first proposed by Abadie and Gardeazabal [ 24 ]. In the context of this study, the MAH system, which has been piloted in 10 Chinese provinces since 2015, is considered a quasi-natural experiment. Based on this, an experimental group and a control group are established to evaluate the impact of the MAH system on the innovation quality of the Chinese pharmaceutical manufacturing industry. The SCM is a data-driven, non-parametric research method that overcomes problems associated with subjective sample selection and endogeneity that have been problematic for traditional regression methods. As a result, it has been widely employed in research investigating policy implementation effect evaluations in a broad range of fields. For example, evaluations of California’s tobacco control law’s impact [ 25 ], the consequences of administrative division adjustments on economic development [ 26 ], the impact of trade liberalization on child mortality in emerging and developing countries [ 27 ], the impact of China’s carbon emission trading pilot policy on the environment and economy [ 28 ], the effects of grassroots environmental protection institutions establishment in China on the environment and economy [ 29 ], the impact of developing environmental protection institutions in China, and the impact of the one-child policy on the Chinese economy [ 30 ]. This method has also been widely applied in the technological innovation research field. For instance, this paper uses provincial panel data to research the impact of China’s carbon emission trading pilot policy on industrial green technology innovation [ 31 ] and how environmental policy affects enterprise innovation ability [ 32 ].

Abadie et al. (2015) pointed out in their study that the application of synthetic control should exclude units that are susceptible to policy interventions or specific shocks from the control group. Guided by this idea, the primary operational steps of the synthetic control method used in this paper are as follows: (1) Identifying the appropriate control variables and allocating dissimilar weights for the control group samples; (2) Based on the diverse predictor variable weights, applying the synthetic control method to fit a counterfactual province with comparable characteristics to each MAH system pilot province (i.e., processing group province); (3) Comparing the differences in innovation quality between MAH system pilot provinces and corresponding counterfactual provinces to evaluate the MAH system’s policy effect.

The innovation quality of the synthetic province’s pharmaceutical manufacturing industry, obtained through weighting, effectively mirrors the innovation quality development level of the pilot province that has not yet implemented the MAH system, known as the "counterfactual". The disparity in innovation quality between the pilot province and its corresponding synthetic province represents the net impact of the MAH system on the province’s pharmaceutical manufacturing industry innovation quality following its implementation.

4.2 Variables

(1) Dependent variable: Innovation quality(IQ). Most scholars approach measuring innovation quality from the perspective of patents. For example, Lanjouw and Schankerman [ 33 ] employed patent quality as an indicator of innovation quality. They gathered relevant metrics, such as forward and backward citations, to form an index representing patent quality. Zhang Gupeng et al. [ 34 ] measured innovation quality using patent grant rate and patent length. As corporations are the primary drivers of innovation, micro-data, such as patent codes and length, are challenging to obtain at a macro level. Therefore, following the practices of Yang Bo et al. [ 35 ] and Ma Yonghong et al. [ 36 ], this study uses the natural logarithm of the average number of invention patent applications per enterprise in the pharmaceutical manufacturing industry to represent innovation quality (IQ) within a given region.

(2) Mediating variable. As per the aforementioned theoretical analysis, the intermediary variable in this paper is R&D capital investment (RDI). The intensity of R&D capital investment within the pharmaceutical manufacturing industry serves as the measurement index.

(3) Control variables. To enhance the fitting effect and result robustness of the synthetic control object, it is essential to include critical factors that may impact the innovation quality of the pharmaceutical manufacturing industry as control variables, as per the practice of Liu Jiayan and Fan Ziying [ 37 ]. Drawing from previous studies [ 37 – 40 ], this paper identifies seven variables that may influence the industry’s innovation quality, which are subsequently added as control variables. Table 1 illustrates the variable names and their respective definitions as used in this paper.

https://doi.org/10.1371/journal.pone.0304056.t001

To eliminate the impacts of price factors, this paper deflates the main business income using the producer price index (PPI) published by the National Bureau of Statistics (2009 = 100); the total profit is deflated using the consumer price index (CPI) published by the same bureau (2009 = 100); while the total assets are deflated using the fixed asset investment index also published by the National Bureau of Statistics (2009 = 100). Meanwhile, Zhu Pingfang et al.’s method [ 41 ] for constant price treatment is adopted for the R&D internal expenditure price index, whereby a fixed investment price index and consumer price index are attributed 0.45 and 0.55, respectively.

Table 2 displays the descriptive statistical results for each variable.

https://doi.org/10.1371/journal.pone.0304056.t002

4.3 Sample selection and data

This paper utilizes panel data from 29 provinces in China’s pharmaceutical manufacturing industry spanning from 2009 to 2019 as research samples. Qinghai and Tibet were excluded due to extensive missing data. Experimental subjects were selected from ten provinces participating in the pilot, namely Beijing, Shanghai, Tianjin, Hebei, Shandong, Jiangsu, Zhejiang, Fujian, Guangdong, and Sichuan. Meanwhile, 19 provinces outside the pilot area were chosen as the control group. The selected research interval holds significance for two reasons: firstly, data on the number of invention patent applications of the pharmaceutical manufacturing industry in various provinces before 2009 were missing, and secondly, the MAH system pilot concluded on December 1, 2019, after which it was rolled out countrywide. All the data utilized in this paper were sourced from the China High-tech Industry Statistical Yearbook and provincial statistical yearbook throughout the years.

5. Empirical results and analysis

5.1 baseline empirical results.

Although the MAH system pilot was launched simultaneously in ten provinces, the heterogeneity in economic development, geographical conditions, and industrial structure among these provinces would inevitably result in variation in the innovative quality of the pharmaceutical manufacturing industry. However, such variability does not pose a threat to the robustness of the empirical analysis results presented in this paper. The synthetic control method is a data-driven empirical analysis tool that permits the examination of the innovation quality of pharmaceutical manufacturing industries in provinces irrespective of their innovation quality levels. As long as synthetic control provinces manifest a better fit to the changing trend of innovation quality in pilot provinces before policy implementation, the effect of policy implementation can be effectively reflected [ 42 ]. Comprehensively utilizing the synth command written by Abadie et al and the synth2 command written by Professor Chen Qiang’s team of Shandong University, corresponding synthetic control provinces for each pilot province in Stata17.0 software were constructed, and the weights for each synthetic control province were computed as indicated in Table 3 . The greater the weight assigned to a particular composite province, the more similar its characteristics are to the corresponding pilot province. A weight of 0, on the other hand, indicates significantly varied features between the composite and pilot provinces. For instance, the synthetic control provinces and their corresponding weights for Zhejiang Province are Yunnan (0.451), Hubei (0.425), Chongqing (0.091), and Heilongjiang (0.033), with all other provinces assigned a weight of 0, resulting in a total weight of 1. This demonstrates that the innovation quality index of the pharmaceutical manufacturing industry in Yunnan, Hubei, Chongqing, and Heilongjiang can be estimated based on their weights of 0.451, 0.425, 0.091, and 0.033, allowing for the estimation of the developmental trend of innovation quality in Zhejiang Province without the need for implementing the MAH system pilot.

https://doi.org/10.1371/journal.pone.0304056.t003

Fig 2 shows the evolution trend of pharmaceutical manufacturing innovation quality in 10 MAH system pilot provinces and their corresponding synthetic control provinces. The solid lines depict the actual development trend of pharmaceutical manufacturing innovation quality in the pilot province, while the dotted lines represent the corresponding development trend in the synthetic control province. The vertical dotted line corresponds to the year of the pilot implementation of the MAH system. It is evident that before the vertical dotted line, the real pharmaceutical manufacturing innovation quality of provinces such as Zhejiang ( Fig 2A ), Shanghai ( Fig 2D ), Shandong ( Fig 2E ), Jiangsu ( Fig 2F ), Guangdong ( Fig 2H ), Fujian ( Fig 2I ) and Beijing ( Fig 2J ) almost coincided with the corresponding synthetic curves of pharmaceutical manufacturing innovation quality development. This implies that the synthetic control method aligns well with the actual development trend of pharmaceutical manufacturing industry innovation quality in these pilot provinces. However, the fitting results in Sichuan ( Fig 2C ) were poor, and the fitting results in Tianjin ( Fig 2B ) and Hebei ( Fig 2G ) were the worst, indicating that the pharmaceutical manufacturing industries in Sichuan, Tianjin and Hebei provinces possessed unique innovation quality characteristics and other related economic characteristics that made it challenging to obtain a perfect fit by computing the weights of other provinces and cities.

https://doi.org/10.1371/journal.pone.0304056.g002

Table 4 shows the R-squared values of the fitted synthetic control method for the aforementioned 10 pilot provinces. As can be seen, except for Tianjin, Sichuan and Hebei, the R-squared values of the other seven pilot provinces are close to unity. This indicates that the synthetic control method’s fitting effect is excellent in these seven provinces: Zhejiang, Shanghai, Shandong, Jiangsu, Guangdong, Fujian and Beijing. Therefore, our subsequent analysis will concentrate on these seven provinces only.

https://doi.org/10.1371/journal.pone.0304056.t004

As depicted in Fig 2 , following the MAH system pilot’s launch, the actual and synthetic pharmaceutical manufacturing innovation quality trends in six provinces, namely Zhejiang, Shanghai, Shandong, Jiangsu, Guangdong and Fujian, gradually diverged. The real change trend line (i.e., the solid line) was above the synthetic change trend line (i.e., the dashed line). In other words, the actual pharmaceutical manufacturing industry innovation quality in these six provinces was higher than the synthetic innovation quality after the pilot implementation of the MAH system, implying a higher innovation quality than what would have been anticipated without the pilot. Although Beijing’s pharmaceutical manufacturing industry innovation quality was slightly lower than that of the first year after the policy implementation, it has exceeded that of Beijing significantly since then. This is likely because Beijing is the political and regulatory center of our country’s pharmaceutical manufacturing industry. As such, the local enterprises and regulatory authorities may exercise greater caution in implementing and promoting the MAH system pilot to minimize the uncertainty risks associated with such innovative policies, as compared to the other pilot provinces.

Moreover, the real innovation quality trend of the pharmaceutical manufacturing industry in each pilot province after the MAH system pilot implementation differs from the innovation quality trend of its synthetic counterpart. Fig 2A shows that the innovation quality of the pharmaceutical manufacturing industry in Zhejiang Province initially fluctuated and declined after the policy implementation, but changed after the MAH policy was introduced. Despite a decline from 2016 to 2018, the industry still exhibited an upward trend thereafter. Contrarily, Fig 2D portrays that the innovation quality of the pharmaceutical manufacturing industry in Shanghai experienced a significant decline from 2014, which persisted until 2018. However, due to the MAH system’s pilot implementation, the downward trend slowed after 2015 and rebounded after 2018, resulting in a steady improvement. In contrast, Fig 2E, 2F and 2I reveal that the innovation quality of the pharmaceutical manufacturing industries in Shandong, Jiangsu, and Fujian continued to grow after the policy implementation. Although the growth trend of Shandong and Jiangsu decelerated slightly from 2016 to 2018, they continued to show a rapid expansion thereafter. As illustrated in Fig 2h , the innovation quality of the pharmaceutical manufacturing industry in Guangdong Province briefly declined within the first year of the policy’s implementation before steadily improving for several years. Despite a brief decline in 2019, the industry still exhibited an overall upward trend. Finally, in Fig 2j, it can be observed that the innovation quality of Beijing’s actual pharmaceutical manufacturing industry briefly declined within the first year after the policy’s implementation before experiencing a lag period of about one year, after which it resumed rapid growth.

To thoroughly examine the effect of the MAH system pilot on the innovative quality of the pharmaceutical manufacturing industry in different provinces, this study quantifies the average innovation quality value in the actual and counterfactual cases of this industry in the seven above-mentioned provinces during the pilot period. The corresponding data is presented in Table 5 . According to the table, for the pilot duration of the MAH system, the innovation quality of the pharmaceutical manufacturing industry in Jiangsu, Guangdong, and Shandong provinces ranked among the top three of all seven pilot provinces. Furthermore, the treatment effect values of these three provinces also placed in the top three, indicating that the pilot implementation of the MAH system in these regions had the most significant impact on improving the innovative quality of the respective pharmaceutical manufacturing industries.

https://doi.org/10.1371/journal.pone.0304056.t005

5.2 Validity checks

(1) Placebo test. Based on the empirical analysis presented above, it is clear that there exists a significant disparity between the actual and synthetic innovation qualities of the pharmaceutical manufacturing industry in Zhejiang, Shanghai, Shandong, Jiangsu, Guangdong, Fujian and Beijing due to the pilot implementation of the MAH system. In order to ascertain that this difference is not attributed to random factors, but rather due to the MAH pilot implementation, this study applies the placebo approach, utilizing the methods employed in previously published literature [ 25 , 43 , 44 ], to analyze the policy impact of the MAH system pilot. The placebo test operates under the fundamental notion: select a non-pilot province as a placebo province, and assume that both the placebo province and an actual pilot province underwent the same policy treatment within the same year. Then, using the synthetic control method to conduct identical analysis. This test verifies that if the calculated policy treatment effect of the placebo province is lesser than that of the corresponding actual pilot province, then the innovation quality variation between the synthetic province of pharmaceutical manufacturing industry and the actual pilot province is genuinely a result of the MAH pilot implementation, that is, the empirical analysis results are valid. On the contrary, if the policy treatment effect of the placebo province is greater than that of the corresponding actual pilot province, the empirical analysis outcomes are deemed ineffective.

In this study, the placebo provinces were selected from the control group provinces with the highest similarity to the pilot provinces, based on the synthetic contribution rate derived from fitting the synthetic control method—these provinces are presented in the second row of Table 3 . As shown in Fig 3 , the corresponding actual and synthetic innovation quality change trend curves of Yunnan, Heilongjiang (SH), Guizhou, Heilongjiang (JS), Heilongjiang (GD), Gansu, Shanxi serve as placebo provinces. The fitting weights of each placebo province are exhibited in Table 6 . For Fujian, despite Gansu and Jiangxi ranking as the top two provinces in terms of weight contribution under synthetic control, they demonstrated limited fitting to the actual situation prior to 2015, resulting in R-squared values of 0.72786 and 0.66745 respectively, thus the policy effect cannot be effectively reflected. Therefore, Guizhou, which had the third-highest contribution weight to synthetic Fujian, was selected as the placebo test province. Similarly, for Beijing, Shanxi was chosen as the placebo test province due to the suboptimal fitting effects observed in Chongqing (R 2 = 0.57767) and Inner Mongolia (R 2 = 0.51229).

https://doi.org/10.1371/journal.pone.0304056.g003

https://doi.org/10.1371/journal.pone.0304056.t006

Illustrated in Fig 3 , with the exception of Heilongjiang, the placebo province of Shanghai showed a slightly lower synthetic path compared to the true path in 2018, while all other placebo provinces had a higher synthetic path. These results suggest that provinces with the highest similarity to the pilot provinces did not experience similar effects after implementation of the MAH system, indicating that the effects observed were not due to random factors, but indeed attributable to the MAH policy.

(2) Ranking test. In order to further test the statistical significance of the effect of the MAH pilot policy, a ranking test method, similar to the one proposed by Abadi et al. [ 25 ], was used in this study. The specific idea behind this method involves randomly selecting a series of non-pilot provinces from the control group and assuming that they were also affected by the same policy impact in 2015. The synthetic control method is then used to construct the innovation quality results of the pharmaceutical manufacturing industry in each of these selected provinces. Next, a series of random "policy effects" are obtained, which represent the difference between the real and synthetic values of each non-pilot province. The policy effect of the corresponding pilot province is then compared with the policy effect of the randomly selected non-pilot provinces. If the policy effect of the pilot province is significantly greater than that of the randomly selected non-pilot province, it indicates that the effect of the policy on the pilot province is significant. It should be noted that the ranking test method requires a good fitting effect of the provinces in the control group prior to policy implementation. If the fitting effect before policy implementation is poor, it suggests that the final "policy effect" may be due to the poor fitting effect and not related to the policy implementation.

This study evaluated the fitting effect of the provinces in the control group prior to policy implementation by calculating and comparing their mean square prediction error (MSPE) values. Provinces in the control group whose MSPE values were 20 times greater than the MSPE values of the pilot provinces were excluded, and the remaining provinces were identified as the provinces in the ranking test. Based on this methodology, a line chart of ranking test results was created for each pilot province, as illustrated in Fig 4 . The chart shows that the policy processing effect of the 7 pilot provinces was significantly higher than that of the non-pilot provinces in their respective control groups. This implies that if the control group provinces intend to achieve the same policy processing effect as the pilot provinces, it is a low probability event, thereby indicating that the policy effect of the MAH system pilot is significant.

https://doi.org/10.1371/journal.pone.0304056.g004

5.3 Robustness checks

To address the potential problem of variations in analysis results due to different control group selections, this study adopts the iterative method in accordance with the approach of Su Zhi and Hu Di et al. [ 45 ] to conduct a robustness test. For each pilot province, one control group province with a positive contribution towards its synthesis is eliminated iteratively, and the pilot province and the composite province are re-estimated. This methodology aims to examine whether the effect of the MAH system on the innovation quality of the pharmaceutical manufacturing industry in each pilot province will be affected due to the lack of control group provinces.

The test results are presented in Fig 5 .

https://doi.org/10.1371/journal.pone.0304056.g005

The results in Fig 5(a)–5(g) demonstrate that after successive changes in the composition of the control group of the 7 pilot provinces, the trend of the broken line is consistent with the fluctuation range and the actual composite provinces. Additionally, the study employed the LOO (leave-one-out) method to quantitatively estimate the results [ 46 ]. If the fitting effect of the original synthetic control is not affected by the elimination of some provinces in the control group, the empirical result can be considered robust. On the contrary, if the fitting results were driven by the eliminated synthetic control province, the results would not be robust.

Table 7 summarizes the LOO estimation results. The real results of the experimental group and its Synthetic control results, i.e., Synthetic Outcome, as well as the synthetic control results after iterative elimination using the LOO method, i.e., synthetic Outcome (LOO), are presented. After applying the LOO method to the synthetic control, stata calculates corresponding results for each province in the control group with a positive weight. However, for clarity, stata only reports the maximum and minimum values corresponding to each year.

https://doi.org/10.1371/journal.pone.0304056.t007

If the value of the Synthetic control result of the experimental group falls between the maximum value and the minimum value obtained after the elimination of LOO iteration, it implies that the elimination of a control group does not affect the synthetic control fitting results of the original experimental group, and the empirical results are thus robust. All values listed in Table 7 satisfy this requirement, which proves that the empirical results are robust. In addition, the process and results of the above empirical analyses and tests are consistent with the results reported in other literatures [ 25 – 32 , 43 – 45 ] that adopt similar analytical methods, which also proves that the empirical results of this paper are reliable and trustworthy, i.e., the implementation of the MAH system has a significant contribution to the quality of innovation of China’s pharmaceutical manufacturing industry, and H1 has been verified up to this point.

6. Further analysis

The above empirical research results demonstrate that the MAH system significantly improves the innovation quality of the pharmaceutical manufacturing industry in pilot provinces. However, the underlying mechanism for this improvement remains to be explored. According to the theoretical analysis described above, the pilot MAH system has the potential to enhance the intensity of R&D capital investment (H1) of enterprises, which can affect the innovation quality of the pharmaceutical manufacturing industry. To test whether this mechanism is valid, empirical evidence will be presented in this section.

6.1 Model specification

In this model, treat it represents the policy dummy variable of the MAH system pilot, which is derived from the multiplication of a dummy variable that represents the processing group and a dummy variable that represents the time point of policy intervention. Specifically, if province i implements the MAH system pilot at time point t, the value of this variable is 1; otherwise, it is 0. M it is the intermediate variable, which represents research and development investment intensity (RDI) and control it represents the combination of control variables.

6.2 Analysis of the mediating effect

Table 8 presents the test results of the mediating effect of the MAH system pilot on promoting the improvement of innovation quality in the pharmaceutical manufacturing industry. Columns (1)—(4) display the regression results of models (7)—(10), respectively. The results in column (1) demonstrate that the coefficient β 1 of the policy dummy variable (Treat) is significantly positive at the 1% level, indicating that the MAH system has a significant promoting effect on the innovation quality of the pharmaceutical manufacturing industry in the pilot provinces. This aligns with the conclusion obtained by the synthesis control method above and provides the premise for further analysis of the mediating effect.

https://doi.org/10.1371/journal.pone.0304056.t008

Using research and development investment intensity (RDI) as the mediating variable, the results in column (2) show that the regression coefficient β 2 of the policy dummy variable (Treat) is significantly positive at the 10% level, implying that the implementation of the MAH system increases R&D investment intensity in pilot provinces. The results in column (3) reveal that the regression coefficient β 3 of the mediating variable is significantly positive at the 1% level, indicating that the R&D investment intensity significantly affects the improvement of innovation quality, suggesting that the indirect effect of the entire model is significant. The results in column (4) show that the regression coefficient β 4 of the policy dummy variable (Treat) is still significant, implying that the direct effect is significant. Furthermore, β 2 , β 3 , β 4 are all positive, indicating that the intensity of R&D capital investment plays a partial mediating effect in the process of improving the innovation quality of the pharmaceutical manufacturing industry by the MAH system, accounting for 12.55% ((0.002*43.490) /0.700) of the total effect. Therefore, the MAH system can enhance the quality of innovation by strengthening the intensity of R&D investment in the pharmaceutical manufacturing industry. Hence, hypothesis H2 is verified.

There are several possible reasons for this finding: First, the implementation of the MAH system enables non-manufacturing enterprises to become the primary applicant for registration and listing of new drugs, establishing a business mechanism of equal investment and benefit, which greatly stimulates the enthusiasm of enterprises to use funds for innovative drug research and development and also promotes the inflow of social and venture capital. Second, the MAH system has promoted the separation of drug research and development and production, which has further released the idle capacity of production enterprises, reduced the operating costs of production enterprises, and can transfer the cost savings to the research and development link. Thus, the MAH system improves the quality of innovation in the pharmaceutical manufacturing industry by enhancing the intensity of investment in research and development funds.

7. Conclusion and implications

The Marketing Authorization Holder system is a mature and well-established drug regulatory framework in the world. It plays a crucial role in promoting rational and integrated utilization of production resources and stimulating innovation vitality in the pharmaceutical manufacturing industry. Utilizing panel data from 29 provinces in China between 2009 and 2019, this study employs the quasi-natural experiment approach to analyze the impact of the pilot policies of MAH system implemented in 10 provinces in 2015 on the innovation quality of the pharmaceutical manufacturing industry using the synthetic control method. Based on the empirical analysis, the following conclusions were drawn: (1) The implementation of the MAH system significantly improved the innovation quality of the pharmaceutical manufacturing industry in the pilot provinces, with Jiangsu Province having the most significant improvement effect, followed by Guangdong Province and Shandong Province. (2) R&D capital investment played a significant intermediary role in the process of improving the innovation quality of the pharmaceutical manufacturing industry through the MAH system, highlighting how the MAH system can boost innovation quality by strengthening the R&D capital investment in the pharmaceutical manufacturing industry.

Based on the above conclusions, the following implications can be drawn:

• ① The Chinese drug regulatory authorities must fully appreciate the pivotal role of the MAH system in advancing innovation quality within the pharmaceutical manufacturing industry. They should clarify its mechanism of action and introduce targeted management rules and incentive measures in accordance with the differentiated development stage of the industry across different provinces. Furthermore, the authorities should leverage effective policy tools to create a robust research environment for MAH, upstream and downstream related enterprises, and ensure that the institutional dividends from the MAH system’s efficacy in enhancing innovation quality are maximized.
• ② Adequate investments in research and development have played an indispensable role in facilitating MAH’s efforts to promote high-quality innovation in the pharmaceutical manufacturing industry. To this end, both enterprises and government departments must coordinate and work in tandem, leveraging a development strategy that incorporates "market leadership" and "government regulation." On the one hand, government regulators should authorize tax subsidies, research and development project grants, and other forms of financial aid to key innovative enterprises while establishing policies that incentivize pharmaceutical research and development innovation. On the other hand, production enterprises or research and development institutions should effectively allocate their investment portfolios, prioritize internal and external new product research and development projects, maximize the MAH system’s efficacy in augmenting enterprise innovation capacity, and ultimately enhance the overall innovation quality of China’s pharmaceutical manufacturing industry.

This study enriches the academic research on the MAH system and is an essential supplement to the existing research results; it can provide crucial theoretical references for the countries that have already implemented the MAH system and those international countries that are about to implement the MAH system. It also provides important theoretical references for the MAH system’s self-improvement and coordinated development with industrial innovation and socio-economics. Therefore, the research results of this paper will be highly concerning to food and drug regulatory agencies, pharmaceutical research institutes, and enterprises of marketing authorization holders in various countries.

This study still has certain limitations. First, due to the limitations of the data, the sample size included in the empirical study of this paper is small; second, this paper only selects the relevant data in China as the research object, which leads to a certain lack of international generalizability of the results of this paper; third, the mediating effect of the variable of the intensity of R&D capital investment in this paper only accounts for 12.55% of the total effect, which indicates that there are still other mediating variables. Due to data limitations, it is not possible to conduct a comprehensive and in-depth investigation of the mechanism of the MAH system affecting the innovation quality of the pharmaceutical manufacturing industry. Therefore, in the subsequent research, it is necessary tto expand the sample size of empirical analysis further and search for more mediating variables to more comprehensively analyze the path of the MAH system’s influence on the innovation quality of the pharmaceutical manufacturing industry; at the same time, as far as possible, the data of different countries should be selected for empirical analysis to improve the generality and practicability of the conclusions of this study.

• 1. Liang JY. Research On Marketing Authorization Holder System .(Master’s Thesis), Liaoning University.2017.
• 2. FDA. Federal Food, Drug, and Cosmetic Act (FD&C Act). https://www.fda.gov/regulatory-information/laws-enforced-fda/federal-food-drug-and-cosmetic-act-fdc-act .
• View Article
• Google Scholar
• 5. Zhang ZR, Zhang P, Yuan FH. Annual report on China’s economic growth(2017–2018).Social Sciences Academic Press, Beijing(China),2018.
• 8. Central People’s Government of the People’s Republic of China. Opinions of The State Council on reforming the examination and approval system for drugs and medical devices . https://www.gov.cn/govweb/zhengce/content/2015-08/18/content_10101.htm .
• 9. Central People’s Government of the People’s Republic of China. The Drug Administration Law of the PRC . https://www.gov.cn/xinwen/2019-08/26/content_5424780.htm .
• 15. Wang J. The development trend and suggestion of pharmaceutical contract manufacturing under China “MAH” system . (Master’s Thesis), Shanghai Jiao Tong University.2017.
• 18. Ekelund Robert B. Jr.. A History of Economic Theory and Method. Beijing: China Renmin University Press;2009.
• 19. Davis Lance E., North Douglass C. Institutional change and American economic growth. Beijing: Peking University Press;2004.
• 20. Lu XX, Zhu QL. New institutional economics. Beijing: Peking University Press;2009.
• PubMed/NCBI
• 37. Chen H. The influence of science and technology policy on the innovation output of Chinese high-tech industry—Empirical research based on the difference in difference model .(Master’s Thesis), Northeast Normal University.2021.