a research normally ends with some

Where does research begin and end, or where should it?

For some time now, I have been pondering the notion of how we need to consider changing where research begins and ends. Traditionally research starts with writing a grant and usually finishes with publishing a paper – it’s funny, I have never thought about the way we currently bookend our research with a bunch of writing to justify ourselves! But, with all the discussion around translation and research impact the goal posts are moving. Knowledge translation is the underpinning process that gives us a pathway to research impact, with impact being the ultimate thing we are trying to create and if you ask funders….measure!

Is translation an add on to the research process or should it be embedded? The answer is yes, no and maybe!

Simply put, if we are to create impact, then we must do research that is relevant, meets a user’s needs and is delivered in a way that is appropriate to those needs and understanding. There are exceptions to this rule – cue happy sigh of the basic scientists. Indeed, in some cases, the use or user may not be known until the end of the process. However, KT still applies, only it will look a little different. You may recall the differences in integrated and end-of-grant KT as shown below.

Integrated KT

Involves collaboration between researchers and knowledge users at every stage of the research process – from shaping the research question, to interpreting the results, to disseminating the research findings into practice. This co-production of research increases the likelihood that the results of a project will be relevant to end-users, thereby improving the possibility of uptake and application

End-of-grant KT

The dissemination of findings generated from research once a project is completed, depending on the extent to which there are mature findings appropriate for dissemination. Researchers who undertake traditional dissemination activities such as publishing in peer-reviewed journals and presenting their research at conferences and workshops are engaging in end-of-grant knowledge translation.

http://www.cihr-irsc.gc.ca/e/45321.html#a3

Relationships, interactions and dialogue between multidisciplinary research groups and stakeholder groups create buy-in are crucial for successful KT. These interactions increase the likely uptake of your knowledge in some form, be it to change behaviour, guide discussions, or for more tangible changes. The inclusion of a variety of stakeholders, from policymakers, planners and managers, private sector industries and consumer groups within different areas of health care and health policy, helps to shape questions and solutions while representing the interests of research user groups ( Sudsawad ). Additionally, the engagement between researchers and research user groups facilitates an understanding of each other environments that help the utilisation process ( Mitton ). This evidence points to the importance of early engagement and a more integrated process of translation, hence altering where the research process begins and ends.

Other types of research may not need consistent engagement, but I would argue that a well thought out KT plan would add significant value to any research endeavour, and this would be considered at the research development stage, hence the “maybe”. Additionally, we should be finding additional ways of sharing new knowledge from our research, be it sharing with an academic audience or a non-academic audience; this would require moving the finish line just a little.

I first started to ponder this question at the Medical Research Future Fund (MRFF) Public Forum held in Melbourne. It was explicitly stated that the fund was to be used to fund research and not translation. However, I think that translation should be considered an integral part of the process of research,l and how we share that work with others. The MRFF has an opportunity to move the goal posts and change where research begins and ends with its funding process, particularly if it is to meet its strategic objectives, all of which are pointed heavily toward translation and impact.

Writing up a Research Report

Stefan Hunziker 3 &
Michael Blankenagel 3
First Online: 04 January 2024

302 Accesses

A research report is one big argument about how and why you came up with your conclusions. To make it a convincing argument, a typical guiding structure has developed. In the different chapters, there are distinct issues that need to be addressed to explain to the reader why your conclusions are valid. The governing principle for writing the report is full disclosure: to explain everything and ensure replicability by another researcher.

This is a preview of subscription content, log in via an institution .

Buying options

Available as PDF
Read on any device
Instant download
Own it forever
Available as EPUB and PDF
Compact, lightweight edition
Dispatched in 3 to 5 business days
Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Barros, L. O. (2016). The only academic phrasebook you’ll ever need . Createspace Independent Publishing Platform.

Google Scholar

Field, A. (2016). An adventure in statistics. The reality enigma . SAGE.

Field, A. (2020). Discovering statistics using IBM SPSS statistics (5th ed.). SAGE.

Früh, M., Keimer, I., & Blankenagel, M. (2019). The impact of Balanced Scorecard excellence on shareholder returns. IFZ Working Paper No. 0003/2019. https://zenodo.org/record/2571603#.YMDUafkzZaQ . Accessed: 9 June 2021.

Pearl, J., & Mackenzie, D. (2018). The book of why: The new science of cause and effect. Basic Books.

Yin, R. K. (2013). Case study research: Design and methods (5th ed.). SAGE.

Download references

Author information

Authors and affiliations.

Wirtschaft/IFZ, Campus Zug-Rotkreuz, Hochschule Luzern, Zug-Rotkreuz, Zug, Switzerland

Stefan Hunziker & Michael Blankenagel

You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Stefan Hunziker .

Rights and permissions

Reprints and permissions

Copyright information

About this chapter

Cite this chapter.

Hunziker, S., Blankenagel, M. (2024). Writing up a Research Report. In: Research Design in Business and Management. Springer Gabler, Wiesbaden. https://doi.org/10.1007/978-3-658-42739-9_4

Download citation

DOI : https://doi.org/10.1007/978-3-658-42739-9_4

Published : 04 January 2024

Publisher Name : Springer Gabler, Wiesbaden

Print ISBN : 978-3-658-42738-2

Online ISBN : 978-3-658-42739-9

eBook Packages : Business and Management Business and Management (R0)

Share this chapter

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

Publish with us

Policies and ethics

Find a journal
Track your research

Princeton Correspondents on Undergraduate Research

Flexibility of Research: What to Do When You Feel like You’ve Hit a Dead End

Research can be a truly thrilling experience–interesting data, new findings, surprising collections. However, research can also be incredibly frustrating, namely when you feel like your work isn’t going anywhere.

If you’ve ever been really excited about a topic, done a load of research, and still found that you aren’t making forward progress, this post is for you. I’m talking about hitting tough obstacles in your research–walls you can’t seem to get over–reaching what seems like a dead end.

I’ve been there before (oh more times than I would like to admit), and what I’ve found is that normally, this frustrating lack of a solution is not an indicator that your research is ‘wrong’ or isn’t worthwhile. In fact, it might actually mean that there is another question buried in your topic that needs to be addressed primarily.

I learned this after experiencing a dead–end–feeling just last year as I wrote my R3 (that’s right! flashback to everyone’s favorite: Writing Sem!*). My original plan had been to interpret a modern still life––picture the abstract still lifes of Stuart Davis and Arthur Dove––using symbols and traditions from realistic still lifes––imagine a Renaissance painting of a fruit bowl and flowers. Specifically, I had selected Morris Grave’s August Still Life which was currently on display in the Princeton University Art Museum in the exhibit The Artist Sees Differently: Modern Still Lifes from the Phillips Collection . I hoped to use a change–over–time comparison between modern and Renaissance still lifes to discover how modern still lifes may or may not showcase the unique characteristics of the modern age.

It was a great plan, but as I dug further into my research, I quickly hit a roadblock…

It was a great plan, but as I dug further into my research, I quickly hit a roadblock: art historians don’t interpret still lifes.

Seriously! I had pulled almost every book in the Princeton library system on still life paintings; they all were focused only on the formal elements of the works (such as compositional balance, technique, color, etc)! I could barely find any instruction on how to interpret a still life painting or any evidence that any one ever had!

I thought back to the exhibit I had picked my painting from; it was entire exhibit of still life paintings. I thought to myself ‘ Well, maybe the curator of this exhibit can tell me a little bit more about still lifes! He designed a whole exhibition of them! Surely there was some method to his interpretation of each work that helped him arrange the works.’

I arranged to meet with T. Butron Thurber, the Associate Director for Collections and Exhibitions who had curated the exhibit I was curious about. We got together and discussed the genre. Turns out he had organized the gallery based on formal relationships and style because he too had worries about interpreting still lifes.

This was initially extremely disappointing, but now I had a new motive for my essay:

Art historians, including Mr. Thurber, were choosing not to engage in the interpretation of still lifes. I needed to explore why this is the case. I realized the problem wasn’t about art historians not wanting to interpret still lifes. It was about a lack of a method. As Thurber expressed, art historians didn’t trust that there was a method that wasn’t over interpretive (making up meaning out of little evidence). By studying this why, I was better able to develop a method for interpreting these overlooked works of art that––in my opinion––addressed the concerns of art historians like Mr.Thurber.

My experience should remind us that research is a flexible process; you must follow where your evidence takes you. Research is variable; you can not control the answers you will receive. Nonetheless, new information can always push you forward. If you are hitting a roadblock in your research, maybe that doesn’t mean you need to pick a new topic but rather there is a different question, hidden in your topic, you need to explore first!

— Raya Ward, Natural Sciences Correspondent

* If you haven’t taken writing sem yet, your R3 is your final research paper in the class where you have almost full flexibility to pick your topic.

Organizing Your Social Sciences Research Paper

The Research Problem/Question
Purpose of Guide
Design Flaws to Avoid
Independent and Dependent Variables
Glossary of Research Terms
Reading Research Effectively
Narrowing a Topic Idea
Broadening a Topic Idea
Extending the Timeliness of a Topic Idea
Academic Writing Style
Applying Critical Thinking
Choosing a Title
Making an Outline
Paragraph Development
Research Process Video Series
Executive Summary
The C.A.R.S. Model
Background Information
Theoretical Framework
Citation Tracking
Content Alert Services
Evaluating Sources
Primary Sources
Secondary Sources
Tiertiary Sources
Scholarly vs. Popular Publications
Qualitative Methods
Quantitative Methods
Insiderness
Using Non-Textual Elements
Limitations of the Study
Common Grammar Mistakes
Writing Concisely
Avoiding Plagiarism
Footnotes or Endnotes?
Further Readings
Generative AI and Writing
USC Libraries Tutorials and Other Guides
Bibliography

A research problem is a definite or clear expression [statement] about an area of concern, a condition to be improved upon, a difficulty to be eliminated, or a troubling question that exists in scholarly literature, in theory, or within existing practice that points to a need for meaningful understanding and deliberate investigation. A research problem does not state how to do something, offer a vague or broad proposition, or present a value question. In the social and behavioral sciences, studies are most often framed around examining a problem that needs to be understood and resolved in order to improve society and the human condition.

Bryman, Alan. “The Research Question in Social Research: What is its Role?” International Journal of Social Research Methodology 10 (2007): 5-20; Guba, Egon G., and Yvonna S. Lincoln. “Competing Paradigms in Qualitative Research.” In Handbook of Qualitative Research . Norman K. Denzin and Yvonna S. Lincoln, editors. (Thousand Oaks, CA: Sage, 1994), pp. 105-117; Pardede, Parlindungan. “Identifying and Formulating the Research Problem." Research in ELT: Module 4 (October 2018): 1-13; Li, Yanmei, and Sumei Zhang. "Identifying the Research Problem." In Applied Research Methods in Urban and Regional Planning . (Cham, Switzerland: Springer International Publishing, 2022), pp. 13-21.

Importance of...

The purpose of a problem statement is to:

Introduce the reader to the importance of the topic being studied . The reader is oriented to the significance of the study.
Anchors the research questions, hypotheses, or assumptions to follow . It offers a concise statement about the purpose of your paper.
Place the topic into a particular context that defines the parameters of what is to be investigated.
Provide the framework for reporting the results and indicates what is probably necessary to conduct the study and explain how the findings will present this information.

In the social sciences, the research problem establishes the means by which you must answer the "So What?" question. This declarative question refers to a research problem surviving the relevancy test [the quality of a measurement procedure that provides repeatability and accuracy]. Note that answering the "So What?" question requires a commitment on your part to not only show that you have reviewed the literature, but that you have thoroughly considered the significance of the research problem and its implications applied to creating new knowledge and understanding or informing practice.

To survive the "So What" question, problem statements should possess the following attributes:

Clarity and precision [a well-written statement does not make sweeping generalizations and irresponsible pronouncements; it also does include unspecific determinates like "very" or "giant"],
Demonstrate a researchable topic or issue [i.e., feasibility of conducting the study is based upon access to information that can be effectively acquired, gathered, interpreted, synthesized, and understood],
Identification of what would be studied, while avoiding the use of value-laden words and terms,
Identification of an overarching question or small set of questions accompanied by key factors or variables,
Identification of key concepts and terms,
Articulation of the study's conceptual boundaries or parameters or limitations,
Some generalizability in regards to applicability and bringing results into general use,
Conveyance of the study's importance, benefits, and justification [i.e., regardless of the type of research, it is important to demonstrate that the research is not trivial],
Does not have unnecessary jargon or overly complex sentence constructions; and,
Conveyance of more than the mere gathering of descriptive data providing only a snapshot of the issue or phenomenon under investigation.

Bryman, Alan. “The Research Question in Social Research: What is its Role?” International Journal of Social Research Methodology 10 (2007): 5-20; Brown, Perry J., Allen Dyer, and Ross S. Whaley. "Recreation Research—So What?" Journal of Leisure Research 5 (1973): 16-24; Castellanos, Susie. Critical Writing and Thinking. The Writing Center. Dean of the College. Brown University; Ellis, Timothy J. and Yair Levy Nova. "Framework of Problem-Based Research: A Guide for Novice Researchers on the Development of a Research-Worthy Problem." Informing Science: the International Journal of an Emerging Transdiscipline 11 (2008); Thesis and Purpose Statements. The Writer’s Handbook. Writing Center. University of Wisconsin, Madison; Thesis Statements. The Writing Center. University of North Carolina; Tips and Examples for Writing Thesis Statements. The Writing Lab and The OWL. Purdue University; Selwyn, Neil. "‘So What?’…A Question that Every Journal Article Needs to Answer." Learning, Media, and Technology 39 (2014): 1-5; Shoket, Mohd. "Research Problem: Identification and Formulation." International Journal of Research 1 (May 2014): 512-518.

Structure and Writing Style

I. Types and Content

There are four general conceptualizations of a research problem in the social sciences:

Casuist Research Problem -- this type of problem relates to the determination of right and wrong in questions of conduct or conscience by analyzing moral dilemmas through the application of general rules and the careful distinction of special cases.
Difference Research Problem -- typically asks the question, “Is there a difference between two or more groups or treatments?” This type of problem statement is used when the researcher compares or contrasts two or more phenomena. This a common approach to defining a problem in the clinical social sciences or behavioral sciences.
Descriptive Research Problem -- typically asks the question, "what is...?" with the underlying purpose to describe the significance of a situation, state, or existence of a specific phenomenon. This problem is often associated with revealing hidden or understudied issues.
Relational Research Problem -- suggests a relationship of some sort between two or more variables to be investigated. The underlying purpose is to investigate specific qualities or characteristics that may be connected in some way.

A problem statement in the social sciences should contain :

A lead-in that helps ensure the reader will maintain interest over the study,
A declaration of originality [e.g., mentioning a knowledge void or a lack of clarity about a topic that will be revealed in the literature review of prior research],
An indication of the central focus of the study [establishing the boundaries of analysis], and
An explanation of the study's significance or the benefits to be derived from investigating the research problem.

NOTE : A statement describing the research problem of your paper should not be viewed as a thesis statement that you may be familiar with from high school. Given the content listed above, a description of the research problem is usually a short paragraph in length.

II. Sources of Problems for Investigation

The identification of a problem to study can be challenging, not because there's a lack of issues that could be investigated, but due to the challenge of formulating an academically relevant and researchable problem which is unique and does not simply duplicate the work of others. To facilitate how you might select a problem from which to build a research study, consider these sources of inspiration:

Deductions from Theory This relates to deductions made from social philosophy or generalizations embodied in life and in society that the researcher is familiar with. These deductions from human behavior are then placed within an empirical frame of reference through research. From a theory, the researcher can formulate a research problem or hypothesis stating the expected findings in certain empirical situations. The research asks the question: “What relationship between variables will be observed if theory aptly summarizes the state of affairs?” One can then design and carry out a systematic investigation to assess whether empirical data confirm or reject the hypothesis, and hence, the theory.

Interdisciplinary Perspectives Identifying a problem that forms the basis for a research study can come from academic movements and scholarship originating in disciplines outside of your primary area of study. This can be an intellectually stimulating exercise. A review of pertinent literature should include examining research from related disciplines that can reveal new avenues of exploration and analysis. An interdisciplinary approach to selecting a research problem offers an opportunity to construct a more comprehensive understanding of a very complex issue that any single discipline may be able to provide.

Interviewing Practitioners The identification of research problems about particular topics can arise from formal interviews or informal discussions with practitioners who provide insight into new directions for future research and how to make research findings more relevant to practice. Discussions with experts in the field, such as, teachers, social workers, health care providers, lawyers, business leaders, etc., offers the chance to identify practical, “real world” problems that may be understudied or ignored within academic circles. This approach also provides some practical knowledge which may help in the process of designing and conducting your study.

Personal Experience Don't undervalue your everyday experiences or encounters as worthwhile problems for investigation. Think critically about your own experiences and/or frustrations with an issue facing society or related to your community, your neighborhood, your family, or your personal life. This can be derived, for example, from deliberate observations of certain relationships for which there is no clear explanation or witnessing an event that appears harmful to a person or group or that is out of the ordinary.

Relevant Literature The selection of a research problem can be derived from a thorough review of pertinent research associated with your overall area of interest. This may reveal where gaps exist in understanding a topic or where an issue has been understudied. Research may be conducted to: 1) fill such gaps in knowledge; 2) evaluate if the methodologies employed in prior studies can be adapted to solve other problems; or, 3) determine if a similar study could be conducted in a different subject area or applied in a different context or to different study sample [i.e., different setting or different group of people]. Also, authors frequently conclude their studies by noting implications for further research; read the conclusion of pertinent studies because statements about further research can be a valuable source for identifying new problems to investigate. The fact that a researcher has identified a topic worthy of further exploration validates the fact it is worth pursuing.

III. What Makes a Good Research Statement?

A good problem statement begins by introducing the broad area in which your research is centered, gradually leading the reader to the more specific issues you are investigating. The statement need not be lengthy, but a good research problem should incorporate the following features:

1. Compelling Topic The problem chosen should be one that motivates you to address it but simple curiosity is not a good enough reason to pursue a research study because this does not indicate significance. The problem that you choose to explore must be important to you, but it must also be viewed as important by your readers and to a the larger academic and/or social community that could be impacted by the results of your study. 2. Supports Multiple Perspectives The problem must be phrased in a way that avoids dichotomies and instead supports the generation and exploration of multiple perspectives. A general rule of thumb in the social sciences is that a good research problem is one that would generate a variety of viewpoints from a composite audience made up of reasonable people. 3. Researchability This isn't a real word but it represents an important aspect of creating a good research statement. It seems a bit obvious, but you don't want to find yourself in the midst of investigating a complex research project and realize that you don't have enough prior research to draw from for your analysis. There's nothing inherently wrong with original research, but you must choose research problems that can be supported, in some way, by the resources available to you. If you are not sure if something is researchable, don't assume that it isn't if you don't find information right away--seek help from a librarian !

NOTE: Do not confuse a research problem with a research topic. A topic is something to read and obtain information about, whereas a problem is something to be solved or framed as a question raised for inquiry, consideration, or solution, or explained as a source of perplexity, distress, or vexation. In short, a research topic is something to be understood; a research problem is something that needs to be investigated.

IV. Asking Analytical Questions about the Research Problem

Research problems in the social and behavioral sciences are often analyzed around critical questions that must be investigated. These questions can be explicitly listed in the introduction [i.e., "This study addresses three research questions about women's psychological recovery from domestic abuse in multi-generational home settings..."], or, the questions are implied in the text as specific areas of study related to the research problem. Explicitly listing your research questions at the end of your introduction can help in designing a clear roadmap of what you plan to address in your study, whereas, implicitly integrating them into the text of the introduction allows you to create a more compelling narrative around the key issues under investigation. Either approach is appropriate.

The number of questions you attempt to address should be based on the complexity of the problem you are investigating and what areas of inquiry you find most critical to study. Practical considerations, such as, the length of the paper you are writing or the availability of resources to analyze the issue can also factor in how many questions to ask. In general, however, there should be no more than four research questions underpinning a single research problem.

Given this, well-developed analytical questions can focus on any of the following:

Highlights a genuine dilemma, area of ambiguity, or point of confusion about a topic open to interpretation by your readers;
Yields an answer that is unexpected and not obvious rather than inevitable and self-evident;
Provokes meaningful thought or discussion;
Raises the visibility of the key ideas or concepts that may be understudied or hidden;
Suggests the need for complex analysis or argument rather than a basic description or summary; and,
Offers a specific path of inquiry that avoids eliciting generalizations about the problem.

NOTE: Questions of how and why concerning a research problem often require more analysis than questions about who, what, where, and when. You should still ask yourself these latter questions, however. Thinking introspectively about the who, what, where, and when of a research problem can help ensure that you have thoroughly considered all aspects of the problem under investigation and helps define the scope of the study in relation to the problem.

V. Mistakes to Avoid

Beware of circular reasoning! Do not state the research problem as simply the absence of the thing you are suggesting. For example, if you propose the following, "The problem in this community is that there is no hospital," this only leads to a research problem where:

The need is for a hospital
The objective is to create a hospital
The method is to plan for building a hospital, and
The evaluation is to measure if there is a hospital or not.

This is an example of a research problem that fails the "So What?" test . In this example, the problem does not reveal the relevance of why you are investigating the fact there is no hospital in the community [e.g., perhaps there's a hospital in the community ten miles away]; it does not elucidate the significance of why one should study the fact there is no hospital in the community [e.g., that hospital in the community ten miles away has no emergency room]; the research problem does not offer an intellectual pathway towards adding new knowledge or clarifying prior knowledge [e.g., the county in which there is no hospital already conducted a study about the need for a hospital, but it was conducted ten years ago]; and, the problem does not offer meaningful outcomes that lead to recommendations that can be generalized for other situations or that could suggest areas for further research [e.g., the challenges of building a new hospital serves as a case study for other communities].

Alvesson, Mats and Jörgen Sandberg. “Generating Research Questions Through Problematization.” Academy of Management Review 36 (April 2011): 247-271 ; Choosing and Refining Topics. Writing@CSU. Colorado State University; D'Souza, Victor S. "Use of Induction and Deduction in Research in Social Sciences: An Illustration." Journal of the Indian Law Institute 24 (1982): 655-661; Ellis, Timothy J. and Yair Levy Nova. "Framework of Problem-Based Research: A Guide for Novice Researchers on the Development of a Research-Worthy Problem." Informing Science: the International Journal of an Emerging Transdiscipline 11 (2008); How to Write a Research Question. The Writing Center. George Mason University; Invention: Developing a Thesis Statement. The Reading/Writing Center. Hunter College; Problem Statements PowerPoint Presentation. The Writing Lab and The OWL. Purdue University; Procter, Margaret. Using Thesis Statements. University College Writing Centre. University of Toronto; Shoket, Mohd. "Research Problem: Identification and Formulation." International Journal of Research 1 (May 2014): 512-518; Trochim, William M.K. Problem Formulation. Research Methods Knowledge Base. 2006; Thesis and Purpose Statements. The Writer’s Handbook. Writing Center. University of Wisconsin, Madison; Thesis Statements. The Writing Center. University of North Carolina; Tips and Examples for Writing Thesis Statements. The Writing Lab and The OWL. Purdue University; Pardede, Parlindungan. “Identifying and Formulating the Research Problem." Research in ELT: Module 4 (October 2018): 1-13; Walk, Kerry. Asking an Analytical Question. [Class handout or worksheet]. Princeton University; White, Patrick. Developing Research Questions: A Guide for Social Scientists . New York: Palgrave McMillan, 2009; Li, Yanmei, and Sumei Zhang. "Identifying the Research Problem." In Applied Research Methods in Urban and Regional Planning . (Cham, Switzerland: Springer International Publishing, 2022), pp. 13-21.

<< Previous: Background Information
Next: Theoretical Framework >>
Last Updated: Apr 19, 2024 11:16 AM
URL: https://libguides.usc.edu/writingguide

6 Moments of Research “Failure” and How to Deal With Them
Coates Library Blog

“The artist, perhaps more than most people, inhabits failure, degrees of failure and accommodation and compromise: but the terms of his failure are generally secret” observes Joyce Carol Oates in her essay, “Notes on Failure” (231).

Substitute “researcher” for “artist” and Oates’ description still rings true. Yet while the failures and secrets of the research process are par for the course, we often overlook this fact and think that our own particular failures are unique and even shamefully so.

For this reason, for all the summer researchers taking on bigger projects than they have experienced in their classes so far, for all the incoming first-years who will soon receive more rigorous research assignments than they have previously encountered, I hope this brief catalog of common research “failures” might, if nothing else, do away with some of the unnecessary secrecy.

6 Moments of Research Failure and how to deal with them

1. Procrastination

Having trouble getting started can feel like failure. If this is you, you may find this NYTimes article helpful. Journalist Charlotte Lieberman interviews several psychology researchers who describe procrastination not as a task or productivity problem, but an emotional one.

If you are struggling to get started, it might be worth taking a minute to ask yourself what exactly you’re afraid to face. Are you unsure of what you’re supposed to do and scared to ask? Worried you’re not smart enough to do the research? Bored or uninterested and therefore avoiding the fact that you don’t like whatever it is you now have to spend the whole summer working on?

Facing these feelings or even talking to your professor about them may help you break through your slump.

2. Finding sources

In doing library or archival research, sometimes it can feel like a failure if the sources you hoped to find don’t immediately appear. In this case, there is one very easy step you can take: meet with a librarian to see if your assessment of the situation is true!

Of course, it’s also possible that what you wanted to find does not exist. In this case, a librarian or professor can still help, by confirming the gap or absence you have noted, and also in helping you figure out what to do with that information (hint: the lack of information/attention on a topic is itself a finding, and as such a starting point, not an ending).

3. Staying focused

If you are anything like me or a number of other Trinity students, this is a common challenge. I often cast my research net wider and wider until it becomes impossible to pull in and sort through. For me, one way to mediate this tendency is to involve others in my process.

Try talking through your ideas and questions with someone else—a friend, professor, or librarian. See if the connections and project scope you see in your mind actually make sense when you say them out loud. Just the act of talking through your ideas may help you make sense of them yourself! Alternatively, talking to someone else can also help you determine whether different components of the project could stand separate projects in their own right.

Writing can be useful at this stage, too. Sometimes the urge to keep researching is really about something else—like avoiding writing and committing your thoughts to paper.

4. Having something to say

This, I think, can be an especially secretive secret. Who wants to admit they don’t have anything compelling to say about their research subject?

However, especially in assigned research or research under a time constraint, you may not always have an immediate or clear response to your research. Or perhaps your findings are not as striking as you hoped, or your hypothesis is not confirmed.

Remember, even the lack of a dramatic finding is still a finding; your research is not, in the end, only about you. It’s also about the community and conversation around your subject. What may seem like a small contribution to you is still part of shaping that community and future research.

5. Organizing and documenting

The failure to organize your research—the process, sources, methods, etc.—can sneak up on you. At least it does for me!

It might be small things, like not writing down the exact way you came up with a particular result, or getting a little sloppy with your citations. And yet, these little details matter. They are what gives your work polish and credibility.

You don’t have to be a “Type A” personality in order to become more organized. Rather, think about organization in research as an ongoing practice. By paying attention to where you’ve missed the mark this time, you can pinpoint small ways to improve the next time.

6. Sharing your work

Usually in the form of writing or presentation, this stage can be laden with anxiety. It also raises the possibility of public failure: what if you submit your work and it’s rejected? What if you bomb your presentation, or you don’t get the grade you hoped for on your paper?

It is difficult not to interpret rejection or an undesirable grade as failure. After all, if success is about meeting a specific expectation, and you do not meet it—that is one definition of failure. And yet, the story you tell yourself about that failure may do more damage than the failure itself.

My own story

Especially when the stakes are high, it’s easy to interpret failure as an ending. As I learned, though, it isn’t. While some failures may indeed mark the end of things going “the way they’re supposed to” or “according to plan,” they are never the final word. In my experience, and in the experiences of many students I’ve worked with on research projects, moments of doubt, fear, and failure often give way to later feelings of accomplishment, pride, and growth. In fact, this pattern is so common that information science researchers have documented it.

Engaging in research will no doubt bring you face to face with some form of failure. But you don’t have to live in that experience alone, in secret. Failure is a normal part of research, learning, and growing as a person.

At Trinity, there are always people ready to help you figure out how to get to the next part of the story. Failure is not an ending.

Kuhlthau, Carol Collier. “Information Search Process.” Rutgers.edu . 5 June 2019, http://wp.comminfo.rutgers.edu/ckuhlthau/information-search-process/

Lieberman, Charlotte. “Why You Procrastinate (It Has Nothing to Do With Self-Control).” New York Times , 25 March 2019, https://www.nytimes.com/2019/03/25/smarter-living/why-you- procrastinate-it-has-nothing-to-do-with-self-control.html

Oates, Joyce Carol. “Notes on Failure.” The Hudson Review , vol. 35, no. 2, 1982, pp. 231–245. JSTOR , www.jstor.org/stable/3850783 .

Self-Defeating Behavior

Self-defeating behavior definition.

For social psychologists, a self-defeating behavior is any behavior that normally ends up with a result that is something the person doing the behavior doesn’t want to happen. If you are trying to accomplish some goal, and something you do makes it less likely that you will reach that goal, then that is a self-defeating behavior. If the goal is reached, but the ways you used to reach the goal cause more bad things to happen than the positive things you get from achieving the goal, that is also self-defeating behavior. Social psychologists have been studying self-defeating behaviors for at least 30 years. And although they have identified several things that seem to lead to self-defeating behaviors, much more can be learned about what self-defeating behaviors have in common, and how to get people to reduce the impact of these behaviors in their lives.

Self-Defeating Behavior Background and History

Academic Writing, Editing, Proofreading, And Problem Solving Services

Get 10% off with 24start discount code.

The group revising the DSM in the 1980s wanted to include a disorder where people showed “a pervasive pattern of self-defeating behaviors.” Some people didn’t want this to be included because they said that there wasn’t enough research to show that a disorder like this really existed; some people didn’t want it to be included because they said that the behaviors that supposedly made up the self-defeating personality disorder were really parts of other personality disorders; and finally, some people didn’t want it to be included because they were afraid that the disorder would be biased against women and would excuse spouse abusers, blaming their victims by claiming that the victims had self-defeating personality disorder.

In the edition of the DSM published in 1987 (called the DSM-III-R), self-defeating personality disorder was included in an appendix and was not considered an official diagnosis. More recent editions of the DSM do not mention the self-defeating personality disorder at all.

Even though social psychologists were inspired by this controversy, they are interested in studying behaviors of normal people, not those of people who are mentally ill. Although some psychiatrists believe that all humans are driven to harm themselves, most people are not motivated in this way. Most humans are interested in accomplishing their goals, not in harming themselves.

Types of Self-Defeating Behavior

Social psychologists have divided self-defeating behaviors into two types. One type is called counterproductive behaviors. A counterproductive behavior happens when people try to get something they want, but the way they try to get it ends up not being a good one. One type of counterproductive behavior occurs when people persevere at something beyond the time that it is realistic for them to achieve the desired outcome. For example, students taking a class, and doing very poorly, sometimes refuse to drop the class. They think that if they stick it out, they will be able to pull their grades up and pass the class. But, it may just be too late for some, or they may not have the ability to really pass the class. Most students’ goals are to get a degree with as high a grade point average as possible, so refusing to drop the class is a self-defeating behavior. Counterproductive behaviors usually happen because the person has a wrong idea either about himself or herself or about the situation the person is in. The students have an incorrect idea about their own abilities; they think they can succeed, but they can’t.

The second type of self-defeating behavior is called trade-offs. We make trade-offs in our behavior all the time. For example, you may decide not to go to a party so you can study for an exam. This is a trade-off: You are trading the fun you will have at the party for the benefit you will get from studying (a better grade).

This example of a trade-off is not self-defeating. You are probably going to come out a winner: The benefit of studying will, in the end, outweigh the benefit of going to the party. But, some kinds of trade-offs are self-defeating: The cost that you have to accept is greater than the benefit that you end up getting. One example is neglecting to take care of yourself physically. When people don’t exercise, go to the dentist, or follow the doctor’s orders, they are risking their health to either avoid some short-term pain or discomfort (such as the discomfort of exercise or the anxiety that the dentist causes).

Another example of a self-defeating trade-off is called self-handicapping. Self-handicapping is when people do something to make their success on a task less likely to happen. People do this so that they will have a built-in excuse if they fail. For example, students may get drunk the night before a big exam. If they do poorly on the exam, they have a built in excuse: They didn’t study and they were hungover. This way they avoid thinking that they don’t have the ability to do well in the class.

Some common self-defeating behaviors represent a combination of counterproductive behaviors and trade-offs. Procrastination is a familiar example. When you think about why people procrastinate, you probably think about it as a trade-off. People want to do something more fun, or something that is less difficult, or something that allows them to grow or develop more, instead of the thing they are putting off. But, sometimes people explain why they procrastinate in another way: That they do better work if they wait until the last minute. If this is really the reason people procrastinate (instead of something people just say to justify their procrastination), then it is a counterproductive strategy; they believe that they will do better work if they wait until the last minute, but that is not usually the case. (Research shows that college students who procrastinate get worse grades, have more stress, and are more likely to get sick.)

Alcohol or drug abuse is another self-defeating behavior. Many people use alcohol and drugs responsibly, and do it to gain pleasure or pain relief. But for addicts, and in some situations for anyone, substance use is surely self-defeating. Substance use may be a trade-off: A person trades the costs of using drugs or alcohol (health risks, addiction, embarrassing or dangerous behavior, legal problems) for benefits (feeling good, not having to think about one’s inadequacies). Usually over the long run, however, the costs are much greater than the benefits.

Even suicide can be looked at as either a self-defeating trade-off or counterproductive behavior. People who commit suicide are trying to escape from negative things in their life. They are trading off the fear of death, and the good things in life, because they think the benefit of no longer feeling the way they do will be greater than what they are giving up. But, suicide can also be thought of as a counterproductive behavior. People may think that taking their life will allow them to reach a certain goal (not having problems).

Causes and Consequences of Self-Defeating Behavior

Causes of different self-defeating behaviors vary; however, most self-defeating behaviors have some things in common. People who engage in self-defeating behaviors often feel a threat to their egos or self-esteem; there is usually some element of bad mood involved in self-defeating behaviors. And, people who engage in self-defeating behaviors often focus on the short-term consequences of their behavior, and ignore or underestimate the long-term consequences.

Procrastination is an example that combines all three of these factors. One reason people procrastinate is that they are afraid that when they do the thing they are putting off, it will show that they are not as good or competent as they want to be or believe they are (threat to self). Also, people procrastinate because the thing they put off causes anxiety (a negative emotion). Finally, people who procrastinate are focusing on the short-term effects of their behavior (it will feel good right now to watch TV instead of do my homework), but they are ignoring the long-term consequences (if I put off my homework, either I’ll get an F or I will have to pull an all-nighter to get it done).

These three common causes are all related to each other. If you have a goal for yourself, or if other people expect certain things from you, and you fail or think you will fail to meet the goal, this is a threat to your self-esteem or ego. That will usually make you feel bad (negative mood). So, ego-threats make you have negative moods.

But, negative moods also can lead to ego threats. When people are in negative moods, they set higher standards or goals for themselves. So, this will make them more likely to fail. Here is a vicious cycle: Failing to meet your goals is a threat to your ego, which leads to negative emotion, which leads you to set higher standards, which makes you fail more. Negative moods also can lead you to think more about the immediate consequences of your actions, instead of the long-term consequences. This, too, can make people do something self-defeating.

References:

Baumeister, R. F. (1997). Esteem threat, self-regulatory breakdown, and emotional distress as factors in self-defeating behavior. Review of General Psychology, 1, 145-174.
Baumeister, R. F., & Scher, S. J. (1988). Self-defeating behavior patterns among normal individuals: Review and analysis of common self-destructive tendencies. Psychological Bulletin, 104, 3-22.
Curtis, R. C. (Ed.). (1989). Self-defeating behaviors: Experimental research, clinical impressions, and practical implications. New York: Plenum.
Fiester, S. J. (1995). Self-defeating personality disorder. In W. J. Livesley (Ed.), The DSM-IV personality disorders (pp. 341-358). New York: Guilford Press.
Widiger, T. A. (1995). Deletion of self-defeating and sadistic personality disorders. In W. J. Livesley (Ed.), The DSM-IV personality disorders (pp. 359-373). New York: Guilford Press.

Spartanburg Community College Library
SCC Research Guides

Evaluating Sources

Domain Endings

Domain endings are the end part of a URL (.com, .org, .edu, etc.). Sometimes the domain ending can give you a clue to a website's purpose. While domain endings can give you some ideas about a website, they should not be the only way you determine if a website is credible or not.

.com stands for commercial sites, but can really be anything. Below are some examples of .com sites you are probably familiar with:

Huffington Post: generally considered to have a liberal slant, but does have articles with legitimate information and credible sources.
NBC : official television network with information about news and current events; not always guaranteed to be accurate, but in general a credible .com website.
The Atlantic: considered to have a moderate worldview, their news articles are considered to be of a higher quality.
Biography : a website for the television channel Biography . Often a good source for newer figures who have not had time to be printed in more formal biographical publications.
Glamour : may have occasional articles that are relevant for specific topics. It depends on the context and the assignment needs. If a student is preparing a speech on a popular culture topic, Glamour may be an acceptable source of information.

.org websites should be organizations, but again, they can really be anything, since the purchase of .org domain names is not restricted. The Modern Language Association, American Cancer Society, and the American Welding Society are all examples of respected, well-known websites and are considered to be "the" organization that drives the discipline.

NPR : an example of a well-known, respected news organization.
Wikipedia : often students are allowed to use .org websites, but not Wikipedia . This creates a little conflict in the information relayed to students. While Wikipedia is a .org website, it can technically be edited by anyone, which is why it is not always the most credible source to use for an assignment.
Martin Luther King : sounds respectable. While not obvious at first glance, but if you look into the publisher of the website, you will find the group, Stormfront, is a white nationalist organization with a very biased opinion of Dr. King.
Institute for Historical Review : this site looks and sounds official. The article, "Context and Perspective in the 'Holocaust' Controversy," is published in a journal with the biography of the author provided. But what do we know about the organization? The journal is actually published by a well-known Holocaust denial group. And if you look at the author's credibility, you find that he is not a historian, but a professor of electrical engineering. To further raise suspicions, most of the citations in the article are to other works by the author.

.edu websites should contain credible materials, at least that is our expectation. The domain .edu is restricted for purchase, however, there are a few sites that have been grandfathered in (not legitimate schools), but the other general information to keep in mind is that schools often offer web space to students for student work. Many schools also offer a digital commons to share student and faculty work, that has not been peer-reviewed. The sites look professional, so it is important to understand what type of information you are looking at.

"Drug Shortages: The Problem of Inadequate Profits": this is a student paper located in the Harvard Digital Commons. The paper looks very similar to any other article you might locate in a database, meaning it may be difficult to determine this is another student's work.
"Researcher Dispels Myth of Dioxins and Plastic Water Bottles": This article on the Johns Hopkins website is written by a professor with a PhD in environmental science and works in the Department of Public Health at the university. There is a measure of credibility to this article, but it has not been formally published or peer-reviewed. You will need to consider whether the information is authoritative for your information need.
"Gender Bias in Microlending: Do Opposite Attract?": this example is a student thesis for a master's program. There should be some oversight by the student's professors, but the authority of the information would be dependent on how the student is using it. Is a master's thesis acceptable material for your assignment?

Miscellaneous Domain

Miscellaneous domain endings have a lot of variety, so it is important to look closely at the source itself, and not the domain ending on it's own:

.int: stands for international. The NATO website and the World Health Organization (WHO) are great examples of credible websites ending with this domain.
.uk: the country code for the United Kingdom. The official website for the UK government, Parliament , uses this domain.
.net: could be any type of website, like the Institute of War and Peace Reporting , a charitable foundation that reports on the safety and events occurring in countries in upheaval around the world.
<< Previous: Evaluation Criteria
Next: Evaluating Sources with Lateral Reading >>
What Makes a Credible Source?
Scholarly Sources vs. Popular Sources
Evaluation Criteria
Evaluating Sources with Lateral Reading
Evaluating Websites Exercises
Helpful Resources

Questions? Ask a Librarian

SCC Librarian and student working together

Last Updated: Jan 17, 2024 9:39 AM
URL: https://libguides.sccsc.edu/evaluatingsources

Giles Campus | 864.592.4764 | Toll Free 866.542.2779 | Contact Us

Info for Library Staff | Guide Search

Return to SCC Website

SUGGESTED TOPICS
The Magazine
Newsletters
Managing Yourself
Managing Teams
Work-life Balance
The Big Idea
Data & Visuals
Reading Lists
Case Selections
HBR Learning
Topic Feeds
Account Settings
Email Preferences

The Research-Backed Benefits of Daily Rituals

Michael I. Norton

A survey of more than 130 HBR readers asked how they use rituals to start their days, psych themselves up for stressful challenges, and transition when the workday is done.

While some may cringe at forced corporate rituals, research shows that personal and team rituals can actually benefit the way we work. The authors’ expertise on the topic over the past decade, plus a survey of nearly 140 HBR readers, explores the ways rituals can set us up for success before work, get us psyched up for important presentations, foster a strong team culture, and help us wind down at the end of the day.

“Give me a W ! Give me an A ! Give me an L ! Give me a squiggly! Give me an M ! Give me an A ! Give me an R ! Give me a T !”

Michael I. Norton is the Harold M. Brierley Professor of Business Administration at the Harvard Business School. He is the author of The Ritual Effect and co-author of Happy Money: The Science of Happier Spending . His research focuses on happiness, well-being, rituals, and inequality. See his faculty page here .

Partner Center

Numbers, Facts and Trends Shaping Your World

Read our research on:

Full Topic List

Regions & Countries

Publications
Our Methods
Short Reads
Tools & Resources

Read Our Research On:

What the data says about abortion in the U.S.

Pew Research Center has conducted many surveys about abortion over the years, providing a lens into Americans’ views on whether the procedure should be legal, among a host of other questions.

In a Center survey conducted nearly a year after the Supreme Court’s June 2022 decision that ended the constitutional right to abortion , 62% of U.S. adults said the practice should be legal in all or most cases, while 36% said it should be illegal in all or most cases. Another survey conducted a few months before the decision showed that relatively few Americans take an absolutist view on the issue .

Find answers to common questions about abortion in America, based on data from the Centers for Disease Control and Prevention (CDC) and the Guttmacher Institute, which have tracked these patterns for several decades:

How many abortions are there in the U.S. each year?

How has the number of abortions in the u.s. changed over time, what is the abortion rate among women in the u.s. how has it changed over time, what are the most common types of abortion, how many abortion providers are there in the u.s., and how has that number changed, what percentage of abortions are for women who live in a different state from the abortion provider, what are the demographics of women who have had abortions, when during pregnancy do most abortions occur, how often are there medical complications from abortion.

This compilation of data on abortion in the United States draws mainly from two sources: the Centers for Disease Control and Prevention (CDC) and the Guttmacher Institute, both of which have regularly compiled national abortion data for approximately half a century, and which collect their data in different ways.

The CDC data that is highlighted in this post comes from the agency’s “abortion surveillance” reports, which have been published annually since 1974 (and which have included data from 1969). Its figures from 1973 through 1996 include data from all 50 states, the District of Columbia and New York City – 52 “reporting areas” in all. Since 1997, the CDC’s totals have lacked data from some states (most notably California) for the years that those states did not report data to the agency. The four reporting areas that did not submit data to the CDC in 2021 – California, Maryland, New Hampshire and New Jersey – accounted for approximately 25% of all legal induced abortions in the U.S. in 2020, according to Guttmacher’s data. Most states, though, do have data in the reports, and the figures for the vast majority of them came from each state’s central health agency, while for some states, the figures came from hospitals and other medical facilities.

Discussion of CDC abortion data involving women’s state of residence, marital status, race, ethnicity, age, abortion history and the number of previous live births excludes the low share of abortions where that information was not supplied. Read the methodology for the CDC’s latest abortion surveillance report , which includes data from 2021, for more details. Previous reports can be found at stacks.cdc.gov by entering “abortion surveillance” into the search box.

For the numbers of deaths caused by induced abortions in 1963 and 1965, this analysis looks at reports by the then-U.S. Department of Health, Education and Welfare, a precursor to the Department of Health and Human Services. In computing those figures, we excluded abortions listed in the report under the categories “spontaneous or unspecified” or as “other.” (“Spontaneous abortion” is another way of referring to miscarriages.)

Guttmacher data in this post comes from national surveys of abortion providers that Guttmacher has conducted 19 times since 1973. Guttmacher compiles its figures after contacting every known provider of abortions – clinics, hospitals and physicians’ offices – in the country. It uses questionnaires and health department data, and it provides estimates for abortion providers that don’t respond to its inquiries. (In 2020, the last year for which it has released data on the number of abortions in the U.S., it used estimates for 12% of abortions.) For most of the 2000s, Guttmacher has conducted these national surveys every three years, each time getting abortion data for the prior two years. For each interim year, Guttmacher has calculated estimates based on trends from its own figures and from other data.

The latest full summary of Guttmacher data came in the institute’s report titled “Abortion Incidence and Service Availability in the United States, 2020.” It includes figures for 2020 and 2019 and estimates for 2018. The report includes a methods section.

In addition, this post uses data from StatPearls, an online health care resource, on complications from abortion.

An exact answer is hard to come by. The CDC and the Guttmacher Institute have each tried to measure this for around half a century, but they use different methods and publish different figures.

The last year for which the CDC reported a yearly national total for abortions is 2021. It found there were 625,978 abortions in the District of Columbia and the 46 states with available data that year, up from 597,355 in those states and D.C. in 2020. The corresponding figure for 2019 was 607,720.

The last year for which Guttmacher reported a yearly national total was 2020. It said there were 930,160 abortions that year in all 50 states and the District of Columbia, compared with 916,460 in 2019.

How the CDC gets its data: It compiles figures that are voluntarily reported by states’ central health agencies, including separate figures for New York City and the District of Columbia. Its latest totals do not include figures from California, Maryland, New Hampshire or New Jersey, which did not report data to the CDC. ( Read the methodology from the latest CDC report .)
How Guttmacher gets its data: It compiles its figures after contacting every known abortion provider – clinics, hospitals and physicians’ offices – in the country. It uses questionnaires and health department data, then provides estimates for abortion providers that don’t respond. Guttmacher’s figures are higher than the CDC’s in part because they include data (and in some instances, estimates) from all 50 states. ( Read the institute’s latest full report and methodology .)

While the Guttmacher Institute supports abortion rights, its empirical data on abortions in the U.S. has been widely cited by groups and publications across the political spectrum, including by a number of those that disagree with its positions .

These estimates from Guttmacher and the CDC are results of multiyear efforts to collect data on abortion across the U.S. Last year, Guttmacher also began publishing less precise estimates every few months , based on a much smaller sample of providers.

The figures reported by these organizations include only legal induced abortions conducted by clinics, hospitals or physicians’ offices, or those that make use of abortion pills dispensed from certified facilities such as clinics or physicians’ offices. They do not account for the use of abortion pills that were obtained outside of clinical settings .

(Back to top)

A line chart showing the changing number of legal abortions in the U.S. since the 1970s.

The annual number of U.S. abortions rose for years after Roe v. Wade legalized the procedure in 1973, reaching its highest levels around the late 1980s and early 1990s, according to both the CDC and Guttmacher. Since then, abortions have generally decreased at what a CDC analysis called “a slow yet steady pace.”

Guttmacher says the number of abortions occurring in the U.S. in 2020 was 40% lower than it was in 1991. According to the CDC, the number was 36% lower in 2021 than in 1991, looking just at the District of Columbia and the 46 states that reported both of those years.

(The corresponding line graph shows the long-term trend in the number of legal abortions reported by both organizations. To allow for consistent comparisons over time, the CDC figures in the chart have been adjusted to ensure that the same states are counted from one year to the next. Using that approach, the CDC figure for 2021 is 622,108 legal abortions.)

There have been occasional breaks in this long-term pattern of decline – during the middle of the first decade of the 2000s, and then again in the late 2010s. The CDC reported modest 1% and 2% increases in abortions in 2018 and 2019, and then, after a 2% decrease in 2020, a 5% increase in 2021. Guttmacher reported an 8% increase over the three-year period from 2017 to 2020.

As noted above, these figures do not include abortions that use pills obtained outside of clinical settings.

Guttmacher says that in 2020 there were 14.4 abortions in the U.S. per 1,000 women ages 15 to 44. Its data shows that the rate of abortions among women has generally been declining in the U.S. since 1981, when it reported there were 29.3 abortions per 1,000 women in that age range.

The CDC says that in 2021, there were 11.6 abortions in the U.S. per 1,000 women ages 15 to 44. (That figure excludes data from California, the District of Columbia, Maryland, New Hampshire and New Jersey.) Like Guttmacher’s data, the CDC’s figures also suggest a general decline in the abortion rate over time. In 1980, when the CDC reported on all 50 states and D.C., it said there were 25 abortions per 1,000 women ages 15 to 44.

That said, both Guttmacher and the CDC say there were slight increases in the rate of abortions during the late 2010s and early 2020s. Guttmacher says the abortion rate per 1,000 women ages 15 to 44 rose from 13.5 in 2017 to 14.4 in 2020. The CDC says it rose from 11.2 per 1,000 in 2017 to 11.4 in 2019, before falling back to 11.1 in 2020 and then rising again to 11.6 in 2021. (The CDC’s figures for those years exclude data from California, D.C., Maryland, New Hampshire and New Jersey.)

The CDC broadly divides abortions into two categories: surgical abortions and medication abortions, which involve pills. Since the Food and Drug Administration first approved abortion pills in 2000, their use has increased over time as a share of abortions nationally, according to both the CDC and Guttmacher.

The majority of abortions in the U.S. now involve pills, according to both the CDC and Guttmacher. The CDC says 56% of U.S. abortions in 2021 involved pills, up from 53% in 2020 and 44% in 2019. Its figures for 2021 include the District of Columbia and 44 states that provided this data; its figures for 2020 include D.C. and 44 states (though not all of the same states as in 2021), and its figures for 2019 include D.C. and 45 states.

Guttmacher, which measures this every three years, says 53% of U.S. abortions involved pills in 2020, up from 39% in 2017.

Two pills commonly used together for medication abortions are mifepristone, which, taken first, blocks hormones that support a pregnancy, and misoprostol, which then causes the uterus to empty. According to the FDA, medication abortions are safe until 10 weeks into pregnancy.

Surgical abortions conducted during the first trimester of pregnancy typically use a suction process, while the relatively few surgical abortions that occur during the second trimester of a pregnancy typically use a process called dilation and evacuation, according to the UCLA School of Medicine.

In 2020, there were 1,603 facilities in the U.S. that provided abortions, according to Guttmacher . This included 807 clinics, 530 hospitals and 266 physicians’ offices.

A horizontal stacked bar chart showing the total number of abortion providers down since 1982.

While clinics make up half of the facilities that provide abortions, they are the sites where the vast majority (96%) of abortions are administered, either through procedures or the distribution of pills, according to Guttmacher’s 2020 data. (This includes 54% of abortions that are administered at specialized abortion clinics and 43% at nonspecialized clinics.) Hospitals made up 33% of the facilities that provided abortions in 2020 but accounted for only 3% of abortions that year, while just 1% of abortions were conducted by physicians’ offices.

Looking just at clinics – that is, the total number of specialized abortion clinics and nonspecialized clinics in the U.S. – Guttmacher found the total virtually unchanged between 2017 (808 clinics) and 2020 (807 clinics). However, there were regional differences. In the Midwest, the number of clinics that provide abortions increased by 11% during those years, and in the West by 6%. The number of clinics decreased during those years by 9% in the Northeast and 3% in the South.

The total number of abortion providers has declined dramatically since the 1980s. In 1982, according to Guttmacher, there were 2,908 facilities providing abortions in the U.S., including 789 clinics, 1,405 hospitals and 714 physicians’ offices.

The CDC does not track the number of abortion providers.

In the District of Columbia and the 46 states that provided abortion and residency information to the CDC in 2021, 10.9% of all abortions were performed on women known to live outside the state where the abortion occurred – slightly higher than the percentage in 2020 (9.7%). That year, D.C. and 46 states (though not the same ones as in 2021) reported abortion and residency data. (The total number of abortions used in these calculations included figures for women with both known and unknown residential status.)

The share of reported abortions performed on women outside their state of residence was much higher before the 1973 Roe decision that stopped states from banning abortion. In 1972, 41% of all abortions in D.C. and the 20 states that provided this information to the CDC that year were performed on women outside their state of residence. In 1973, the corresponding figure was 21% in the District of Columbia and the 41 states that provided this information, and in 1974 it was 11% in D.C. and the 43 states that provided data.

In the District of Columbia and the 46 states that reported age data to the CDC in 2021, the majority of women who had abortions (57%) were in their 20s, while about three-in-ten (31%) were in their 30s. Teens ages 13 to 19 accounted for 8% of those who had abortions, while women ages 40 to 44 accounted for about 4%.

The vast majority of women who had abortions in 2021 were unmarried (87%), while married women accounted for 13%, according to the CDC , which had data on this from 37 states.

A pie chart showing that, in 2021, majority of abortions were for women who had never had one before.

In the District of Columbia, New York City (but not the rest of New York) and the 31 states that reported racial and ethnic data on abortion to the CDC , 42% of all women who had abortions in 2021 were non-Hispanic Black, while 30% were non-Hispanic White, 22% were Hispanic and 6% were of other races.

Looking at abortion rates among those ages 15 to 44, there were 28.6 abortions per 1,000 non-Hispanic Black women in 2021; 12.3 abortions per 1,000 Hispanic women; 6.4 abortions per 1,000 non-Hispanic White women; and 9.2 abortions per 1,000 women of other races, the CDC reported from those same 31 states, D.C. and New York City.

For 57% of U.S. women who had induced abortions in 2021, it was the first time they had ever had one, according to the CDC. For nearly a quarter (24%), it was their second abortion. For 11% of women who had an abortion that year, it was their third, and for 8% it was their fourth or more. These CDC figures include data from 41 states and New York City, but not the rest of New York.

A bar chart showing that most U.S. abortions in 2021 were for women who had previously given birth.

Nearly four-in-ten women who had abortions in 2021 (39%) had no previous live births at the time they had an abortion, according to the CDC . Almost a quarter (24%) of women who had abortions in 2021 had one previous live birth, 20% had two previous live births, 10% had three, and 7% had four or more previous live births. These CDC figures include data from 41 states and New York City, but not the rest of New York.

The vast majority of abortions occur during the first trimester of a pregnancy. In 2021, 93% of abortions occurred during the first trimester – that is, at or before 13 weeks of gestation, according to the CDC . An additional 6% occurred between 14 and 20 weeks of pregnancy, and about 1% were performed at 21 weeks or more of gestation. These CDC figures include data from 40 states and New York City, but not the rest of New York.

About 2% of all abortions in the U.S. involve some type of complication for the woman , according to an article in StatPearls, an online health care resource. “Most complications are considered minor such as pain, bleeding, infection and post-anesthesia complications,” according to the article.

The CDC calculates case-fatality rates for women from induced abortions – that is, how many women die from abortion-related complications, for every 100,000 legal abortions that occur in the U.S . The rate was lowest during the most recent period examined by the agency (2013 to 2020), when there were 0.45 deaths to women per 100,000 legal induced abortions. The case-fatality rate reported by the CDC was highest during the first period examined by the agency (1973 to 1977), when it was 2.09 deaths to women per 100,000 legal induced abortions. During the five-year periods in between, the figure ranged from 0.52 (from 1993 to 1997) to 0.78 (from 1978 to 1982).

The CDC calculates death rates by five-year and seven-year periods because of year-to-year fluctuation in the numbers and due to the relatively low number of women who die from legal induced abortions.

In 2020, the last year for which the CDC has information , six women in the U.S. died due to complications from induced abortions. Four women died in this way in 2019, two in 2018, and three in 2017. (These deaths all followed legal abortions.) Since 1990, the annual number of deaths among women due to legal induced abortion has ranged from two to 12.

The annual number of reported deaths from induced abortions (legal and illegal) tended to be higher in the 1980s, when it ranged from nine to 16, and from 1972 to 1979, when it ranged from 13 to 63. One driver of the decline was the drop in deaths from illegal abortions. There were 39 deaths from illegal abortions in 1972, the last full year before Roe v. Wade. The total fell to 19 in 1973 and to single digits or zero every year after that. (The number of deaths from legal abortions has also declined since then, though with some slight variation over time.)

The number of deaths from induced abortions was considerably higher in the 1960s than afterward. For instance, there were 119 deaths from induced abortions in 1963 and 99 in 1965 , according to reports by the then-U.S. Department of Health, Education and Welfare, a precursor to the Department of Health and Human Services. The CDC is a division of Health and Human Services.

Note: This is an update of a post originally published May 27, 2022, and first updated June 24, 2022.

Support for legal abortion is widespread in many countries, especially in Europe

Nearly a year after roe’s demise, americans’ views of abortion access increasingly vary by where they live, by more than two-to-one, americans say medication abortion should be legal in their state, most latinos say democrats care about them and work hard for their vote, far fewer say so of gop, positive views of supreme court decline sharply following abortion ruling, most popular.

1615 L St. NW, Suite 800 Washington, DC 20036 USA (+1) 202-419-4300 | Main (+1) 202-857-8562 | Fax (+1) 202-419-4372 | Media Inquiries

Research Topics

Age & Generations
Coronavirus (COVID-19)
Economy & Work
Family & Relationships
Gender & LGBTQ
Immigration & Migration
International Affairs
Internet & Technology
Methodological Research
News Habits & Media
Non-U.S. Governments
Other Topics
Politics & Policy
Race & Ethnicity
Email Newsletters

ABOUT PEW RESEARCH CENTER Pew Research Center is a nonpartisan fact tank that informs the public about the issues, attitudes and trends shaping the world. It conducts public opinion polling, demographic research, media content analysis and other empirical social science research. Pew Research Center does not take policy positions. It is a subsidiary of The Pew Charitable Trusts .

Terms & Conditions

Cookie Settings

Reprints, Permissions & Use Policy

International edition
Australia edition
Europe edition

Drug shortages, now normal in UK, made worse by Brexit, report warns

Some shortages are so serious they are imperilling the health and even lives of patients with serious illnesses, pharmacy bosses say

Drug shortages are a “new normal” in the UK and are being exacerbated by Brexit , a report by the Nuffield Trust health thinktank has warned. A dramatic recent spike in the number of drugs that are unavailable has created serious problems for doctors, pharmacists, the NHS and patients, it found.

The number of warnings drug companies have issued about impending supply problems for certain products has more than doubled from 648 in 2020 to 1,634 last year.

Mark Dayan, the report’s lead author and the Nuffield Trust’s Brexit programme lead, said: “The rise in shortages of vital medicines from rare to commonplace has been a shocking development that few would have expected a decade ago.”

The UK has been struggling since last year with major shortages of drugs to treat ADHD, type 2 diabetes and epilepsy. Three ADHD drugs that were in short supply were meant to be back in normal circulation by the end of 2023 but remain hard to obtain.

Some medicine shortages are so serious that they are imperilling the health and even lives of patients with serious illnesses, pharmacy bosses warned.

Health charities have seen a sharp rise in calls from patients unable to obtain their usual medication. Nicola Swanborough, head of external affairs at the Epilepsy Society, said: “Our helpline has been inundated with calls from desperate people who are having to travel miles, often visiting multiple pharmacies to try and access their medication.”

Paul Rees, the chief executive of the National Pharmacy Association, which represents most of the UK’s 7,000 independently owned pharmacies, said: “Supply shortages are a real and present danger to those patients who rely on life-saving medicines for their wellbeing. Pharmacy teams have seen the problems get worse in this country over recent years, putting more patients at risk.

“Pharmacists … are spending hours a day hunting down stock, yet too often have to turn patients away. It’s distressing when pharmacy teams find themselves unable to provide a prompt medicines services, through no fault of their own.”

Global manufacturing problems linked to Covid, inflation, the war in Ukraine and global instability have helped cause the UK’s unprecedented inability to ensure patients can access drugs.

But Britain’s departure from the EU in 2020 has significantly aggravated the problem, laid bare the “fragility” of the country’s medicines supply networks and could lead to the situation worsening, the report said.

“A clear picture emerged of underlying fragilities at a global and UK level, not fundamentally rooted in Brexit but exacerbated by it in some specific ways, especially through some companies removing the UK from their supply chains,” it said.

The UK’s exit from the single market has disrupted the previously smooth supply of drugs, for example through the creation of a requirement for customs checks at the border, as has Britain’s decision to leave the EU’s European Medicines Agency and start approving drugs itself. The UK is now much slower than the EU at making new drugs available, the report found.

Post-Brexit red tape has prompted some firms to stop supplying to the UK altogether.

The fact that the fall in sterling’s value after the Brexit vote in 2016 coincided with drugs being in much shorter supply globally due to pharmaceutical firms experiencing shortages of ingredients, which drove up prices, has also played a key role in creating the shortages.

That has forced the Department of Health and Social Care (DHSC) to agree to pay above the usual price for drugs that are scarce to try to ensure continuity of supply far more often than it used to. “Price concessions” rose tenfold from about 20 a month before 2016 to 199 a month in late 2022, and cost the NHS in England £220m in 2022-23, the thinktank found.

The report is based on Freedom of Information requests to health bodies as well as interviews and a roundtable discussion with key figures in the drugs industry, senior DHSC civil servants and European health bodies.

It warned that Brexit had created “further risks … for the UK”. The Nuffield Trust said drug shortages could get worse because the EU’s 27 countries have recently decided to act as a unified bloc to try to minimise the impact of global scarcity, which could leave supplying Britain even less of a priority for drug companies.

Dr Andrew Hill, an expert in the drugs industry at Liverpool University, said: “With this background stress on global supplies, the UK is now more vulnerable to drug shortages. The UK is now stuck behind the US and Europe in the queue for essential drugs. Other countries offer high prices and easier access, with simpler regulations for supply.”

Ministers should agree to pay more for generic medicines, which are usually much cheaper than branded ones, to help tackle shortages, Hill added.

The Royal Pharmaceutical Society, which represents pharmacists, urged ministers to amend the law to allow community pharmacists to circumvent shortages by giving patients slightly different prescriptions, as their counterparts in hospitals already do.

“At present, if a liquid version of a medicine is available but tablets have been prescribed and are out of stock, the pharmacist cannot provide the liquid version,” said James Davies, the society’s director for England. “The patient has no choice but to return to the prescriber for a new prescription, which causes unnecessary workload for GPs and delay for the patient.”

DHSC said most drugs remained available. “Concessionary prices can arise for various reasons and cannot be linked to shortages,” a DHSC spokesperson said.

“Our priority is to ensure patients continue to get the treatments they need. There are around 14,000 licensed medicines and the overwhelming majority are in good supply.”

European Union

More on this story

Sunak rejects offer of youth mobility scheme between EU and UK

Brexit plans in ‘complete disarray’ as EU import checks delayed, say businesses

UK to delay start of health and safety checks on EU imports – report

Brussels proposes return to pre-Brexit mobility for UK and EU young people

Labour aiming to draw closer to Europe on foreign and security issues

Garden centres in UK stockpile plants before new Brexit checks

Even Europe’s far-right firebrands seem to sense Brexit is a disaster

‘It’s catastrophic’: Italian restaurants in London struggle to find staff post-Brexit

New Brexit checks will cause food shortages in UK, importers warn

UK and EU ‘within kissing distance’ of post-Brexit Gibraltar border deal

Randomization Does Not Help Much, Comparability Does

Uwe saint-mont.

Nordhausen University of Applied Sciences, Nordhausen, Germany

Conceived and designed the experiments: USM. Performed the experiments: USM. Analyzed the data: USM. Contributed reagents/materials/analysis tools: USM. Wrote the paper: USM.

According to R.A. Fisher, randomization “relieves the experimenter from the anxiety of considering innumerable causes by which the data may be disturbed.” Since, in particular, it is said to control for known and unknown nuisance factors that may considerably challenge the validity of a result, it has become very popular. This contribution challenges the received view. First, looking for quantitative support, we study a number of straightforward, mathematically simple models. They all demonstrate that the optimism surrounding randomization is questionable: In small to medium-sized samples, random allocation of units to treatments typically yields a considerable imbalance between the groups, i.e., confounding due to randomization is the rule rather than the exception. In the second part of this contribution, the reasoning is extended to a number of traditional arguments in favour of randomization. This discussion is rather non-technical, and sometimes touches on the rather fundamental Frequentist/Bayesian debate. However, the result of this analysis turns out to be quite similar: While the contribution of randomization remains doubtful, comparability contributes much to a compelling conclusion. Summing up, classical experimentation based on sound background theory and the systematic construction of exchangeable groups seems to be advisable.

1 The logic of the experiment

Randomization, the allocation of subjects to experimental conditions via a random procedure, was introduced by eminent statistician R.A. Fisher [ 1 ]. Arguably, it has since become the most important statistical technique. In particular, statistical experiments are defined by the use of randomization [ 2 , 3 ], and many applied fields, such as evidence based medicine , draw a basic distinction between randomized and non-randomized evidence.

In order to explain randomization’s eminent role, one may refer to the logic of the experiment, largely based on J. S. Mill’s method of difference [ 4 ]: If one compares two groups of subjects (Treatment T versus Control C , say) and observes a salient contrast in the end (e.g. X ‾ T > X ‾ C ), that difference must be due to the experimental manipulation—IF the groups were equivalent at the very beginning of the experiment.

In other words, since the difference between treatment and control (i.e. the experimental manipulation) is the only perceivable reason that can explain the variation in the observations, it must be the cause of the observed effect (the difference in the end). The situation is quite altered, however, if the two groups already differed substantially at the beginning. Then (see Table 1 below), there are two possible explanations of an effect:

2 Comparability

Thus, for the logic of the experiment, it is of paramount importance to ensure equivalence of the groups at the beginning of the experiment. The groups, or even the individuals involved, must not be systematically different; one has to compare like with like. Alas, in the social sciences exact equality of units, e.g. human individuals, cannot be maintained. Therefore one must settle for comparable subjects or groups ( T ≈ C ).

2.1 Defining comparability

In practice, it is straightforward to define comparability with respect to the features or properties of the experimental units involved. In a typical experimental setup, statistical units (e.g. persons) are represented by their corresponding vectors of attributes (properties, variables) such as gender, body height, age, etc.

If the units are almost equal in as many properties as possible, they should be comparable, i.e., the remaining differences shouldn’t alter the experimental outcome substantially. However, since, in general, vectors have to be compared, there is not a single measure of similarity. Rather, there are quite a lot of measures available, depending on the kind of data at hand. An easily accessible and rather comprehensive overview may be found here: reference.wolfram.com/mathematica/guide/DistanceAndSimilarityMeasures.html

As an example, suppose a unit i is represented by a binary vector a i = ( a i 1 , …, a im ). The Hamming distance d (⋅,⋅) between two such vectors is the number of positions at which the corresponding symbols are different. In other words, it is the minimum number of substitutions required to change one vector into the other. Let a 1 = (0,0,1,0), a 2 = (1,1,1,0), and a 3 = (1,1,1,1). Therefore d ( a 1 , a 2 ) = 2, d ( a 1 , a 3 ) = 3, d ( a 2 , a 3 ) = 1, and d ( a i , a i ) = 0. Having thus calculated a reasonable number for the “closeness” of two experimental units, one next has to consider what level of deviance from perfect equality may be tolerable.

Due to this, coping with similarities is a tricky business. Typically many properties (covariates) are involved and conscious (subjective) judgement seems to be inevitable. An even more serious question concerns the fact that relevant factors may not have been recorded or might be totally unknown. In the worst case, similarity with respect to some known factors has been checked, but an unnoticed nuisance variable is responsible for the difference between the outcome in the two groups.

Moreover, comparability depends on the phenomenon studied. A clearly visible difference, such as gender, is likely to be important with respect to life expectancy, and can influence some physiological and psychological variables such as height or social behaviour, but it is independent of skin color or blood type. In other words, experimental units do not need to be twins in any respect; it suffices that they be similar with respect to the outcome variable under study.

Given a unique sample it is easy to think about a reference set of other samples that are alike in all relevant respects to the one observed. However, even Fisher could not give these words a precise formal meaning [ 5 ]. Thus De Finetti [ 6 ] proposed exchangeability , i.e. “instead of judging whether two groups are similar, the investigator is instructed to imagine a hypothetical exchange of the two groups … and then judge whether the observed data under the swap would be distinguishable from the actual data” (see [ 7 ], p. 196). Barnard [ 8 ] gives some history on this idea and suggests the term permutability , “which conveys the idea of replacing one thing by another similar thing.” Nowadays, epidemiologists say that “the effect of treatment is unconfounded if the treated and untreated groups resemble each other in all relevant features” [ 7 ], p. 196.

2.2 Experimental techniques to achieve comparability

There are a number of strategies to achieve comparability. Starting with the experimental units, it is straightforward to match similar individuals, i.e., to construct pairs of individuals that are alike in many (most) respects. Looking at the group level ( T and C ), another straightforward strategy is to balance all relevant variables when assigning units to groups. Many approaches of this kind are discussed in [ 9 ], minimization being the most prominent among them. Treasure and MacRae [ 10 ] explain:

In our study of aspirin versus placebo … we chose age, sex, operating surgeon, number of coronary arteries affected, and left ventricular function. But in trials in other diseases those chosen might be tumour type, disease stage, joint mobility, pain score, or social class.

At the point when it is decided that a patient is definitely to enter a trial, these factors are listed. The treatment allocation is then made, not purely by chance, but by determining in which group inclusion of the patient would minimise any differences in these factors. Thus, if group A has a higher average age and a disproportionate number of smokers, other things being equal, the next elderly smoker is likely to be allocated to group B. The allocation may rely on minimisation alone, or still involve chance but ‘with the dice loaded’ in favour of the allocation which minimises the differences.

However, apart from being cumbersome and relying on the experimenter’s expertise (in particular in choosing and weighing the factors), these strategies are always open to the criticism that unknown nuisance variables may have had a substantial impact on the result. Therefore Fisher [ 1 ], pp. 18–20, advised strongly against treating every conceivable factor explicitly. Instead, he taught that “the random choice of the objects to be treated in different ways [guarantees] the validity of the test of significance … against corruption by the causes of disturbance which have not been eliminated.” More explicitly, Berger [ 11 ], pp. 9–10, explains:

The idea of randomization is to overlay a sequence of units (subjects, or patients) onto a sequence of treatment conditions. If neither sequence can influence the other, then there should be no bias in the assignment of the treatments, and the comparison groups should be comparable.

2.3 Randomization vs. comparability

Historically, Fisher’s idea proved to be a great success [ 12 ]. Randomized controlled trials (RCTs), as much as the randomized evidence they produce became the gold standard in a number of fields, and watchwords highlighting randomization’s leading part spread, e.g. “randomization controls for all possible confounders, known and unknown.”

Nevertheless, there have always been reservations about randomization. Putting the basic logic of the experiment in first place, randomization is a lower-ranking tool, employed towards the end of comparability. Moreover, it would be rather problematic if randomization failed to reliably yield similar groups, since non-comparable groups offer a straightforward alternative explanation, undermining experimental validity. To this end, Greenland [ 13 ], see Table 2 , came up with “the smallest possible controlled trial” illustrating that randomization does not prevent confounding:

That is, he flips a coin once in order to assign two patients to T and C , respectively: If heads, the first patient is assigned to T , and the second to C ; if tails, the first patient is assigned to C , and the second to T . Suppose X ‾ T > X ‾ C , what is the reason for the observed effect? Due to the experimental design, there are two alternatives: either the treatment condition differed from the control condition, or patient P 1 was not comparable to patient P 2 . However, as each patient is only observed under either the treatment or control (the left hand side or the right hand side of the above table), one cannot distinguish between the patient’s and the treatment’s impact on the observed result. Therefore Greenland concludes that “no matter what the outcome of randomization, the study will be completely confounded.” This effect has been observed on many occasions, for similar remarks see [ 10 , 14 – 22 ]. In total generality, Berger [ 11 ], p. 9, states:

While it is certainly true that randomization is used for the purpose of ensuring comparability between or among comparison groups, … it is categorically not true that this goal is achieved.

Suppose the patients are perfect twins with the exception of a single difference. Then Greenland’s example shows that randomization cannot even balance a single nuisance factor. To remedy the defect, it is straightforward to increase n . However, no quantitative advice is given here or elsewhere. Thus it should be worthwhile studying a number of explicit and straightforward models, quantifying the effects of randomization. Moreover, quite early, statisticians—in particular of the Bayesian persuasion—put forward several rather diverse arguments against randomization [ 23 – 29 ]. At this point, it is not necessary to delve into delicate philosophical matters or the rather violent Bayesian-Frequentist debate (however, see Section 5), since fairly elementary probabilistic arguments suffice to demonstrate that the above criticism hits its target: By its very nature a random mechanism provokes fluctuations in the composition of T and C , making these groups (rather often) non-comparable.

The subsequent reasoning has the advantage of being straightforward, mathematical, and not primarily “foundational”. Its flavour is Bayesian in the sense that we are comparing the actual groups produced by randomization which is the “posterior view” preferred by that school. At the same time its flavour is Frequentist, since we are focusing on the properties of a certain random procedure which is the “design view” preferred by this school. There are not just two, but (at least) three, competing statistical philosophies, and “in many ways the Bayesian and frequentist philosophies stand at opposite poles from each other, with Fisher’s ideas being somewhat of a compromise” [ 30 ]. Since randomization is a Fisherian proposal, a neutral quantitative analysis of his approach seems to be appropriate, acceptable to all schools, and, in a sense, long overdue. To a certain degree, philosophy is a matter of opinion. However, it is difficult to argue with mathematical facts of actual performance (see [ 31 ], p. xxii).

3 Random confounding

The overall result of the following calculations is thus [ 32 ]:

Despite randomization, imbalance in prognostic factors as a result of chance (chance imbalance) may still arise, and with small to moderate sample sizes such imbalance may be substantial.

3.1 Dichotomous factors

Suppose there is a nuisance factor X taking the value 1 if present and 0 if absent. One may think of X as a genetic aberration, a medical condition, a psychological disposition or a social habit. Assume that the factor occurs with probability p in a certain person (independent of anything else). Given this, 2 n persons are randomized into two groups of equal size by a chance mechanism independent of X .

Let S 1 and S 2 count the number of persons with the trait in the first and the second group respectively. S 1 and S 2 are independent random variables, each having a binomial distribution with parameters n and p . A natural way to measure the extent of imbalance between the groups is D = S 1 − S 2 . Obviously, ED = 0 and

Iff D = 0, the two groups are perfectly balanced with respect to factor X . In the worst case ∣ D ∣ = n , that is, in one group all units possess the characteristic, whereas it is completely absent in the other. For fixed n , let the two groups be comparable if ∣ D ∣ ≤ n / i with some i ∈ {1, …, n }. Iff i = 1, the groups will always be considered comparable. However, the larger i , the smaller the number of cases we classify as comparable. In general, n / i defines a proportion of the range of ∣ D ∣ that seems to be acceptable. Since n / i is a positive number, and S 1 = S 2 ⇔ ∣ D ∣ = 0, the set of comparable groups is never empty.

Given some constant i (< n ), the value n / i grows at a linear rate in n , whereas σ ( D ) = 2 n p ( 1 − p ) grows much more slowly. Due to continuity, there is a single point n ( i , k ), where the line intersects with k times the standard deviation of D . Beyond this point, i.e. for all n ≥ n ( i , k ), at least as many realizations of ∣ D ∣ will be within the acceptable range [0, n / i ]. Straightforward algebra gives,

A typical choice could be i = 10 and k = 3, which specifies the requirement that most samples be located within a rather tight acceptable range. In this case, one has to consider the functions n /10 and f p ( n ) = 3 2 p ( 1 − p ) n . These functions of n are shown in the following figure ( Fig 1 ):

An external file that holds a picture, illustration, etc.
Object name is pone.0132102.g001.jpg

Thus, depending on p , the following numbers of subjects are needed per group (and twice this number altogether):

Relaxing the criterion of comparability (i.e. a smaller value of i ) decreases the number of subjects necessary:

The same happens if one decreases the number of standard deviations k :

This shows that randomization works, if the number of subjects ranges in the hundreds or if the probability p is rather low. (By symmetry, the same conclusion holds if p is close to one.) Otherwise there is hardly any guarantee that the two groups will be comparable. Rather, they will differ considerably due to random fluctuations.

The distribution of D is well known ([ 33 ], pp. 142–143). For d = − n , …, n ,

Therefore, it is also possible to compute the probability q = q ( i , n , p ) that two groups, constructed by randomization, will be comparable. If i = 5, i.e., if one fifth of the range of ∣ D ∣ is judged to be comparable, we obtain:

Thus, it is rather difficult to control a factor that has a probability of about 1/2 in the population. However, even if the probability of occurrence is only about 1/10, one needs more than 25 people per group to have reasonable confidence that the factor has not produced a substantial imbalance.

Several factors

The situation becomes worse if one takes more than one nuisance factor into account. Given m independent binary factors, each of them occurring with probability p , the probability that the groups will be balanced with respect to all nuisance variables is q m . Numerically, the above results yield:

Accordingly, given m independent binary factors, each occurring with probability p j (and corresponding q j = q ( i , n , p j )), the probabilities closest to 1/2 will dominate 1 − q 1 ⋯ q m , which is the probability that the two groups are not comparable due to an imbalance in at least one variable. In a typical study with 2 n = 100 persons, for example, it does not matter if there are one, two, five or even ten factors, if each of them occurs with probability of 1/100. However, if some of the factors are rather common (e.g. 1/5 < p j < 4/5), this changes considerably. In a smaller study with fewer than 2 n = 50 participants, a few such factors suffice to increase the probability that the groups constructed by randomization won’t be comparable to 50%. With only a few units per group, one can be reasonably sure that some undetected, but rather common, nuisance factor(s) will make the groups non-comparable. Altogether our conclusion based on an explicit quantitative analysis coincides with the qualitative argument given by [ 23 ], p. 91 (my emphasis):

Suppose we had, say, thirty fur-bearing animals of which some were junior and some senior, some black and some brown, some fat and some thin, some of one variety and some of another, some born wild and some in captivity, some sluggish and some energetic, and some long-haired and some short-haired. It might be hard to base a convincing assay of a pelt-conditioning vitamin on an experiment with these animals, for every subset of fifteen might well contain nearly all of the animals from one side or another of one of the important dichotomies …

Thus contrary to what I think I was taught, and certainly used to believe, it does not seem possible to base a meaningful experiment on a small heterogenous group .

Interactions

The situation deteriorates considerably if there are interactions between the variables that may also yield convincing alternative explanations for an observed effect. It is possible that all factors considered in isolation are reasonably balanced (which is often checked in practice), but that a certain combination of them affects the observed treatment effect. For the purpose of illustration (see Table 3 below), suppose four persons (being young or old, and male or female) are investigated:

Although gender and (dichotomized) age are perfectly balanced between T and C , the young woman has been allocated to the first group. Therefore a property of young women (e.g. pregnancy) may serve as an explanation for an observed effect, e.g. X ‾ T > X ‾ C .

Given m factors, there are m ( m − 1)/2 possible interactions between just two of the factors, and ( m ν ) possible interactions between ν of them. Thus, there is a high probability that some considerable imbalance occurs in at least one of these numerous interactions, in small groups in particular. For a striking early numerical study see [ 34 ]. Detected or undetected, such imbalances provide excellent alternative explanations of an observed effect.

In the light of this, one can only hope for some ‘benign’ dependence structure among the factors, i.e., a reasonable balance in one factor improving the balance in (some of) the others. Given such a tendency, a larger number of nuisance factors may be controlled, since it suffices to focus on only a few. Independent variables possess a ‘neutral’ dependence structure in that the balance in one factor does not influence the balance in others. Yet, there may be a ‘malign’ dependence structure, such that balancing one factor tends to actuate imbalances in others. We will make this argument more precise in Section 4. However, a concrete example will illustrate the idea: Given a benign dependence structure, catching one cow (balancing one factor) makes it easier to catch others. Therefore it is easy to lead a herd into an enclosure: Grab some of the animals by their horns (balance some of the factors) and the others will follow. However, in the case of a malign dependence structure the same procedure tends to stir up the animals, i.e., the more cows are caught (the more factors are being balanced), the less controllable the remaining herd becomes.

3.2 Ordered random variables

In order to show that our conclusions do not depend on some specific model, let us next consider ordered random variables. To begin with, look at four units with ranks 1 to 4. If they are split into two groups of equal size, such that the best (1) and the worst (4) are in one group, and (2) and (3) are in the other, both groups have the same rank sum and are thus comparable. However, if the best and the second best constitute one group and the third and the fourth the other group, their rank sums (3 versus 7) differ by the maximum amount possible, and they do not seem to be comparable. If the units with ranks 1 and 3 are in the first group and the units with ranks 2 and 4 are in the second one, the difference in rank sums is ∣6 − 4∣ = 2 and it seems to be a matter of personal judgement whether or not one thinks of them as comparable.

Given two groups, each having n members, the total sum of ranks is r = 2 n (2 n +1)/2 = n (2 n +1). If, in total analogy to the last section, S 1 and S 2 are the sum of the ranks in the first and the second group, respectively, S 2 = r − S 1 . Therefore it suffices to consider S 1 , which is the test statistic of Wilcoxon’s test. Again, a natural way to measure the extent of imbalance between the groups is D = S 1 − S 2 = 2 S 1 − r . Like before ED = 0, and because σ 2 ( S 1 ) = n 2 (2 n +1)/12 we have

Moreover, n ( n +1)/2 ≤ S j ≤ n (3 n +1)/2 ( j = 1,2) yields − n 2 ≤ D ≤ n 2 . Thus, in this case, n 2 / i ( i ∈ {1, …, n 2 }) determines a proportion of the range of ∣ D ∣ that may be used to define comparability. Given a fixed i (< n 2 ), the quantity n 2 / i is growing at a quadratic rate in n , whereas σ ( D ) = n ( 2 n + 1 ) / 3 is growing at a slower pace. Like before, there is a single point n ( i , k ), where n 2 / i = kσ ( D ). Straightforward algebra gives,

Again, we see that large numbers of observations are needed to ensure comparability:

As before, it is possible to work with the distribution of D explicitly. That is, given i and n , one may calculate the probability q = q ( i , n ) that two groups, constructed by randomization, are comparable. If ∣ D ∣ ≤ n 2 / i is considered comparable, it is possible to obtain, using the function pwilcox() in R:

These results for ordered random variables are perfectly in line with the conclusions drawn from the binary model. Moreover, the same argument as before shows that the situation becomes (considerably) worse if several factors may influence the final result.

3.3 A continuous model

Finally, we consider a continuous model. Suppose there is just one factor X ∼ N ( μ , σ ). One may think of X as a normally distributed personal ability, person i having individual ability x i . As before, assume that 2 n persons are randomized into two groups of equal size by a chance mechanism independent of the persons’ abilities.

Suppose that also in this model S 1 and S 2 measure the total amount of ability in the first and the second group respectively. Obviously, S 1 and S 2 are independent random variables, each having a normal distribution N ( n μ , n σ ) . A straightforward way to measure the absolute extent of imbalance between the groups is

Due to independence, obviously D ∼ N ( 0 , 2 n σ ) .

Let the two groups be comparable if ∣ D ∣ ≤ lσ , i.e., if the difference between the abilities assembled in the two groups does not differ by more than l standard deviations of the ability X in a single unit. The larger l , the more cases are classified as comparable. For every fixed l , lσ is a constant, whereas σ ( D ) = 2 n σ is growing slowly. Owing to continuity, there is yet another single point n ( l ), where l σ = σ ( D ) = 2 n σ . Straightforward algebra gives,

In particular, we have:

In other words, the two groups become non-comparable very quickly. It is almost impossible that two groups of 500 persons each, for example, could be close to one another with respect to total (absolute) ability.

However, one may doubt if this measure of non-comparability really makes sense. Given two teams with a hundred or more subjects, it does not seem to matter whether the total ability in the first one is within a few standard deviations of the other. Therefore it is reasonable to look at the relative advantage of group 1 with respect to group 2, i.e. Q = D / n . Why divide by n and not by some other function of n ? First, due to Eq (1) , exactly n comparisons X 1, i − X 2, i have to be made. Second, since

Q may be interpreted in a natural way, i.e., being the difference between the typical (mean) representative of group 1 (treatment) and the typical representative of group 2 (control). A straightforward calculation yields Q ∼ N ( 0 , σ 2 / n ) .

Let the two groups be comparable if ∣ Q ∣ ≤ lσ . If one wants to be reasonably sure (three standard deviations of Q ) that comparability holds, we have l σ ≥ 3 σ 2 / n ⇔ n ≥ 18 / l 2 . Thus, at least the following numbers of subjects are required per group:

If one standard deviation is considered a large effect [ 35 ], three dozen subjects are needed to ensure that such an effect will not be produced by chance. To avoid a small difference between the groups due to randomization (one quarter of a standard deviation, say), the number of subjects needed goes into the hundreds.

In general, if k standard deviations of Q are desired, we have,

Thus, for k = 1,2 and 5, the following numbers of subjects n k are required in each group:

These are just the results for one factor. As before, the situation deteriorates considerably if one sets out to control several nuisance variables by means of randomization.

3.4 Intermediate conclusions

The above models have deliberately been kept as simple as possible. Their results are straightforward and they agree: If n is small, it is almost impossible to control for a trait that occurs frequently at the individual level, or for a larger number of confounders, via randomization. It is of paramount importance to understand that random fluctuations lead to considerable differences between small or medium-sized groups, making them very often non-comparable, thus undermining the basic logic of experimentation. That is, ‘blind’ randomization does not create equivalent groups, but rather provokes imbalances and subsequent artifacts. Even in larger samples one needs considerable luck to succeed in creating equivalent groups: p close to 0 or 1, a small number of nuisance factors m , or a favourable dependence structure that balances all factors, including their relevant interactions, if only some crucial factors are to be balanced by chance.

4 Unknown factors

Had the trial not used random assignment, had it instead assigned patients one at a time to balance [some] covariates, then the balance might well have been better [for those covariates], but there would be no basis for expecting other unmeasured variables to be similarly balanced ([ 2 ], p. 21)

This is a straightforward and popular argument in favour of randomization. Since randomization treats known and unknown factors alike, it is quite an asset that one may thus infer from the observed to the unobserved without further assumptions. However, this argument backfires immediately since, for exactly the same reason, an imbalance in an observed variable cannot be judged as harmless. Quite the contrary: With random assignment there is some basis for expecting other unmeasured variables to be similarly unbalanced. An observed imbalance hints at further undetectable imbalances in unobserved variables.

Moreover, treating known and unknown factors equivalently is cold comfort compared to the considerable amount of imbalance evoked by randomization. Fisher’s favourite method always comes with the cost that it introduces additional variability, whereas a systematic schema at least balances known factors. In subject areas haunted by heterogeneity it seems intuitively right to deliberately work in favour of comparability, and rather odd to introduce further variability.

In order to sharpen these qualitative arguments, let us look at an observed factor X , an unobserved factor Y , and their dependence structure in more detail. Without loss of generality let all functions d be positive in the following. Having constructed two groups of equal size via randomization, suppose d R ( X ) = X ‾ T − X ‾ C > 0 is the observed difference between the groups with respect to variable X . Using a systematic scheme instead, i.e., distributing the units among T and C in a more balanced way, this may be reduced to d S ( X ).

The crucial question is how such a manipulation affects d R ( Y ), the balance between the groups with respect to another variable. Now there are three types of dependence structures:

A benign dependence structure may be characterized by d S ( Y ) < d R ( Y ). In other words, the effort of balancing X pays off, since the increased comparability in this variable carries over to Y . For example, given today’s occupational structures with women earning considerably less than men, balancing for gender should also even out differences in income.

If balancing in X has no effect on Y , d S ( Y ) ≈ d R ( Y ), no harm is done. For example, balancing for gender should not affect the distribution of blood type in the observed groups, since blood type is independent of gender.

Only in the pathological case when increasing the balance in X has the opposite effect on Y , one may face troubles. As an example, let there be four pairs ( x 1 , y 1 ) = (1, 4); ( x 2 , y 2 ) = (2, 2); ( x 3 , y 3 ) = (3, 1); and ( x 4 , y 4 ) = (4, 3). Putting units 1 and 4 in one group, and units 2 and 3 in another, yields a perfect balance in the first variable, but the worst imbalance possible in the second.

However, suppose d (⋅) < c where the constant (threshold) c defines comparability. Then, in the randomized case, the groups are comparable if both d R ( X ) and d R ( Y ) are smaller than c . By construction, d S ( X ) ≤ d R ( X ) < c , i.e., the systematically composed groups are also comparable with respect to X . Given a malign dependence structure, d S ( Y ) increases and may exceed d R ( Y ). Yet d S ( Y ) < c may still hold, since, in this case, the “safety margin” c − d R ( Y ) may prevent the systematically constructed groups from becoming non-comparable with respect to property Y . In large samples, c − d R (⋅) is considerable for both variables. Therefore, in most cases, consciously constructed samples will (still) be comparable. Moreover, the whole argument easily extends to more than two factors.

In a nutshell, endeavouring to balance relevant variables pays off. A conscious balancing schema equates known factors better than chance and may have some positive effect on related, but unknown, variables. If the balancing schema has no effect on an unknown factor, the latter is treated as if randomization were interfering—i.e. in a completely nonsystematic, ‘neutral’ way. Only if there is a malign dependence structure, when systematically balancing some variable yields (considerable) “collateral damage”, might randomization be preferable.

This is where sample size comes in. In realistic situations with many unknown nuisance factors, randomization only works if n is (really) large. Yet if n is large, so are the “safety margins” in the variables, and even an unfortunate dependence structure won’t do any harm. If n is smaller, the above models show that systematic efforts, rather than randomization, may yield comparability. Given a small number of units, both approaches only have a chance of succeeding if there are hardly any unknown nuisance factors, or if there is a benign dependence structure, i.e., if a balance in some variable (no matter how achieved) has a positive effect on others. In particular, if the number of relevant nuisance factors and interactions is small, it pays to isolate and control for a handful of obviously influential variables, which is a crucial ingredient of experimentation in the classical natural sciences. Our overall conclusion may thus be summarized in the following table ( Table 4 ):

5 More principled questions

Since randomization has been a core point of dispute between the major philosophical schools of statistics, it seems necessary and appropriate to address these issues here.

5.1 The Frequentist position

Possibly the most important, some would say outstanding argument in favour of randomization is the view that the major “function of randomization is to generate the sample space and hence provide the basis for estimates of error and tests of significance” [ 36 ]. It has been proposed and defended by prominent statisticians, once dominated the field of statistics, and still has a stronghold in certain quarters, in particular medical statistics, where randomized controlled trials have been the gold standard.

In a statistical experiment one controls the random mechanism, thus the experimenter knows the sample space and the distribution in question. This constructed and therefore “valid” framework keeps nuisance variables at bay, and sound reasoning within the paradigm leads to correct results. Someone following this “Frequentist” train of thought could therefore state—and several referees of this contribution have indeed done so—that the above models underline the rather well-known fact that randomization can have difficulties in constructing similar groups (achieving exchangeablility/comparability, balancing covariates), in particular if n is small. However, this goal is quite subordinate to the major goal of establishing a known distribution on which quantitative statistical conclusions can be based. More precisely,

randomization in design … is supposed to provide the grounds for replacing uncertainty about the possible effects of nuisance factors with a probability statement about error ([ 37 ], p. 214, my emphasis).

In other words, because of randomization, all effects of a large (potentially infinite) number of nuisance factors can be captured by a single probability statement. How is this remarkable goal achieved?

Any analytical procedure, e.g., a statistical test, is an algorithm, transferring some numerical input into a certain output which, in the simplest case, is just a number. Given the same data, it yields exactly the same result. The procedure does not go any further: In general, there are no semantics or convincing story associated with a bare numerical result that could increase the latter’s impact. In other words, a strong interpretation needs to be based on the framework in which the calculations are embedded.

Now, since randomization treats all variables (known and unknown) alike, the analytical procedure is able to “catch” them all and their effects show up in the output. For example, a confidence interval, so the story goes, gives a quantitative estimate of all of the variables’ impact. One can thus numerically assess how strong this influence is, and one has, in a sense, achieved explicit quantitative control. In particular, if the total influence of all nuisance factors (including random fluctuations due to randomization) is numerically small, one may conclude with some confidence that a substantial difference between T and C should be due to the experimental intervention.

Following this line of argument, owing to randomization, a statistical experiment gives a valid result in the sense that it allows for far-reaching, in particular causal, conclusions. Thus, from a Frequentist point of view, one should distinguish between two very different kinds of input: (randomized) experimental data on the one hand and (non-randomized) non-experimental data on the other. Moreover, since randomization seems to be crucial—at least sufficient—for a causal conclusion, some are convinced that it is also necessary. For example, the frequently heard remark that “only randomization can break a causal link” ([ 38 ], p. 200) echoes the equally famous statement that there is “no causation without manipulation” [ 39 ].

This train of thought is supplemented by the observation that random assignment is easy to implement, and that hardly any (questionable) assumptions are needed in order to get a strong conclusion. For example, Pawitan [ 40 ] says:

A new eye drug was tested against an old one on 10 subjects. The drugs were randomly assigned to both eyes of each person. In all cases the new drug performed better than the old drug. The P-value from the observed data is 2 −10 = 0.001, showing that what we observe is not likely due to chance alone, or that it is very likely the new drug is better than the old one … Such simplicity is difficult to beat. Given that a physical randomization was actually used, very little extra assumption is needed to produce a valid conclusion.

Finally, one finds a rather broad range of verbal arguments why randomization should be employed, e.g. “valid” conclusions, either based on the randomization distribution [ 41 ] or some normal-theory approximation [ 42 ], removal of investigator bias [ 43 ], face validity, fairness, and simple analysis [ 44 ], justification of inductive steps, in particular generalizations from the observed results to all possible results [ 45 ].

5.2 Bayesian opposition

Traditionally, criticism of the Frequentist line of argument in general, and randomization in particular, has come from the Bayesian school of statistics. While Frequentist statistics is much concerned with the way data is collected, focusing on the design of experiments, the corresponding sample space and sampling distribution, Bayesian statistics is rather concerned with the data actually obtained. Its focus is on learning from the(se) data—in particular with the help of Bayes’ theorem—and the parameter space.

In a sense, both viewpoints are perfectly natural and do not contradict one other. However, the example of randomization shows that this cannot be the final word: For the pre-data view, randomization is essential; it constitutes the difference between a real statistical experiment and any kind of quasi-experiment. For the post-data view, however, randomization does not add much to the information at hand, and is ancillary or just a nuisance.

The crucial and rather fundamental issue therefore becomes how far-reaching the conclusions of each of these styles of inference are. To cut a long story short, despite a “valid” framework and mathematically sound conclusions a Frequentist train of thought may easily miss its target or might even go astray. (The long story, containing a detailed philosophical discussion, is told in [ 46 ].) After decades of Frequentist—Bayesian comparisons, it has become obvious that in many important situations the numerical results of Frequentist and Bayesian arguments (almost) coincide. However, the two approaches are conceptually completely different, and it has also become apparent that simple calculations within the sampling framework lead to reasonable answers to post-data questions only because of “lucky” coincidences (e.g., the existence of sufficient statistics for the normal distribution). Of course, in general, such symmetries do not exist, and pre-data results cannot be transferred to post-data situations. In particular, purely Frequentist arguments fail if the sampling distribution does not belong to the “exponential family”, if there are several nuisance parameters, if there is important prior information, or if the number of parameters is much larger than the number of observations ( p ≫ n ).

5.3 A formal as well as an informal framework

The idea that the influence of many nuisance factors—even unknown ones—may be caught by a simple experimental device and some probability theory is a bold claim. Therefore it should come as no surprise that some Frequentist statisticians, many scientists and most Bayesians have questioned it. For example, towards the end of his article, Basu [ 26 ] writes quite categorically: “The randomization exercise cannot generate any information on its own. The outcome of the exercise is an ancillary statistic. Fisher advised us to hold the ancillary statistic fixed, did he not?” Basing our inferences on the distribution that randomization creates seems to be the exact reverse.

Even by the 1970s, members of the classical school noted that, upon using randomization and the distribution it entails, we are dealing with “the simplest hypothesis, that our treatment … has absolutely no effect in any instance”, and that “under this very tight hypothesis this calculation is obviously logically sound” ([ 47 ], my emphasis). Contemporary criticism can be found in Heckman [ 19 ] who complains that “a large statistical community” idealizes randomization, “implicitly appeal[s] to a variety of conventions rather than presenting rigorous models”, and that “crucial assumptions about sources of randomness are kept implicit.”

As for the sources of randomness, one should at least distinguish between natural variation and artificially introduced variability. A straightforward question then surely is, how inferences based on the “man-made” portion bear on the “natural” part. To this end, [ 26 ], pp. 579–581, compares a scientist, following the logic we described in Section 1, and a statistician who counts on randomization. It turns out that they are talking at cross-purposes. While the foremost goal of the scientist is to make the groups comparable, the statistician focuses on the randomization distribution. Moreover, the scientist asks repeatedly to include important information, but with his inquiry falling on deaf ears, he disputes this statistician’s analysis altogether.

Heckman’s criticism deploring a lack of explicit models and assumptions has been repeated by many (e.g. [ 31 ], [ 7 ]). In the natural sciences, mathematical arguments have always been more important than verbal reasoning. Typically, the thrust of an argument consists of formulae and their implications, with words of explanation surrounding the formal nucleus. Other fields like economics have followed suit and have learnt—often the hard way—that seemingly very convincing heuristic arguments can be wrong or misleading. Causality is no exception to that rule. In the last twenty years or so, causal graphs and causal calculus have formalized this field. And, as was to be expected, increased rigor straightforwardly demonstrated that certain “reasonable” beliefs and rather “obvious” time-honored conventions do not work as expected (see [ 7 ], in particular Chapter 6 and p. 341).

Our analysis above fits in nicely: The standard phrase “if n is not too small” is a verbal statement, implicitly appealing to the central limit theorem. Owing to the latter theorem, groups created by random assignment tend to become similar. The informal assurance, affirming that this happens fast, ranks among the most prominent conventions of traditional statistics. However, explicit numerical models underline that our intuition needs to be corrected. Rather straightforward calculations suffice to show that fluctuations cannot be dismissed easily, even if n is large.

Worse still, the crucial part of the Frequentist’s main argument in favour of randomization is informal in a rather principled way: In an experimental, as well as in a similar non-experimental situation, the core formal machinery, i.e. the data at hand, the explicit analytical procedure (e.g. a statistical test), and the final numerical result may be identical. In other words, it is just the narrative prior to the data that makes such a tremendous difference in the end. Since heuristic arguments have a certain power of persuasion which is certainly weaker than a crisp formal derivation or a strict mathematical proof, it seems to be no coincidence that opinion on this matter has remained divided. Followers of Fisher believed in his intuition and trusted randomization; critics did not. And since, sociologically speaking, the Frequentist school dominated the field for decades, so did randomization.

It is also no coincidence, but rather sheer necessity, that a narrow formal line of argument needs to be supplemented with much intuition and heuristics. So, on the one hand, an orthodox author may claim that “randomization, instrumental variables, and so forth have clear statistical definitions”; yet, on the other hand, he has to concede at once that “there is a long tradition of informal —but systematic and successful—causal inference in the medical sciences” ([ 7 ], p. 387, my emphasis). Without doubt, such a mixture is difficult to understand, to use and to criticize, and could be one of the main reasons for the reputation of statistics as an opaque subject. The narrow formal framework also partly explains why there is such a wide variety of verbal arguments in favour of randomization (see the end of Section 5.1).

5.4 Pragmatical eclecticism

From a Frequentist point of view, randomization is crucial since it “provides a known distribution for the assignment variables; statistical inferences are based on this distribution” [ 48 ]. Thus, the “known-distribution argument” is perhaps the single most important argument in favour of randomization.

How is it applied? If the result of a random allocation is extreme (e.g. all women are assigned to T, and all men to C), everybody—Fisher included—seems to be prepared to dismiss this realization: “It should in fairness be mentioned that, when randomization leads to a bad-looking experiment or sample, Fisher said that the experimenter should, with discretion and judgement, put the sample aside and draw another” [ 24 ].

The latter concession isn’t just a minor inconvenience, but runs contrary to the very principle of the Frequentist viewpoint: First, an informal correction is wide open to subjective judgement. (Already) “bad-looking” to person A may be (still) “fine-looking” to person B. Second, what’s the randomization distribution actually being used when dismissing some samples? A vague selection procedure will inevitably lead to a badly defined distribution. Third, why reject certain samples at all? If the crucial feature of randomization is to provide a “valid” distribution (on which all further inference is based), one should not give away this advantage unhesitatingly. At the very least, it is inconsistent to praise the argument of the known framework in theoretical work, and to turn a blind eye to it in practice.

As a matter of fact, in applications, the exact permutation distribution created by some particular randomization process plays a rather subordinate role. Much more frequently, randomization is used as a rationale for common statistical procedures. Here is one of them: Randomization guarantees independence and if many small uncorrelated (and also often unknown) factors contribute to the distribution of some observable variable X , this distribution should be normal—at least approximately, if n is not too small. Therefore, in a statistical experiment, it seems to be justified to compare X ‾ T and X ‾ C , using these means and the sample variance as estimators of their corresponding population parameters. Thus we have given an informal derivation of the t-test. Both the test and the numerical results it yields are supported by randomization. However, it may be noted that Student’s famous test was introduced much earlier [ 49 ] and worked quite well without randomization’s assistance.

How should experimental data be analyzed? If the known distribution were of paramount importance, there should be a unanimous vote, at least by Frequentist statisticians. However, only a minority, perfectly in line with the received position, advise leaving the data as it is. Freedman [ 50 ] argues thus (for similar comments see [ 48 ], [ 7 ], p. 340, and [ 38 ], pp. 250–253.):

Regression adjustments are often made to experimental data. Since randomization does not justify the models, almost anything can happen … The simulations, like the analytic results, indicate a wide range of possible behavior. For instance, adjustment may help or hurt.

Yet a majority have a different opinion (e.g. [ 2 , 3 , 45 ]). Tu et al. [ 51 ] explain that the first reason why they opt for an “adjustment of treatment effect for covariates in clinical trials” is to “improve the credibility of the trial results by demonstrating that any observed treatment effect is not accounted for by an imbalance in patient characteristics.”

5.5 Once again: randomization vs. comparability

Apart from the rather explicit rhetoric of a “valid framework”, there is also always the implicit logic of the experiment. Thus, although the received theory emphasizes that “actual balance has nothing to do with validity of statistical inference; it is an issue of efficiency only” [ 41 ]; comparability turns out to be crucial:

Many, if not most, of those supporting randomization rush to mention that it promotes similar groups. Nowadays, only a small minority bases its inferences on the known permutation distribution created by the process of randomization; but an overwhelming majority checks for comparability. Reviewers of experimental studies routinely request that authors provide randomization checks, that is, statistical tests designed to substantiate the equivalence of T and C . At least, in almost every article a list of covariates—with their groupwise means and standard errors—can be found.

A narrow, restricted framework is only able to support weak conclusions; there is no such thing as a “free lunch.” Therefore, upon reaching a strong conclusion, there must be implicit, hidden assumptions (cf. [ 19 ], pp. 139, 155). In particular, a second look at the “little-assumption” argument reveals that it is the hidden assumption of comparability that carries much of the burden of evidence: It is no coincidence that in Pawitan’s example an eye drug was tested. Suppose one had tested a liver drug instead. The same numerical result would be almost as convincing if such a drug had been applied to twins. However, if the liver drug had been administered to a heterogenous set of persons or if it had been given to a different biological species (mice instead of men, say), exactly the same formal result would not be convincing at all; since, rather obviously, a certain discrepancy a priori may cause a remarkable difference a posteriori.

Savage’s example is quite similar. No matter how one splits a small heterogenous group into two, the latter groups will always be systematically different. Randomization does not help: If you assign randomly and detect a large effect in the end, still, your experimental intervention or the initial difference between T and C may have caused it. All “valid” inferential statistics is, in a sense, an illusion, since it cannot exclude the straightforward second explanation. Instead, it’s the initial exchangeability of the groups that turns out to be decisive; similarity of T and C rules out the second explanation and leaves the experimental intervention as the only cause.

In conclusion, comparability, much more than randomization, keeps alternative explanations at bay. Since it is our endeavour to achieve similar groups, minimization is not just some supplementary technique to improve efficiency. Rather, it is a straightforward and elaborate device to enhance comparability, i.e., to consciously construct similar groups. (The influence of unknown factors is discussed in Section 4.) Though, at times, we fail, e.g. “it does not seem possible to base a meaningful experiment on a small heterogenous group” [ 23 ], there can hardly be any doubt that “the purpose of randomization is to achieve homogeneity in the sample units. [Thus] it should be spelled out that stability and homogeneity are the foundation of the statistical solution, not the other way around” [ 52 ], p. 70 (my emphasis).

In a nutshell, nobody, not even Fisher, follows “pure Frequentist logic”, in particular the distribution that randomization generates. In a strict sense, there is no logic at all, rather a certain kind of mathematical reasoning plus—since the formal framework is restricted to sampling—a fairly large set of conventions; rigid “pure” arguments being readily complemented by applied “flexibility”, consisting of time-honored informal reasoning and shibboleth, but also outright concessions. Bayesians noticed long ago [ 28 , 31 ] that “Frequentist theory is shot full of contradictions” [ 53 ], and during the last few decades, efforts to overcome the received framework have gained momentum.

6 A broader perspective

In the 20th century, R.A. Fisher (1890–1962) was the most influential statistician. However, while his early work on mathematical statistics is highly respected in all quarters, hardly anybody relies on his later ideas, in particular fiducial inference [ 24 ]. Randomization lies in-between, and, quite fittingly, public opinion on this formal technique has remained divided.

6.1 Replicate!

Since the most important application of random assignment can be found in clinical trials, it is straightforward to ask how strong the evidence produced by a randomized controlled trial is. Vis-à-vis the rather anecdotal and qualitative research that preceded today’s RCTs, the latter surely constituted real progress. Strict design and standardized analysis has raised the bar and has fostered consensus among researchers. However, many classical experiments in the natural sciences are deterministic and have an even better reputation. If in doubt, physicists do not randomize, but replicate. Fisher [ 54 ], p. 58, gave similar advice.

Alas, a large number of important biomedical findings (RCTs included) failed this examination and turned out to be non-replicable [ 55 – 57 ], so that the National Institute of Health was forced to launch the “Replication of Key Clinical Trials Initiative” [ 58 ]. The same with experimental psychology which has relied on (small) randomized trials for decades. Now, it is lamenting a “replicability crisis” that has proved to be so severe that an unprecedented “reproducibility project” needed to be launched [ 59 , 60 ]. (Similar initiatives are collected in http://validation.scienceexchange.com . One may also consult www.nature.com/nature/focus/reproducibility on this matter.)

The logic of Section 1 offers a straightforward explanation for this unfortunate state of affairs. Given (almost) equal initial conditions, the same boundary conditions thereafter, and a well-defined experimental intervention, an effect once observed must re-occur. That’s how classical experiments work, which reliably and thus repeatedly hit their target. During the experiment, a controlled environment keeps disturbing factors at bay. Thus, if an effect cannot be replicated, the constructional flaw should be sought at the very beginning of the endeavour. At this point, it is conspicuous that today’s studies do not focus explicitly on the crucial idea of comparability. With other issues—possibly rather irrelevant or even misleading—being at least as important, initial imbalances are the rule and not the exception. At the very least, with randomization, the starting point of researcher 2, trying to repeat the result of researcher 1, will always differ from the latter’s point of origin. Therefore, if an effect cannot be replicated, this may well be due to the additional variability introduced by the “R” in RCT, yielding unequal initial conditions, and “drowning” the interesting phenomenon in a sea of random fluctuation. In a similar but more positive vein, Abel and Koch [ 16 ] underline that experimental control (the “C” in RCT) is crucial:

Apart from the control of imbalance in prognostic factors, randomized studies have several qualities that … follow from the fact that [they] belong to a larger class of high-quality studies, namely, prospective parallel comparisons with a written protocol specifying important aspects of patient enrollment, treatment, observation, analysis, and other procedures.

6.2 Learning to cope with variability

Chance has a Janus face. The idea that many (the more the better), small and rather uncorrelated random influences sum up to a “mild” distribution originated in the 19th century, culminating in the famous central limit theorem on which much of classical statistics is built. However, this was not the end of the story. Studying complex systems, physicists soon encountered “wild” distributions, in particular power laws ([ 61 ], p. 104). It is well within this newer set of ideas that a single random event may have a major impact that cannot be neglected (e.g. the energy released by a particularly strong earthquake or the area devastated by a single large flood). The fact that the process of randomization can produce a major fluctuation, i.e., a pronounced imbalance in a covariate (and thereby between T and C ), exerting a tremendous influence on the final result of an RCT is in line with this more recent portrait of chance.

Randomization has tremendous prestige in orthodox statistics, downgrading all designs without a random element to quasi-experiments. One even distinguishes between truly random allocation and “haphazard assignment, that is, a procedure that is not formally random but has no obvious bias” ([ 3 ], p. 302). Honoring thus the classical philosophical distinction between “deterministic” and “random‘”, one readily neglects the fact that modern dynamical systems theory sees a continuum of increasing complexity between perfect (deterministic) order and “randomness [which] can be thought of as an extreme form of chaos” [ 62 ].

With the core technique (or rather dogma) of randomization, Fisher’s conception of experiments could even develop into a “cult of the single study” ([ 63 ], p. 262), and catch-phrases highlighting randomization’s outstanding role became increasingly popular [ 38 , 39 ]. However, this determined point of view has also blocked progress, and innovative solutions have been developed elsewhere: Econometrician J.J. Heckman, earning a Nobel prize for his contributions in 2000, explains that Holland’s claim that there can be no causal effect of gender on earnings (because we cannot randomly assign gender) “conflates the act of definition of the causal effect … with empirical difficulties in estimating it” [ 19 ]. Moreover, he complains that “this type of reasoning is prevalent in statistics.” Epidemiology, not following Fisher [ 64 ] but the Advisory Committee to the Surgeon General of the Public Health Service [ 65 ], has made its way to causal graphs. The latter formalization, mainly developed by computer scientist J. Pearl, has given crucial concepts a sound basis, but it also tells a “tale of statistical agony” [ 7 ].

In a more positive vein, computer scientist J. Rissannen [ 66 ] showed how Fisher’s finest ideas may be reformulated and extended within a modern, fine-tuned mathematical framework. In his work one finds a logically sound and general unifying theory of hypothesis testing, estimation and modeling; yet there is no link to randomization. Instead, in this contemporary theory the basic concept is Kolmogorov complexity, allowing to express the idea that a regular sequence r (e.g. “1,0” repeated 13 times) is less complex than a sequence like s = (1, 0, 0, 1, 0, 0, 1, 1, 0, 1, 1, 0, 0, 0, 1, 1, 1, 0, 1, 1, 0, 1, 0, 0, 0, 0) in a mathematically strict way. (The sequence s can be found in [ 67 ], p. 48; a book including a chapter on “nonprobabilistic statistics.”) It also turns out that stochastic processes typically produce complex sequences. However, contrary to the fundamental distinction (deterministic vs. random) mentioned above, given a certain sequence like s , it is not possible to tell whether the process that generated s was systematic or not. s could be the output of a “(pseudo-)random number generator”, i.e., a deterministic algorithm designed to produce chaotic output, or of a “truly random” mechanism. (Whatever that is. For example, strictly speaking, coin tossing—being a part of classical physics—is not “truly” random.)

Interpreting s as a particular assignment of units to groups (1 → T , and 0 → C , say), the above fundamental distinction between “‘haphazard” and “random” assignment processes seems to be exaggerated, some might even question whether it is relevant at all. However, it is difficult to deny that the (non-)regularity of the concrete mapping matters. Just compare r to s : Since r invites a straightforward alternative explanation, most people would prefer s . In today’s terminology, Fisher could have been looking for maximally complex sequences, i.e., allocations without any regularity. In his time, a simple stochastic process typically yielding an “irregular” sequence was a straightforward and convenient solution to this problem.

6.3 Conclusion: Good experimental practice

In sum, Fisher’s idea of randomization is still alive. However, on the whole it looks more like a backward-looking tool than like the indispensable key to the future of statistics. Though there are many claims in its favour, they can all be seriously questioned:

Yes, but only if n is large. Otherwise, randomization rather provokes imbalances (Section 3).

Conversely, imbalances in observed variables hint at imbalances in unobserved variables. Moreover, a more detailed study of dependence structures reveals that consciously working in favour of similar groups typically pays off (Section 4).

The formal Frequentist framework thus defined is narrow, yielding weak conclusions that have to be supplemented with a lot of informal reasoning (Section 5).

No; comparability and strict experimental control are crucial for internally valid, replicable studies (Section 6).

In a nutshell, contrary to what Fisher [ 1 ] thought, randomization does not “relieve the experimenter from the anxiety of considering and estimating the magnitude of the innumerable causes by which the data may be disturbed.” Quantitative arguments and formalized theories demonstrate that it is no philosopher’s stone, almost effortlessly lifting experimental procedures in the medical and social sciences to the level of classical experiments in the natural sciences. Rather, random assignment may lull researchers into a false sense of security.

It is true that chance, in the guise of randomization, by and large supports comparability. However, since the former is blind with respect to the concrete factors and relevant interactions that may be present, it needs a large number of experimental units to do so. The intuition behind this result is easy to grasp: Without knowledge of the subject matter, randomization has to protect against every conceivable nuisance factor. Such unsystematic protection is provided by number and builds up slowly. Thus, a huge number of randomly allocated subjects is needed to shield against a moderate number of potential confounders. And, of course, no finite procedure such as the flip of a coin is able to control for an infinite number of nuisance variables.

Therefore, it seems much more advisable to use background knowledge in order to minimize the difference between groups with respect to known factors or specific threats to experimental validity. As of today, minimization seems to operationalize this idea best. At the end of such a conscious construction process, randomization finds its proper place. Only if no reliable context information exists is unrestricted randomization the method of choice. It must be clear, however, that it is a weak guard against confounding, yet the only one available in such inconvenient situations.

Summing up, the above analysis strongly recommends traditional experimentation, thoroughly selecting, balancing and controlling factors and subjects with respect to known relevant variables, thereby using broader context information—i.e., substantial scientific knowledge. I agree with Penston [ 68 ], pp. 76–77, who says:

… it is the existence of sound background theory which is crucial for the success of science. It is the framework against which observations are made, it allows strict definition of the items involved, it is the source of information about possible relevant variables and allows for the identification of homogeneous reference classes that ensure regularity and, hence, reliable causal inference.

Cumulative science is the result of a successful series of such experiments, each of them focusing on the crucial ingredients, like precise research questions, convincing operationalizations, explicit control, quantitative measures of effect, and—last but not least—comparability.

Funding Statement

The author has no support or funding to report.

IMAGES

Explain Research Starts With a Problem and Ends With Another
Presenting Your Research
Essential Things to Do Before Starting Your Research Study
How to Write a Research Introduction: 10 Steps (with Pictures)
Typical Preliminary Research Proposal
The Ends of IS Research

VIDEO

Rezertful but it's 10 minutes
How to write a research paper conclusion

COMMENTS

When does research end? The emotional labour of researching abjection
Yet, I have argued that research never ends, so we must continually reflect on the ethics and politics of the research we do. While the research project continues to haunt me, I do not want the reader to conflate haunting with negativity or damage. Some memories make me cringe and feel ashamed, yet others fill me with happiness and pride.
Where does research begin and end, or where should it?
For some time now, I have been pondering the notion of how we need to consider changing where research begins and ends. Traditionally research starts with writing a grant and usually finishes with publishing a paper - it's funny, I have never thought about the way we currently bookend our research with a bunch of writing to justify ourselves!
Organizing Your Social Sciences Research Paper
The conclusion is intended to help the reader understand why your research should matter to them after they have finished reading the paper. A conclusion is not merely a summary of the main topics covered or a re-statement of your research problem, but a synthesis of key points derived from the findings of your study and, if applicable, where you recommend new areas for future research.
How to Conduct Responsible Research: A Guide for Graduate Students
Abstract. Researchers must conduct research responsibly for it to have an impact and to safeguard trust in science. Essential responsibilities of researchers include using rigorous, reproducible research methods, reporting findings in a trustworthy manner, and giving the researchers who contributed appropriate authorship credit.
Organizing Your Social Sciences Research Paper
The introduction leads the reader from a general subject area to a particular topic of inquiry. It establishes the scope, context, and significance of the research being conducted by summarizing current understanding and background information about the topic, stating the purpose of the work in the form of the research problem supported by a hypothesis or a set of questions, explaining briefly ...
Writing a Research Paper Conclusion
Step 1: Restate the problem. The first task of your conclusion is to remind the reader of your research problem. You will have discussed this problem in depth throughout the body, but now the point is to zoom back out from the details to the bigger picture. While you are restating a problem you've already introduced, you should avoid phrasing ...
Comparison between Research Methodologies: Experiments, Surveys, and
Every experiment, research or analysis ends with some type of report that is prepared for a specific audience. Experiments and research are normally done with some type of end goal in mind whether to provide supporting data, contradictory data, unique data or merely providing some type of information via observation or interviews.
What Is Research, and Why Do People Do It?
Abstractspiepr Abs1. Every day people do research as they gather information to learn about something of interest. In the scientific world, however, research means something different than simply gathering information. Scientific research is characterized by its careful planning and observing, by its relentless efforts to understand and explain ...
The Conclusion: How to End a Scientific Report in Style
reiterates the importance of the research project. a. i and ii. b. i and iii. c. ii and iii. d. iii and iv. 3. The second rhetorical move of the Conclusion section is reflection of outcome(s). It … i. summarizes all the findings of the research project. ii. synthesizes outcomes of the research project. iii. is a repeat of important ideas ...
Writing up a Research Report
The research design section usually starts with the research aim. The research aim is the intellectual contribution you want to achieve (i.e., the argument you like to make). The research aim is based on the research gap at the literature review's end.
Flexibility of Research: What to Do When You Feel like You've Hit a
Research can be a truly thrilling experience-interesting data, new findings, surprising collections. However, research can also be incredibly frustrating, namely when you feel like your work isn't going anywhere.. If you've ever been really excited about a topic, done a load of research, and still found that you aren't making forward progress, this post is for you.
Organizing Your Social Sciences Research Paper
Some generalizability in regards to applicability and bringing results into general use, ... Given the content listed above, a description of the research problem is usually a short paragraph in length. ... Explicitly listing your research questions at the end of your introduction can help in designing a clear roadmap of what you plan to ...
The Use of Research Methods in Psychological Research: A Systematised
Introduction. Psychology is an ever-growing and popular field (Gough and Lyons, 2016; Clay, 2017).Due to this growth and the need for science-based research to base health decisions on (Perestelo-Pérez, 2013), the use of research methods in the broad field of psychology is an essential point of investigation (Stangor, 2011; Aanstoos, 2014).Research methods are therefore viewed as important ...
6 Moments of Research "Failure" and How to Deal With Them
5. Organizing and documenting. The failure to organize your research—the process, sources, methods, etc.—can sneak up on you. At least it does for me! It might be small things, like not writing down the exact way you came up with a particular result, or getting a little sloppy with your citations.
Common Pitfalls In The Research Process
Conducting research from planning to publication can be a very rewarding process. However, multiple preventable setbacks can occur within each stage of research. While these inefficiencies are an inevitable part of the research process, understanding common pitfalls can limit those hindrances. Many issues can present themselves throughout the research process. It has been said about academics ...
PDF Defining Normal
Defining Normal Katie Segura Abstract Based on the available research, normal has traditionally been defined in terms of the abnormal. In other words, each term is used as a benchmark to describe the other, without which neither term could be defined on its own. This raises questions in the psychological community about the
Self-Defeating Behavior
Self-Defeating Behavior Definition. For social psychologists, a self-defeating behavior is any behavior that normally ends up with a result that is something the person doing the behavior doesn't want to happen. If you are trying to accomplish some goal, and something you do makes it less likely that you will reach that goal, then that is a ...
Rules mcqs
(75) A research normally ends with some . A) Recommendations B) Conclusions C) Summary D) Both A & B E) Only C (76) Expansion of knowledge is possible & the result of research. A) True research B) Systematic research C) Complete research D) All A, B & C E) Only A (77) Step by step & proper method is called _____. A) Systematic approach B) Non ...
Why do some open-ended survey questions result in higher item
With such a wide range of item nonresponse rates for open ends, we sought to understand what drives these differential rates, and if possible, figure out how to encourage greater response. Some research suggests that the general question type — such as whether it is asking about a fact, attitude or behavior — can influence the item ...
SCC Research Guides: Evaluating Sources: Domain Endings
Domain Endings. Domain endings are the end part of a URL (.com, .org, .edu, etc.). Sometimes the domain ending can give you a clue to a website's purpose. While domain endings can give you some ideas about a website, they should not be the only way you determine if a website is credible or not.
How Does Research Start?
Clinical research aims to deliver healthcare advancements that are safe, beneficial, and cost-effective ( Ford & Norrie, 2016 ). Research requires a methodical approach to develop studies that generate high-quality evidence to support changes in clinical practice. The method is a step-wise process that attempts to limit the chances of errors ...
The Research-Backed Benefits of Daily Rituals
The authors' expertise on the topic over the past decade, plus a survey of nearly 140 HBR readers, explores the ways rituals can set us up for success before work, get us psyched up for ...
What the data says about abortion in the U.S.
Since 1997, the CDC's totals have lacked data from some states (most notably California) for the years that those states did not report data to the agency. The four reporting areas that did not submit data to the CDC in 2021 - California, Maryland, New Hampshire and New Jersey - accounted for approximately 25% of all legal induced ...
Drug shortages, now normal in UK, made worse by Brexit, report warns
Some shortages are so serious they are imperilling the health and even lives of patients with serious illnesses, pharmacy bosses say Denis Campbell Health policy editor Wed 17 Apr 2024 19.01 EDT ...
Randomization Does Not Help Much, Comparability Does
Randomization does not help: If you assign randomly and detect a large effect in the end, still, your experimental intervention or the initial difference between T and C may have caused it. All "valid" inferential statistics is, in a sense, an illusion, since it cannot exclude the straightforward second explanation.
Chaos in Dubai as UAE records heaviest rainfall in 75 years
Chaos ensued in the United Arab Emirates after the country witnessed the heaviest rainfall in 75 years, with some areas recording more than 250 mm of precipitation in fewer than 24 hours, the ...