hypothesis space of

• Machine Learning Tutorial
• Data Analysis Tutorial
• Python - Data visualization tutorial
• Machine Learning Projects
• Machine Learning Interview Questions
• Machine Learning Mathematics
• Deep Learning Tutorial
• Deep Learning Project
• Deep Learning Interview Questions
• Computer Vision Tutorial
• Computer Vision Projects
• NLP Project
• NLP Interview Questions
• Statistics with Python
• 100 Days of Machine Learning
• ML | Active Learning
• Difference between Supervised and Unsupervised Learning
• Overview of ROBERTa model
• What is Saliency Map?
• Intuition behind Adagrad Optimizer
• Differentiate between Support Vector Machine and Logistic Regression
• Logistic Regression using Python
• Cost function in Logistic Regression in Machine Learning
• CNN | Introduction to Padding
• GrowNet: Gradient Boosting Neural Networks
• Orthogonal Projections
• Multiclass classification using scikit-learn
• Getting started with Dialogflow
• Rule-Based Classifier - Machine Learning
• Introduction to Speech Separation Based On Fast ICA
• Python | Create Test DataSets using Sklearn
• Early Stopping for Regularisation in Deep Learning
• ALBERT - A Light BERT for Supervised Learning

Hypothesis in Machine Learning

The concept of a hypothesis is fundamental in Machine Learning and data science endeavors. In the realm of machine learning, a hypothesis serves as an initial assumption made by data scientists and ML professionals when attempting to address a problem. Machine learning involves conducting experiments based on past experiences, and these hypotheses are crucial in formulating potential solutions.

It’s important to note that in machine learning discussions, the terms “hypothesis” and “model” are sometimes used interchangeably. However, a hypothesis represents an assumption, while a model is a mathematical representation employed to test that hypothesis. This section on “Hypothesis in Machine Learning” explores key aspects related to hypotheses in machine learning and their significance.

A hypothesis in machine learning is the model’s presumption regarding the connection between the input features and the result. It is an illustration of the mapping function that the algorithm is attempting to discover using the training set. To minimize the discrepancy between the expected and actual outputs, the learning process involves modifying the weights that parameterize the hypothesis. The objective is to optimize the model’s parameters to achieve the best predictive performance on new, unseen data, and a cost function is used to assess the hypothesis’ accuracy.

What is Hypothesis Testing?

Researchers must consider the possibility that their findings could have happened accidentally before interpreting them. The systematic process of determining whether the findings of a study validate a specific theory that pertains to a population is known as hypothesis testing.

To assess a hypothesis about a population, hypothesis testing is done using sample data. A hypothesis test evaluates the degree of unusualness of the result, determines whether it is a reasonable chance variation, or determines whether the result is too extreme to be attributed to chance.

How does a Hypothesis work?

In most supervised machine learning algorithms, our main goal is to find a possible hypothesis from the hypothesis space that could map out the inputs to the proper outputs. The following figure shows the common method to find out the possible hypothesis from the Hypothesis space:

Hypothesis Space (H)

Hypothesis space is the set of all the possible legal hypothesis. This is the set from which the machine learning algorithm would determine the best possible (only one) which would best describe the target function or the outputs.

Hypothesis (h)

A hypothesis is a function that best describes the target in supervised machine learning. The hypothesis that an algorithm would come up depends upon the data and also depends upon the restrictions and bias that we have imposed on the data.

The Hypothesis can be calculated as:

• m = slope of the lines
• b = intercept

To better understand the Hypothesis Space and Hypothesis consider the following coordinate that shows the distribution of some data:

Say suppose we have test data for which we have to determine the outputs or results. The test data is as shown below:

We can predict the outcomes by dividing the coordinate as shown below:

So the test data would yield the following result:

But note here that we could have divided the coordinate plane as:

The way in which the coordinate would be divided depends on the data, algorithm and constraints.

• All these legal possible ways in which we can divide the coordinate plane to predict the outcome of the test data composes of the Hypothesis Space.
• Each individual possible way is known as the hypothesis.

Hence, in this example the hypothesis space would be like:

Hypothesis in Statistics

In statistics , a hypothesis refers to a statement or assumption about a population parameter. It is a proposition or educated guess that helps guide statistical analyses. There are two types of hypotheses: the null hypothesis (H0) and the alternative hypothesis (H1 or Ha).

• Null Hypothesis(H 0 ): This hypothesis suggests that there is no significant difference or effect, and any observed results are due to chance. It often represents the status quo or a baseline assumption.
• Aternative Hypothesis(H 1 or H a ): This hypothesis contradicts the null hypothesis, proposing that there is a significant difference or effect in the population. It is what researchers aim to support with evidence.

Frequently Asked Questions (FAQs)

1. how does the training process use the hypothesis.

The learning algorithm uses the hypothesis as a guide to minimise the discrepancy between expected and actual outputs by adjusting its parameters during training.

2. How is the hypothesis’s accuracy assessed?

Usually, a cost function that calculates the difference between expected and actual values is used to assess accuracy. Optimising the model to reduce this expense is the aim.

3. What is Hypothesis testing?

Hypothesis testing is a statistical method for determining whether or not a hypothesis is correct. The hypothesis can be about two variables in a dataset, about an association between two groups, or about a situation.

4. What distinguishes the null hypothesis from the alternative hypothesis in machine learning experiments?

The null hypothesis (H0) assumes no significant effect, while the alternative hypothesis (H1 or Ha) contradicts H0, suggesting a meaningful impact. Statistical testing is employed to decide between these hypotheses.

Please Login to comment...

• Machine Learning
• 10 Best Screaming Frog Alternatives in 2024
• 10 Best Serpstat Alternatives in 2024
• Top 15 Fastest Roller Coasters in the World
• 10 Best Mint Alternatives in 2024 (Free)
• 30 OOPs Interview Questions and Answers (2024)

Improve your Coding Skills with Practice

What kind of Experience do you want to share?

Programmathically

Introduction to the hypothesis space and the bias-variance tradeoff in machine learning.

In this post, we introduce the hypothesis space and discuss how machine learning models function as hypotheses. Furthermore, we discuss the challenges encountered when choosing an appropriate machine learning hypothesis and building a model, such as overfitting, underfitting, and the bias-variance tradeoff.

The hypothesis space in machine learning is a set of all possible models that can be used to explain a data distribution given the limitations of that space. A linear hypothesis space is limited to the set of all linear models. If the data distribution follows a non-linear distribution, the linear hypothesis space might not contain a model that is appropriate for our needs.

To understand the concept of a hypothesis space, we need to learn to think of machine learning models as hypotheses.

The Machine Learning Model as Hypothesis

Generally speaking, a hypothesis is a potential explanation for an outcome or a phenomenon. In scientific inquiry, we test hypotheses to figure out how well and if at all they explain an outcome. In supervised machine learning, we are concerned with finding a function that maps from inputs to outputs.

But machine learning is inherently probabilistic. It is the art and science of deriving useful hypotheses from limited or incomplete data. Our functions are not axioms that explain the data perfectly, and for most real-life problems, we will never have all the data that exists. Accordingly, we will not find the one true function that perfectly describes the data. Instead, we find a function through training a model to map from known training input to known training output. This way, the model gradually approximates the assumed true function that describes the distribution of the data. So we treat our model as a hypothesis that needs to be tested as to how well it explains the output from a given input. We do this using a test or validation data set.

The Hypothesis Space

During the training process, we select a model from a hypothesis space that is subject to our constraints. For example, a linear hypothesis space only provides linear models. We can approximate data that follows a quadratic distribution using a model from the linear hypothesis space.

Of course, a linear model will never have the same predictive performance as a quadratic model, so we can adjust our hypothesis space to also include non-linear models or at least quadratic models.

The Data Generating Process

The data generating process describes a hypothetical process subject to some assumptions that make training a machine learning model possible. We need to assume that the data points are from the same distribution but are independent of each other. When these requirements are met, we say that the data is independent and identically distributed (i.i.d.).

Independent and Identically Distributed Data

How can we assume that a model trained on a training set will perform better than random guessing on new and previously unseen data? First of all, the training data needs to come from the same or at least a similar problem domain. If you want your model to predict stock prices, you need to train the model on stock price data or data that is similarly distributed. It wouldn’t make much sense to train it on whether data. Statistically, this means the data is identically distributed . But if data comes from the same problem, training data and test data might not be completely independent. To account for this, we need to make sure that the test data is not in any way influenced by the training data or vice versa. If you use a subset of the training data as your test set, the test data evidently is not independent of the training data. Statistically, we say the data must be independently distributed .

Overfitting and Underfitting

We want to select a model from the hypothesis space that explains the data sufficiently well. During training, we can make a model so complex that it perfectly fits every data point in the training dataset. But ultimately, the model should be able to predict outputs on previously unseen input data. The ability to do well when predicting outputs on previously unseen data is also known as generalization. There is an inherent conflict between those two requirements.

If we make the model so complex that it fits every point in the training data, it will pick up lots of noise and random variation specific to the training set, which might obscure the larger underlying patterns. As a result, it will be more sensitive to random fluctuations in new data and predict values that are far off. A model with this problem is said to overfit the training data and, as a result, to suffer from high variance .

To avoid the problem of overfitting, we can choose a simpler model or use regularization techniques to prevent the model from fitting the training data too closely. The model should then be less influenced by random fluctuations and instead, focus on the larger underlying patterns in the data. The patterns are expected to be found in any dataset that comes from the same distribution. As a consequence, the model should generalize better on previously unseen data.

But if we go too far, the model might become too simple or too constrained by regularization to accurately capture the patterns in the data. Then the model will neither generalize well nor fit the training data well. A model that exhibits this problem is said to underfit the data and to suffer from high bias . If the model is too simple to accurately capture the patterns in the data (for example, when using a linear model to fit non-linear data), its capacity is insufficient for the task at hand.

When training neural networks, for example, we go through multiple iterations of training in which the model learns to fit an increasingly complex function to the data. Typically, your training error will decrease during learning the more complex your model becomes and the better it learns to fit the data. In the beginning, the training error decreases rapidly. In later training iterations, it typically flattens out as it approaches the minimum possible error. Your test or generalization error should initially decrease as well, albeit likely at a slower pace than the training error. As long as the generalization error is decreasing, your model is underfitting because it doesn’t live up to its full capacity. After a number of training iterations, the generalization error will likely reach a trough and start to increase again. Once it starts to increase, your model is overfitting, and it is time to stop training.

Ideally, you should stop training once your model reaches the lowest point of the generalization error. The gap between the minimum generalization error and no error at all is an irreducible error term known as the Bayes error that we won’t be able to completely get rid of in a probabilistic setting. But if the error term seems too large, you might be able to reduce it further by collecting more data, manipulating your model’s hyperparameters, or altogether picking a different model.

Bias Variance Tradeoff

We’ve talked about bias and variance in the previous section. Now it is time to clarify what we actually mean by these terms.

Understanding Bias and Variance

In a nutshell, bias measures if there is any systematic deviation from the correct value in a specific direction. If we could repeat the same process of constructing a model several times over, and the results predicted by our model always deviate in a certain direction, we would call the result biased.

Variance measures how much the results vary between model predictions. If you repeat the modeling process several times over and the results are scattered all across the board, the model exhibits high variance.

In their book “Noise” Daniel Kahnemann and his co-authors provide an intuitive example that helps understand the concept of bias and variance. Imagine you have four teams at the shooting range.

Team B is biased because the shots of its team members all deviate in a certain direction from the center. Team B also exhibits low variance because the shots of all the team members are relatively concentrated in one location. Team C has the opposite problem. The shots are scattered across the target with no discernible bias in a certain direction. Team D is both biased and has high variance. Team A would be the equivalent of a good model. The shots are in the center with little bias in one direction and little variance between the team members.

Generally speaking, linear models such as linear regression exhibit high bias and low variance. Nonlinear algorithms such as decision trees are more prone to overfitting the training data and thus exhibit high variance and low bias.

A linear model used with non-linear data would exhibit a bias to predict data points along a straight line instead of accomodating the curves. But they are not as susceptible to random fluctuations in the data. A nonlinear algorithm that is trained on noisy data with lots of deviations would be more capable of avoiding bias but more prone to incorporate the noise into its predictions. As a result, a small deviation in the test data might lead to very different predictions.

To get our model to learn the patterns in data, we need to reduce the training error while at the same time reducing the gap between the training and the testing error. In other words, we want to reduce both bias and variance. To a certain extent, we can reduce both by picking an appropriate model, collecting enough training data, selecting appropriate training features and hyperparameter values. At some point, we have to trade-off between minimizing bias and minimizing variance. How you balance this trade-off is up to you.

The Bias Variance Decomposition

Mathematically, the total error can be decomposed into the bias and the variance according to the following formula.

Remember that Bayes’ error is an error that cannot be eliminated.

Our machine learning model represents an estimating function \hat f(X) for the true data generating function f(X) where X represents the predictors and y the output values.

Now the mean squared error of our model is the expected value of the squared difference of the output produced by the estimating function \hat f(X) and the true output Y.

The bias is a systematic deviation from the true value. We can measure it as the squared difference between the expected value produced by the estimating function (the model) and the values produced by the true data-generating function.

Of course, we don’t know the true data generating function, but we do know the observed outputs Y, which correspond to the values generated by f(x) plus an error term.

The variance of the model is the squared difference between the expected value and the actual values of the model.

Now that we have the bias and the variance, we can add them up along with the irreducible error to get the total error.

A machine learning model represents an approximation to the hypothesized function that generated the data. The chosen model is a hypothesis since we hypothesize that this model represents the true data generating function.

We choose the hypothesis from a hypothesis space that may be subject to certain constraints. For example, we can constrain the hypothesis space to the set of linear models.

When choosing a model, we aim to reduce the bias and the variance to prevent our model from either overfitting or underfitting the data. In the real world, we cannot completely eliminate bias and variance, and we have to trade-off between them. The total error produced by a model can be decomposed into the bias, the variance, and irreducible (Bayes) error.

About Author

Related Posts

Best Guesses: Understanding The Hypothesis in Machine Learning

• February 22, 2024
• General , Supervised Learning , Unsupervised Learning

Machine learning is a vast and complex field that has inherited many terms from other places all over the mathematical domain.

It can sometimes be challenging to get your head around all the different terminologies, never mind trying to understand how everything comes together.

In this blog post, we will focus on one particular concept: the hypothesis.

While you may think this is simple, there is a little caveat regarding machine learning.

The statistics side and the learning side.

Don’t worry; we’ll do a full breakdown below.

You’ll learn the following:

What Is a Hypothesis in Machine Learning?

• Is This any different than the hypothesis in statistics?
• What is the difference between the alternative hypothesis and the null?
• Why do we restrict hypothesis space in artificial intelligence?
• Example code performing hypothesis testing in machine learning

In machine learning, the term ‘hypothesis’ can refer to two things.

First, it can refer to the hypothesis space, the set of all possible training examples that could be used to predict or answer a new instance.

Second, it can refer to the traditional null and alternative hypotheses from statistics.

Since machine learning works so closely with statistics, 90% of the time, when someone is referencing the hypothesis, they’re referencing hypothesis tests from statistics.

Is This Any Different Than The Hypothesis In Statistics?

In statistics, the hypothesis is an assumption made about a population parameter.

The statistician’s goal is to prove it true or disprove it.

This will take the form of two different hypotheses, one called the null, and one called the alternative.

Usually, you’ll establish your null hypothesis as an assumption that it equals some value.

For example, in Welch’s T-Test Of Unequal Variance, our null hypothesis is that the two means we are testing (population parameter) are equal.

This means our null hypothesis is that the two population means are the same.

We run our statistical tests, and if our p-value is significant (very low), we reject the null hypothesis.

This would mean that their population means are unequal for the two samples you are testing.

Usually, statisticians will use the significance level of .05 (a 5% risk of being wrong) when deciding what to use as the p-value cut-off.

What Is The Difference Between The Alternative Hypothesis And The Null?

The null hypothesis is our default assumption, which we are trying to prove correct.

The alternate hypothesis is usually the opposite of our null and is much broader in scope.

For most statistical tests, the null and alternative hypotheses are already defined.

You are then just trying to find “significant” evidence we can use to reject our null hypothesis.

These two hypotheses are easy to spot by their specific notation. The null hypothesis is usually denoted by H₀, while H₁ denotes the alternative hypothesis.

Example Code Performing Hypothesis Testing In Machine Learning

Since there are many different hypothesis tests in machine learning and data science, we will focus on one of my favorites.

This test is Welch’s T-Test Of Unequal Variance, where we are trying to determine if the population means of these two samples are different.

There are a couple of assumptions for this test, but we will ignore those for now and show the code.

You can read more about this here in our other post, Welch’s T-Test of Unequal Variance .

We see that our p-value is very low, and we reject the null hypothesis.

What Is The Difference Between The Biased And Unbiased Hypothesis Spaces?

The difference between the Biased and Unbiased hypothesis space is the number of possible training examples your algorithm has to predict.

The unbiased space has all of them, and the biased space only has the training examples you’ve supplied.

Since neither of these is optimal (one is too small, one is much too big), your algorithm creates generalized rules (inductive learning) to be able to handle examples it hasn’t seen before.

Here’s an example of each:

Example of The Biased Hypothesis Space In Machine Learning

The Biased Hypothesis space in machine learning is a biased subspace where your algorithm does not consider all training examples to make predictions.

This is easiest to see with an example.

Let’s say you have the following data:

Happy  and  Sunny  and  Stomach Full  = True

Whenever your algorithm sees those three together in the biased hypothesis space, it’ll automatically default to true.

This means when your algorithm sees:

Sad  and  Sunny  And  Stomach Full  = False

It’ll automatically default to False since it didn’t appear in our subspace.

This is a greedy approach, but it has some practical applications.

Example of the Unbiased Hypothesis Space In Machine Learning

The unbiased hypothesis space is a space where all combinations are stored.

We can use re-use our example above:

This would start to breakdown as

Happy  = True

Happy  and  Sunny  = True

Happy  and  Stomach Full  = True

Let’s say you have four options for each of the three choices.

This would mean our subspace would need 2^12 instances (4096) just for our little three-word problem.

This is practically impossible; the space would become huge.

So while it would be highly accurate, this has no scalability.

More reading on this idea can be found in our post, Inductive Bias In Machine Learning .

Why Do We Restrict Hypothesis Space In Artificial Intelligence?

We have to restrict the hypothesis space in machine learning. Without any restrictions, our domain becomes much too large, and we lose any form of scalability.

This is why our algorithm creates rules to handle examples that are seen in production. 

This gives our algorithms a generalized approach that will be able to handle all new examples that are in the same format.

Other Quick Machine Learning Tutorials

At EML, we have a ton of cool data science tutorials that break things down so anyone can understand them.

Below we’ve listed a few that are similar to this guide:

• Instance-Based Learning in Machine Learning
• Types of Data For Machine Learning
• Verbose in Machine Learning
• Generalization In Machine Learning
• Epoch In Machine Learning
• Inductive Bias in Machine Learning
• Understanding The Hypothesis In Machine Learning
• Zip Codes In Machine Learning
• get_dummies() in Machine Learning
• Bootstrapping In Machine Learning
• X and Y in Machine Learning
• F1 Score in Machine Learning
• Recent Posts

• How BI Systems Differ from Transaction Processing Systems [Unlock the Key Differences] - March 20, 2024
• What Does Computing Mean? Discover Its Empowering Role Today [Uncover the Hidden Power] - March 20, 2024
• How to Master Database Management System Tutorial for Beginners [Boost Your Skills Now] - March 20, 2024

Trending now

Encyclopedia of Systems Biology pp 931 Cite as

Hypothesis Space

• Eyke Hüllermeier 5 ,
• Thomas Fober 5 &
• Marco Mernberger 5  
• Reference work entry

123 Accesses

In machine learning, the goal of a supervised learning algorithm is to perform induction, i.e., to generalize a (finite) set of observations (the training data) into a general model of the domain. In this regard, the hypothesis space is defined as the set of candidate models considered by the algorithm.

More specifically, consider the problem of learning a mapping (model) \( f \in F = Y^X \) from an input space X to an output space Y , given a set of training data \( D = \left\{ {\left( {{x_1},{y_1}} \right),...,\left( {{x_n},{y_n}} \right)} \right\} \subset X \times Y \) . A learning algorithm A takes D as an input and produces a function (model, hypothesis) f ∈ H ⊂ F as an output, where H is the hypothesis space. This subset is determined by the formalism used to represent models (e.g., as logical formulas, linear functions, or non-linear functions implemented as artificial neural networks or decision trees ). Thus, the choice of the hypothesis space produces a representation...

This is a preview of subscription content, log in via an institution .

Buying options

• Available as PDF
• Read on any device
• Instant download
• Own it forever
• Available as EPUB and PDF
• Durable hardcover edition
• Dispatched in 3 to 5 business days
• Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Author information

Authors and affiliations.

Philipps-Universität Marburg, Hans-Meerwein-Straße, Marburg, Germany

Eyke Hüllermeier, Thomas Fober & Marco Mernberger

You can also search for this author in PubMed   Google Scholar

Corresponding author

Correspondence to Eyke Hüllermeier .

Editor information

Editors and affiliations.

Biomedical Sciences Research Institute, University of Ulster, Coleraine, UK

Werner Dubitzky

Department of Computer Science, University of Rostock, Rostock, Germany

Olaf Wolkenhauer

Department of Bio and Brain Engineering, Korea Advanced Institute of Science and Technology (KAIST), Daejeon, Republic of Korea

Kwang-Hyun Cho

Department of Biomedical Engineering, Rensselaer Polytechnic Institute, Troy, NY, USA

Hiroki Yokota

Rights and permissions

Reprints and permissions

Copyright information

© 2013 Springer Science+Business Media, LLC

About this entry

Cite this entry.

Hüllermeier, E., Fober, T., Mernberger, M. (2013). Hypothesis Space. In: Dubitzky, W., Wolkenhauer, O., Cho, KH., Yokota, H. (eds) Encyclopedia of Systems Biology. Springer, New York, NY. https://doi.org/10.1007/978-1-4419-9863-7_926

Download citation

DOI : https://doi.org/10.1007/978-1-4419-9863-7_926

Publisher Name : Springer, New York, NY

Print ISBN : 978-1-4419-9862-0

Online ISBN : 978-1-4419-9863-7

eBook Packages : Biomedical and Life Sciences Reference Module Biomedical and Life Sciences

Share this entry

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Publish with us

Policies and ethics

• Find a journal
• Track your research

Machine Learning

Artificial Intelligence

Control System

Supervised Learning

Classification, miscellaneous, related tutorials.

Interview Questions

• Send your Feedback to [email protected]

Help Others, Please Share

Learn Latest Tutorials

Transact-SQL

Reinforcement Learning

R Programming

React Native

Python Design Patterns

Python Pillow

Python Turtle

Preparation

Verbal Ability

Company Questions

Trending Technologies

Cloud Computing

Data Science

B.Tech / MCA

Data Structures

Operating System

Computer Network

Compiler Design

Computer Organization

Discrete Mathematics

Ethical Hacking

Computer Graphics

Software Engineering

Web Technology

Cyber Security

C Programming

Data Mining

Data Warehouse

ID3 Algorithm and Hypothesis space in Decision Tree Learning

The collection of potential decision trees is the hypothesis space searched by ID3. ID3 searches this hypothesis space in a hill-climbing fashion, starting with the empty tree and moving on to increasingly detailed hypotheses in pursuit of a decision tree that properly classifies the training data.

In this blog, we’ll have a look at the Hypothesis space in Decision Trees and the ID3 Algorithm. 

ID3 Algorithm: 

The ID3 algorithm (Iterative Dichotomiser 3) is a classification technique that uses a greedy approach to create a decision tree by picking the optimal attribute that delivers the most Information Gain (IG) or the lowest Entropy (H).

What is Information Gain and Entropy?  

Information gain: .

The assessment of changes in entropy after segmenting a dataset based on a characteristic is known as information gain.

It establishes how much information a feature provides about a class.

We divided the node and built the decision tree based on the value of information gained.

The greatest information gain node/attribute is split first in a decision tree method, which always strives to maximize the value of information gain. 

The formula for Information Gain: 

Entropy is a metric for determining the degree of impurity in a particular property. It denotes the unpredictability of data. The following formula may be used to compute entropy:

S stands for “total number of samples.”

P(yes) denotes the likelihood of a yes answer.

P(no) denotes the likelihood of a negative outcome.

• Calculate the dataset’s entropy.
• For each feature/attribute.

Determine the entropy for each of the category values.

Calculate the feature’s information gain.

• Find the feature that provides the most information.
• Repeat it till we get the tree we want.

Characteristics of ID3: 

• ID3 takes a greedy approach, which means it might become caught in local optimums and hence cannot guarantee an optimal result.
• ID3 has the potential to overfit the training data (to avoid overfitting, smaller decision trees should be preferred over larger ones).
• This method creates tiny trees most of the time, however, it does not always yield the shortest tree feasible.
• On continuous data, ID3 is not easy to use (if the values of any given attribute are continuous, then there are many more places to split the data on this attribute, and searching for the best value to split by takes a lot of time).

Over Fitting:  

Good generalization is the desired property in our decision trees (and, indeed, in all classification problems), as we noted before. 

This implies we want the model fit on the labeled training data to generate predictions that are as accurate as they are on new, unseen observations.

Capabilities and Limitations of ID3:

• In relation to the given characteristics, ID3’s hypothesis space for all decision trees is a full set of finite discrete-valued functions.
• As it searches across the space of decision trees, ID3 keeps just one current hypothesis. This differs from the prior version space candidate Elimination approach, which keeps the set of all hypotheses compatible with the training instances provided.
• ID3 loses the capabilities that come with explicitly describing all consistent hypotheses by identifying only one hypothesis. It is unable to establish how many different decision trees are compatible with the supplied training data.
• One benefit of incorporating all of the instances’ statistical features (e.g., information gain) is that the final search is less vulnerable to faults in individual training examples.
• By altering its termination criterion to allow hypotheses that inadequately match the training data, ID3 may simply be modified to handle noisy training data.
• In its purest form, ID3 does not go backward in its search. It never goes back to evaluate a choice after it has chosen an attribute to test at a specific level in the tree. As a result, it is vulnerable to the standard dangers of hill-climbing search without backtracking, resulting in local optimum but not globally optimal solutions.
• At each stage of the search, ID3 uses all training instances to make statistically based judgments on how to refine its current hypothesis. This is in contrast to approaches that make incremental judgments based on individual training instances (e.g., FIND-S or CANDIDATE-ELIMINATION ).

Hypothesis Space Search by ID3: 

• ID3 climbs the hill of knowledge acquisition by searching the space of feasible decision trees.
• It looks for all finite discrete-valued functions in the whole space. Every function is represented by at least one tree.
• It only holds one theory (unlike Candidate-Elimination). It is unable to inform us how many more feasible options exist.
• It’s possible to get stranded in local optima.
• At each phase, all training examples are used. Errors have a lower impact on the outcome.

Concept Learning in Machine Learning

Download final year projects, concept learning in machine learning – 17cs73.

The problem of inducing general functions from specific training examples is central to learning.

Concept learning can be formulated as a problem of searching through a predefined space of potential hypotheses for the hypothesis that best fits the training examples.

What is Concept Learning…?

“A task of acquiring potential hypothesis (solution) that best fits the given training examples.”

Consider the example task of learning the target concept “days on which my friend Prabhas enjoys his favorite water sport.”

Below Table describes a set of example days, each represented by a set of attributes. The attribute EnjoySport indicates whether or not Prabhas enjoys his favorite water sport on this day. The task is to learn to predict the value of EnjoySport for an arbitrary day, based on the values of its other attributes.

What hypothesis representation shall we provide to the learner in this case?

Let us begin by considering a simple representation in which each hypothesis consists of a conjunction of constraints on the instance attributes.

In particular, let each hypothesis be a vector of six constraints, specifying the values of the six attributes Sky, AirTemp , Humidity, Wind, Water, and Forecast.

For each attribute, the hypothesis will either

• indicate by a “?’ that any value is acceptable for this attribute,
• specify a single required value (e.g., Warm) for the attribute, or
• indicate by a “ø” that no value is acceptable.

If some instance x satisfies all the constraints of hypothesis h, then h classifies x as a positive example (h(x) = 1).

To illustrate, the hypothesis that Prabhas enjoys his favorite sport only on cold days with high humidity (independent of the values of the other attributes) is represented by the expression

(?, Cold, High, ?, ?, ?)

Most General and Specific Hypothesis

The most general hypothesis-that every day is a positive example-is represented by

(?, ?, ?, ?, ?, ?)

and the most specific possible hypothesis-that no day is a positive example-is represented by

(ø, ø, ø, ø, ø, ø)

A CONCEPT LEARNING TASK – Search

Concept learning can be viewed as the task of searching through a large space of hypotheses implicitly defined by the hypothesis representation.

The goal of this search is to find the hypothesis that best fits the training examples.

It is important to note that by selecting a hypothesis representation, the designer of the learning algorithm implicitly defines the space of all hypotheses that the program can ever represent and therefore can ever learn.

Instance Space

Consider, for example, the instances X and hypotheses H in the EnjoySport learning task.

Given that the attribute Sky has three possible values, and that AirTemp, Humidity, Wind, Water, and Forecast each have two possible values, the instance space X contains exactly 3 . 2 . 2 . 2 . 2 . 2 = 96 distinct instances.

Let’s assume there are two features F1 and F2 with F1 has A and B as possibilities and F2 as X and Y as possibilities.

F1  – > A, B

F2  – > X, Y

Instance Space: (A, X), (A, Y), (B, X), (B, Y) – 4 Examples

Hypothesis Space: (A, X), (A, Y), (A, ø ), (A, ?), (B, X), (B, Y), (B, ø ), (B, ?), ( ø , X), ( ø , Y), ( ø , ø ), ( ø , ?), ( ? , X), ( ? , Y), ( ? , ø ), ( ? , ?)  – 16

Hypothesis Space: (A, X), (A, Y), (A, ?), (B, X), (B, Y), (B, ?), ( ? , X), ( ? , Y ( ? , ?) – 10

Hypothesis Space

Similarly there are 5 . 4 . 4 . 4 . 4 . 4 = 5120 syntactically distinct hypotheses within H.

Notice, however, that every hypothesis containing one or more “ ø ” symbols represents the empty set of instances; that is, it classifies every instance as negative.

Therefore, the number of semantically distinct hypotheses is only 1 + ( 4 . 3 . 3 . 3 . 3 . 3 ) = 973.

Our EnjoySport example is a very simple learning task, with a relatively small, finite hypothesis space.

General-to-Specific Ordering of Hypotheses

To illustrate the general-to-specific ordering, consider the two hypotheses

h1 = (Sunny, ?, ?, Strong, ?, ?)

h2 = (Sunny, ?, ?, ?, ?, ?)

Now consider the sets of instances that are classified positive by hl and by h2. Because h2 imposes fewer constraints on the instance, it classifies more instances as positive.

In fact, any instance classified positive by h1 will also be classified positive by h2. Therefore, we say that h2 is more general than h1.

For any instance x in X and hypothesis h in H, we say that x satisjies h if and only if h(x) = 1.

We define the more_general_than_or_equal e_to relation in terms of the sets of instances that satisfy the two hypotheses.

This tutorial discusses the Concept Learning Tasks in Machine Learning. If you like the tutorial share with your friends. Like the Facebook page for regular updates and YouTube channel for video tutorials.

Related Posts

1 thought on “concept learning in machine learning”.

A very pleasant and comprehensible article.

Leave a Comment Cancel Reply

Your email address will not be published. Required fields are marked *

• Share full article

The Comet Strike Theory That Just Won’t Die

Mainstream science has done its best to debunk the notion, but a belief in a world-changing series of prehistoric impacts continues to gain momentum.

Credit... Photo illustration by Ricardo Tomás

Supported by

By Zach St. George

• Published March 5, 2024 Updated March 8, 2024

In 2007, a group of researchers, led by a nuclear physicist named Richard Firestone, announced an astonishing discovery. They had uncovered evidence, they said, that 12,900 years ago, a comet — or possibly a whole fleet of comets — struck Earth and changed the course of history. For the preceding two and a half million years, through the Pleistocene Epoch, the planet’s climate fluctuated between frozen stretches, called glacials, and warm interglacials. At that time, Earth was warming again, and the ice sheets that covered much of North America, Europe and Asia were in retreat. Mammoths, steppe bison, wild horses and other enormous mammals still wandered the Americas, pursued by bands of humans wielding spears with fluted stone blades. Suddenly, somewhere over the Upper Midwest — an explosion.

Listen to this article, read by Robert Petkoff

Open this article in the New York Times Audio app on iOS.

Presenting their claim in the Proceedings of the National Academy of Sciences, a top scientific journal, the researchers took the sober tone characteristic of such publications. But in “The Cycle of Cosmic Catastrophes,” a book published around the same time, two of the researchers described the scene more vividly. The impact caused the ground to shake and the sky to glow, they wrote. A hail of tiny molten particles sank into flesh and set forests ablaze. Soot blotted out the sun. Earth’s magnetic field wavered, and living things were bombarded by cosmic rays, confounding the navigational senses of turtles and porpoises, which beached themselves en masse. Addled birds plummeted from the sky.

Most disastrous of all, the impact shattered the ice dam holding back Lake Agassiz, a vast expanse of glacial meltwater that stretched across Manitoba, Ontario, Saskatchewan, Wisconsin and Minnesota. The lake cascaded into the Atlantic Ocean, where the freshwater pooled over the denser seawater, disrupting the convection current carrying warm water north from the tropics. The Northern Hemisphere plunged back into full-glacial cold.

For decades, scientists had puzzled over the cause of this rapid climatic reversal, which they marked by, among other things, the reappearance in southerly fossil deposits of tundra plants. These included the wildflower Dryas integrifolia, which gives the 1,200-year time span its name: the Younger Dryas. Here was an explanation: The impact caused the sudden cooling, the Firestone team argued, and contributed to the demise of the mammoths, steppe bison and other large Pleistocene mammals, along with the people who pursued them.

Researchers later claimed that the Younger Dryas impact prompted a turn toward agriculture in Eurasia and eventually, civilization. It might even have influenced, in surprising ways, the outlines of our current nation-states. James Kennett, a member of the Firestone team, told me that if the impact had not led to the extinction of horses in the Americas, Native Americans surely would have domesticated them and would therefore have presented more formidable opposition to European conquistadors; perhaps they would even have been conquistadors themselves. “So the whole setup of human culture would have been very different,” he says.

Drawing out this counterfactual, we might imagine that the people of Europe came to speak a dialect of Lakota or Nahuatl or Yanomamo, that Siegfried and Roy performed with saber-toothed tigers and cave lions. Absent the climate-changing effects of agriculture and industry, the world might now be tipping back into an ice age. In short, without the Younger Dryas impact, nearly everything would be different. “Our modern way of life had its genesis,” according to the authors of “The Cycle of Cosmic Catastrophes,” “in the thunderous flash of crashing comets.”

This cometary origin story, with its mix of ancient humans, vanished megafauna and global cataclysm, quickly spread beyond the confines of scientific journals. Media outlets around the world covered the Younger Dryas impact hypothesis. It has been the subject of two more books and multiple documentaries, including one produced by PBS NOVA. Joe Rogan has discussed the hypothesis a dozen times on his podcast, and it provided the scientific underpinnings for Netflix’s 2022 hit series “Ancient Apocalypse.” But even as the hypothesis wormed its way into the public imagination, an important question persisted: Was any of it true?

Many geologists, astronomers, archaeologists, paleoecologists and other scientists with relevant expertise were immediately skeptical and soon published studies of their own rebutting the Firestone team, which responded with rebuttals to the rebuttals. While heated debate is typical in science, the back-and-forth quickly veered outside the usual bounds. Today some proponents of the impact hypothesis insist that skeptics make up a small but vocal minority desperately trying to prevent the inevitable acceptance of the hypothesis as fact. Others take a darker view, suggesting that the ongoing resistance to the hypothesis is a result of a coordinated coverup by the scientific mainstream. Still other observers hear in the stubborn persistence of the Younger Dryas impact hypothesis an echo of some unsettling trends in the public discourse: a tendency toward tribalism and distrust, the conspiratorial embrace of supposedly forbidden knowledge, the seeming triumphs of narrative over truth.

William Topping, an archaeologist, was studying a site on Michigan’s Upper Peninsula where ancient North Americans quarried the raw materials to make their stone tools, when he encountered a puzzle in the late 1990s: Anthropological and geological evidence indicated that the layer of soil containing the ancient artifacts dated to around 12,900 years ago, just before the start of the Younger Dryas. But radiocarbon dating suggested that the layer was only 2,900 years old. Topping sent an email about his problem to Richard Firestone, an expert on radioactive isotopes who was then working at the Lawrence Berkeley National Laboratory. Firestone, who told me that he has always had an interest in side projects, agreed to help.

Their investigation progressed slowly at first. Then, in the early 2000s, Firestone and Topping were joined by Allen West, who had recently left his career as a geophysical consultant to the oil and mining industries. “I was retired and bored,” West says, so he “decided to write a book.” A small asteroid had recently made headlines when it passed between Earth and the moon. His literary agent suggested he write a book about the hazard of asteroid impacts. After West came across an article that Firestone and Topping had published about their early research, he wrote to Firestone and proposed that they join forces.

The planet’s surface is a churn of deposition and erosion, uplift and subsidence, which tend to erase from sight evidence of even the most consequential events. The asteroid thought to have killed off the dinosaurs some 66 million years ago, for instance, remains apparent to us primarily as a thin, iridium-rich layer of rock found around the world and as a faint inscription on the edge of the Yucatán Peninsula, where it struck. Evidence of a much earlier collision between Earth and a Mars-size body, which scientists think hewed away the material that became the moon, is even more subtle, derived from orbital models and chemical comparisons of Earth rocks and moon rocks.

Firestone, Topping and West pursued a similarly faint set of proxies. They were particularly interested in metallic spheres — so-called microspherules, each a fraction of the width of a human hair — that Topping discovered in abundance at the Michigan quarry and other archaeological sites. Topping found few of these orbs below the soil layer with the errant radiocarbon dates, and few after. But the anomalous soil, a carbon-rich layer that other archaeologists called the black mat, was full of them. The trio suspected that the orbs were a result of an impact, which, they thought, could have also reset the radiocarbon dates at the quarry site.

West began traveling the country, visiting archaeological sites that dated to the beginning of the Younger Dryas. At many of the sites he visited, he found the same black mat, containing the same types of microspherules. He reached out to other experts, and some of them joined the effort. Ted Bunch, a retired NASA chief of exobiology and an expert on meteorites, was one. Another was James Kennett, who was based at the University of California, Santa Barbara, and who had helped found the scientific field of paleo-oceanography. In 2000, he was elected to the National Academy of Sciences, “one of the highest honors that a scientist can receive,” according to the academy’s website. Kennett had studied the Younger Dryas climatic reversal for decades and even mused upon the possibility that it resulted from a cosmic impact. When West approached him, Kennett says, “I grabbed onto this immediately.”

The growing team exerted a kind of gravity, drawing other scientists into the effort: archaeologists, impact specialists, chemists, geologists, a polar explorer. Each was adept at recognizing and interpreting different kinds of scientific proxies. Led by Firestone, West and Kennett, the group put together its hypothesis the way that contractors build a house, with masons, carpenters, plumbers, electricians and drywallers each playing distinct but complementary roles. A result was a hypothesis of sweeping scope but also one that, paradoxically, West notes, was beyond the full grasp of any single scientist.

By 2007, the team was ready to publish its work. In Proceedings of the National Academy of Sciences, Firestone, Topping, West, Kennett and 22 co-authors laid out the evidence from 25 sites scattered across North America. Along with the magnetic and carbon microspherules, they had discovered granules with unusual concentrations of iridium, bits of glasslike carbon containing “nanodiamonds” and “fullerenes with ET helium.” Together, these proxies were a clear indication of extraterrestrial impact, they argued. Each reached its highest concentration in the black mat, which in turn was consistently found to be 12,900 years old — the start of the Younger Dryas.

At a news conference that several members of the Younger Dryas impact-research team gave just before the journal released its study, Kennett seemed to anticipate some of the debate to come. “I think it’s going to be very hard for the skeptics — and there will be a lot of skeptics for this, as there should be, it’s a big discovery — that there was an impact of this proportion,” he said. “It’s going to be very hard for the skeptics to take this range of evidence.”

When the paper came out, Jacquelyn Gill was working on her dissertation at the University of Wisconsin, Madison, studying the Pleistocene mammal populations of the Upper Midwest through the proxies of ancient pollen, charcoal and fungal spores. The impact hypothesis almost perfectly overlapped with her research. “They’re talking about fire, they’re talking about vegetation, they’re talking about megafauna,” she says — all of it centered on the Great Lakes. But Gill, now a paleoecologist at the University of Maine, was dubious. The hypothesis depended on synchronicity, on all the various lines of evidence aligning perfectly 12,900 years ago and pointing to a sudden, disastrous event. Gill thought the Firestone team had significantly overstated this alignment, at least in regard to the impact’s purported ecological effects. In the lake cores from the Younger Dryas that she was studying, there was no abrupt spike in charcoal to suggest catastrophic fires. Pollen evidence indicated changes to the vegetation consistent with a quickly cooling climate, not an impact. The fossils of bones and the spores of dung-dwelling fungi each suggested that many large mammals were already in decline or even extinct thousands of years before the onset of the Younger Dryas; many others lingered long after. “None of this is lining up,” Gill says she thought.

Other scientists reached similar conclusions. One group of researchers reported that they couldn’t find the nanodiamonds described by the Firestone team. Another group discovered no signs of a continentwide fire that coincided with the beginning of the Younger Dryas — or even a unique degree of burning at that time. Yet another group announced that, after an extensive search for microspherules and other proxies, it was “unable to reproduce any result of the Firestone et al. study” and found “no support for Younger Dryas extraterrestrial impact.” A review paper published in February 2011 summed up these efforts: Outside scientists had been unable to reproduce seven of the Firestone team’s 12 original lines of evidence for an impact; the other five lines of evidence resulted from ordinary earthly geological processes. The article was titled — in what would turn out to be wishful thinking — “The Younger Dryas Impact Hypothesis: A Requiem.”

As they tried to replicate the Firestone team’s findings, the skeptics noticed numerous odd details that seemed to hover around the hypothesis. There was, for example, “The Cycle of Cosmic Catastrophes,” which came out just before the Proceedings of the National Academy of Sciences study. The book’s publisher was a division of Inner Traditions, which, according to its website, is “devoted exclusively to the subjects of spirituality, the occult, ancient mysteries, new science, holistic health and natural medicine.” The book, written by West and Firestone, intersperses a breathless account of their work with the “astonishingly similar stories” of floods and celestial conflagrations from dozens of ancient cultures, including the tale of the “Long-Tailed-Heavenly-Climbing-Star,” attributed to the Ojibwa. “It clearly wasn’t a science book,” says Jennifer Marlon, a paleoecologist at Yale who read the book soon after seeing the PNAS study. “I just thought, Well, this is kind of silly.”

In 2011, an article by Rex Dalton in the magazine Pacific Standard revealed the strangest detail yet. Shortly before West first reached out to Firestone, he was convicted in California, under his given name, Allen Whitt, “for masquerading as a state-licensed geologist” while conducting groundwater surveys across the state. Not long before that, Dalton reported, a “new age” business that West owned in Sedona went under, and his geosciences business went bankrupt.

When Mark Boslough learned about West’s past — first revealed publicly in the Pacific Standard article, titled “Comet Theory Comes Crashing to Earth” — he says it “was kind of when a lightbulb went on over my head.” Boslough, an impact physicist at Los Alamos National Laboratory, helped pioneer the study of cosmic airbursts — meteors that explode in the atmosphere instead of striking the planet’s surface. He was skeptical of the Firestone team’s hypothesis from the beginning, but he says that he viewed them as essentially ordinary, if misguided, scientists. “Until that point, I took their evidence at face value,” he says. “I never believed that again.” To Boslough, West’s background was a sign of rot at the center of the impact hypothesis. He expected that West’s colleagues would abandon both him and the hypothesis, he says. “And the opposite happens.”

“The Cycle of Cosmic Catastrophes” “was never intended to be a scientific book,” Allen West says. “It’s strictly a popular attempt to talk about impacts.” The “new age” business was in fact a self-help organization, he told me. And his fraud conviction was really more of a bureaucratic mix-up, he says, resulting from his mistakenly failing to fill out the proper forms. Indeed, not long before Dalton’s article was published, a judge revised West’s verdict to “not guilty” and expunged the conviction from his record.

West suggests that the failure of many scientists to replicate the Firestone team’s results was because of a lack of understanding, improper methodology, obstinacy or even jealousy. He points to the long history of groundbreaking scientific hypotheses meeting with initial resistance. Galileo was committed to house arrest for his public backing of heliocentricity. Darwin was engulfed in controversy after he proposed evolution by natural selection. Alfred Wegener was ridiculed in the early 1900s when he suggested that continents drift. “That doesn’t mean that everything that scientists object to is true,” West says, “but it does mean that that’s the typical response to something new.”

Some of Firestone and West’s co-authors did distance themselves from the effort, but other scientists took their places. In 2016, West and several colleagues formed Comet Research Group Inc., which, according to its website, “cooperates with and provides funding for selected impact research scientists around the world.” The organization is a division of Rising Light Group, an Arizona-based nonprofit that “promotes public awareness and tolerance in a variety of fields, including religion, philosophy and science.” To skeptics of the impact hypothesis, this affiliation was another sign that something was amiss. But West, listed as a director of Rising Light Group, dismisses any suggestion that religion or mysticism has seeped into the scientific research on the Younger Dryas impact hypothesis. “We have scientists in our group of all kinds of religious persuasions, and to my knowledge, none of their beliefs have gotten into our papers,” he says. “Any scientist who judges the beliefs of a scientist outside of that paper, to me, that’s not good science.”

Joined by a growing cohort of collaborators, the Comet Research Group churned out new research, presenting such evidence as shock-synthesized hexagonal nanodiamonds from Santa Rosa Island, Calif.; siliceous scoria-like objects from Melrose, Pa., Blackville, S.C., and Abu Hureyra, Syria, as well as corundum, mullite, sessile and lechatelierite; elevated levels of chromium, iridium, copper, nickel and ruthenium in the sediments of western Russia’s Lake Medvedeskoye; planar deformation features, orthoclase and monazite in the northwestern Venezuelan Andes; and suggestive patterns in the eubacterial and paleosol chronosequences in the Mount Viso catchment of the Cottian Alps. What Topping and Firestone first uncovered at a single archaeological site in Michigan had expanded into, as one researcher put it, a “global cosmic catastrophe.”

These elements, minerals and geological forms are real. What many outside scientists continued to dispute was the hypothesizers’ interpretations of what these things meant. To the nonscientist, this back-and-forth is impenetrable. “It is very difficult for laypeople to assess whether something is true or not,” says Tiffany Morisseau, a social cognitive scientist at the University Paris Cité. She was part of an interdisciplinary team of experts commissioned by the European Union in the wake of the pandemic to investigate the decline of trust in experts. The group thought that, in a complicated world, there is no choice but to rely on experts. After all, everyone is a layperson in some facets of their existence. The plumber must at times place trust in the veterinarian, who at times relies on the engineer.

Looking to experts is one way that people employ what psychologists call “epistemic vigilance” — a kind of immune system for our individual conceptions of reality, allowing us to parse truth and falsehood. But this defense can be confounded in cases of contested expertise, with rows of Ph.D.s arrayed on each side, offering conflicting accounts. In such a situation, Morisseau says, a person might be tipped toward one understanding over another by how closely it aligns with previously held beliefs or political or cultural affiliations. A compelling story might make the difference.

In a recent paper, two psychologists at the University of California, Santa Barbara, Spencer Mermelstein and Tamsin German, have argued that pseudoscientific beliefs, which range from the relatively harmless (astrology, dowsing) to the deeply malignant (eugenics, Holocaust denial), tend to find cultural success when they hit a sweet spot of strangeness: too outlandish, and the epistemological immune system will reject it; too banal, and no one passes it on. What is most likely to take hold, Mermelstein says, is something that adds an intriguing twist to a person’s current sense of the world. The idea that a comet impact shaped many details of the modern world is not just surprising and interesting, he says; it also roughly fits most people’s previous understanding of Earth’s geologic past. And it’s simpler and more satisfying than alternative explanations for the events of the Younger Dryas. “It’s just like, one big cause, one big outcome,” Mermelstein says. “We can move on, right?”

There are now many dozens of videos about the Younger Dryas impact hypothesis on YouTube. Some YouTubers doubt the hypothesis or even try to debunk it, but many more treat it as true. In their retellings, the hypothesis takes on the sheen of legend, with new embellishments, new twists, new conclusions. Some YouTubers use the impact and its supposed connection to rapid climatic cooling to challenge the importance of modern anthropogenic contributions to climate change. Others tie the impact to biblical events. Skeptics of the hypothesis, meanwhile, swell into villains — members of the “scientific cabal,” as one YouTuber describes them, or victims of groupthink.

“What’s crazy is that this evidence has already existed for years but has been shunned by the mainstream scientific and academic communities,” says Jimmy Corsetti, who runs the YouTube channel “Bright Insight,” in one video. “They don’t want to talk about it, and the reason is, is because this is people’s livelihoods. A lot of people in the scientific community have become very wealthy.” Reporters, too, are complicit. “The failure to properly report evidence for the Younger Dryas Impact will one day be understood as the worst intellectual crime in the history of science journalism,” writes the Comet Research Group member George Howard, who describes himself as an “avocational expert” and “noncredentialed scientist,” on his blog, “The Cosmic Tusk.”

‘People are always looking for justification for their beliefs. If they can find archaeological evidence for something that happens in Genesis, brilliant.’

In November 2022, the impact hypothesis reached its biggest stage yet, in the hit Netflix documentary series “Ancient Apocalypse.” The show is hosted by Graham Hancock, a former correspondent for The Economist who has long been devoted to exploring what he calls “historical mysteries.” In the series, Hancock argues that a Pleistocene civilization possessed surprisingly advanced capabilities, including a detailed understanding of astronomical phenomena and the ability to accurately calculate longitude, a skill not mastered until the 18th century. This civilization — whose existence is rejected by mainstream archaeologists, whom Hancock refers to as “so-called experts” — was decimated by a cataclysm. As Hancock explains in the series finale, this disaster is now known as the Younger Dryas impact. In that episode, Allen West takes Hancock into the field to see the black mat and other evidence of the impact.

Netflix has reported that viewers spent almost 25 million hours watching “Ancient Apocalypse” during the first full week of its release, ranking it among the Top 10 English-language TV shows on Netflix in 31 countries. Within days of its debut, the Society for American Archaeology published an open letter urging the heads of Netflix to reclassify the show from “docuseries” to “science fiction,” insisting that the society’s members were not ignoring or suppressing credible evidence, as Hancock claimed. The society also argued that Hancock’s idea of an advanced ice-age civilization echoed and promoted “dangerous racist thinking.” (Hancock posted a detailed denial of these claims on his website.)

Articles rebutting (or ridiculing) the show appeared in The Guardian, Slate, The Nation and a host of other left-leaning publications. Conservative media outlets ran glowing reviews. “The propaganda press may not care about science, but they do care about controlling the public discourse for the benefit of the political left,” a reporter wrote in The Federalist, in an article titled “The Lying Media Told Me Not to Watch Netflix’s ‘Ancient Apocalypse,’ So I Did.” The Daily Caller, the conservative website co-founded by Tucker Carlson, declared the Society for American Archaeology an “elitist, closed-minded cabal,” linking its unchecked power to the “collapse of the American idea.” The debate over the show focused largely on Hancock’s lost civilization, including his discussion of Atlantis, which was wiped out, he said, during the Younger Dryas.

The widespread interest in the impact hypothesis outside academia can appear difficult to understand, says Tristan Sturm, a geographer at Queen’s University Belfast, who studies apocalyptic narratives and conspiracy theories. “Archaeology is not a superpopular topic,” he points out. Nor does grasping the truth about the impact hypothesis have obvious importance for the average person. But he says that it has a clear resonance with multiple cultural strains. Notably, it echoes and affirms Christian apocalyptic narratives, including that of Noah’s flood. “People are always looking for justification for their beliefs,” he says. “If they can find archaeological evidence for something that happens in Genesis, brilliant.”

More broadly, the hypothesis’ fringe status appeals to those who are experiencing what Sturm calls “conspiracism,” the reflexive distrust of authority figures, including politicians, journalists and, increasingly, scientists. A tendency toward conspiracism does not necessarily mean someone subscribes to actual conspiracy theories, Sturm says; rather, it is a gap in the epistemological immune system through which conspiracy theories enter.

About a year ago, Sturm’s colleague in the geography department, Maarten Blaauw, a paleoecologist, approached him about co-supervising a Ph.D. student who wanted to write a dissertation on the Younger Dryas impact hypothesis as a conspiracy theory. (Blaauw, who wrote a rebuttal to one early study purporting to provide evidence of the impact, says that over the past few years he has noticed that his first-year students arrive at class assuming the impact hypothesis to be true. “It’s TikTok,” he says.) Sturm stops short of calling the Younger Dryas impact hypothesis a conspiracy theory, but he says that it is an example, at least, of what he calls post-consensus: a rejection of objective truth, not dissimilar to what Kellyanne Conway, an official in the Trump administration, memorably called “alternative facts.”

When I asked Hancock to account for the success of his show, he told me: “I think a very large number of people are deeply interested in the human past and are deeply unsatisfied by the sterilized, anodyne, nuts-and-bolts, weigh-it, measure-it and count-it version of the human past that is provided and taught by archaeology.”

He went on: “People don’t like to be told what to think. They like to have a selection of views to pick and choose from. And I just think that the views I present resonate with a large number of people.”

Over the past three years, the Comet Research Group has had what it considers several notable triumphs. In July 2021, Martin Sweatman, a theoretical physicist at the University of Edinburgh and a blogger (“Prehistory Decoded”), published a long paper titled “The Younger Dryas Impact Hypothesis: Review of the Impact Evidence” in the journal Earth-Science Reviews. He concluded that the impact was “essentially confirmed” and that the hypothesis “should now be called a ‘theory’” — in effect, placing it alongside the theories of evolution, relativity and other foundations of scientific understanding. Sweatman rejected arguments against the hypothesis. These, he wrote, stemmed from a “small cohort of researchers” whose arguments were, “in general, poorly constructed.”

Soon after that, in September, in the journal Scientific Reports, members of the Comet Research Group published the results of their investigation into a more recent cosmic disaster: the destruction, roughly 3,600 years ago, of Tall el-Hammam, a city in what is now Jordan. Based on an array of proxies, they concluded that Tall el-Hammam had been destroyed by the airburst of a meteorite whose power, they calculated, was approximately equal to 1,000 Hiroshima-size nuclear bombs. The dig there was led by a self-described biblical archaeologist. I first learned about the study on Drudge Report, where it ran under the headline, “Ruins of Biblical City Sodom Found?”

In 2022, Eugene Jhong, a philanthropist who was an early Google employee, donated $1.25 million to two universities to fund the Comet Research Group’s efforts. Jhong told me by email that he learned about the impact hypothesis from one of Hancock’s books or lectures and was inspired to donate. Aided by this funding, the Comet Research Group last year published a series of papers presenting what some of its members believe to be among the best evidence yet for an impact.

In the course of publishing this work, though, members of the Comet Research Group say they have encountered signs that their opponents have moved from simply voicing skepticism to actively trying to suppress their research. Despite receiving several favorable peer reviews on a paper submitted to a scientific journal, group leaders told me, the journal’s editor summarily rejected it. In response, they started their own scientific journal, called Airbursts and Cratering Impacts, whose editors include West and two other Comet Research Group members. All three assured me that submissions to the journal are peer-reviewed according to the usual best practices; so far, the journal has published six papers from the group.

While they acknowledge that the journal might appear, to some people, to be self-dealing, group members that I spoke with insisted that it was, in some cases, a necessary step toward bringing their work to the public. Malcolm LeCompte, a retired solar-system astrophysicist and one of the Comet Research Group members who edits the new journal, told me in an email that he and some of his colleagues thought that a small number of “very influential scientists” had been actively working as “Gate Keepers, to prevent our publications from being fairly reviewed.”

When we later spoke over the phone, I asked LeCompte if he could name some of these gatekeepers. He noted the longtime skepticism of Vance Holliday, an archaeologist and geologist at the University of Arizona, Tucson, and the lead author of a recent paper titled “Comprehensive Refutation of the Younger Dryas Impact Hypothesis.” But upon further reflection, he said he didn’t think Holliday deserved that label. Indeed, the Comet Research Group members I talked with only felt comfortable pointing to one publicly: Mark Boslough, the Los Alamos impact physicist, who has spent years working to refute the Younger Dryas impact hypothesis. Boslough seemed amused when I asked him about LeCompte’s characterization. But he suggested there was no organized opposition to the impact hypothesis, only a large number of unaffiliated scientists who simply remained skeptical.

For more than a year, Boslough had been keeping me apprised of his activities, which included posting rebuttals to the Comet Research Group’s papers on social media and science forums; sending letters to journal editors; and writing popular articles for The Skeptical Inquirer and other outlets. Last September, Boslough told me that he had gotten wind of what he said was an active inquiry that would discredit the impact hypothesis. More recently, though, he told me it could be months or even years before any results became public.

I began to wonder if, in trying to draw connections between the various oddities that swirled around the Comet Research Group, Boslough was himself falling into a kind of conspiratorial thinking. “I have indeed asked myself that question,” he told me. But after careful consideration, he had concluded that he was not.

Comet Research Group members predicted to me that skeptics like Boslough could never be persuaded, only waited out. “You know that old saying,” West told me. “ ‘Science advances one funeral at a time.’” During one of my conversations with him, I asked — as I did of nearly everyone I spoke with, on both sides of the issue — whether he ever harbored any doubts. Was there any kind of evidence that might convince him that he was wrong?

In a sense, what West and his collaborators think now hardly matters. The hypothesis has already penetrated deeply, and perhaps indelibly, into the public imagination, seemingly on its way to becoming less a matter of truth than a matter of personal and group identity. Nobody I spoke with seemed to think it would go away soon, if ever. West, though, took a measured view. “All we can say is this is a hypothesis,” he said. “It’s still a debate. We may be wrong; we may be right. But only time will tell.”

Landscape: source material from Old Books Images/Alamy

An earlier version of this article referred incorrectly to an organization that urged Netflix to reclassify its “Ancient Apocalypse” series as science fiction. It is the Society for American Archaeology, not the Society for American Archaeologists.

An earlier version of this article misspelled the surname of a scientist at the University Paris Cité. She is Tiffany Morisseau, not Morriseau.

How we handle corrections

What’s Up in Space and Astronomy

Keep track of things going on in our solar system and all around the universe..

Never miss an eclipse, a meteor shower, a rocket launch or any other 2024 event  that’s out of this world with  our space and astronomy calendar .

A nova named T Coronae Borealis lit up the night about 80 years ago. Astronomers say it’s expected to put on another show  in the coming months.

Voyager 1, the 46-year-old first craft in interstellar space which flew by Jupiter and Saturn in its youth, may have gone dark .

Two spacecraft have ended up askew on the moon this year, illustrating that it’s not so easy to land upright on the lunar surface. Here is why .

What do you call a galaxy without stars? In addition to dark matter and dark energy, we now have dark galaxies  — collections of stars so sparse and faint that they are all but invisible.

Is Pluto a planet? And what is a planet, anyway? Test your knowledge here .

Advertisement

What is emergent gravity, and will it rewrite physics?

The idea is still new and requires a lot of assumptions in its calculations to make it work. Over the years, experimental results have been mixed.

In 2009, theoretical physicist Erik Verlinde proposed a radical reformulation of gravity. In his theory, gravity is not a fundamental force but rather a manifestation of deeper hidden processes. But in the 15 years since then, there hasn't been much experimental support for the idea. So where do we go next?

Emergence is common throughout physics. The property of temperature, for example, isn't an intrinsic property of gases. Instead, it's the emergent result of countless microscopic collisions. We have the tools to match those microscopic collisions to temperature; indeed, there is an entire branch of physics, known as statistical mechanics, that makes these connections known.

In other areas, the connections between microscopic behaviors and emergent properties aren't so clear. For example, while we understand the simple mechanisms behind superconductivity, we do not know how microscopic interactions lead to the emergence of high-temperature superconductors.

Related: Why Einstein must be wrong: In search of the theory of gravity

Verlinde's theory is based on what Stephen Hawking and Jacob Bekenstein observed in the 1970s: Many properties of black holes can be expressed in terms of the laws of thermodynamics. However, the laws of thermodynamics are themselves emergent from microscopic processes. To Verlinde, this was more than a mere coincidence and indicated that what we perceive as gravity may be emerging from some deeper physical process.

In 2009, he published the first version of his theory . Crucially, we do not need to know what those deeper processes are, since we already have the tool kit — statistical mechanics — for describing emergent properties. So Verlinde applied these techniques to gravity and arrived at an alternate formulation of gravity. And because gravity is also tied to our concepts of motion, inertia, space and time, this means our entire universe is also emergent from those same deeper processes. 

At first, not much came of this; rewriting a known law of physics, while interesting, doesn't necessarily provide deeper insights. But in 2016, Verlinde expanded his theory by discovering that a universe containing dark energy naturally leads to a new emergent property of space, thus allowing it to push inward on itself in regions of low density.

Get the Space.com Newsletter

Breaking space news, the latest updates on rocket launches, skywatching events and more!

This discovery led to a flurry of excitement, as it provided an alternative explanation for dark matter . Currently, astronomers believe that dark matter is a mysterious, invisible substance that makes up the bulk of all the mass of every galaxy . While that hypothesis has been able to explain a vast wealth of observations, from the rotation rates of stars within galaxies to the evolution of the largest structures in the cosmos, we have yet to identify the mysterious particle.

In Verlinde's picture of emergent gravity, as soon as you enter low-density regions — basically, anything outside the solar system — gravity behaves differently than we would expect from Einstein's theory of general relativity . At large scales, there is a natural inward pull to space itself, which forces matter to clump up more tightly than it otherwise would.

This idea was exciting because it allowed astronomers to find a way to test this new theory. Observers could take this new theory of gravity and put it in models of galaxy structure and evolution to find differences between it and models of dark matter.

Over the years, however, the experimental results have been mixed. Some early tests favored emergent gravity over dark matter when it came to the rotation rates of stars. But more recent observations haven't found an advantage. And dark matter can also explain much more than galaxy rotation rates; tests within galaxy clusters have found emergent gravity coming up short.

— Is the origin of dark matter gravity itself?

— Why is gravity so weak? The answer may lie in the very nature of space-time

— 'Quantum gravity' could help unite quantum mechanics with general relativity at last  

This isn't the end of emergent gravity. The idea is still new and requires a lot of assumptions in its calculations to make it work. Without a fully realized theory, it's hard to tell if the predictions it makes for the behavior of galaxies and clusters accurately represent what emergent gravity would tell us. And astronomers are still trying to develop more stringent tests, like using data from the cosmic microwave background , to really put the theory through its paces.

Emergent gravity remains an intriguing idea. If it's correct, we would have to radically reshape our understanding of the natural world and see gravity and motion — and even more fundamental concepts, like time and space — through a lens of emergence from deeper, more complicated interactions. But for right now, it remains simply an intriguing idea. Only time and extensive observational testing will tell us if we're on the right track.

Join our Space Forums to keep talking space on the latest missions, night sky and more! And if you have a news tip, correction or comment, let us know at: [email protected].

Paul M. Sutter is an astrophysicist at SUNY Stony Brook and the Flatiron Institute in New York City. Paul received his PhD in Physics from the University of Illinois at Urbana-Champaign in 2011, and spent three years at the Paris Institute of Astrophysics, followed by a research fellowship in Trieste, Italy, His research focuses on many diverse topics, from the emptiest regions of the universe to the earliest moments of the Big Bang to the hunt for the first stars. As an "Agent to the Stars," Paul has passionately engaged the public in science outreach for several years. He is the host of the popular "Ask a Spaceman!" podcast, author of "Your Place in the Universe" and "How to Die in Space" and he frequently appears on TV — including on The Weather Channel, for which he serves as Official Space Specialist.

Scientists find galaxy supercluster as massive as 26 quadrillion suns

3D map of over 1 million black holes traces where the universe's dark matter lies

SpaceX's Starship could fly again as soon as early May

Most Popular

By Mike Wall March 20, 2024

By Robert Lea March 20, 2024

By Josh Dinner March 20, 2024

By Mike Wall March 19, 2024

By Joe Rao March 19, 2024

By Meredith Garofalo March 19, 2024

By Josh Dinner March 19, 2024

By Jeff Spry March 19, 2024

By Robert Lea March 19, 2024

By Keith Cooper March 19, 2024

• 2 China launches satellite to support future moon missions (video)
• 3 1 in 12 stars might have swallowed a planet
• 4 Whee! Zip down from the launch tower in SpaceX's new emergency-escape slide (video)
• 5 Wage interplanetary war in upcoming space strategy game 'Falling Frontier' (video)

Why Have a Telescope in Space?

Hubble's unique characteristics – like its broad wavelength coverage, its position beyond Earth's atmosphere, and its assortment of science instruments – have made it one of astronomy's most important tools.

Quick Facts

Hubble can see astronomical objects with an angular size of 0.05 arcsecond, which is like seeing a pair of fireflies in Tokyo that are less than 10 feet (3 m) apart from Washington, D.C.

Hubble's detectors are so sensitive that they could spot a four-watt night-light on the Moon from Earth.

Hubble can see wavelengths of light ranging from the ultraviolet, through the visible, and into the near-infrared, including some you can't see from the ground.

Hubble was designed as a general purpose observatory, meant to explore the universe in visible, ultraviolet, and infrared wavelengths. To date, the telescope has studied a vast array of cosmic objects, providing views that astronomers were unable to capture from the ground.

In addition to blocking certain wavelengths of light altogether, Earth’s atmosphere is made up of shifting pockets of air that cause the twinkling appearance of stars in the night sky. This motion blurs images captured by telescopes on the ground. Hubble was placed into orbit above the atmosphere to avoid these effects.

As the telescope orbits Earth, its mirror gathers light from the cosmos, collecting images and data. For some of deepest images, the telescope has stared at the same point in the sky for days to capture the dim glow of the distant universe.

Hubble Advantages

Distortion-free.

When you look up at the stars in the sky, they seem to twinkle. That’s because shifting pockets of air in Earth’s atmosphere distort your view, even on the clearest nights. When astronomers try to take a picture of a cosmic object from the ground, that picture ends up blurry. Hubble’s position above the atmosphere lets it observe steady, unwavering light from cosmic objects and thus achieve higher resolution.

More Wavelengths

Earth’s atmosphere stops certain wavelengths of light. This is good for life on our planet, since it keeps away some dangerous radiation, but it also blinds us to some of the light emitted by cosmic objects. Hubble is designed to view some ultraviolet and infrared wavelengths, which are blocked or partially blocked by the atmosphere, as well as visible light. Hubble’s perceptiveness to this broad range of wavelengths captures details that would otherwise be invisible, revealing a wealth of information about cosmic objects.

Hubble has extremely high angular resolution, or ability to distinguish between two objects that are very close together. If your eyes had Hubble’s resolution, you could read the date on a dime two miles away. For Hubble, this means it can see fine details in star-forming nebulae, galaxies, and other cosmic objects.

Orbiting 320 miles (515 km) above the surface, Hubble doesn’t have to contend with any form of light pollution or weather conditions. This gives it a permanent, clear dark sky, which, combined with its sharp vision, allows it to see objects 10 times fainter than those that can be observed from Earth even by the largest telescopes.

Serviceability

Hubble was positioned just above Earth’s atmosphere, in what’s called a low-Earth orbit, so it would be unaffected by Earth’s atmosphere. That position only hundreds of miles above the surface also meant that Hubble could be repaired and upgraded by astronauts, enabling the repair of its original mirror flaw, keeping the telescope in orbit for more than three decades and continuously renewing it with the newest technology.

Hubble has multiple science instruments dedicated to probing the universe in unique ways. Hubble’s cameras observe the universe in a variety of wavelengths, including those invisible to the human eye. Its spectrometers dissect light into its component colors, exposing details like chemical composition and temperature. Its interferometers are used for aiming the telescope but can also measure the relative positions and brightnesses of stars.

Technology Benefits

Hubble technological advancements have benefited science but also has benefited people on Earth in unique ways, from cancer screening to wildlife preservation.

Hubble Discoveries

Browse through highlights of Hubble's scientific journey, see stunning images of Hubble's discoveries, and learn the science behind those iconic images.

Hubble's Online Activities

Find your Hubble birthday image, take interactive tours, play games, and more.

Explore More

Hubble Views a Galaxy Under Pressure

Hubble Tracks Jupiter’s Stormy Weather

NASA’s Roman Team Selects Survey to Map Our Galaxy’s Far Side

NASA’s Nancy Grace Roman Space Telescope team has announced plans for an unprecedented survey of the plane of our Milky Way galaxy. It will peer deeper into this region than any other survey, mapping more of our galaxy’s stars than all previous observations combined. “There’s a really broad range of science we can explore with […]

Mysterious 'unparticles' may be pushing the universe apart, new theoretical study suggests

New theoretical research suggests that a mysterious form of matter called "unparticles" could be the driving force behind the expansion of the universe.

The ever-accelerating expansion of the universe may be driven by a mysterious form of matter called "unparticles," which do not obey the Standard Model of particle physics , a new theoretical paper suggests.

Scientists widely acknowledge that the universe is expanding , though the cause of that expansion remains elusive. One of the most popular proposed explanations is a mysterious entity called dark energy in the form of a cosmological constant, which leads to expansion at a rate independent of the age of the universe and the temperature of matter and radiation. However, recent astronomical observations challenge this hypothesis , prompting physicists to explore alternatives to what dark energy could be.

Now, in a new paper, researchers analyzed the idea that dark energy is instead made of a theoretical form of matter called unparticles. They found that this theory aligns better with observations than the prevalent standard cosmological model, which assumes a cosmological constant.

"Observationally, discrepancies arise in the values of the universe's expansion rate and the growth of large-scale structures [galaxies and galactic clusters] between measurements," study co-author Utkarsh Kumar , a cosmologist at Ariel University, told Live Science in an email. "Various observations, including Cosmic Microwave Background measurements, dimming of supernovae and many others, contribute to this tension."

Related: There may be a 'dark mirror' universe within ours where atoms failed to form, new study suggests

Quantities such as the Hubble constant, which determines the rate of expansion, and the so-called S8, which contains information about the formation of large-scale structures, are not measured directly. Instead, they are calculated from observations of the cosmic microwave background (leftover radiation from the Big Bang ) and distant stars and galaxies, using mathematical  theories. However, different theories yield different values of these parameters from the same data, posing a huge tension in cosmology .

To address this problem, the authors of the new study, published in December 2023 in the Journal of Cosmology and Astroparticle Physics , suggest that the expansion of the universe is driven not by a cosmological constant but by unparticles, which had previously been considered in the context of particle physics.

Sign up for the Live Science daily newsletter now

Get the world’s most fascinating discoveries delivered straight to your inbox.

"The idea of unparticles was introduced by [theoretical physicist Howard] Georgi over a decade ago," lead study author Ido Ben-Dayan , also of Ariel University, told Live Science in an email. "In fundamental physics, we usually discuss fields, like the electric field, where particles are excitations of that field. In the electric field case, these are the photons," or packets of light. In almost all cases, Ben-Dayan added, particles are excitations with a well-defined mass and momentum.

However, "unparticles are the result of a set of fields that their excitations do not have a well-defined momentum and mass," Ben-Dayan said. "Thus, at the macroscopic level, they behave as a fluid. A special outcome of this property is that their equation of state, describing the ratio between the pressure they exert and their energy density, depends on temperature."

This equation of state strongly resembles the equation for the cosmological constant. Moreover, the very weak interaction of unparticles with “regular” matter, which is predicted by all theoretical models of the substance, makes it an excellent candidate for dark energy.

Unparticles untangled

In their work, Ben-Dayan and Kumar used the unparticle hypothesis instead of the cosmological constant and combined it with observational data collected from many experiments. They found that, unlike the values calculated using the standard cosmological model, the values of the Hubble constant and the S8 parameter deduced from these experiments were consistent with each other when they used the unparticle theory.

"Moreover, their model reduced the discrepancy between the measurements of the Hubble constant and S8, thus restoring the agreement between the different measurements, Kumar said. 

For now, there is no empirical evidence to back up this theory.  However, the authors are confident that, in the next decade or so, the accuracy of astronomical measurements will improve enough to determine whether the unparticle theory is correct.

"Our model is tested by constantly improving cosmological observations," Ben-Dayan said. "If it is correct, future Cosmic Microwave Background experiments should [confirm it]."

Experiments to measure the nature of dark energy are currently being developed, but will require telescopes to "probe further back in time " than they currently do, Ben-Dayan added.

— The James Webb telescope may have found some of the very 1st stars in the universe

— James Webb telescope detects oldest 'dead' galaxy in the known universe — and its death could challenge cosmology

— James Webb telescope reveals collection of ancient galaxies that 'transformed the entire universe'

Moreover, the physicists plan to increase the accuracy of their calculations and look for possible manifestations of unparticles in more familiar experiments with elementary particles in accelerators, which could be affected by the presence of unparticles.

"We plan to consider interactions between unparticles and the Standard Model of elementary particles," Kumar said. "This can further test our model. We will further study some extensions of our model and their cosmological consequences."

Andrey got his B.Sc. and M.Sc. degrees in elementary particle physics from Novosibirsk State University in Russia, and a Ph.D. in string theory from the Weizmann Institute of Science in Israel. He works as a science writer, specializing in physics, space, and technology. His articles have been published in  Elements ,  N+1 , and  AdvancedScienceNews .

World's smallest particle accelerator is 54 million times smaller than the Large Hadron Collider, and it works

What is the strong force?

2,000-year-old carvings of celestial bodies and animals discovered on rocky cliffs in Brazil

Most Popular

By Robert Lea March 19, 2024

By Jennifer Nalewicki March 19, 2024

By Brandon Specktor March 19, 2024

By Stephanie Pappas March 18, 2024

By Robert Lea March 18, 2024

By Kelly Carroll March 18, 2024

By Harry Baker March 18, 2024

By Elise Poore March 18, 2024

By Keumars Afifi-Sabet March 18, 2024

By Sharmila Kuthunur March 18, 2024

• 2 Mass grave of plague victims may be largest ever found in Europe, archaeologists say
• 3 India's evolutionary past tied to huge migration 50,000 years ago and to now-extinct human relatives
• 4 1,900-year-old coins from Jewish revolt against the Romans discovered in the Judaen desert
• 5 Dying SpaceX rocket creates glowing, galaxy-like spiral in the middle of the Northern Lights
• 2 'Flow state' uncovered: We finally know what happens in the brain when you're 'in the zone'
• 3 12 surprising facts about pi to chew on this Pi Day
• 4 James Webb telescope confirms there is something seriously wrong with our understanding of the universe

March 8, 2024

The Scale of Space Will Break Your Brain

The scale of the cosmos exceeds the bounds of human comprehension. But that doesn’t mean the universe is beyond our understanding

By Phil Plait

Hasbi Sahin/Getty Images

Space is big. That’s why we call it space .

But how big is “big”?

That’s relative. When an astronomer says something is nearby, they might mean a few million kilometers (if they’re talking asteroids) or a few tens of trillions (for stars) or a few tens of quintillions (for galaxies).

On supporting science journalism

If you're enjoying this article, consider supporting our award-winning journalism by subscribing . By purchasing a subscription you are helping to ensure the future of impactful stories about the discoveries and ideas shaping our world today.

No matter the destination, it’s a long walk. We make it easier on ourselves by using huge units to measure distance, such as a light-year, the distance traveled in a year by light—the fastest thing in the universe. A light-year is about 10 trillion kilometers. But that’s still pretty abstract to the casual person reading about “nearby” exoplanets or “distant” galaxies.

One way to help you grasp this scale is to take it step by step.

The moon is the closest astronomical object to us in the entire universe. On average it’s about 380,000 km from Earth. That’s already a pretty long way; nearly 30 Earths could fit side-by-side over that distance! Or think of it this way: the Apollo astronauts, traveling faster than any human before them, still took three days to reach the moon’s vicinity.

And that’s the closest heavenly body.

The sun is about 400 times farther away from us than the moon is: 150 million km. How far is that? If you could pave a road between Earth and the sun, at highway speeds, it would take you about 170 years to drive there. Better pack a lunch. A commercial jet would be better—it would take a mere 17 years.

When working with objects inside the solar system, it’s convenient to use the Earth-sun distance as a sort of cosmic meter stick. We call it the astronomical unit, or AU, and it’s defined by the International Astronomical Union (the keepers of all astronomical numbers, names and other agreed-upon conventions) to be exactly 149,597,870.7 km. In these terms, Mercury is about 0.4 AU from the sun and Venus about 0.7. Their distances from Earth depend on where all the planets are in their orbit—and increase when respective planets are on opposite sides of the sun—so Venus actually ranges from about 0.3 to 1.7 AU from Earth.

Neptune , the farthest major planet from the sun, is 4.5 billion km out, or 30 AU. Pluto’s at roughly the same distance, and it’s a long way from us. The New Horizons spacecraft took more than nine years to get there despite moving at speeds of more 50,000 km per hour.

These numbers are still difficult to grasp. When I used to travel to schools to give demonstrations to kids about astronomy, one of my favorite props was the solar system rope: a hefty 50-foot cord that represented the average sun-Pluto distance. The students were given printouts of planet photographs, and we’d place them at the proper scaled distance from the sun. The inner four planets were so close together that the kids were practically on top of one another, but the outer planets were spread out a long way; we had to either find a long hallway or go outside for the demo.

That demo proved so popular that I created a spreadsheet allowing anyone to calculate the solar system to scale . It’s based on the size of the sun, so you can change it from the default of one meter to, say, the size of a grape and find out how big and how far-off the planets become. (You have to download it to change the values; the link goes to a read-only version.) It’s fun—and eye-opening.

But it’s useful , too, to consider the separation between objects in terms of their size. For example, the sun is 1.4 million km wide. The nearest star system to the sun is Alpha Centauri, which is 41 trillion km away. If we divide the two numbers, Alpha Centauri is about 30 million “suns” away. Stars are very small compared to the distance between them, and that is one reason why you really don’t need to worry about one ever colliding with our sun!

That’s also why we use light-years to measure these distances; it’s a more palatable unit when dealing with interstellar journeys. Alpha Centauri is 4.3 light-years away. The Orion Nebula is about 1,250 light-years from the sun. The center of the Milky Way is 26,000 light-years away, and the galaxy itself is a flattish disk some 120,000 light-years across .

The nearest big galaxy to the Milky Way is Andromeda, which is 2.5 million light years from us. That’s an interesting number because it’s “only” 20 times the size of the Milky Way itself. On the scale of galaxy sizes, galaxies are actually pretty close together.

Inside galaxies, collisions between stars are extremely rare because they’re so far apart relative to their size. But galaxies themselves are more crowded, so it’s not too big a surprise that galaxy collisions are not only common, they’re ubiquitous. The Milky Way grew to its huge size by colliding and merging with other galaxies, and in fact every big galaxy has undergone multiple collisions.

The Milky Way and Andromeda are the two biggest galaxies in a clutch of about 100 galaxies that we call the Local Group. It’s about 10 million light-years across. Bigger and more populous groups exist called galaxy clusters. The nearest big one is the Virgo Cluster, with well more than 1,000 galaxies in it, located about 50 million light-years from us, though smaller groups exist that are closer to us.

Galaxy clusters are held together by the gravity of their members and can be tens of millions of light-years wide. But we’re not done! Clusters can clump up in the cosmos to form clusters of clusters called superclusters. The Virgo Cluster and the Local Group are part of one called the Laniakea supercluster , which may have more than 100,000 galaxies in it and stretches for 500 million light-years.

The universe is 13.8 billion years old, so you might think the most distant objects we can see are roughly that distance away in light-years. But the cosmos is expanding, and in the time it’s taken for the light from distant objects to reach us, that expansion has swept them farther from us. Because of this, the observable universe is estimated to be more like 90 or so billion light-years across!

After all that, I’ll let you in on a secret: even astronomers can’t truly grasp these scales. We work with them and we can do the math and physics with them, but our ape brain still struggles to comprehend even the distance to the moon—and the universe is 2 million trillion times bigger than that.

So yeah—space is big . And it’s true that we are very, very small. These scales can seem crushing. But I’ll leave you with this: while the cosmos is immense beyond what we can grasp, using math and physics and our brain, we can actually understand it.

And that makes us pretty big, too.