research paper about calculus pdf

Accessibility Links

• Skip to content
• Skip to search IOPscience
• Skip to Journals list
• Accessibility help
• Accessibility Help

Click here to close this panel.

Purpose-led Publishing is a coalition of three not-for-profit publishers in the field of physical sciences: AIP Publishing, the American Physical Society and IOP Publishing.

Together, as publishers that will always put purpose above profit, we have defined a set of industry standards that underpin high-quality, ethical scholarly communications.

We are proudly declaring that science is our only shareholder.

Analysis of Calculus Learning Beliefs and Students' Understanding about Limit of Functions: A Case Study

Usman 1 , RM Bambang 1 , S, M. Hasbi 1 and MZ Mardhiah 1

Published under licence by IOP Publishing Ltd Journal of Physics: Conference Series , Volume 1462 , The 6th Annual International Seminar on Trends in Science and Science Education 16–17 October 2019, North Sumatera Province, Indonesia Citation Usman et al 2020 J. Phys.: Conf. Ser. 1462 012024 DOI 10.1088/1742-6596/1462/1/012024

Article metrics

725 Total downloads

Share this article

Author e-mails.

[email protected]

Author affiliations

1 Syiah Kuala University, Jl. Teuku Nyak Arief Darussalam, Banda Aceh 23111, Indonesia

Buy this article in print

Belief is an individual construction result based on learning experiences. This study aimed to describe students' calculus learning beliefs and its relations with an understanding of solving limit of functions problems. This study used a qualitative descriptive approach. The subjects of this research were two Mathematics education students of Syiah Kuala University. The data collected by giving calculus learning belief questionnaire, the limit of functions understanding test, and interview guidelines. The data were analyzed by displaying data, analyzing, interpreting, and concluding. The result showed that both students who had high-beliefs in calculus learning were able to solve the limit of functions problems by using concepts, principles, and procedures.

Export citation and abstract BibTeX RIS

Content from this work may be used under the terms of the Creative Commons Attribution 3.0 licence . Any further distribution of this work must maintain attribution to the author(s) and the title of the work, journal citation and DOI.

• Math Careers

Search form

• MAA Centennial
• Spotlight: Archives of American Mathematics
• MAA Officers
• MAA to the Power of New
• Council and Committees
• MAA Code of Conduct
• Policy on Conflict of Interest
• Statement about Conflict of Interest
• Recording or Broadcasting of MAA Events
• Policy for Establishing Endowments and Funds
• Avoiding Implicit Bias
• Copyright Agreement
• Principal Investigator's Manual
• Planned Giving
• The Icosahedron Society
• Our Partners
• Advertise with MAA
• Employment Opportunities
• Staff Directory
• 2022 Impact Report
• In Memoriam
• Membership Categories
• Become a Member
• Membership Renewal
• MERCER Insurance
• MAA Member Directories
• New Member Benefits
• The American Mathematical Monthly
• Mathematics Magazine
• The College Mathematics Journal
• How to Cite
• Communications in Visual Mathematics
• About Convergence
• What's in Convergence?
• Convergence Articles
• Mathematical Treasures
• Portrait Gallery
• Paul R. Halmos Photograph Collection
• Other Images
• Critics Corner
• Problems from Another Time
• Conference Calendar
• Guidelines for Convergence Authors
• Math Horizons
• Submissions to MAA Periodicals
• Guide for Referees
• Scatterplot
• Math Values
• MAA Book Series
• MAA Press (an imprint of the AMS)
• MAA Library Recommendations
• Additional Sources for Math Book Reviews
• About MAA Reviews
• Mathematical Communication
• Information for Libraries
• Author Resources
• MAA MathFest
• Proposal and Abstract Deadlines
• MAA Policies
• Invited Paper Session Proposals
• Contributed Paper Session Proposals
• Panel, Poster, Town Hall, and Workshop Proposals
• Minicourse Proposals
• MAA Section Meetings
• Virtual Programming
• Joint Mathematics Meetings
• Calendar of Events
• MathFest Programs Archive
• MathFest Abstract Archive
• Historical Speakers
• Information for School Administrators
• Information for Students and Parents
• Registration
• Getting Started with the AMC
• AMC Policies
• AMC Administration Policies
• Important AMC Dates
• Competition Locations
• Invitational Competitions
• Putnam Competition Archive
• AMC International
• Curriculum Inspirations
• Sliffe Award
• MAA K-12 Benefits
• Mailing List Requests
• Statistics & Awards
• Submit an NSF Proposal with MAA
• MAA Distinguished Lecture Series
• Common Vision
• CUPM Curriculum Guide
• Instructional Practices Guide
• Möbius MAA Placement Test Suite
• META Math Webinar May 2020
• Progress through Calculus
• Survey and Reports
• "Camp" of Mathematical Queeries
• DMEG Awardees
• National Research Experience for Undergraduates Program (NREUP)
• Neff Outreach Fund Awardees
• Tensor SUMMA Grants
• Tensor Women & Mathematics Grants
• Grantee Highlight Stories
• "Best Practices" Statements
• CoMInDS Summer Workshop 2023
• MAA Travel Grants for Project ACCCESS
• 2024 Summer Workshops
• Minority Serving Institutions Leadership Summit
• Previous Workshops
• Frequently Asked Questions
• Course Resources
• Industrial Math Case Studies
• Participating Faculty
• 2020 PIC Math Student Showcase
• Previous PIC Math Workshops on Data Science
• Dates and Locations
• Past Programs
• Leadership Team
• Support Project NExT
• Section NExT
• Section Officers Meeting History
• Preparations for Section Meetings
• Bylaws Template
• Editor Lectures Program
• MAA Section Lecturer Series
• Officer Election Support
• Section Awards
• Section Liaison Programs
• Section Visitors Program
• Expense Reimbursement
• Guidelines for Bylaw Revisions
• Guidelines for Local Arrangement Chair and/or Committee
• Guidelines for Section Webmasters
• MAA Logo Guidelines
• MAA Section Email Policy
• Section Newsletter Guidelines
• Statement on Federal Tax ID and 501(c)3 Status
• Communication Support
• Guidelines for the Section Secretary and Treasurer
• Legal & Liability Support for Section Officers
• Section Marketing Services
• Section in a Box
• Subventions and Section Finances
• Web Services
• Joining a SIGMAA
• Forming a SIGMAA
• History of SIGMAA
• SIGMAA Officer Handbook
• MAA Connect
• Meetings and Conferences for Students
• Opportunities to Present
• Information and Resources
• MAA Undergraduate Student Poster Session
• Undergraduate Research Resources
• MathFest Student Paper Sessions
• Research Experiences for Undergraduates
• Student Poster Session FAQs
• High School
• A Graduate School Primer
• Reading List
• Student Chapters
• Awards Booklets
• Carl B. Allendoerfer Awards
• Regulations Governing the Association's Award of The Chauvenet Prize
• Trevor Evans Awards
• Paul R. Halmos - Lester R. Ford Awards
• Merten M. Hasse Prize
• George Pólya Awards
• David P. Robbins Prize
• Beckenbach Book Prize
• Euler Book Prize
• Daniel Solow Author’s Award
• Henry L. Alder Award
• Deborah and Franklin Tepper Haimo Award
• Certificate of Merit
• Gung and Hu Distinguished Service
• JPBM Communications Award
• Meritorious Service
• MAA Award for Inclusivity
• T. Christine Stevens Award
• Dolciani Award Guidelines
• Morgan Prize Information
• Selden Award Eligibility and Guidelines for Nomination
• Selden Award Nomination Form
• AMS-MAA-SIAM Gerald and Judith Porter Public Lecture
• Etta Zuber Falconer
• Hedrick Lectures
• James R. C. Leitzel Lecture
• Pólya Lecturer Information
• Putnam Competition Individual and Team Winners
• D. E. Shaw Group AMC 8 Awards & Certificates
• Maryam Mirzakhani AMC 10 A Awards & Certificates
• Two Sigma AMC 10 B Awards & Certificates
• Jane Street AMC 12 A Awards & Certificates
• Akamai AMC 12 B Awards & Certificates
• High School Teachers
• MAA Social Media

You are here

Mathematical treasure: leibniz's papers on calculus.

• Mathematical Treasure: Leibniz's Papers on Calculus - Differential Calculus
• Mathematical Treasure: Leibniz's Papers on Calculus - Integral Calculus
• Mathematical Treasure: Leibniz's Papers on Calculus - Fundamental Theorem

Gottfried Wilhelm Leibniz (1646-1716) was a true polymath recognized for his excellence in many fields, particularly philosophy, theology, mathematics, and logic. He is considered a cofounder, along with Isaac Newton, of the Calculus. In 1682, Leibniz, together with a fellow German philosopher and scientist, Otto Mencke (1644-1703), founded a scholarly journal, Acta Eruditorum [ Reports of Scholars ], in Leipzig. The journal was intended for the German-speaking regions of Europe, despite being written almost entirely in Latin. Acta Eruditorum , a monthly journal, would become the vehicle for much of the mathematical publication of Leibniz and the Bernoullis and would eventually be the forum through which Leibniz defended his priority in the development of calculus.

According to J. J. O’Connor’s and E. F. Robertson’s biography in the MacTutor History of Mathematics Archive, Leibniz “developed the basic features of his version of the calculus” while living in Paris during the 1670s:

In 1673 he was still struggling to develop a good notation for his calculus and his first calculations were clumsy. On 21 November 1675 he wrote a manuscript using the \(\int f(x)\,dx\) notation for the first time. In the same manuscript the product rule for differentiation is given. By autumn 1676 Leibniz discovered the familiar \(d(x^n)=nx^{n-1}dx\) for both integral and fractional \(n.\)

Leibniz began publishing his calculus results during the 1680s. We present on the following pages three famous articles on the Calculus, published by Leibniz in  Acta Eruditorum in 1684, 1686, and 1693:

• " Nova Methodus pro Maximis et Minimis… " (1684)
• “ De geometria recondite et analysi indivisibilium atque infinitorum... ” (1686)
• “ Supplementum Geometrie Dimensorie… ” (1693)

The portrait of Leibniz above is from the Convergence Portrait Gallery . It appears there courtesy of the Dibner Library of Science and Technology, The Smithsonian Institution Libraries, and its usage must conform to the Library’s rules and standards. The images in the following three pages of this article are used through the courtesy of the Lilly Library, Indiana University, Bloomington, Indiana. You may use them in your classroom; for all other purposes, please seek permission from the Lilly Library.

J. J. O’Connor and E. F. Robertson, “ Gottfried Wilhelm von Leibniz ,” MacTutor History of Mathematics Archive .

Index to Mathematical Treasures

Frank J. Swetz (The Pennsylvania State University), "Mathematical Treasure: Leibniz's Papers on Calculus," Convergence (June 2015)

• Printer-friendly version

Dummy View - NOT TO BE DELETED

Mathematical Treasure: Leibniz's Papers on Calculus

Maa publications.

• Images for Classroom Use
• MAA Reviews
• MAA History
• Policies and Procedures
• Support MAA
• Member Discount Programs
• Propose a Session
• MathFest Archive
• Putnam Competition
• AMC Resources
• Curriculum Resources
• Outreach Initiatives
• Professional Development
• Communities

Connect with MAA

Mathematical Association of America P: (800) 331-1622 F: (240) 396-5647 Email: [email protected]

Copyright © 2024

• Terms of Use
• Privacy Policy
• Mobile Version

A Study of Students’ Readiness to Learn Calculus

• Published: 16 September 2015
• Volume 1 , pages 209–233, ( 2015 )

Cite this article

• Marilyn P. Carlson 1 ,
• Bernard Madison 2 &
• Richard D. West 3  

19k Accesses

34 Citations

1 Altmetric

Explore all metrics

The Calculus Concept Readiness (CCR) instrument assesses foundational understandings and reasoning abilities that have been documented to be essential for learning calculus. The CCR Taxonomy describes the understandings and reasoning abilities assessed by CCR. The CCR is a 25-item multiple-choice instrument that can be used as a placement test for entry into calculus and to assess the effectiveness of precalculus level instruction. Results from administering the CCR to first semester calculus students at the beginning of the semester revealed severe weaknesses in students’ foundational knowledge and reasoning abilities for learning calculus. Correlating CCR results with course grades revealed that students with higher CCR scores are better prepared to succeed in beginning calculus. The CCR data further identified specific ways of thinking and concepts for which precalculus instruction could be improved to influence student learning and preparation for calculus.

Similar content being viewed by others

Ideas foundational to calculus learning and their links to students’ difficulties

Patrick W. Thompson & Guershon Harel

What meanings are assessed in collegiate calculus in the United States?

Michael A. Tallman, Zackery Reed, … Marilyn P. Carlson

Factors contributing to students and instructors experiencing a lack of time in college calculus

Jessica Ellis Hagman, Estrella Johnson & Bailey K. Fosdick

Avoid common mistakes on your manuscript.

Introduction

Precalculus courses in the United States (US) are not achieving their educational potential, especially with regard to preparing students to succeed in calculus (e.g., Breidenbach et al. 1992 ; Carlson 1998 ; Moore 2012 ; Moore and Carlson 2012 ). One consequence of this is high attrition from precalculus to calculus, but the major consequence is the lost learning opportunities that would benefit precalculus and calculus students. The attrition and missed educational opportunities are enormously expensive because they delay STEM students’ progress toward degrees, lower the learning in some degrees, or cause students to drop out of STEM altogether. Research over the past few decades point to ways precalculus and calculus courses can be strengthened to address this alarming situation (e.g., Carlson and Rasmussen 2008 ). This research has informed the development of the Calculus Concept Readiness (CCR) instrument, Footnote 1 while curriculum and instruction in precalculus level courses have not been noticeably impacted by research on learning key ideas of precalculus level mathematics.

This article provides an overview of literature that has identified foundational precalculus level reasoning abilities and understandings that students need for understanding key ideas of calculus. The Calculus Concept Readiness (CCR) Taxonomy is presented to detail the specific abilities assessed by the CCR instrument. We conclude by sharing results from administering the CCR to US college students prior to taking beginning calculus. Footnote 2

Over the past 25 years many mathematics education researchers have found that student difficulties in understanding key ideas of calculus are rooted in their weak understanding of the function concept (e.g., Breidenbach et al. 1992 ; Carlson 1998 ; Carlson et al. 2010 ; Tall and Vinner 1981 ; Tall 1992 , 1996 ; Thompson 1994a ; Smith 2008 ; Zandieh 2000 ). Early studies of students’ understandings of the function concept revealed common misconceptions among students (e.g., Monk 1992 ; Sierpinska 1992 ; Vinner and Dreyfus 1989 ) including their: i) strong tendency to view a graph as a picture of an event (Monk 1992 ), rather than a representation of how two quantities change together; and ii) viewing a function as a recipe for getting an answer instead of as a process that maps input values to output values (Breidenbach et al. 1992 ; Carlson 1998 ). These weaknesses in students’ understanding of the function concept are contributing to students being unprepared to understand ideas in beginning calculus.

Other studies have investigated student thinking in the context of curriculum tasks designed to develop student understanding of ideas of function (Dubinsky and Harel 1992 ; Moore and Carlson 2012 ), function composition and inverse (Engelke et al. 2005 ; Engelke 2007 ), quantity (Moore and Carlson 2012 ; Moore 2012 , 2014 ), exponential growth (Castillo-Garsow 2010 ; Strom 2008 ), and central ideas of trigonometry (Moore 2013 , 2014 ). These studies consistently report that when students conceptualize a function as a process that maps input values from a function’s domain to output values in a function’s range, they are able to understand and use the idea of function, function composition and function inverse to solve novel problems (Engelke 2007 ; Moore and Carlson 2012 ). As students begin to reason about how the input values and output values of a function change together they are able to distinguish between different function types, explain the meaning of a concave up graph and conceptualize the covarying quantities related by trigonometric functions in precalculus (Moore 2012 , 2014 ) and the Fundamental Theorem of Calculus (e.g., Carlson et al. 2003 ; Oehrtman 2008 ; Smith 2008 ; Thompson 1994a ). This is encouraging since it suggests that student learning in precalculus level mathematics can be affected by instructional interventions that support students in understanding and reasoning with the function concept.

Foundational Reasoning Abilities and Understandings for Learning Calculus

This section provides a more detailed description of the reasoning abilities and understandings that students need to develop prior to beginning a course in calculus. In particular we describe what is involved in conceptualizing quantities and make an argument for the importance of covariational reasoning in defining meaningful functions to model relationships in word problems. We discuss the importance of developing a process view of function and what it means. We also describe complexities of engaging in proportional reasoning and conclude by discussing understandings of other key ideas (e.g., constant rate of change, average rate of change) that are needed for learning calculus.

Covariational Reasoning

In the context of mathematics it is common that students need to conceptualize quantities in a problem situation and to consider how those quantities are related and change together (Carlson et al. 2002 ; Thompson 1994b ). This ability to both conceptualize a situation and imagine the measurable attributes of the objects (quantities) in a situation is referred to as quantitative reasoning (Smith and Thompson 2007 ; Thompson 1993 , 2011 , 2012 ). For example, when watching a race one might initially observe runners and a starting line, and then when the starting gun is fired be interested in how a quantity such as distance of a runner from the starting line changes as the runner is moving down the track. One might also notice that the elapsed time since the runner started the race is increasing and that the length of the race is 100 m. The observer has conceptualized two varying quantities, the elapsed time since the start of the race and the distance of the runner from the starting line, and one fixed quantity, the length of the race. Footnote 3 There are many other varying and fixed quantities that could be conceptualized in this situation (e.g., the height of the runner or the distance of the runner from the finish line). However, it is important that students learn to focus on and conceptualize the quantities that are relevant for the line of inquiry they are pursuing. Footnote 4

The mental process of relating two varying quantities requires that students think about how the two quantities are changing together. Gaining clarity about how the runner’s distance from the starting line is changing with the elapsed time since the runner started running might involve considering fixed amounts of change in one quantity while considering how much the other quantity is changing. In this example we might consider fixed amounts of time (e.g., ½ sec) while considering how much distance the runner traverses for each successive ½ sec of the race. If we were to observe that the runner was traveling a greater distance, over some interval of time for each successive ½ sec since starting the race, we might conclude that the runner’s rate of change of distance with respect to time is increasing over that interval of time.

Carlson ( 1998 ) documented that both precalculus and second semester calculus students had difficulty creating a graph to represent the height of water in a spherical bottle as a function of the amount of water in the bottle. Carlson et al. ( 2002 ) later described 5 mental actions associated with covariational reasoning in the context of making or interpreting a graph of two quantities that change together. These mental actions include: i) conceptualizing the quantities in the situation that are to be related; ii) imagining how the direction of the two quantities change together (e.g., as the elapsed time since the race started increases the distance of the runner from the finish line decreases) iii) imagining how the amount of change of one quantity changes while considering contiguous fixed amounts of the other quantity on intervals of that quantity; and iv) imagining how the average rate of change of the output variable with respect to the input variable is changing on small contiguous intervals of the input variable.

A student who considers how two quantities in a dynamic situation change together is said to be engaging in covariational reasoning (Carlson et al. 2002 ; Thompson 1994b ). This is a foundational way of thinking that is needed to construct meaningful formulas and graphs to model relationships in applied contexts (Carlson et al. 2002 ; Moore and Carlson 2012 ). Covariational reasoning has also been documented to be an essential reasoning ability for understanding and using ideas in beginning calculus (Carlson et al. 2003 ; Engelke 2007 ; Smith 2008 ; Thompson 1994b ; Zandieh 2000 ).

When students first encounter word problems in precalculus, regardless of the type of function model that is needed, they must first conceptualize the quantities to be related (e.g., length of the radius of an expanding sphere and volume of the expanding sphere, angle measure and vertical distance, ambient temperature and the temperature of an object, amount of time since making an investment, the value of the investment). Once the relevant quantities in a word problem have been conceptualized students are able to think about how the quantities are related and how they change together. These conceptualizations describe the reasoning that students must engage in to construct a formula to model the relationship between two quantities in an applied context (Moore and Carlson 2012 ).

Understanding the Function Concept

Carlson and colleagues (Carlson 1998 ; Carlson et al. 2002 ) found that many precalculus level students have difficulty using and interpreting function notation, and many are not clear on what it means to express one quantity as a function of another. Precalculus students sometimes confuse the output of a function with the function name and interpret the equal sign as a statement of equivalence rather than as a means of defining a relationship between two quantities that change together (e.g., Carlson 1997 , 1998 ). Students who exhibit a process view of function are better prepared to understand and use function notation to relate values from a function’s domain to its range. This is because they see both a function’s defining formula and graph as specifying how to process the input values to produce output values. This image supports students in thinking about function composition as the stringing together of two function processes for the purpose of relating two quantities that cannot be directly related by a single formula. Students who have a process view of function are also able to understand the idea of function inverse as a new mapping that reverses the process of the original function.

Carlson ( 1995 ) investigated precalculus level students’ understanding of function. She found that when function names such as f, g , and h were used, students were unclear as to what the letters meant and that some students believed that the letters represented variables. This is not surprising because up to this point in students’ experiences in algebra, letters had been used to define variables. This conception does not support students in viewing a function that has been named with a letter and defined with a formula as representing a process that maps input values in a function’s domain to output values in a function’s range.

Extending the process view of function to reason about how the values in a function’s domain covary with values in a function’s range involves considering how function values change over a continuum of values, rather than just imagining one input value being mapped to one output value one value at a time. This extends the process view of a function so that students are able to both construct meaningful graphs, and to describe what a graph’s concavity conveys about how the function’s output value is changing relative to its input value. Several studies (e.g., Carlson 1998 ; Carlson et al. 2002 ; Carlson et al. 2010 ; Moore and Carlson 2012 ) revealed that attention to the quantities being modeled in a situation, and the ability to think about how one quantity is changing while imagining fixed incremental changes of the other quantity, enables students to understand and use ideas of constant rate of change and to interpret and construct meaningful function formulas and graphs to represent specific linear, exponential, quadratic, rational, and periodic growth using functions.

The idea of average rate of change on an interval of a function’s domain is also a key idea of precalculus that is needed for learning calculus. It is common for precalculus level students to associate the word average with a procedure to add numbers in a list and then divide by the number of items added (Carlson et al. 2010 ). In contrast, one accurate and meaningful understanding of the idea of average rate of change of a function on an interval, is that the average rate of change is the constant rate that would achieve the same change in both the input and output quantities as the actual function on the interval of interest (Thompson 1994b ). Studies (e.g., Carlson et al. 2002 ; Carlson et al. 2010 ) have consistently revealed that the vast majority of students at many different universities across the US exit precalculus courses without understanding the idea of average rate of change.

Proportional Relationships

The ability to recognize situations in which two varying quantities are related proportionally has been documented to be problematic for precalculus level students (Castillo-Garsow 2010 ). They might be able to solve for x when given a situation in which x /y = 3/7 and, but if given a problem that requires students to recognize that the ratio of two varying quantities is a constant, or that two varying quantities are related by a constant multiple, the vast majority of precalculus level students have difficulty recognizing proportional relationships in dynamic situations (Carlson et al. 2010 ). For example, when administering the rain-gauge problem (Fig.  1 ) to 1205 precalculus students at the end of the semester, only 43 % of these students provided the correct response of \( 7\frac{1}{3} \) (Carlson et al. 2010 ). Most of these students recognized that when the wide cylinder had 6 inches, the narrow cylinder had 4 inches. However, they failed to recognize that as the amount of rain increased, the ratio of the number of inches of rain in the wide cylinder to the number of inches of rain in the narrow cylinder had to remain in a constant ratio of 4:6. It was common for students who provided an incorrect response to reason that, if the water rises to the 11th mark on the narrow cylinder it would rise to the 9th mark on the wide cylinder since the amount of water originally in the wide cylinder was 2 less than what was in the narrow cylinder.

The rain-gauge problem (Piaget et al. 1977 ; Lawson 1978 )

To the right are drawings of a wide and a narrow cylinder. The cylinders have equally spaced marks on them. Water is poured into the wide cylinder up to the 4th mark (see A). This water rises to the 6th mark when poured into the narrow cylinder (see B). Both cylinders are emptied, and water is poured into the narrow cylinder up to the 7th mark. How high would this water rise if it were poured into the empty wide cylinder?

Recent work to improve student learning in precalculus (Carlson et al. 2015 ) has highlighted the important role that proportional reasoning plays in understanding and using the idea of constant rate of change. If two quantities are changing at a constant rate of change, the changes in the two quantities are proportional. It is this understanding that is needed to determine a new value for one quantity when the constant rate of change and a value of the other quantity is known. Many applied problems in precalculus and beginning calculus require use of proportional reasoning. Recognizing proportionality of quantities and using proportional reasoning is also key to understanding and using the idea of angle measure in trigonometry (Moore 2013 ).

Angle Measure and Sine Function

The ideas of angle measure and trigonometric functions have been documented to be under-developed in inservice teachers (Thompson 2008 ; Thompson et al. 2007 ) and precalculus level students (Moore 2012 , 2013 ) and preservice teachers (Moore 2012 ; Moore et al. 2012 ). Students often do not conceptualize an angle measure as an amount of openness between two rays with a common endpoint; nor do they recognize the need to use an arc of a circle (with its center at the angle’s vertex) to measure an angle’s openness.

According to Moore ( 2013 ), approaches that help students view an angle’s measure in terms of the relative length of circle’s arc that the angle subtends, are able to understand the idea of radian measure and effectively use the radius of a circle as a unit for measuring angles. Examination of student thinking during a teaching experiment designed to support student understanding of trigonometric functions revealed that when students are able to reason about how an angle’s measure and the vertical coordinate of the arc’s terminus (measured in units of the arc’s radius) covary, they are better able to understand the sine function and use it meaningfully to model periodic motion (Moore 2012 ). This image of the sine function in the unit circle context also was useful for students in connecting their unit circle conceptions of the sine and cosine functions to their conceptions of these functions in the triangle trigonometry context. These students came to understand specific right triangles as corresponding to specific input–output pairs of the trigonometric functions.

The CCR Taxonomy

The CCR Taxonomy (Table 1 ) includes three primary reasoning abilities that are foundational for learning and using key ideas of calculus. The taxonomy includes understandings of various function types that emerge from examining growth patterns in data, and other understandings that have been identified in research studies to be essential for either constructing or interpreting meaningful function formulas and graphs. Lastly, the taxonomy has a category that describes the trigonometric ideas that are needed to model periodic growth and to understand and connect unit circle and triangle trigonometry.

The CCR is a 25 item multiple-choice exam with each question having five answer choices. Eighteen of the twenty-five CCR items assess or rely on student understanding of the function concept. Five items assess student understanding or use of trigonometric functions, and four items assess student understanding or ability to use exponential functions. Ten items are situated in an applied (or word problem) context and require students to reason about quantities and use ideas of function, function composition, or function inverse to represent how the quantities change together. There are other items that provide information about such things as students’ understanding of notational issues, their ability to interpret the meaning of an absolute value inequality such as | x  − 3| < 5, and whether they can determine the inverse function of an exponential function. In some problems students are expected to recognize equivalent expressions, perform translations on a known function, and/or use structural equivalence in their reasoning, such as recognizing that 3( x +2) 2 –4( x +2)+7 = 0 is a quadratic function in ( x +2).

The Process of Developing the CCR

The literature review and CCR taxonomy informed the development of items produced by a committee of five mathematicians and one mathematics educator. Four of the mathematicians have participated in developing and scoring items for Advanced Placement Calculus, one has worked in placement testing for over 20 years, and the mathematics educator had led the development of the Precalculus Concept Assessment (PCA) instrument (Carlson et al. 2010 ).

Following methods of instrument development used to develop the Precalculus Concept Assessment (PCA) (Carlson et al. 2010 ), the Mechanics Baseline Test (Hestenes and Wells 1992 ) and the Force Concept Inventory (Hestenes et al. 1992 ), open-ended questions were designed to assess the reasoning abilities and understandings that had been revealed to be most critical for learning calculus. According to Lissitz and Samuelsen ( 2007 ), the development of a valid examination should always begin by identifying the constructs worthy of assessment. Question wording and item distractors for the multiple-choice items were based on student interview data that illustrated common student thinking when responding to each open-ended item. Further interviews were conducted with students after the initial multiple-choice items were developed to identify or refine item distractors so that they were representative of five common student responses, and to verify that questions and answer choices were interpreted as we intended. The taxonomy and CCR items went through multiple cycles of refinement. Clinical interviews with students were conducted repeatedly until each CCR item had been validated to: i) be consistently interpreted, ii) assess the knowledge intended by the item designer, and iii) have distractors that were representative of student thinking as revealed during the interviews.

After validating that the CCR items assess what we claimed, we administered the CCR exam to 631 students at three public universities and one private university. The data was subsequently analyzed to examine trends in the CCR data.

Analyses of CCR Results

We administered CCR to 601 Calculus 1 students at three different universities during the first week of the fall 2009 semester. Our primary goal was to determine how well CCR scores predicted grades in Calculus 1 courses. We administered CCR to thirty precalculus students in the last week of classes, comparing student course grades with their CCR scores. We used these quantitative data to: (1) estimate the reliability of the test as a whole; (2) measure how each of the 25 items was functioning, individually and as a part of the test; and (3) measure validity of the instrument as a measure of readiness for success in learning calculus.

Reliability is a measure of consistency of scores on repeated administrations of the test. Since repeated administrations pose logistical difficulties, reliability estimates from a single administration of the test are made from correlations of scores on subsets of the test. For example, split-half reliability is a correlation of scores on halves of the test. The estimate of reliability we used is an extension of the split-half reliability, called Cronbach’s alpha. To measure item functioning, we computed a difficulty index (the percent correct), a discrimination index, and the point-biserial coefficient for each item. The point-biserial coefficient of an item is the correlation between the scores on the item and the scores on the entire test with the item deleted.

We measured validity of CCR as a predictor of course grades in Calculus 1 (predictive validity), correlated CCR scores with the American College Testing (ACT) Footnote 5 mathematics scores (concurrent validity), and compared grades in a prerequisite precalculus course to CCR scores (criterion validity). Each of these analyses indicated that CCR scores are useful when deciding on readiness for success in the study of calculus.

Establishing CCR Validity Footnote 6 and Predictive Potential

During the fall semester of 2009, CCR was administered to 215 Footnote 7 Calculus 1 students at a large public university during the first week of class. Approximately 70 % of these students were freshmen (149 freshmen and 66 upperclassmen) and were placed in the course by virtue of having an ACT mathematics test score of at least 26. Most of the 66 upperclassmen were in the class by virtue of passing a prerequisite course. Final course grades were obtained for 214 of these 215 students. The statistics for these students are given in Table 2 .

Table 3 provides the Pearson correlation coefficients between ACT mathematics scores (most recent and maximum in 2007–2009), the score on CCR, and the course grade (0–4 with withdrawal (W) being 0). These are all significant at the 0.0001 level under the null hypothesis that the correlations are 0. The number of records for each correlation is given as N.

The above results (Table 3 ) indicate that course grades are moderately correlated to CCR scores and ACT, while the correlation of grades with ACT scores is somewhat weaker. This raised the question of using both ACT scores and CCR scores to explain variation in course grades. The results of two multiple regressions are given in Tables  4 and 5 .

These results indicate that using an ACT score (most recent or maximum in 2007–2009) in conjunction with the CCR score does not add much to the explanation of the variation of course grades. There is considerable overlap in the predictive information from CCR scores and ACT scores, but the predictive power of CCR is better, with CCR scores contributing significantly to the explanation of the variation of course grades.

We provide a brief description of the 25 items, the percent correct (P), a measure of discrimination (D), and the point-biserial coefficient (PBS) (see Appendix A ). The percent correct is a measure of item difficulty. The Discrimination Index (D) (Kelly et al. 2002 ) is computed from equal-sized (27 %) high and low scoring groups on the test by subtracting the number of successes by the low group on the item from the number of successes by the high group, and dividing this difference by the size of a group. The range of this index is +1 to −1, and values of 0.4 and above are regarded as high and less than 0.2 as low discrimination. We observed that 12 of the items have high discrimination indexes and four (20, 21, 23, 24) have low indexes. Not surprising these four items with low discrimination indexes correspond to the ones on which our interviews revealed severe weakness in most students. Three of these four are trigonometry items, corroborating our findings from the clinical interviews that even high performing students had weak understanding of ideas of angle measure and trigonometric functions in the unit circle context.

The Pearson point-biserial for each test item is a correlation of the scores on that item (dichotomous at 0 or 1) and the scores on the test with that item deleted. A point-biserial of at least 0.15 is recommended but “good” items have point-biserials greater than 0.25 (Varma 2012 ). It is noted that all of the items have biserial coefficients greater than 0.15 and nineteen of them have coefficients greater than 0.25 and moderate to high discrimination indexes. The six items 18, 19, 20, 21, 23 and 24 with a point-biserial lower than 0.25 are items on which the percentages of correct answers were less than 20 %, similar to what one would expect if students were selecting an answer without even reading the question.

Reliability Measure

The Cronbach coefficient, alpha, is an estimate of test reliability, i.e., the internal consistency of the test. The raw Cronbach coefficient from this test administration is 0.665 ( p  < 0.0001); standardized it is 0.658. The raw Cronback alpha of 0.665 increases slightly with items 18, 19, 20, 21, 23, and 24 deleted (one at a time), consistent with the low correlation of these items with the rest of the test as indicated by the point-biserial coefficients. These data provide further support of qualitative studies that have revealed severe weaknesses of students’ understanding of trigonometry. Since our the research literature and our own qualitative data support that the understandings assessed by these items are important for learning key ideas of calculus we are retaining these items on CCR to raise awareness of the need for greater curriculum focus with ideas of angle measure (T1 and T2), function translations (U7), and trigonometric functions (T3 and T4). We are also hopeful that shifts in the conceptual focus of curriculum will lead to improved results on these items.

Validity Measures

Our analysis of CCR scores from assessing 215 Calculus 1 students at the beginning of the course shows a reasonably strong connection between levels of CCR scores and success in their Calculus 1 course, as measured by course grades. The mean CCR score of students whose course grades were A, B, or C is 11.83, while the mean CCR score for students whose course grades were D, W, or F is 8.49. Tables  6 and 7 give the grades sorted by CCR score and the mean CCR scores for students with grades of A, B, C, D, F, and W. These tables indicate a reasonably strong correlation between CCR scores and course grades, and it is noted that each of the 15 students with CCR scores of 17 or more earned a grade of A or B. This finding suggests that CCR items assess prerequisite knowledge for learning key of ideas of calculus.

Factor Analysis

A conceptual analysis of what is assessed by individual items revealed that each item assesses a unique combination of reasoning abilities, understandings, and notational issues, and that this uniqueness of individual items results in low correlations between items. Carlson et al. ( 2010 ) report similar findings that four PCA items that primarily assessed students’ ability to use function composition were weakly correlated. Their clinical interviews revealed differences in the complexity of the items that might have attributed to the low correlation.

Criterion Validity

Another type of validity study involved comparison of the CCR results to the outcome of the prerequisite precalculus course at the university where this study was conducted. Since students who earn grades of A, B, or C in a precalculus course are presumably ready to learn the content taught in a Calculus 1 course, testing precalculus students at the end of the course should reveal a correspondence between success in the course and a CCR score. Data from 30 precalculus students (Tables  8 and 9 ) at the same university were tested at the end of their course. The correlation between CCR test scores and course grades in precalculus was 0.58 with the following results that, when combined with the predictive study, yields reasonably consistent results.

Our data analysis illustrates two approaches for validating a placement instrument. One is to compare placement test scores to an existing criterion for placement in calculus, namely successfully completing a precalculus course. Since this placement testing is administered at the end of a precalculus course, students who have dropped out are not available for the study. The second approach for validating a placement instrument is to compare students’ placement test scores with their grades in calculus. There are also limitations to this approach since factors other than knowledge of and facility with precalculus concepts influence grades in calculus. As a result, it is unlikely that a test administered at the beginning of a course will predict accurately all students’ course grades. Since students in this study were placed in Calculus 1 by using ACT mathematics scores, and in one institution, a procedurally oriented placement exam, these conditions likely reduced the variation in the CCR scores of our sample, resulting in the likelihood that the predictive power of CCR scores was reduced.

These results of this analyses revealed that students who receive higher CCR scores generally performed better in Calculus 1. In fact, students who scored 11 or higher on CCR passed calculus (grade of A, B, or C) at rates of 66 %, 86 % and 91 % at the three universities participating in this study. This finding is consistent with the results reported by Carlson et al. ( 2010 ) using the Precalculus Concept Assessment (PCA). The authors reported that 77 % of 248 Calculus I students who took PCA at the beginning of a fall semester, and received a score of 13 (out of 25) or higher, were awarded a course grade of A, B, or C. However, the 0.51 correlation coefficient between the initial CCR score and final course grade was slightly higher than the correlation coefficient of 0.47 between PCA scores, also administered at the beginning of a fall semester, and final course grade. The success rate (at least a grade of C) of 95 % or greater for students who scored a 15 or higher on CCR suggests that collectively the items on CCR assess essential understandings that are used in beginning calculus. However, a success rate of 27 % for those with CCR scores less than 9 indicates that some students are achieving grades of A, B, or C in Calculus 1 without initially understanding many fundamental function concepts. This finding might be due to the strong procedural emphasis in this calculus curriculum, rather than an indication of the overall efficacy of CCR to detect strengths and weaknesses in students’ foundational knowledge for calculus.

Beginning Calculus Students are Not Prepared to Understand Calculus

An examination of the CCR response patterns of the 601 students completing CCR at the beginning of calculus revealed severe weaknesses in these calculus students’ understandings and reasoning abililities of ideas on which calculus is built. The majority of students were unable to answer both proportional reaosning questions correctly and only 9 % of the 631 students answered all three function word problems (See Appendix A , Items 2, 7, 12), suggesting weakness in their ability to construct meaningful formulas by examining the quantities in a dynamic word problem context. Another area of difficulty was in students’ ability to compose two functions. Only 28 % of students provided a correct response to the item that asked students to define the area A of a circle in terms of its circumference C (Fig.  2 ) and only 29 % of the students selected the correct answer to a question that asked them to determine the area of a circular oil spill that traveled outward from the center of the spill at a speed of 2 feet per second. In another function composition item that provides a table of values for the functions f and g and asks students to determine the value of f ( g (3)), only 37 % of the students chose the correct answer. Interviews with students on this item revealed that students who selected the correct answer spoke about the functions as a means of processing input values to produce output values, while students who selected incorrect answers did not. They appeared to use the table as a way to look up answers without any guiding principles for how quantities in one column were mapped to quantities in another column. We also observed that the students in this study did not perform any better on items that required them to execute standard procedures or remember common definitions. Only 21 % of the students selected the correct answer to the item that asked them to solve f ( t ) = 100 t for t , while the answer f −1 ( t ) = 1/100 t was selected by 53 % of the students.

Area-circumference item

An examination of student responses on the four trigonometry function questions, 3 of which were asked in the context of unit circle trigonometry ( Appendix A , Items 21, 23, 24), further revealed severe weaknesses in their covariational reasoning abilities, with 16, 21 and 17 % respectively of the students selecting the correct answers. On a fourth more traditional trigonometry item that asked students to identify the formula for the graph of a cosine function with a period 2π/3, only 21 % of the students selected the correct formula.

The following sections provide a more detailed description of student thinking on three CCR items. These descriptions reveal the nature of students’ weakness and specific reasoning abilities and understandings assessed by these CCR items.

Proportional Reasoning

Two varying quantities are related proportionally if their ratios remain constant or if they are related by a constant multiple. The CCR assesses whether students can recognize this structure in an applied context, and then set up and solve an equation stating the proportional relationship when the value of one of the proportional quantities is known. The CCR has two items that assess students’ ability to apply proportional reasoning. One item is in the form of a capture-release problem (item #1, Appendix A ) with 61 % of the students in our study selecting the correct response. Footnote 8 The other item asked students to construct a formula to represent the driving distance d on a road in terms of the number of centimeters n between two points on a map, given that 3 cm on the map corresponds to 114 km of actual driving distance. We expected that almost all beginning calculus students would select the correct response and were suprised that only 65 % of beginning calculus students selected the correct answer of d  = 38 x The most common incorrect choices were \( d=\frac{1}{38}x \) and d  = 114 x . These students were clearly not imagining the quantities in the situation and how they are related and change together.

Function as Process

The CCR contains items that require students to compose two functions in a graphical, tabular, and word problem context. One of these items was a function composition word problem (Fig.  2 ) that prompted students to define the area of a circle in terms of its circumference. Interviews conducted with students who answered this question in an open-ended format revealed that the item assesses students’ ability to identify the quantities to be related (U1) in a word problem. The item also assesses student ability to interpret the phrase, “express area A as a function of circumference C ” as a prompt to construct a formula of the form, A = <some expression that contains C> (A5). Students must recall the formulas for the area and circumference of a circle, and view these formulas/functions as processes that map values of one quantity to values of another quantity (R2). To obtain the function that relates the area and circumference of a circle, students must recognize that the two formulas can be combined it they first invert the formula that defines a circle’s circumference in terms of its radius (U5). They also recognize that by composing (U4) the area formula with the inverted circumference formula they are able to define a new formula that defines the circle’s area in terms of its circumference.

Follow-up interviews were conducted with 19 students, 7 students had provided the correct answer and 3 students had constructed each of the four most common incorrect responses. Analysis of interview data with students who provided a correct answer revealed a common approach of verbalizing the problem goal (to express area in terms of circumference). These students typically wrote the formulas A  =  πr 2 and C  =  πr 2 , and eventually recognized that they needed to re-express the circumference formula by solving for r (or determining the inverse of the circumference formula) to obtain r  =  C /(2π). When students were prompted to explain how they knew to solve the circumference formula for r , a common response as articulated by one student was,

I know that A  = π r 2 and I need a formula for area in terms of circumference. I then need to solve C  = 2π r for r so I can put r = C/ (2π) in for r . This will give me a formula that takes values of C and computes values of r . When I put C/ (2π) in for r we get a formula that takes values of the circumference C and computes the area A .

The students who selected this response appeared to be thinking of formulas as processes for determining values of one quantity when values of another quantity are known. Various misconceptions and impoverished ways of thinking led students to select the incorrect responses. Students selecting answers (c) and (d) were unable to say what it means to express one quantity in terms of another and students who selected answer choices (b) and (e) appeared to view the letters in the formulas as something to solve for. Interview data further revealed that students who selected the incorrect responses did not view C/ (2π) as an expression that determines values for r when values of C are known. It was interesting that many students wrote the formula C  = 2 πr but did not recognize that they could reverse the process of the formula to obtain r  =  C /(2π) to express r in terms of C .

When administering the item to 631 students who completed CCR, only 28 % of these students selected the correct response. Since at least 13 % of the students (out of 631) selected each answer choice, this quantitative data (Table 10 ) further supports the findings from analyzing the qualitative data that the five answer choices are representative of common student answers.

Idea of Quantity and Covariational Reasoning

Ten of the 25 CCR items (1, 2, 6, 7, 8, 11, 12, 17, 20, 23) require students to conceptualize quantities as a first step in responding to the question. In each of these items the student must first imagine some measurable attribute of a situation and subsequently define a variable to represent the values that this attribute’s measure can assume. Such an item is often followed by a request to define a formula to relate two quantities. These items are typically applied problems that also assess whether students have a process view of function. An item that assesses whether students are able to engage in covariational reasoning asks that students think about how the two quantities in a situation or applied context are changing together. One example of this type of problem is the bottle that asks students to construct a graph of the height of water in the bottle in terms of the volume (Carlson 1998 ).

Another item on quantity and covariational reasoning is a question that asks students to describe how the top of a 10-foot ladder, leaned in a vertical position against a wall, changes as the distance of the bottom of the ladder is pulled away from the wall at a constant rate (Carlson 1998 ; Kaput 1992 ). It is noteworthy that some items that require students to use covariational reasoning might not require that students initially identify quantities in the problem context. One example is a CCR item in which students are asked to describe the behavior of the rational function, f defined by f ( x ) =  x 2 /( x – 2). In this question students need to think about how the values of the output f ( x ) change while imagining changes in x . They might begin by reasoning that, as x decreases from 4 to 3, f ( x ) increases from 8 to 9, as x decreases from 3 to 2.5, f ( x ) increases from 9 to 12.5, as x decreases from 2.5 to 2.4, f ( x ) increases from 12.5 to 57.6, etc. CCR has 7 items (4, 6, 7, 11, 16, 21, 23) in which students need to reason covariationally to provide the correct response.

A Quantitative Reasoning and Slope Item

The following item (Fig.  3 ) requires that students conceptualize two quantities (the distance of the top of the ladder from the floor and the distance of the base of the ladder from the wall) (U1). They then need to imagine how the distance of the base of the ladder from the wall changes as the top of the ladder increases to twice its original distance from the floor (R3). As they imagine how these measurements change together (engage in covariational reasoning), they also need to think about how the quotient of the ratio of the changes in these two quantities (slope) (U3) changes as the distance of the ladder from the wall decreases (R3). Another correct justification involved imagining how the ratio that represents the slope of a line changes if the algebraic form of the slope is changed by doubling the value of the numerator and decreasing the value of the denominator. Students who used this approach verbalized that doubling the numerator would produce a slope twice what it was before, but doubling the numerator and decreasing the value of the denominator would produce a slope that is more than twice as large as what it was before. These students were able to imagine how concurrent changes in two lengths on the illustration impacted the values of the numerator and denominator and how these values impacted the value of the quotient.

Slope of ladder item

Only 27 % of 631 students who completed CCR (Table 11 ) provided a correct response to this item. The most common incorrect response was choice (b), exactly twice what it was. Interviews with seven students who selected this choice revealed that these students were not conceptualizing the slope of the straight line (ladder) as a ratio of two quantities.

Follow-up interviews revealed that most students who chose answer (b) were only focusing on the amount of increase of the top of the ladder. When prompted to elaborate on their rationale for this choice, one student explained, “since the top of the ladder is two times higher when the ladder is repositioned, the slope will be twice as large.” The student failed to consider the effect of the shortened distance of the base of the ladder from the wall and never considered the slope as representing a ratio of the two distances. Students who selected answer (e), not enough information to determine, thought they needed exact numbers for the distances to be able to compute and compare the slopes. Students who thought the slope was less than twice what it was (answer (a)) thought that the smaller denominator made the slope less than twice what it was. Students who indicated that the ladder’s slope had not changed (answer (d)) indicated that the ladder itself did not change because the positioning of the ladder on the wall had no effect on the shape of the ladder. These students were not thinking about the slope as representing a ratio of two quantities.

A Trigonometry Item: Assessing Ideas of Angle Measure and Sine Function

The following trigonometry item (Fig.  4 ) relies on students having developed robust conceptions of function (R2), angle measure (T1), radian as a unit (T2) and the sine function (T3), in addition to being able to imagine and fluently reason about how two quantities change in tandem (an angle measure and distance) (R3). Responding to this item requires that students first have a conception of the sine function as representing the covariation of an angle measure and the distance of a point on the unit circle (positioned at the end of the terminal side of the angle) from a horizontal line through the circle’s center (T3, R3). Determining the function that expresses d in terms of k also requires that students conceptualize an angle measure in relation to arc length that is cut off by the rays of an angle positioned at the center of a circle and measured in lengths of the circle’s radius (T1 and T3) ( k needs to be divided by 47). As students imagine k varying, they also imagine how d varies (R3) and recognize that, since the output of the sine function is measured in lengths of the radius (radian), this value needs to be multiplied by 47 to express the value of d in feet.

Periodic motion item

Only 21 % of 631 students who completed CCR selected the correct answer, (e) (Table 12 ). As noted above, responding to this question requires that students understand ideas of angle measure, radian and sine function. They must also reason about quantities, their variation and their covariation—how the values of two quantities, k and d , change in tandem.

Students who selected answer (a) did not understand the idea of angle measure and that an angle measure can be expressed in lengths of the radius. The interview data also revealed that these students did not understand that the input to the sine function must be expressed as an angle measure. Students who selected answer (b) did not understand the idea of angle measure. Students who chose answer (c) or (d) had a weak understanding of angle measure and radian as a measure of the number of radius lengths subtended by the rays of an angle with its vertex positioned at a circle’s center.

Concluding Remarks

A CCR cut score of 11 out of 25 (44 %) is a relatively low score on an exam that assesses fundamental ideas and reasoning abilities for calculus, suggesting that many students are succeeding in Calculus 1 without the prerequisite knowledge. The CCR break points of 11 and 9 that our data suggests could be used to advise students relative to whether they should (or should not) enroll in Calculus 1 are separated by only two CCR items. This implies small differences in the initial knowledge base of students who pass Calculus 1, and those who fail Calculus 1. This observation could be explained by the fact that the Calculus 1 courses in this study did not rely on student understanding of foundational knowledge. Our observation that 27 % of students who received a nine or below were able to pass calculus with an A, B, or C also raises questions about the conceptual focus of these Calculus 1 courses.

These data in combination with our examination of the percentages of correct CCR answers further supports that many student are taking and passing first semester calculus with severe deficiencies in their reasoning abilities and knowledge base. One explanation for this finding is suggested in a recent study (Tallman and Carlson under review ) that examined 150 randomly chosen Calculus 1 final examinations selected from 246 Calculus 1 final examinations administered at institutions of higher learning across the United States in the fall of 2010. They found that the exams were highly procedural in their focus—87 % of the items were coded as recall a fact or carry out a procedure . Our results combined with Tallman and Carlson’s findings strongly suggest a potentially serious shortcoming in the conceptual focus of Calculus 1. They also point to the need for further investigation of the content focus and student learning in both precalculus and beginning calculus in the United States. Another noteworthy finding of this study is that all students who received a 17 or higher on CCR received an A or B in Calculus 1. This finding supports that CCR is assessing relevant knowledge for succeeding in calculus, whether the course has a conceptual focus or not.

Examination of student responses on the collection of items that assess students’ function conception revealed that the vast majority of students in our study did not view a function as a process. This finding corroborates results that have been previously reported in the literature (e.g., Breidenbach et al. 1992 ; Carlson 1998 ; Dubinsky and Harel 1992 ). We found similar data trends for the collection of items that require students to use covariational reasoning to consider growth patterns in two quantities changing together. Students were asked to describe the behavior of the function f defined by f ( x ) = 1/( x –2) 2 . Only 37 % of students selected the correct answer—as the value of x gets larger, the value of f decreases, and as the value of x approaches 2, the value of f increases. Another noteworthy result is the high percentages of students who selected incorrect answers for the proportional reasoning and exponential growth items. These findings suggest a need for higher standards for curriculum and courses prior to calculus in terms of the degree to which they support students’ development of fundamental reasoning abilities and understandings needed for learning and using central ideas of calculus.

Our analysis of the CCR data suggests that it is useful as a tool to assess the effectiveness of a precalculus course or curriculum in preparing students for calculus. It can also be used to advise students about their readiness for calculus. We expect that CCR correlations with success in calculus will be higher when administered as a pre-test to students enrolled in calculus courses that emphasize understanding (making connections) and reasoning with ideas. Even though calculus courses vary in the amount of emphasis placed on skills, techniques, and understanding and using key concepts, we believe that CCR is a good measure of whether students are prepared to learn and understand calculus. However, we encourage those who administer CCR to Calculus 1 students to use the cut scores that we have suggested as advisory, and to consider local constraints and current curriculum foci in precalculus and beginning calculus to adjust break points accordingly.

The CCR instrument is part of the Placement Testing Suite of the Mathematical Association of America that is delivered by Maplesoft. Parrallel forms of the initial CCR have been developed and are being disseminated to inform precalculus instruction.

Many students who graduate from US high schools and attend US colleges and universities are not prepared to take beginning calculus, resulting in their enrolling in a course in precalculus as their first college level math course.

Students can engage in covariational reasoning prior to being formally introduced to the function concept.

Most applied problems in precalculus and beginning calculus request that students define a formula or function to express one quantity in terms of other formulas.

The ACT college readiness assessment is a standardized test for high school achievement and college admissions in the United States produced by ACT, Inc.

The authors are indebted to Charles Stegman and Clay Johnson of the National Office for Research on Measurement and Evaluation Systems and Joon Jin Song of the Statistical Consulting Institute of the University of Arkansas for assistance in analyzing these data.

These 215 students were a subset of the 601 beginning calculus students who completed CCR.

The data for the Carlson, Oehrtman & Engelke study was collected at the end of precalculus and the data for our study was collected during the first week of a beginning calculus. As a result, the samples are not comparable.

Breidenbach, D., Dubinsky, E., Hawks, J., & Nichols, D. (1992). Development of the process conception of function. Educational Studies in Mathematics, 23 (3), 247–285.

Article   Google Scholar  

Carlson, M. (1995). A cross-sectional investigation of the development of the function concept (Unpublished doctoral dissertation). Lawrence: University of Kansas.

Google Scholar  

Carlson, M. (1997). Obstacles for college algebra students in understanding functions: what do high performing students really know? AMATYC Review, 19 (1), 48–59.

Carlson, M. (1998). A cross-sectional investigation of the development of the function concept. In E. Dubinsky, A. H. Schoenfeld, & J. J. Kaput (Eds.), Research in collegiate mathematics education III (pp. 114–162). Providence: American Mathematical Society.

Carlson, M., & Rasmussen, C. (Eds.). (2008). Making the connection: Research and teaching in undergraduate mathematics education. MAA Notes 73 . Washington, DC: Mathematical Association of America.

Carlson, M., Jacobs, S., Coe, E., Larsen, S., & Hsu, E. (2002). Applying covariational reasoning while modeling dynamic events: a framework and a study. Journal for Research in Mathematics Education, 33 (5), 352–378.

Carlson, M., Smith, N., Persson, J. (2003). Developing and connecting calculus students’ notions of rate-of-change and accumulation: The fundamental theorem of calculus. Proceedings of the 2003 Joint Meeting of PME and PME-NA (Vol. 2, pp. 165–172). Honolulu, HI.

Carlson, M., Oehrtman, M., & Engelke, N. (2010). The precalculus concept assessment (PCA) instrument: a tool for assessing reasoning patterns, understandings and knowledge of precalculus level students. Cognition and Instruction, 28 (2), 113–145.

Carlson, M., Oehrtman, M., Moore, K. (2015). Pathways to calculus: A problem solving approach . Gilbert: Rational Reasoning. https://www.rationalreasoning.net .

Castillo-Garsow, C. C. (2010). Teaching the Verhulst model: A teaching experiment in covariational reasoning and exponential growth (Unpublished doctoral dissertation). Tempe: Arizona State University.

Dubinsky, E., & Harel, G. (1992). The nature of the process conception of function. In G. Harel & E. Dubinsky (Eds.), The concept of function: Aspects of epistemology and pedagogy. MAA Notes 25 (pp. 85–106). Washington, DC: Mathematical Association of America.

Engelke, N. (2007). Students’ understanding of related rates problems in calculus (Unpublished doctoral dissertation). Tempe: Arizona State University.

Engelke, N., Carlson, M., & Oehrtman, M. (2005). Composition of functions: Precalculus students’ understandings. In G. M. Lloyd, M. R. Wilson, J. L. M. Wilkins, & S. L. Behm (Eds.), Proceedings of the 27th Annual Meeting of the North American Chapter of the International Group for the Psychology of Mathematics Education [CD-ROM] . Eugene: All Academic.

Hestenes, D., & Wells, M. (1992). Mechanics baseline test. The Physics Teacher, 30 , 159–169.

Hestenes, D., Wells, M., & Swackhamer, G. (1992). Force concept inventory. The Physics Teacher, 30 , 141–158.

Kaput, J. (1992). Patterns in students’ formalization of quantitative patterns. In G. Harel & E. Dubinsky (Eds.), The concept of function: Aspects of epistemology and pedagogy. MAA Notes 25 (pp. 290–318). Washington, DC: Mathematical Association of America.

Kelly, T., Ebel, R., & Linacre, J. M. (2002). Item discrimination indices. Rasch Measurement Transactions, 16 (3), 883–884.

Lawson, A. E. (1978). The development and validation of a classroom test of formal reasoning. Journal of Research in Science Teaching, 15 , 11–24. doi: 10.1002/tea.3660150103 .

Lissitz, R., & Samuelsen, K. (2007). A suggested change in terminology and emphasis regarding validity and education. Educational Researcher, 36 (8), 437–448.

Monk, S. (1992). Students’ understanding of a function given by a physical model. In G. Harel & E. Dubinsky (Eds.), The concept of function: Aspects of epistemology and pedagogy. MAA Notes 25 (pp. 175–193). Washington, DC: Mathematical Association of America.

Moore, K. C. (2012). Coherence, quantitative reasoning, and the trigonometry of students. In R. Mayes & L. L. Hatfield (Eds.), Quantitative reasoning and mathematical modeling: A driver for STEM integrated education and teaching in context (pp. 75–92). Laramie: University of Wyoming.

Moore, K. C. (2013). Making sense by measuring arcs: a teaching experiment in angle measure. Educational Studies in Mathematics, 83 (2), 225–245.

Moore, K. C. (2014). Quantitative reasoning and the sine function: the case of Zac. Journal for Research in Mathematics Education, 45 (1), 102–138.

Moore, K. C., & Carlson, M. P. (2012). Students’ images of problem contexts when solving applied problems. Journal of Mathematical Behavior, 31 , 48–59.

Moore, K. C., LaForest, K. R., & Kim, H. J. (2012). The unit circle and unit conversions. In S. Brown, S. Larsen, K. Marrongelle, & M. Oehrtman (Eds.), Proceedings of the fifteenth annual conference on research in undergraduate mathematics education (pp. 16–31). Portland: Portland State University.

Oehrtman, M. (2008). Collapsing dimensions, physical limitation, and other student metaphors for limit concepts. Journal for Research in Mathematics Education, 40 (4), 396–426.

Piaget, J., Blaise-Grize, J., Szeminska, A., & Bang, V. (1977). Epistemology and psychology of functions . Dordrecht: Reidel.

Book   Google Scholar  

Sierpinska, A. (1992). On understanding the notion of function. In G. Harel & E. Dubinsky (Eds.), The concept of function: Aspects of epistemology and pedagogy. MAA Notes 25 (pp. 25–58). Washington, DC: The Mathematical Association of America.

Smith, N. (2008). Students’ emergent conceptions of the fundamental theorem of calculus (Unpublished doctoral dissertation). Tempe: Arizona State University.

Smith, J., & Thompson, P. W. (2007). Quantitative reasoning and the development of algebraic reasoning. In J. J. Kaput, D. W. Carraher, & M. L. Blanton (Eds.), Algebra in the early grades (pp. 95–132). New York: Erlbaum.

Strom, A. (2008). A case study of a secondary mathematics teacher’s understanding of exponential function: An emerging theoretical framework. (Unpublished doctoral dissertation) . Tempe: Arizona State University.

Tall, D. (1992). The transition to advanced mathematical thinking: Functions, infinity, and proof. In D. A. Grouws (Ed.), Handbook of research on mathematics teaching and learning (pp. 495–511). New York: Macmillan.

Tall, D. (1996). Functions and calculus. In A. Bishop, K. Clements, C. Keitel, J. Kilpatrick, & C. Laborde (Eds.), International handbook of mathematics education, part 1 (pp. 289–325). Dordrecht: Kluwer Academic Publishers.

Tall, D., & Vinner, S. (1981). Concept image and concept definition in mathematics with particular reference to limits and continuity. Educational Studies in Mathematics, 12 (2), 151–169.

Tallman, M., & Carlson, M. P. (under review). A characterization of calculus I final exams in U.S. Colleges and Universities .

Thompson, P. W. (1993). Quantitative reasoning, complexity, and additive structures. Educational Studies in Mathematics, 25 (3), 165–208.

Thompson, P. W. (1994a). Images of rate and operational understanding of the fundamental theorem of calculus. Educational Studies in Mathematics, 26 , 229–274.

Thompson, P. W. (1994b). Students, functions, and the undergraduate mathematics curriculum. In E. Dubinsky, A. H. Schoenfeld, & J. J. Kaput (Eds.), Research in collegiate mathematics education I (pp. 21–44). Providence: American Mathematical Society.

Thompson, P. W. (2008). Conceptual analysis of mathematical ideas: Some spadework at the foundations of mathematics education. In: Figueras, O., Cortina, J. L., Alatorre, S., Rojano, T., and Sépulveda, A. (Eds.), Proceedings of the Annual Meeting of the International Group for the Psychology of Mathematics Education Vol 1 (pp. 31–49). Morélia, Mexico: PME. Retrieved from http://bit.ly/10YE9al .

Thompson, P. W. (2011). Quantitative reasoning and mathematical modeling. In S. Chamberlin, L. L. Hatfield, & S. Belbase (Eds.), New perspectives and directions for collaborative research in mathematics education: Papers from a planning conference for WISDOM e (pp. 33–57). Laramie: University of Wyoming.

Thompson, P. W. (2012). Advances in research on quantitative reasoning. In: Mayes, R., Bonillia, R., L., Hatfield, L., Belbase, S. (Eds.). Quantitative reasoning: Current state of understanding , WISDOM e Monographs (Vol. 2, pp. 143–148). Laramie, WY: University of Wyoming. Retrieved from http://bit.ly/13FomiX .

Thompson, P. W., Carlson, M. P., & Silverman, J. (2007). The design of tasks in support of teachers' development of coherent mathematical meanings. Journal of Mathematics Teacher Education, 10 , 415–432.

Varma, S. (2012). Preliminary item statistics using point-biserial correlations and p-values , Educational Data Systems, Morgan Hill, CA. http://www.eddata.com/resources/publications/EDS_Point_Biserial.pdf . Accessed January 9, 2012.

Vinner, S., & Dreyfus, T. (1989). Images and definitions for the concept of function. Journal for Research in Mathematics Education, 20 , 356–366.

Zandieh, M. (2000). A theoretical framework for analyzing student understanding of the concept of derivative. In E. Dubinsky, A. H. Schoenfeld, & J. Kaput (Eds.), Research in collegiate mathematics education IV (pp. 103–127). Providence: American Mathematical Society.

Download references

Acknowledgments

We wish to thank Pat Thompson for his reviews and many helpful comments in refining this paper. This research was supported by Grant 1122965 from the National Science Foundation (NSF). Any opinions expressed in the article are those of the authors and do not necessarily reflect the views of NSF.

Author information

Authors and affiliations.

School of Mathematical and Statistical Sciences, Arizona State University, Tempe, AZ, 85287-1804, USA

Marilyn P. Carlson

University of Arkansas, Fayetteville, AR, 72701, USA

Bernard Madison

Francis Marion University, Florence, SC, 29502, USA

Richard D. West

You can also search for this author in PubMed   Google Scholar

Corresponding author

Correspondence to Marilyn P. Carlson .

Rights and permissions

Reprints and permissions

About this article

Carlson, M.P., Madison, B. & West, R.D. A Study of Students’ Readiness to Learn Calculus. Int. J. Res. Undergrad. Math. Ed. 1 , 209–233 (2015). https://doi.org/10.1007/s40753-015-0013-y

Download citation

Published : 16 September 2015

Issue Date : July 2015

DOI : https://doi.org/10.1007/s40753-015-0013-y

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Precalculus
• Placement exam
• Find a journal
• Publish with us
• Track your research

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

• View all journals
• My Account Login
• Explore content
• About the journal
• Publish with us
• Sign up for alerts
• Open access
• Published: 26 March 2024

Predicting and improving complex beer flavor through machine learning

• Michiel Schreurs   ORCID: orcid.org/0000-0002-9449-5619 1 , 2 , 3   na1 ,
• Supinya Piampongsant 1 , 2 , 3   na1 ,
• Miguel Roncoroni   ORCID: orcid.org/0000-0001-7461-1427 1 , 2 , 3   na1 ,
• Lloyd Cool   ORCID: orcid.org/0000-0001-9936-3124 1 , 2 , 3 , 4 ,
• Beatriz Herrera-Malaver   ORCID: orcid.org/0000-0002-5096-9974 1 , 2 , 3 ,
• Christophe Vanderaa   ORCID: orcid.org/0000-0001-7443-5427 4 ,
• Florian A. Theßeling 1 , 2 , 3 ,
• Łukasz Kreft   ORCID: orcid.org/0000-0001-7620-4657 5 ,
• Alexander Botzki   ORCID: orcid.org/0000-0001-6691-4233 5 ,
• Philippe Malcorps 6 ,
• Luk Daenen 6 ,
• Tom Wenseleers   ORCID: orcid.org/0000-0002-1434-861X 4 &
• Kevin J. Verstrepen   ORCID: orcid.org/0000-0002-3077-6219 1 , 2 , 3  

Nature Communications volume  15 , Article number:  2368 ( 2024 ) Cite this article

39k Accesses

749 Altmetric

Metrics details

• Chemical engineering
• Gas chromatography
• Machine learning
• Metabolomics
• Taste receptors

The perception and appreciation of food flavor depends on many interacting chemical compounds and external factors, and therefore proves challenging to understand and predict. Here, we combine extensive chemical and sensory analyses of 250 different beers to train machine learning models that allow predicting flavor and consumer appreciation. For each beer, we measure over 200 chemical properties, perform quantitative descriptive sensory analysis with a trained tasting panel and map data from over 180,000 consumer reviews to train 10 different machine learning models. The best-performing algorithm, Gradient Boosting, yields models that significantly outperform predictions based on conventional statistics and accurately predict complex food features and consumer appreciation from chemical profiles. Model dissection allows identifying specific and unexpected compounds as drivers of beer flavor and appreciation. Adding these compounds results in variants of commercial alcoholic and non-alcoholic beers with improved consumer appreciation. Together, our study reveals how big data and machine learning uncover complex links between food chemistry, flavor and consumer perception, and lays the foundation to develop novel, tailored foods with superior flavors.

Similar content being viewed by others

BitterSweet: Building machine learning models for predicting the bitter and sweet taste of small molecules

Rudraksh Tuwani, Somin Wadhwa & Ganesh Bagler

Sensory lexicon and aroma volatiles analysis of brewing malt

Xiaoxia Su, Miao Yu, … Tianyi Du

Predicting odor from molecular structure: a multi-label classification approach

Kushagra Saini & Venkatnarayan Ramanathan

Introduction

Predicting and understanding food perception and appreciation is one of the major challenges in food science. Accurate modeling of food flavor and appreciation could yield important opportunities for both producers and consumers, including quality control, product fingerprinting, counterfeit detection, spoilage detection, and the development of new products and product combinations (food pairing) 1 , 2 , 3 , 4 , 5 , 6 . Accurate models for flavor and consumer appreciation would contribute greatly to our scientific understanding of how humans perceive and appreciate flavor. Moreover, accurate predictive models would also facilitate and standardize existing food assessment methods and could supplement or replace assessments by trained and consumer tasting panels, which are variable, expensive and time-consuming 7 , 8 , 9 . Lastly, apart from providing objective, quantitative, accurate and contextual information that can help producers, models can also guide consumers in understanding their personal preferences 10 .

Despite the myriad of applications, predicting food flavor and appreciation from its chemical properties remains a largely elusive goal in sensory science, especially for complex food and beverages 11 , 12 . A key obstacle is the immense number of flavor-active chemicals underlying food flavor. Flavor compounds can vary widely in chemical structure and concentration, making them technically challenging and labor-intensive to quantify, even in the face of innovations in metabolomics, such as non-targeted metabolic fingerprinting 13 , 14 . Moreover, sensory analysis is perhaps even more complicated. Flavor perception is highly complex, resulting from hundreds of different molecules interacting at the physiochemical and sensorial level. Sensory perception is often non-linear, characterized by complex and concentration-dependent synergistic and antagonistic effects 15 , 16 , 17 , 18 , 19 , 20 , 21 that are further convoluted by the genetics, environment, culture and psychology of consumers 22 , 23 , 24 . Perceived flavor is therefore difficult to measure, with problems of sensitivity, accuracy, and reproducibility that can only be resolved by gathering sufficiently large datasets 25 . Trained tasting panels are considered the prime source of quality sensory data, but require meticulous training, are low throughput and high cost. Public databases containing consumer reviews of food products could provide a valuable alternative, especially for studying appreciation scores, which do not require formal training 25 . Public databases offer the advantage of amassing large amounts of data, increasing the statistical power to identify potential drivers of appreciation. However, public datasets suffer from biases, including a bias in the volunteers that contribute to the database, as well as confounding factors such as price, cult status and psychological conformity towards previous ratings of the product.

Classical multivariate statistics and machine learning methods have been used to predict flavor of specific compounds by, for example, linking structural properties of a compound to its potential biological activities or linking concentrations of specific compounds to sensory profiles 1 , 26 . Importantly, most previous studies focused on predicting organoleptic properties of single compounds (often based on their chemical structure) 27 , 28 , 29 , 30 , 31 , 32 , 33 , thus ignoring the fact that these compounds are present in a complex matrix in food or beverages and excluding complex interactions between compounds. Moreover, the classical statistics commonly used in sensory science 34 , 35 , 36 , 37 , 38 , 39 require a large sample size and sufficient variance amongst predictors to create accurate models. They are not fit for studying an extensive set of hundreds of interacting flavor compounds, since they are sensitive to outliers, have a high tendency to overfit and are less suited for non-linear and discontinuous relationships 40 .

In this study, we combine extensive chemical analyses and sensory data of a set of different commercial beers with machine learning approaches to develop models that predict taste, smell, mouthfeel and appreciation from compound concentrations. Beer is particularly suited to model the relationship between chemistry, flavor and appreciation. First, beer is a complex product, consisting of thousands of flavor compounds that partake in complex sensory interactions 41 , 42 , 43 . This chemical diversity arises from the raw materials (malt, yeast, hops, water and spices) and biochemical conversions during the brewing process (kilning, mashing, boiling, fermentation, maturation and aging) 44 , 45 . Second, the advent of the internet saw beer consumers embrace online review platforms, such as RateBeer (ZX Ventures, Anheuser-Busch InBev SA/NV) and BeerAdvocate (Next Glass, inc.). In this way, the beer community provides massive data sets of beer flavor and appreciation scores, creating extraordinarily large sensory databases to complement the analyses of our professional sensory panel. Specifically, we characterize over 200 chemical properties of 250 commercial beers, spread across 22 beer styles, and link these to the descriptive sensory profiling data of a 16-person in-house trained tasting panel and data acquired from over 180,000 public consumer reviews. These unique and extensive datasets enable us to train a suite of machine learning models to predict flavor and appreciation from a beer’s chemical profile. Dissection of the best-performing models allows us to pinpoint specific compounds as potential drivers of beer flavor and appreciation. Follow-up experiments confirm the importance of these compounds and ultimately allow us to significantly improve the flavor and appreciation of selected commercial beers. Together, our study represents a significant step towards understanding complex flavors and reinforces the value of machine learning to develop and refine complex foods. In this way, it represents a stepping stone for further computer-aided food engineering applications 46 .

To generate a comprehensive dataset on beer flavor, we selected 250 commercial Belgian beers across 22 different beer styles (Supplementary Fig.  S1 ). Beers with ≤ 4.2% alcohol by volume (ABV) were classified as non-alcoholic and low-alcoholic. Blonds and Tripels constitute a significant portion of the dataset (12.4% and 11.2%, respectively) reflecting their presence on the Belgian beer market and the heterogeneity of beers within these styles. By contrast, lager beers are less diverse and dominated by a handful of brands. Rare styles such as Brut or Faro make up only a small fraction of the dataset (2% and 1%, respectively) because fewer of these beers are produced and because they are dominated by distinct characteristics in terms of flavor and chemical composition.

Extensive analysis identifies relationships between chemical compounds in beer

For each beer, we measured 226 different chemical properties, including common brewing parameters such as alcohol content, iso-alpha acids, pH, sugar concentration 47 , and over 200 flavor compounds (Methods, Supplementary Table  S1 ). A large portion (37.2%) are terpenoids arising from hopping, responsible for herbal and fruity flavors 16 , 48 . A second major category are yeast metabolites, such as esters and alcohols, that result in fruity and solvent notes 48 , 49 , 50 . Other measured compounds are primarily derived from malt, or other microbes such as non- Saccharomyces yeasts and bacteria (‘wild flora’). Compounds that arise from spices or staling are labeled under ‘Others’. Five attributes (caloric value, total acids and total ester, hop aroma and sulfur compounds) are calculated from multiple individually measured compounds.

As a first step in identifying relationships between chemical properties, we determined correlations between the concentrations of the compounds (Fig.  1 , upper panel, Supplementary Data  1 and 2 , and Supplementary Fig.  S2 . For the sake of clarity, only a subset of the measured compounds is shown in Fig.  1 ). Compounds of the same origin typically show a positive correlation, while absence of correlation hints at parameters varying independently. For example, the hop aroma compounds citronellol, and alpha-terpineol show moderate correlations with each other (Spearman’s rho=0.39 and 0.57), but not with the bittering hop component iso-alpha acids (Spearman’s rho=0.16 and −0.07). This illustrates how brewers can independently modify hop aroma and bitterness by selecting hop varieties and dosage time. If hops are added early in the boiling phase, chemical conversions increase bitterness while aromas evaporate, conversely, late addition of hops preserves aroma but limits bitterness 51 . Similarly, hop-derived iso-alpha acids show a strong anti-correlation with lactic acid and acetic acid, likely reflecting growth inhibition of lactic acid and acetic acid bacteria, or the consequent use of fewer hops in sour beer styles, such as West Flanders ales and Fruit beers, that rely on these bacteria for their distinct flavors 52 . Finally, yeast-derived esters (ethyl acetate, ethyl decanoate, ethyl hexanoate, ethyl octanoate) and alcohols (ethanol, isoamyl alcohol, isobutanol, and glycerol), correlate with Spearman coefficients above 0.5, suggesting that these secondary metabolites are correlated with the yeast genetic background and/or fermentation parameters and may be difficult to influence individually, although the choice of yeast strain may offer some control 53 .

Spearman rank correlations are shown. Descriptors are grouped according to their origin (malt (blue), hops (green), yeast (red), wild flora (yellow), Others (black)), and sensory aspect (aroma, taste, palate, and overall appreciation). Please note that for the chemical compounds, for the sake of clarity, only a subset of the total number of measured compounds is shown, with an emphasis on the key compounds for each source. For more details, see the main text and Methods section. Chemical data can be found in Supplementary Data  1 , correlations between all chemical compounds are depicted in Supplementary Fig.  S2 and correlation values can be found in Supplementary Data  2 . See Supplementary Data  4 for sensory panel assessments and Supplementary Data  5 for correlation values between all sensory descriptors.

Interestingly, different beer styles show distinct patterns for some flavor compounds (Supplementary Fig.  S3 ). These observations agree with expectations for key beer styles, and serve as a control for our measurements. For instance, Stouts generally show high values for color (darker), while hoppy beers contain elevated levels of iso-alpha acids, compounds associated with bitter hop taste. Acetic and lactic acid are not prevalent in most beers, with notable exceptions such as Kriek, Lambic, Faro, West Flanders ales and Flanders Old Brown, which use acid-producing bacteria ( Lactobacillus and Pediococcus ) or unconventional yeast ( Brettanomyces ) 54 , 55 . Glycerol, ethanol and esters show similar distributions across all beer styles, reflecting their common origin as products of yeast metabolism during fermentation 45 , 53 . Finally, low/no-alcohol beers contain low concentrations of glycerol and esters. This is in line with the production process for most of the low/no-alcohol beers in our dataset, which are produced through limiting fermentation or by stripping away alcohol via evaporation or dialysis, with both methods having the unintended side-effect of reducing the amount of flavor compounds in the final beer 56 , 57 .

Besides expected associations, our data also reveals less trivial associations between beer styles and specific parameters. For example, geraniol and citronellol, two monoterpenoids responsible for citrus, floral and rose flavors and characteristic of Citra hops, are found in relatively high amounts in Christmas, Saison, and Brett/co-fermented beers, where they may originate from terpenoid-rich spices such as coriander seeds instead of hops 58 .

Tasting panel assessments reveal sensorial relationships in beer

To assess the sensory profile of each beer, a trained tasting panel evaluated each of the 250 beers for 50 sensory attributes, including different hop, malt and yeast flavors, off-flavors and spices. Panelists used a tasting sheet (Supplementary Data  3 ) to score the different attributes. Panel consistency was evaluated by repeating 12 samples across different sessions and performing ANOVA. In 95% of cases no significant difference was found across sessions ( p  > 0.05), indicating good panel consistency (Supplementary Table  S2 ).

Aroma and taste perception reported by the trained panel are often linked (Fig.  1 , bottom left panel and Supplementary Data  4 and 5 ), with high correlations between hops aroma and taste (Spearman’s rho=0.83). Bitter taste was found to correlate with hop aroma and taste in general (Spearman’s rho=0.80 and 0.69), and particularly with “grassy” noble hops (Spearman’s rho=0.75). Barnyard flavor, most often associated with sour beers, is identified together with stale hops (Spearman’s rho=0.97) that are used in these beers. Lactic and acetic acid, which often co-occur, are correlated (Spearman’s rho=0.66). Interestingly, sweetness and bitterness are anti-correlated (Spearman’s rho = −0.48), confirming the hypothesis that they mask each other 59 , 60 . Beer body is highly correlated with alcohol (Spearman’s rho = 0.79), and overall appreciation is found to correlate with multiple aspects that describe beer mouthfeel (alcohol, carbonation; Spearman’s rho= 0.32, 0.39), as well as with hop and ester aroma intensity (Spearman’s rho=0.39 and 0.35).

Similar to the chemical analyses, sensorial analyses confirmed typical features of specific beer styles (Supplementary Fig.  S4 ). For example, sour beers (Faro, Flanders Old Brown, Fruit beer, Kriek, Lambic, West Flanders ale) were rated acidic, with flavors of both acetic and lactic acid. Hoppy beers were found to be bitter and showed hop-associated aromas like citrus and tropical fruit. Malt taste is most detected among scotch, stout/porters, and strong ales, while low/no-alcohol beers, which often have a reputation for being ‘worty’ (reminiscent of unfermented, sweet malt extract) appear in the middle. Unsurprisingly, hop aromas are most strongly detected among hoppy beers. Like its chemical counterpart (Supplementary Fig.  S3 ), acidity shows a right-skewed distribution, with the most acidic beers being Krieks, Lambics, and West Flanders ales.

Tasting panel assessments of specific flavors correlate with chemical composition

We find that the concentrations of several chemical compounds strongly correlate with specific aroma or taste, as evaluated by the tasting panel (Fig.  2 , Supplementary Fig.  S5 , Supplementary Data  6 ). In some cases, these correlations confirm expectations and serve as a useful control for data quality. For example, iso-alpha acids, the bittering compounds in hops, strongly correlate with bitterness (Spearman’s rho=0.68), while ethanol and glycerol correlate with tasters’ perceptions of alcohol and body, the mouthfeel sensation of fullness (Spearman’s rho=0.82/0.62 and 0.72/0.57 respectively) and darker color from roasted malts is a good indication of malt perception (Spearman’s rho=0.54).

Heatmap colors indicate Spearman’s Rho. Axes are organized according to sensory categories (aroma, taste, mouthfeel, overall), chemical categories and chemical sources in beer (malt (blue), hops (green), yeast (red), wild flora (yellow), Others (black)). See Supplementary Data  6 for all correlation values.

Interestingly, for some relationships between chemical compounds and perceived flavor, correlations are weaker than expected. For example, the rose-smelling phenethyl acetate only weakly correlates with floral aroma. This hints at more complex relationships and interactions between compounds and suggests a need for a more complex model than simple correlations. Lastly, we uncovered unexpected correlations. For instance, the esters ethyl decanoate and ethyl octanoate appear to correlate slightly with hop perception and bitterness, possibly due to their fruity flavor. Iron is anti-correlated with hop aromas and bitterness, most likely because it is also anti-correlated with iso-alpha acids. This could be a sign of metal chelation of hop acids 61 , given that our analyses measure unbound hop acids and total iron content, or could result from the higher iron content in dark and Fruit beers, which typically have less hoppy and bitter flavors 62 .

Public consumer reviews complement expert panel data

To complement and expand the sensory data of our trained tasting panel, we collected 180,000 reviews of our 250 beers from the online consumer review platform RateBeer. This provided numerical scores for beer appearance, aroma, taste, palate, overall quality as well as the average overall score.

Public datasets are known to suffer from biases, such as price, cult status and psychological conformity towards previous ratings of a product. For example, prices correlate with appreciation scores for these online consumer reviews (rho=0.49, Supplementary Fig.  S6 ), but not for our trained tasting panel (rho=0.19). This suggests that prices affect consumer appreciation, which has been reported in wine 63 , while blind tastings are unaffected. Moreover, we observe that some beer styles, like lagers and non-alcoholic beers, generally receive lower scores, reflecting that online reviewers are mostly beer aficionados with a preference for specialty beers over lager beers. In general, we find a modest correlation between our trained panel’s overall appreciation score and the online consumer appreciation scores (Fig.  3 , rho=0.29). Apart from the aforementioned biases in the online datasets, serving temperature, sample freshness and surroundings, which are all tightly controlled during the tasting panel sessions, can vary tremendously across online consumers and can further contribute to (among others, appreciation) differences between the two categories of tasters. Importantly, in contrast to the overall appreciation scores, for many sensory aspects the results from the professional panel correlated well with results obtained from RateBeer reviews. Correlations were highest for features that are relatively easy to recognize even for untrained tasters, like bitterness, sweetness, alcohol and malt aroma (Fig.  3 and below).

RateBeer text mining results can be found in Supplementary Data  7 . Rho values shown are Spearman correlation values, with asterisks indicating significant correlations ( p  < 0.05, two-sided). All p values were smaller than 0.001, except for Esters aroma (0.0553), Esters taste (0.3275), Esters aroma—banana (0.0019), Coriander (0.0508) and Diacetyl (0.0134).

Besides collecting consumer appreciation from these online reviews, we developed automated text analysis tools to gather additional data from review texts (Supplementary Data  7 ). Processing review texts on the RateBeer database yielded comparable results to the scores given by the trained panel for many common sensory aspects, including acidity, bitterness, sweetness, alcohol, malt, and hop tastes (Fig.  3 ). This is in line with what would be expected, since these attributes require less training for accurate assessment and are less influenced by environmental factors such as temperature, serving glass and odors in the environment. Consumer reviews also correlate well with our trained panel for 4-vinyl guaiacol, a compound associated with a very characteristic aroma. By contrast, correlations for more specific aromas like ester, coriander or diacetyl are underrepresented in the online reviews, underscoring the importance of using a trained tasting panel and standardized tasting sheets with explicit factors to be scored for evaluating specific aspects of a beer. Taken together, our results suggest that public reviews are trustworthy for some, but not all, flavor features and can complement or substitute taste panel data for these sensory aspects.

Models can predict beer sensory profiles from chemical data

The rich datasets of chemical analyses, tasting panel assessments and public reviews gathered in the first part of this study provided us with a unique opportunity to develop predictive models that link chemical data to sensorial features. Given the complexity of beer flavor, basic statistical tools such as correlations or linear regression may not always be the most suitable for making accurate predictions. Instead, we applied different machine learning models that can model both simple linear and complex interactive relationships. Specifically, we constructed a set of regression models to predict (a) trained panel scores for beer flavor and quality and (b) public reviews’ appreciation scores from beer chemical profiles. We trained and tested 10 different models (Methods), 3 linear regression-based models (simple linear regression with first-order interactions (LR), lasso regression with first-order interactions (Lasso), partial least squares regressor (PLSR)), 5 decision tree models (AdaBoost regressor (ABR), extra trees (ET), gradient boosting regressor (GBR), random forest (RF) and XGBoost regressor (XGBR)), 1 support vector regression (SVR), and 1 artificial neural network (ANN) model.

To compare the performance of our machine learning models, the dataset was randomly split into a training and test set, stratified by beer style. After a model was trained on data in the training set, its performance was evaluated on its ability to predict the test dataset obtained from multi-output models (based on the coefficient of determination, see Methods). Additionally, individual-attribute models were ranked per descriptor and the average rank was calculated, as proposed by Korneva et al. 64 . Importantly, both ways of evaluating the models’ performance agreed in general. Performance of the different models varied (Table  1 ). It should be noted that all models perform better at predicting RateBeer results than results from our trained tasting panel. One reason could be that sensory data is inherently variable, and this variability is averaged out with the large number of public reviews from RateBeer. Additionally, all tree-based models perform better at predicting taste than aroma. Linear models (LR) performed particularly poorly, with negative R 2 values, due to severe overfitting (training set R 2  = 1). Overfitting is a common issue in linear models with many parameters and limited samples, especially with interaction terms further amplifying the number of parameters. L1 regularization (Lasso) successfully overcomes this overfitting, out-competing multiple tree-based models on the RateBeer dataset. Similarly, the dimensionality reduction of PLSR avoids overfitting and improves performance, to some extent. Still, tree-based models (ABR, ET, GBR, RF and XGBR) show the best performance, out-competing the linear models (LR, Lasso, PLSR) commonly used in sensory science 65 .

GBR models showed the best overall performance in predicting sensory responses from chemical information, with R 2 values up to 0.75 depending on the predicted sensory feature (Supplementary Table  S4 ). The GBR models predict consumer appreciation (RateBeer) better than our trained panel’s appreciation (R 2 value of 0.67 compared to R 2 value of 0.09) (Supplementary Table  S3 and Supplementary Table  S4 ). ANN models showed intermediate performance, likely because neural networks typically perform best with larger datasets 66 . The SVR shows intermediate performance, mostly due to the weak predictions of specific attributes that lower the overall performance (Supplementary Table  S4 ).

Model dissection identifies specific, unexpected compounds as drivers of consumer appreciation

Next, we leveraged our models to infer important contributors to sensory perception and consumer appreciation. Consumer preference is a crucial sensory aspects, because a product that shows low consumer appreciation scores often does not succeed commercially 25 . Additionally, the requirement for a large number of representative evaluators makes consumer trials one of the more costly and time-consuming aspects of product development. Hence, a model for predicting chemical drivers of overall appreciation would be a welcome addition to the available toolbox for food development and optimization.

Since GBR models on our RateBeer dataset showed the best overall performance, we focused on these models. Specifically, we used two approaches to identify important contributors. First, rankings of the most important predictors for each sensorial trait in the GBR models were obtained based on impurity-based feature importance (mean decrease in impurity). High-ranked parameters were hypothesized to be either the true causal chemical properties underlying the trait, to correlate with the actual causal properties, or to take part in sensory interactions affecting the trait 67 (Fig.  4A ). In a second approach, we used SHAP 68 to determine which parameters contributed most to the model for making predictions of consumer appreciation (Fig.  4B ). SHAP calculates parameter contributions to model predictions on a per-sample basis, which can be aggregated into an importance score.

A The impurity-based feature importance (mean deviance in impurity, MDI) calculated from the Gradient Boosting Regression (GBR) model predicting RateBeer appreciation scores. The top 15 highest ranked chemical properties are shown. B SHAP summary plot for the top 15 parameters contributing to our GBR model. Each point on the graph represents a sample from our dataset. The color represents the concentration of that parameter, with bluer colors representing low values and redder colors representing higher values. Greater absolute values on the horizontal axis indicate a higher impact of the parameter on the prediction of the model. C Spearman correlations between the 15 most important chemical properties and consumer overall appreciation. Numbers indicate the Spearman Rho correlation coefficient, and the rank of this correlation compared to all other correlations. The top 15 important compounds were determined using SHAP (panel B).

Both approaches identified ethyl acetate as the most predictive parameter for beer appreciation (Fig.  4 ). Ethyl acetate is the most abundant ester in beer with a typical ‘fruity’, ‘solvent’ and ‘alcoholic’ flavor, but is often considered less important than other esters like isoamyl acetate. The second most important parameter identified by SHAP is ethanol, the most abundant beer compound after water. Apart from directly contributing to beer flavor and mouthfeel, ethanol drastically influences the physical properties of beer, dictating how easily volatile compounds escape the beer matrix to contribute to beer aroma 69 . Importantly, it should also be noted that the importance of ethanol for appreciation is likely inflated by the very low appreciation scores of non-alcoholic beers (Supplementary Fig.  S4 ). Despite not often being considered a driver of beer appreciation, protein level also ranks highly in both approaches, possibly due to its effect on mouthfeel and body 70 . Lactic acid, which contributes to the tart taste of sour beers, is the fourth most important parameter identified by SHAP, possibly due to the generally high appreciation of sour beers in our dataset.

Interestingly, some of the most important predictive parameters for our model are not well-established as beer flavors or are even commonly regarded as being negative for beer quality. For example, our models identify methanethiol and ethyl phenyl acetate, an ester commonly linked to beer staling 71 , as a key factor contributing to beer appreciation. Although there is no doubt that high concentrations of these compounds are considered unpleasant, the positive effects of modest concentrations are not yet known 72 , 73 .

To compare our approach to conventional statistics, we evaluated how well the 15 most important SHAP-derived parameters correlate with consumer appreciation (Fig.  4C ). Interestingly, only 6 of the properties derived by SHAP rank amongst the top 15 most correlated parameters. For some chemical compounds, the correlations are so low that they would have likely been considered unimportant. For example, lactic acid, the fourth most important parameter, shows a bimodal distribution for appreciation, with sour beers forming a separate cluster, that is missed entirely by the Spearman correlation. Additionally, the correlation plots reveal outliers, emphasizing the need for robust analysis tools. Together, this highlights the need for alternative models, like the Gradient Boosting model, that better grasp the complexity of (beer) flavor.

Finally, to observe the relationships between these chemical properties and their predicted targets, partial dependence plots were constructed for the six most important predictors of consumer appreciation 74 , 75 , 76 (Supplementary Fig.  S7 ). One-way partial dependence plots show how a change in concentration affects the predicted appreciation. These plots reveal an important limitation of our models: appreciation predictions remain constant at ever-increasing concentrations. This implies that once a threshold concentration is reached, further increasing the concentration does not affect appreciation. This is false, as it is well-documented that certain compounds become unpleasant at high concentrations, including ethyl acetate (‘nail polish’) 77 and methanethiol (‘sulfury’ and ‘rotten cabbage’) 78 . The inability of our models to grasp that flavor compounds have optimal levels, above which they become negative, is a consequence of working with commercial beer brands where (off-)flavors are rarely too high to negatively impact the product. The two-way partial dependence plots show how changing the concentration of two compounds influences predicted appreciation, visualizing their interactions (Supplementary Fig.  S7 ). In our case, the top 5 parameters are dominated by additive or synergistic interactions, with high concentrations for both compounds resulting in the highest predicted appreciation.

To assess the robustness of our best-performing models and model predictions, we performed 100 iterations of the GBR, RF and ET models. In general, all iterations of the models yielded similar performance (Supplementary Fig.  S8 ). Moreover, the main predictors (including the top predictors ethanol and ethyl acetate) remained virtually the same, especially for GBR and RF. For the iterations of the ET model, we did observe more variation in the top predictors, which is likely a consequence of the model’s inherent random architecture in combination with co-correlations between certain predictors. However, even in this case, several of the top predictors (ethanol and ethyl acetate) remain unchanged, although their rank in importance changes (Supplementary Fig.  S8 ).

Next, we investigated if a combination of RateBeer and trained panel data into one consolidated dataset would lead to stronger models, under the hypothesis that such a model would suffer less from bias in the datasets. A GBR model was trained to predict appreciation on the combined dataset. This model underperformed compared to the RateBeer model, both in the native case and when including a dataset identifier (R 2  = 0.67, 0.26 and 0.42 respectively). For the latter, the dataset identifier is the most important feature (Supplementary Fig.  S9 ), while most of the feature importance remains unchanged, with ethyl acetate and ethanol ranking highest, like in the original model trained only on RateBeer data. It seems that the large variation in the panel dataset introduces noise, weakening the models’ performances and reliability. In addition, it seems reasonable to assume that both datasets are fundamentally different, with the panel dataset obtained by blind tastings by a trained professional panel.

Lastly, we evaluated whether beer style identifiers would further enhance the model’s performance. A GBR model was trained with parameters that explicitly encoded the styles of the samples. This did not improve model performance (R2 = 0.66 with style information vs R2 = 0.67). The most important chemical features are consistent with the model trained without style information (eg. ethanol and ethyl acetate), and with the exception of the most preferred (strong ale) and least preferred (low/no-alcohol) styles, none of the styles were among the most important features (Supplementary Fig.  S9 , Supplementary Table  S5 and S6 ). This is likely due to a combination of style-specific chemical signatures, such as iso-alpha acids and lactic acid, that implicitly convey style information to the original models, as well as the low number of samples belonging to some styles, making it difficult for the model to learn style-specific patterns. Moreover, beer styles are not rigorously defined, with some styles overlapping in features and some beers being misattributed to a specific style, all of which leads to more noise in models that use style parameters.

Model validation

To test if our predictive models give insight into beer appreciation, we set up experiments aimed at improving existing commercial beers. We specifically selected overall appreciation as the trait to be examined because of its complexity and commercial relevance. Beer flavor comprises a complex bouquet rather than single aromas and tastes 53 . Hence, adding a single compound to the extent that a difference is noticeable may lead to an unbalanced, artificial flavor. Therefore, we evaluated the effect of combinations of compounds. Because Blond beers represent the most extensive style in our dataset, we selected a beer from this style as the starting material for these experiments (Beer 64 in Supplementary Data  1 ).

In the first set of experiments, we adjusted the concentrations of compounds that made up the most important predictors of overall appreciation (ethyl acetate, ethanol, lactic acid, ethyl phenyl acetate) together with correlated compounds (ethyl hexanoate, isoamyl acetate, glycerol), bringing them up to 95 th percentile ethanol-normalized concentrations (Methods) within the Blond group (‘Spiked’ concentration in Fig.  5A ). Compared to controls, the spiked beers were found to have significantly improved overall appreciation among trained panelists, with panelist noting increased intensity of ester flavors, sweetness, alcohol, and body fullness (Fig.  5B ). To disentangle the contribution of ethanol to these results, a second experiment was performed without the addition of ethanol. This resulted in a similar outcome, including increased perception of alcohol and overall appreciation.

Adding the top chemical compounds, identified as best predictors of appreciation by our model, into poorly appreciated beers results in increased appreciation from our trained panel. Results of sensory tests between base beers and those spiked with compounds identified as the best predictors by the model. A Blond and Non/Low-alcohol (0.0% ABV) base beers were brought up to 95th-percentile ethanol-normalized concentrations within each style. B For each sensory attribute, tasters indicated the more intense sample and selected the sample they preferred. The numbers above the bars correspond to the p values that indicate significant changes in perceived flavor (two-sided binomial test: alpha 0.05, n  = 20 or 13).

In a last experiment, we tested whether using the model’s predictions can boost the appreciation of a non-alcoholic beer (beer 223 in Supplementary Data  1 ). Again, the addition of a mixture of predicted compounds (omitting ethanol, in this case) resulted in a significant increase in appreciation, body, ester flavor and sweetness.

Predicting flavor and consumer appreciation from chemical composition is one of the ultimate goals of sensory science. A reliable, systematic and unbiased way to link chemical profiles to flavor and food appreciation would be a significant asset to the food and beverage industry. Such tools would substantially aid in quality control and recipe development, offer an efficient and cost-effective alternative to pilot studies and consumer trials and would ultimately allow food manufacturers to produce superior, tailor-made products that better meet the demands of specific consumer groups more efficiently.

A limited set of studies have previously tried, to varying degrees of success, to predict beer flavor and beer popularity based on (a limited set of) chemical compounds and flavors 79 , 80 . Current sensitive, high-throughput technologies allow measuring an unprecedented number of chemical compounds and properties in a large set of samples, yielding a dataset that can train models that help close the gaps between chemistry and flavor, even for a complex natural product like beer. To our knowledge, no previous research gathered data at this scale (250 samples, 226 chemical parameters, 50 sensory attributes and 5 consumer scores) to disentangle and validate the chemical aspects driving beer preference using various machine-learning techniques. We find that modern machine learning models outperform conventional statistical tools, such as correlations and linear models, and can successfully predict flavor appreciation from chemical composition. This could be attributed to the natural incorporation of interactions and non-linear or discontinuous effects in machine learning models, which are not easily grasped by the linear model architecture. While linear models and partial least squares regression represent the most widespread statistical approaches in sensory science, in part because they allow interpretation 65 , 81 , 82 , modern machine learning methods allow for building better predictive models while preserving the possibility to dissect and exploit the underlying patterns. Of the 10 different models we trained, tree-based models, such as our best performing GBR, showed the best overall performance in predicting sensory responses from chemical information, outcompeting artificial neural networks. This agrees with previous reports for models trained on tabular data 83 . Our results are in line with the findings of Colantonio et al. who also identified the gradient boosting architecture as performing best at predicting appreciation and flavor (of tomatoes and blueberries, in their specific study) 26 . Importantly, besides our larger experimental scale, we were able to directly confirm our models’ predictions in vivo.

Our study confirms that flavor compound concentration does not always correlate with perception, suggesting complex interactions that are often missed by more conventional statistics and simple models. Specifically, we find that tree-based algorithms may perform best in developing models that link complex food chemistry with aroma. Furthermore, we show that massive datasets of untrained consumer reviews provide a valuable source of data, that can complement or even replace trained tasting panels, especially for appreciation and basic flavors, such as sweetness and bitterness. This holds despite biases that are known to occur in such datasets, such as price or conformity bias. Moreover, GBR models predict taste better than aroma. This is likely because taste (e.g. bitterness) often directly relates to the corresponding chemical measurements (e.g., iso-alpha acids), whereas such a link is less clear for aromas, which often result from the interplay between multiple volatile compounds. We also find that our models are best at predicting acidity and alcohol, likely because there is a direct relation between the measured chemical compounds (acids and ethanol) and the corresponding perceived sensorial attribute (acidity and alcohol), and because even untrained consumers are generally able to recognize these flavors and aromas.

The predictions of our final models, trained on review data, hold even for blind tastings with small groups of trained tasters, as demonstrated by our ability to validate specific compounds as drivers of beer flavor and appreciation. Since adding a single compound to the extent of a noticeable difference may result in an unbalanced flavor profile, we specifically tested our identified key drivers as a combination of compounds. While this approach does not allow us to validate if a particular single compound would affect flavor and/or appreciation, our experiments do show that this combination of compounds increases consumer appreciation.

It is important to stress that, while it represents an important step forward, our approach still has several major limitations. A key weakness of the GBR model architecture is that amongst co-correlating variables, the largest main effect is consistently preferred for model building. As a result, co-correlating variables often have artificially low importance scores, both for impurity and SHAP-based methods, like we observed in the comparison to the more randomized Extra Trees models. This implies that chemicals identified as key drivers of a specific sensory feature by GBR might not be the true causative compounds, but rather co-correlate with the actual causative chemical. For example, the high importance of ethyl acetate could be (partially) attributed to the total ester content, ethanol or ethyl hexanoate (rho=0.77, rho=0.72 and rho=0.68), while ethyl phenylacetate could hide the importance of prenyl isobutyrate and ethyl benzoate (rho=0.77 and rho=0.76). Expanding our GBR model to include beer style as a parameter did not yield additional power or insight. This is likely due to style-specific chemical signatures, such as iso-alpha acids and lactic acid, that implicitly convey style information to the original model, as well as the smaller sample size per style, limiting the power to uncover style-specific patterns. This can be partly attributed to the curse of dimensionality, where the high number of parameters results in the models mainly incorporating single parameter effects, rather than complex interactions such as style-dependent effects 67 . A larger number of samples may overcome some of these limitations and offer more insight into style-specific effects. On the other hand, beer style is not a rigid scientific classification, and beers within one style often differ a lot, which further complicates the analysis of style as a model factor.

Our study is limited to beers from Belgian breweries. Although these beers cover a large portion of the beer styles available globally, some beer styles and consumer patterns may be missing, while other features might be overrepresented. For example, many Belgian ales exhibit yeast-driven flavor profiles, which is reflected in the chemical drivers of appreciation discovered by this study. In future work, expanding the scope to include diverse markets and beer styles could lead to the identification of even more drivers of appreciation and better models for special niche products that were not present in our beer set.

In addition to inherent limitations of GBR models, there are also some limitations associated with studying food aroma. Even if our chemical analyses measured most of the known aroma compounds, the total number of flavor compounds in complex foods like beer is still larger than the subset we were able to measure in this study. For example, hop-derived thiols, that influence flavor at very low concentrations, are notoriously difficult to measure in a high-throughput experiment. Moreover, consumer perception remains subjective and prone to biases that are difficult to avoid. It is also important to stress that the models are still immature and that more extensive datasets will be crucial for developing more complete models in the future. Besides more samples and parameters, our dataset does not include any demographic information about the tasters. Including such data could lead to better models that grasp external factors like age and culture. Another limitation is that our set of beers consists of high-quality end-products and lacks beers that are unfit for sale, which limits the current model in accurately predicting products that are appreciated very badly. Finally, while models could be readily applied in quality control, their use in sensory science and product development is restrained by their inability to discern causal relationships. Given that the models cannot distinguish compounds that genuinely drive consumer perception from those that merely correlate, validation experiments are essential to identify true causative compounds.

Despite the inherent limitations, dissection of our models enabled us to pinpoint specific molecules as potential drivers of beer aroma and consumer appreciation, including compounds that were unexpected and would not have been identified using standard approaches. Important drivers of beer appreciation uncovered by our models include protein levels, ethyl acetate, ethyl phenyl acetate and lactic acid. Currently, many brewers already use lactic acid to acidify their brewing water and ensure optimal pH for enzymatic activity during the mashing process. Our results suggest that adding lactic acid can also improve beer appreciation, although its individual effect remains to be tested. Interestingly, ethanol appears to be unnecessary to improve beer appreciation, both for blond beer and alcohol-free beer. Given the growing consumer interest in alcohol-free beer, with a predicted annual market growth of >7% 84 , it is relevant for brewers to know what compounds can further increase consumer appreciation of these beers. Hence, our model may readily provide avenues to further improve the flavor and consumer appreciation of both alcoholic and non-alcoholic beers, which is generally considered one of the key challenges for future beer production.

Whereas we see a direct implementation of our results for the development of superior alcohol-free beverages and other food products, our study can also serve as a stepping stone for the development of novel alcohol-containing beverages. We want to echo the growing body of scientific evidence for the negative effects of alcohol consumption, both on the individual level by the mutagenic, teratogenic and carcinogenic effects of ethanol 85 , 86 , as well as the burden on society caused by alcohol abuse and addiction. We encourage the use of our results for the production of healthier, tastier products, including novel and improved beverages with lower alcohol contents. Furthermore, we strongly discourage the use of these technologies to improve the appreciation or addictive properties of harmful substances.

The present work demonstrates that despite some important remaining hurdles, combining the latest developments in chemical analyses, sensory analysis and modern machine learning methods offers exciting avenues for food chemistry and engineering. Soon, these tools may provide solutions in quality control and recipe development, as well as new approaches to sensory science and flavor research.

Beer selection

250 commercial Belgian beers were selected to cover the broad diversity of beer styles and corresponding diversity in chemical composition and aroma. See Supplementary Fig.  S1 .

Chemical dataset

Sample preparation.

Beers within their expiration date were purchased from commercial retailers. Samples were prepared in biological duplicates at room temperature, unless explicitly stated otherwise. Bottle pressure was measured with a manual pressure device (Steinfurth Mess-Systeme GmbH) and used to calculate CO 2 concentration. The beer was poured through two filter papers (Macherey-Nagel, 500713032 MN 713 ¼) to remove carbon dioxide and prevent spontaneous foaming. Samples were then prepared for measurements by targeted Headspace-Gas Chromatography-Flame Ionization Detector/Flame Photometric Detector (HS-GC-FID/FPD), Headspace-Solid Phase Microextraction-Gas Chromatography-Mass Spectrometry (HS-SPME-GC-MS), colorimetric analysis, enzymatic analysis, Near-Infrared (NIR) analysis, as described in the sections below. The mean values of biological duplicates are reported for each compound.

HS-GC-FID/FPD

HS-GC-FID/FPD (Shimadzu GC 2010 Plus) was used to measure higher alcohols, acetaldehyde, esters, 4-vinyl guaicol, and sulfur compounds. Each measurement comprised 5 ml of sample pipetted into a 20 ml glass vial containing 1.75 g NaCl (VWR, 27810.295). 100 µl of 2-heptanol (Sigma-Aldrich, H3003) (internal standard) solution in ethanol (Fisher Chemical, E/0650DF/C17) was added for a final concentration of 2.44 mg/L. Samples were flushed with nitrogen for 10 s, sealed with a silicone septum, stored at −80 °C and analyzed in batches of 20.

The GC was equipped with a DB-WAXetr column (length, 30 m; internal diameter, 0.32 mm; layer thickness, 0.50 µm; Agilent Technologies, Santa Clara, CA, USA) to the FID and an HP-5 column (length, 30 m; internal diameter, 0.25 mm; layer thickness, 0.25 µm; Agilent Technologies, Santa Clara, CA, USA) to the FPD. N 2 was used as the carrier gas. Samples were incubated for 20 min at 70 °C in the headspace autosampler (Flow rate, 35 cm/s; Injection volume, 1000 µL; Injection mode, split; Combi PAL autosampler, CTC analytics, Switzerland). The injector, FID and FPD temperatures were kept at 250 °C. The GC oven temperature was first held at 50 °C for 5 min and then allowed to rise to 80 °C at a rate of 5 °C/min, followed by a second ramp of 4 °C/min until 200 °C kept for 3 min and a final ramp of (4 °C/min) until 230 °C for 1 min. Results were analyzed with the GCSolution software version 2.4 (Shimadzu, Kyoto, Japan). The GC was calibrated with a 5% EtOH solution (VWR International) containing the volatiles under study (Supplementary Table  S7 ).

HS-SPME-GC-MS

HS-SPME-GC-MS (Shimadzu GCMS-QP-2010 Ultra) was used to measure additional volatile compounds, mainly comprising terpenoids and esters. Samples were analyzed by HS-SPME using a triphase DVB/Carboxen/PDMS 50/30 μm SPME fiber (Supelco Co., Bellefonte, PA, USA) followed by gas chromatography (Thermo Fisher Scientific Trace 1300 series, USA) coupled to a mass spectrometer (Thermo Fisher Scientific ISQ series MS) equipped with a TriPlus RSH autosampler. 5 ml of degassed beer sample was placed in 20 ml vials containing 1.75 g NaCl (VWR, 27810.295). 5 µl internal standard mix was added, containing 2-heptanol (1 g/L) (Sigma-Aldrich, H3003), 4-fluorobenzaldehyde (1 g/L) (Sigma-Aldrich, 128376), 2,3-hexanedione (1 g/L) (Sigma-Aldrich, 144169) and guaiacol (1 g/L) (Sigma-Aldrich, W253200) in ethanol (Fisher Chemical, E/0650DF/C17). Each sample was incubated at 60 °C in the autosampler oven with constant agitation. After 5 min equilibration, the SPME fiber was exposed to the sample headspace for 30 min. The compounds trapped on the fiber were thermally desorbed in the injection port of the chromatograph by heating the fiber for 15 min at 270 °C.

The GC-MS was equipped with a low polarity RXi-5Sil MS column (length, 20 m; internal diameter, 0.18 mm; layer thickness, 0.18 µm; Restek, Bellefonte, PA, USA). Injection was performed in splitless mode at 320 °C, a split flow of 9 ml/min, a purge flow of 5 ml/min and an open valve time of 3 min. To obtain a pulsed injection, a programmed gas flow was used whereby the helium gas flow was set at 2.7 mL/min for 0.1 min, followed by a decrease in flow of 20 ml/min to the normal 0.9 mL/min. The temperature was first held at 30 °C for 3 min and then allowed to rise to 80 °C at a rate of 7 °C/min, followed by a second ramp of 2 °C/min till 125 °C and a final ramp of 8 °C/min with a final temperature of 270 °C.

Mass acquisition range was 33 to 550 amu at a scan rate of 5 scans/s. Electron impact ionization energy was 70 eV. The interface and ion source were kept at 275 °C and 250 °C, respectively. A mix of linear n-alkanes (from C7 to C40, Supelco Co.) was injected into the GC-MS under identical conditions to serve as external retention index markers. Identification and quantification of the compounds were performed using an in-house developed R script as described in Goelen et al. and Reher et al. 87 , 88 (for package information, see Supplementary Table  S8 ). Briefly, chromatograms were analyzed using AMDIS (v2.71) 89 to separate overlapping peaks and obtain pure compound spectra. The NIST MS Search software (v2.0 g) in combination with the NIST2017, FFNSC3 and Adams4 libraries were used to manually identify the empirical spectra, taking into account the expected retention time. After background subtraction and correcting for retention time shifts between samples run on different days based on alkane ladders, compound elution profiles were extracted and integrated using a file with 284 target compounds of interest, which were either recovered in our identified AMDIS list of spectra or were known to occur in beer. Compound elution profiles were estimated for every peak in every chromatogram over a time-restricted window using weighted non-negative least square analysis after which peak areas were integrated 87 , 88 . Batch effect correction was performed by normalizing against the most stable internal standard compound, 4-fluorobenzaldehyde. Out of all 284 target compounds that were analyzed, 167 were visually judged to have reliable elution profiles and were used for final analysis.

Discrete photometric and enzymatic analysis

Discrete photometric and enzymatic analysis (Thermo Scientific TM Gallery TM Plus Beermaster Discrete Analyzer) was used to measure acetic acid, ammonia, beta-glucan, iso-alpha acids, color, sugars, glycerol, iron, pH, protein, and sulfite. 2 ml of sample volume was used for the analyses. Information regarding the reagents and standard solutions used for analyses and calibrations is included in Supplementary Table  S7 and Supplementary Table  S9 .

NIR analyses

NIR analysis (Anton Paar Alcolyzer Beer ME System) was used to measure ethanol. Measurements comprised 50 ml of sample, and a 10% EtOH solution was used for calibration.

Correlation calculations

Pairwise Spearman Rank correlations were calculated between all chemical properties.

Sensory dataset

Trained panel.

Our trained tasting panel consisted of volunteers who gave prior verbal informed consent. All compounds used for the validation experiment were of food-grade quality. The tasting sessions were approved by the Social and Societal Ethics Committee of the KU Leuven (G-2022-5677-R2(MAR)). All online reviewers agreed to the Terms and Conditions of the RateBeer website.

Sensory analysis was performed according to the American Society of Brewing Chemists (ASBC) Sensory Analysis Methods 90 . 30 volunteers were screened through a series of triangle tests. The sixteen most sensitive and consistent tasters were retained as taste panel members. The resulting panel was diverse in age [22–42, mean: 29], sex [56% male] and nationality [7 different countries]. The panel developed a consensus vocabulary to describe beer aroma, taste and mouthfeel. Panelists were trained to identify and score 50 different attributes, using a 7-point scale to rate attributes’ intensity. The scoring sheet is included as Supplementary Data  3 . Sensory assessments took place between 10–12 a.m. The beers were served in black-colored glasses. Per session, between 5 and 12 beers of the same style were tasted at 12 °C to 16 °C. Two reference beers were added to each set and indicated as ‘Reference 1 & 2’, allowing panel members to calibrate their ratings. Not all panelists were present at every tasting. Scores were scaled by standard deviation and mean-centered per taster. Values are represented as z-scores and clustered by Euclidean distance. Pairwise Spearman correlations were calculated between taste and aroma sensory attributes. Panel consistency was evaluated by repeating samples on different sessions and performing ANOVA to identify differences, using the ‘stats’ package (v4.2.2) in R (for package information, see Supplementary Table  S8 ).

Online reviews from a public database

The ‘scrapy’ package in Python (v3.6) (for package information, see Supplementary Table  S8 ). was used to collect 232,288 online reviews (mean=922, min=6, max=5343) from RateBeer, an online beer review database. Each review entry comprised 5 numerical scores (appearance, aroma, taste, palate and overall quality) and an optional review text. The total number of reviews per reviewer was collected separately. Numerical scores were scaled and centered per rater, and mean scores were calculated per beer.

For the review texts, the language was estimated using the packages ‘langdetect’ and ‘langid’ in Python. Reviews that were classified as English by both packages were kept. Reviewers with fewer than 100 entries overall were discarded. 181,025 reviews from >6000 reviewers from >40 countries remained. Text processing was done using the ‘nltk’ package in Python. Texts were corrected for slang and misspellings; proper nouns and rare words that are relevant to the beer context were specified and kept as-is (‘Chimay’,’Lambic’, etc.). A dictionary of semantically similar sensorial terms, for example ‘floral’ and ‘flower’, was created and collapsed together into one term. Words were stemmed and lemmatized to avoid identifying words such as ‘acid’ and ‘acidity’ as separate terms. Numbers and punctuation were removed.

Sentences from up to 50 randomly chosen reviews per beer were manually categorized according to the aspect of beer they describe (appearance, aroma, taste, palate, overall quality—not to be confused with the 5 numerical scores described above) or flagged as irrelevant if they contained no useful information. If a beer contained fewer than 50 reviews, all reviews were manually classified. This labeled data set was used to train a model that classified the rest of the sentences for all beers 91 . Sentences describing taste and aroma were extracted, and term frequency–inverse document frequency (TFIDF) was implemented to calculate enrichment scores for sensorial words per beer.

The sex of the tasting subject was not considered when building our sensory database. Instead, results from different panelists were averaged, both for our trained panel (56% male, 44% female) and the RateBeer reviews (70% male, 30% female for RateBeer as a whole).

Beer price collection and processing

Beer prices were collected from the following stores: Colruyt, Delhaize, Total Wine, BeerHawk, The Belgian Beer Shop, The Belgian Shop, and Beer of Belgium. Where applicable, prices were converted to Euros and normalized per liter. Spearman correlations were calculated between these prices and mean overall appreciation scores from RateBeer and the taste panel, respectively.

Pairwise Spearman Rank correlations were calculated between all sensory properties.

Machine learning models

Predictive modeling of sensory profiles from chemical data.

Regression models were constructed to predict (a) trained panel scores for beer flavors and quality from beer chemical profiles and (b) public reviews’ appreciation scores from beer chemical profiles. Z-scores were used to represent sensory attributes in both data sets. Chemical properties with log-normal distributions (Shapiro-Wilk test, p  <  0.05 ) were log-transformed. Missing chemical measurements (0.1% of all data) were replaced with mean values per attribute. Observations from 250 beers were randomly separated into a training set (70%, 175 beers) and a test set (30%, 75 beers), stratified per beer style. Chemical measurements (p = 231) were normalized based on the training set average and standard deviation. In total, three linear regression-based models: linear regression with first-order interaction terms (LR), lasso regression with first-order interaction terms (Lasso) and partial least squares regression (PLSR); five decision tree models, Adaboost regressor (ABR), Extra Trees (ET), Gradient Boosting regressor (GBR), Random Forest (RF) and XGBoost regressor (XGBR); one support vector machine model (SVR) and one artificial neural network model (ANN) were trained. The models were implemented using the ‘scikit-learn’ package (v1.2.2) and ‘xgboost’ package (v1.7.3) in Python (v3.9.16). Models were trained, and hyperparameters optimized, using five-fold cross-validated grid search with the coefficient of determination (R 2 ) as the evaluation metric. The ANN (scikit-learn’s MLPRegressor) was optimized using Bayesian Tree-Structured Parzen Estimator optimization with the ‘Optuna’ Python package (v3.2.0). Individual models were trained per attribute, and a multi-output model was trained on all attributes simultaneously.

Model dissection

GBR was found to outperform other methods, resulting in models with the highest average R 2 values in both trained panel and public review data sets. Impurity-based rankings of the most important predictors for each predicted sensorial trait were obtained using the ‘scikit-learn’ package. To observe the relationships between these chemical properties and their predicted targets, partial dependence plots (PDP) were constructed for the six most important predictors of consumer appreciation 74 , 75 .

The ‘SHAP’ package in Python (v0.41.0) was implemented to provide an alternative ranking of predictor importance and to visualize the predictors’ effects as a function of their concentration 68 .

Validation of causal chemical properties

To validate the effects of the most important model features on predicted sensory attributes, beers were spiked with the chemical compounds identified by the models and descriptive sensory analyses were carried out according to the American Society of Brewing Chemists (ASBC) protocol 90 .

Compound spiking was done 30 min before tasting. Compounds were spiked into fresh beer bottles, that were immediately resealed and inverted three times. Fresh bottles of beer were opened for the same duration, resealed, and inverted thrice, to serve as controls. Pairs of spiked samples and controls were served simultaneously, chilled and in dark glasses as outlined in the Trained panel section above. Tasters were instructed to select the glass with the higher flavor intensity for each attribute (directional difference test 92 ) and to select the glass they prefer.

The final concentration after spiking was equal to the within-style average, after normalizing by ethanol concentration. This was done to ensure balanced flavor profiles in the final spiked beer. The same methods were applied to improve a non-alcoholic beer. Compounds were the following: ethyl acetate (Merck KGaA, W241415), ethyl hexanoate (Merck KGaA, W243906), isoamyl acetate (Merck KGaA, W205508), phenethyl acetate (Merck KGaA, W285706), ethanol (96%, Colruyt), glycerol (Merck KGaA, W252506), lactic acid (Merck KGaA, 261106).

Significant differences in preference or perceived intensity were determined by performing the two-sided binomial test on each attribute.

Reporting summary

Further information on research design is available in the  Nature Portfolio Reporting Summary linked to this article.

Data availability

The data that support the findings of this work are available in the Supplementary Data files and have been deposited to Zenodo under accession code 10653704 93 . The RateBeer scores data are under restricted access, they are not publicly available as they are property of RateBeer (ZX Ventures, USA). Access can be obtained from the authors upon reasonable request and with permission of RateBeer (ZX Ventures, USA).  Source data are provided with this paper.

Code availability

The code for training the machine learning models, analyzing the models, and generating the figures has been deposited to Zenodo under accession code 10653704 93 .

Tieman, D. et al. A chemical genetic roadmap to improved tomato flavor. Science 355 , 391–394 (2017).

Article   ADS   CAS   PubMed   Google Scholar  

Plutowska, B. & Wardencki, W. Application of gas chromatography–olfactometry (GC–O) in analysis and quality assessment of alcoholic beverages – A review. Food Chem. 107 , 449–463 (2008).

Article   CAS   Google Scholar  

Legin, A., Rudnitskaya, A., Seleznev, B. & Vlasov, Y. Electronic tongue for quality assessment of ethanol, vodka and eau-de-vie. Anal. Chim. Acta 534 , 129–135 (2005).

Loutfi, A., Coradeschi, S., Mani, G. K., Shankar, P. & Rayappan, J. B. B. Electronic noses for food quality: A review. J. Food Eng. 144 , 103–111 (2015).

Ahn, Y.-Y., Ahnert, S. E., Bagrow, J. P. & Barabási, A.-L. Flavor network and the principles of food pairing. Sci. Rep. 1 , 196 (2011).

Article   CAS   PubMed   PubMed Central   Google Scholar  

Bartoshuk, L. M. & Klee, H. J. Better fruits and vegetables through sensory analysis. Curr. Biol. 23 , R374–R378 (2013).

Article   CAS   PubMed   Google Scholar  

Piggott, J. R. Design questions in sensory and consumer science. Food Qual. Prefer. 3293 , 217–220 (1995).

Article   Google Scholar  

Kermit, M. & Lengard, V. Assessing the performance of a sensory panel-panellist monitoring and tracking. J. Chemom. 19 , 154–161 (2005).

Cook, D. J., Hollowood, T. A., Linforth, R. S. T. & Taylor, A. J. Correlating instrumental measurements of texture and flavour release with human perception. Int. J. Food Sci. Technol. 40 , 631–641 (2005).

Chinchanachokchai, S., Thontirawong, P. & Chinchanachokchai, P. A tale of two recommender systems: The moderating role of consumer expertise on artificial intelligence based product recommendations. J. Retail. Consum. Serv. 61 , 1–12 (2021).

Ross, C. F. Sensory science at the human-machine interface. Trends Food Sci. Technol. 20 , 63–72 (2009).

Chambers, E. IV & Koppel, K. Associations of volatile compounds with sensory aroma and flavor: The complex nature of flavor. Molecules 18 , 4887–4905 (2013).

Pinu, F. R. Metabolomics—The new frontier in food safety and quality research. Food Res. Int. 72 , 80–81 (2015).

Danezis, G. P., Tsagkaris, A. S., Brusic, V. & Georgiou, C. A. Food authentication: state of the art and prospects. Curr. Opin. Food Sci. 10 , 22–31 (2016).

Shepherd, G. M. Smell images and the flavour system in the human brain. Nature 444 , 316–321 (2006).

Meilgaard, M. C. Prediction of flavor differences between beers from their chemical composition. J. Agric. Food Chem. 30 , 1009–1017 (1982).

Xu, L. et al. Widespread receptor-driven modulation in peripheral olfactory coding. Science 368 , eaaz5390 (2020).

Kupferschmidt, K. Following the flavor. Science 340 , 808–809 (2013).

Billesbølle, C. B. et al. Structural basis of odorant recognition by a human odorant receptor. Nature 615 , 742–749 (2023).

Article   ADS   PubMed   PubMed Central   Google Scholar  

Smith, B. Perspective: Complexities of flavour. Nature 486 , S6–S6 (2012).

Pfister, P. et al. Odorant receptor inhibition is fundamental to odor encoding. Curr. Biol. 30 , 2574–2587 (2020).

Moskowitz, H. W., Kumaraiah, V., Sharma, K. N., Jacobs, H. L. & Sharma, S. D. Cross-cultural differences in simple taste preferences. Science 190 , 1217–1218 (1975).

Eriksson, N. et al. A genetic variant near olfactory receptor genes influences cilantro preference. Flavour 1 , 22 (2012).

Ferdenzi, C. et al. Variability of affective responses to odors: Culture, gender, and olfactory knowledge. Chem. Senses 38 , 175–186 (2013).

Article   PubMed   Google Scholar  

Lawless, H. T. & Heymann, H. Sensory evaluation of food: Principles and practices. (Springer, New York, NY). https://doi.org/10.1007/978-1-4419-6488-5 (2010).

Colantonio, V. et al. Metabolomic selection for enhanced fruit flavor. Proc. Natl. Acad. Sci. 119 , e2115865119 (2022).

Fritz, F., Preissner, R. & Banerjee, P. VirtualTaste: a web server for the prediction of organoleptic properties of chemical compounds. Nucleic Acids Res 49 , W679–W684 (2021).

Tuwani, R., Wadhwa, S. & Bagler, G. BitterSweet: Building machine learning models for predicting the bitter and sweet taste of small molecules. Sci. Rep. 9 , 1–13 (2019).

Dagan-Wiener, A. et al. Bitter or not? BitterPredict, a tool for predicting taste from chemical structure. Sci. Rep. 7 , 1–13 (2017).

Pallante, L. et al. Toward a general and interpretable umami taste predictor using a multi-objective machine learning approach. Sci. Rep. 12 , 1–11 (2022).

Malavolta, M. et al. A survey on computational taste predictors. Eur. Food Res. Technol. 248 , 2215–2235 (2022).

Lee, B. K. et al. A principal odor map unifies diverse tasks in olfactory perception. Science 381 , 999–1006 (2023).

Mayhew, E. J. et al. Transport features predict if a molecule is odorous. Proc. Natl. Acad. Sci. 119 , e2116576119 (2022).

Niu, Y. et al. Sensory evaluation of the synergism among ester odorants in light aroma-type liquor by odor threshold, aroma intensity and flash GC electronic nose. Food Res. Int. 113 , 102–114 (2018).

Yu, P., Low, M. Y. & Zhou, W. Design of experiments and regression modelling in food flavour and sensory analysis: A review. Trends Food Sci. Technol. 71 , 202–215 (2018).

Oladokun, O. et al. The impact of hop bitter acid and polyphenol profiles on the perceived bitterness of beer. Food Chem. 205 , 212–220 (2016).

Linforth, R., Cabannes, M., Hewson, L., Yang, N. & Taylor, A. Effect of fat content on flavor delivery during consumption: An in vivo model. J. Agric. Food Chem. 58 , 6905–6911 (2010).

Guo, S., Na Jom, K. & Ge, Y. Influence of roasting condition on flavor profile of sunflower seeds: A flavoromics approach. Sci. Rep. 9 , 11295 (2019).

Ren, Q. et al. The changes of microbial community and flavor compound in the fermentation process of Chinese rice wine using Fagopyrum tataricum grain as feedstock. Sci. Rep. 9 , 3365 (2019).

Hastie, T., Friedman, J. & Tibshirani, R. The Elements of Statistical Learning. (Springer, New York, NY). https://doi.org/10.1007/978-0-387-21606-5 (2001).

Dietz, C., Cook, D., Huismann, M., Wilson, C. & Ford, R. The multisensory perception of hop essential oil: a review. J. Inst. Brew. 126 , 320–342 (2020).

CAS   Google Scholar  

Roncoroni, Miguel & Verstrepen, Kevin Joan. Belgian Beer: Tested and Tasted. (Lannoo, 2018).

Meilgaard, M. Flavor chemistry of beer: Part II: Flavor and threshold of 239 aroma volatiles. in (1975).

Bokulich, N. A. & Bamforth, C. W. The microbiology of malting and brewing. Microbiol. Mol. Biol. Rev. MMBR 77 , 157–172 (2013).

Dzialo, M. C., Park, R., Steensels, J., Lievens, B. & Verstrepen, K. J. Physiology, ecology and industrial applications of aroma formation in yeast. FEMS Microbiol. Rev. 41 , S95–S128 (2017).

Article   PubMed   PubMed Central   Google Scholar  

Datta, A. et al. Computer-aided food engineering. Nat. Food 3 , 894–904 (2022).

American Society of Brewing Chemists. Beer Methods. (American Society of Brewing Chemists, St. Paul, MN, U.S.A.).

Olaniran, A. O., Hiralal, L., Mokoena, M. P. & Pillay, B. Flavour-active volatile compounds in beer: production, regulation and control. J. Inst. Brew. 123 , 13–23 (2017).

Verstrepen, K. J. et al. Flavor-active esters: Adding fruitiness to beer. J. Biosci. Bioeng. 96 , 110–118 (2003).

Meilgaard, M. C. Flavour chemistry of beer. part I: flavour interaction between principal volatiles. Master Brew. Assoc. Am. Tech. Q 12 , 107–117 (1975).

Briggs, D. E., Boulton, C. A., Brookes, P. A. & Stevens, R. Brewing 227–254. (Woodhead Publishing). https://doi.org/10.1533/9781855739062.227 (2004).

Bossaert, S., Crauwels, S., De Rouck, G. & Lievens, B. The power of sour - A review: Old traditions, new opportunities. BrewingScience 72 , 78–88 (2019).

Google Scholar  

Verstrepen, K. J. et al. Flavor active esters: Adding fruitiness to beer. J. Biosci. Bioeng. 96 , 110–118 (2003).

Snauwaert, I. et al. Microbial diversity and metabolite composition of Belgian red-brown acidic ales. Int. J. Food Microbiol. 221 , 1–11 (2016).

Spitaels, F. et al. The microbial diversity of traditional spontaneously fermented lambic beer. PLoS ONE 9 , e95384 (2014).

Blanco, C. A., Andrés-Iglesias, C. & Montero, O. Low-alcohol Beers: Flavor Compounds, Defects, and Improvement Strategies. Crit. Rev. Food Sci. Nutr. 56 , 1379–1388 (2016).

Jackowski, M. & Trusek, A. Non-Alcohol. beer Prod. – Overv. 20 , 32–38 (2018).

Takoi, K. et al. The contribution of geraniol metabolism to the citrus flavour of beer: Synergy of geraniol and β-citronellol under coexistence with excess linalool. J. Inst. Brew. 116 , 251–260 (2010).

Kroeze, J. H. & Bartoshuk, L. M. Bitterness suppression as revealed by split-tongue taste stimulation in humans. Physiol. Behav. 35 , 779–783 (1985).

Mennella, J. A. et al. A spoonful of sugar helps the medicine go down”: Bitter masking bysucrose among children and adults. Chem. Senses 40 , 17–25 (2015).

Wietstock, P., Kunz, T., Perreira, F. & Methner, F.-J. Metal chelation behavior of hop acids in buffered model systems. BrewingScience 69 , 56–63 (2016).

Sancho, D., Blanco, C. A., Caballero, I. & Pascual, A. Free iron in pale, dark and alcohol-free commercial lager beers. J. Sci. Food Agric. 91 , 1142–1147 (2011).

Rodrigues, H. & Parr, W. V. Contribution of cross-cultural studies to understanding wine appreciation: A review. Food Res. Int. 115 , 251–258 (2019).

Korneva, E. & Blockeel, H. Towards better evaluation of multi-target regression models. in ECML PKDD 2020 Workshops (eds. Koprinska, I. et al.) 353–362 (Springer International Publishing, Cham, 2020). https://doi.org/10.1007/978-3-030-65965-3_23 .

Gastón Ares. Mathematical and Statistical Methods in Food Science and Technology. (Wiley, 2013).

Grinsztajn, L., Oyallon, E. & Varoquaux, G. Why do tree-based models still outperform deep learning on tabular data? Preprint at http://arxiv.org/abs/2207.08815 (2022).

Gries, S. T. Statistics for Linguistics with R: A Practical Introduction. in Statistics for Linguistics with R (De Gruyter Mouton, 2021). https://doi.org/10.1515/9783110718256 .

Lundberg, S. M. et al. From local explanations to global understanding with explainable AI for trees. Nat. Mach. Intell. 2 , 56–67 (2020).

Ickes, C. M. & Cadwallader, K. R. Effects of ethanol on flavor perception in alcoholic beverages. Chemosens. Percept. 10 , 119–134 (2017).

Kato, M. et al. Influence of high molecular weight polypeptides on the mouthfeel of commercial beer. J. Inst. Brew. 127 , 27–40 (2021).

Wauters, R. et al. Novel Saccharomyces cerevisiae variants slow down the accumulation of staling aldehydes and improve beer shelf-life. Food Chem. 398 , 1–11 (2023).

Li, H., Jia, S. & Zhang, W. Rapid determination of low-level sulfur compounds in beer by headspace gas chromatography with a pulsed flame photometric detector. J. Am. Soc. Brew. Chem. 66 , 188–191 (2008).

Dercksen, A., Laurens, J., Torline, P., Axcell, B. C. & Rohwer, E. Quantitative analysis of volatile sulfur compounds in beer using a membrane extraction interface. J. Am. Soc. Brew. Chem. 54 , 228–233 (1996).

Molnar, C. Interpretable Machine Learning: A Guide for Making Black-Box Models Interpretable. (2020).

Zhao, Q. & Hastie, T. Causal interpretations of black-box models. J. Bus. Econ. Stat. Publ. Am. Stat. Assoc. 39 , 272–281 (2019).

Article   MathSciNet   Google Scholar  

Hastie, T., Tibshirani, R. & Friedman, J. The Elements of Statistical Learning. (Springer, 2019).

Labrado, D. et al. Identification by NMR of key compounds present in beer distillates and residual phases after dealcoholization by vacuum distillation. J. Sci. Food Agric. 100 , 3971–3978 (2020).

Lusk, L. T., Kay, S. B., Porubcan, A. & Ryder, D. S. Key olfactory cues for beer oxidation. J. Am. Soc. Brew. Chem. 70 , 257–261 (2012).

Gonzalez Viejo, C., Torrico, D. D., Dunshea, F. R. & Fuentes, S. Development of artificial neural network models to assess beer acceptability based on sensory properties using a robotic pourer: A comparative model approach to achieve an artificial intelligence system. Beverages 5 , 33 (2019).

Gonzalez Viejo, C., Fuentes, S., Torrico, D. D., Godbole, A. & Dunshea, F. R. Chemical characterization of aromas in beer and their effect on consumers liking. Food Chem. 293 , 479–485 (2019).

Gilbert, J. L. et al. Identifying breeding priorities for blueberry flavor using biochemical, sensory, and genotype by environment analyses. PLOS ONE 10 , 1–21 (2015).

Goulet, C. et al. Role of an esterase in flavor volatile variation within the tomato clade. Proc. Natl. Acad. Sci. 109 , 19009–19014 (2012).

Article   ADS   CAS   PubMed   PubMed Central   Google Scholar  

Borisov, V. et al. Deep Neural Networks and Tabular Data: A Survey. IEEE Trans. Neural Netw. Learn. Syst. 1–21 https://doi.org/10.1109/TNNLS.2022.3229161 (2022).

Statista. Statista Consumer Market Outlook: Beer - Worldwide.

Seitz, H. K. & Stickel, F. Molecular mechanisms of alcoholmediated carcinogenesis. Nat. Rev. Cancer 7 , 599–612 (2007).

Voordeckers, K. et al. Ethanol exposure increases mutation rate through error-prone polymerases. Nat. Commun. 11 , 3664 (2020).

Goelen, T. et al. Bacterial phylogeny predicts volatile organic compound composition and olfactory response of an aphid parasitoid. Oikos 129 , 1415–1428 (2020).

Article   ADS   Google Scholar  

Reher, T. et al. Evaluation of hop (Humulus lupulus) as a repellent for the management of Drosophila suzukii. Crop Prot. 124 , 104839 (2019).

Stein, S. E. An integrated method for spectrum extraction and compound identification from gas chromatography/mass spectrometry data. J. Am. Soc. Mass Spectrom. 10 , 770–781 (1999).

American Society of Brewing Chemists. Sensory Analysis Methods. (American Society of Brewing Chemists, St. Paul, MN, U.S.A., 1992).

McAuley, J., Leskovec, J. & Jurafsky, D. Learning Attitudes and Attributes from Multi-Aspect Reviews. Preprint at https://doi.org/10.48550/arXiv.1210.3926 (2012).

Meilgaard, M. C., Carr, B. T. & Carr, B. T. Sensory Evaluation Techniques. (CRC Press, Boca Raton). https://doi.org/10.1201/b16452 (2014).

Schreurs, M. et al. Data from: Predicting and improving complex beer flavor through machine learning. Zenodo https://doi.org/10.5281/zenodo.10653704 (2024).

Download references

Acknowledgements

We thank all lab members for their discussions and thank all tasting panel members for their contributions. Special thanks go out to Dr. Karin Voordeckers for her tremendous help in proofreading and improving the manuscript. M.S. was supported by a Baillet-Latour fellowship, L.C. acknowledges financial support from KU Leuven (C16/17/006), F.A.T. was supported by a PhD fellowship from FWO (1S08821N). Research in the lab of K.J.V. is supported by KU Leuven, FWO, VIB, VLAIO and the Brewing Science Serves Health Fund. Research in the lab of T.W. is supported by FWO (G.0A51.15) and KU Leuven (C16/17/006).

Author information

These authors contributed equally: Michiel Schreurs, Supinya Piampongsant, Miguel Roncoroni.

Authors and Affiliations

VIB—KU Leuven Center for Microbiology, Gaston Geenslaan 1, B-3001, Leuven, Belgium

Michiel Schreurs, Supinya Piampongsant, Miguel Roncoroni, Lloyd Cool, Beatriz Herrera-Malaver, Florian A. Theßeling & Kevin J. Verstrepen

CMPG Laboratory of Genetics and Genomics, KU Leuven, Gaston Geenslaan 1, B-3001, Leuven, Belgium

Leuven Institute for Beer Research (LIBR), Gaston Geenslaan 1, B-3001, Leuven, Belgium

Laboratory of Socioecology and Social Evolution, KU Leuven, Naamsestraat 59, B-3000, Leuven, Belgium

Lloyd Cool, Christophe Vanderaa & Tom Wenseleers

VIB Bioinformatics Core, VIB, Rijvisschestraat 120, B-9052, Ghent, Belgium

Łukasz Kreft & Alexander Botzki

AB InBev SA/NV, Brouwerijplein 1, B-3000, Leuven, Belgium

Philippe Malcorps & Luk Daenen

You can also search for this author in PubMed   Google Scholar

Contributions

S.P., M.S. and K.J.V. conceived the experiments. S.P., M.S. and K.J.V. designed the experiments. S.P., M.S., M.R., B.H. and F.A.T. performed the experiments. S.P., M.S., L.C., C.V., L.K., A.B., P.M., L.D., T.W. and K.J.V. contributed analysis ideas. S.P., M.S., L.C., C.V., T.W. and K.J.V. analyzed the data. All authors contributed to writing the manuscript.

Corresponding author

Correspondence to Kevin J. Verstrepen .

Ethics declarations

Competing interests.

K.J.V. is affiliated with bar.on. The other authors declare no competing interests.

Peer review

Peer review information.

Nature Communications thanks Florian Bauer, Andrew John Macintosh and the other, anonymous, reviewer(s) for their contribution to the peer review of this work. A peer review file is available.

Additional information

Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary information

Supplementary information, peer review file, description of additional supplementary files, supplementary data 1, supplementary data 2, supplementary data 3, supplementary data 4, supplementary data 5, supplementary data 6, supplementary data 7, reporting summary, source data, source data, rights and permissions.

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ .

Reprints and permissions

About this article

Cite this article.

Schreurs, M., Piampongsant, S., Roncoroni, M. et al. Predicting and improving complex beer flavor through machine learning. Nat Commun 15 , 2368 (2024). https://doi.org/10.1038/s41467-024-46346-0

Download citation

Received : 30 October 2023

Accepted : 21 February 2024

Published : 26 March 2024

DOI : https://doi.org/10.1038/s41467-024-46346-0

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

By submitting a comment you agree to abide by our Terms and Community Guidelines . If you find something abusive or that does not comply with our terms or guidelines please flag it as inappropriate.

Quick links

• Explore articles by subject
• Guide to authors
• Editorial policies

Sign up for the Nature Briefing: Translational Research newsletter — top stories in biotechnology, drug discovery and pharma.