research paper on blended learning

• Research article
• Open access
• Published: 06 February 2017

Blended learning effectiveness: the relationship between student characteristics, design features and outcomes

• Mugenyi Justice Kintu   ORCID: orcid.org/0000-0002-4500-1168 1 , 2 ,
• Chang Zhu 2 &
• Edmond Kagambe 1  

International Journal of Educational Technology in Higher Education volume  14 , Article number:  7 ( 2017 ) Cite this article

778k Accesses

245 Citations

36 Altmetric

Metrics details

This paper investigates the effectiveness of a blended learning environment through analyzing the relationship between student characteristics/background, design features and learning outcomes. It is aimed at determining the significant predictors of blended learning effectiveness taking student characteristics/background and design features as independent variables and learning outcomes as dependent variables. A survey was administered to 238 respondents to gather data on student characteristics/background, design features and learning outcomes. The final semester evaluation results were used as a measure for performance as an outcome. We applied the online self regulatory learning questionnaire for data on learner self regulation, the intrinsic motivation inventory for data on intrinsic motivation and other self-developed instruments for measuring the other constructs. Multiple regression analysis results showed that blended learning design features (technology quality, online tools and face-to-face support) and student characteristics (attitudes and self-regulation) predicted student satisfaction as an outcome. The results indicate that some of the student characteristics/backgrounds and design features are significant predictors for student learning outcomes in blended learning.

Introduction

The teaching and learning environment is embracing a number of innovations and some of these involve the use of technology through blended learning. This innovative pedagogical approach has been embraced rapidly though it goes through a process. The introduction of blended learning (combination of face-to-face and online teaching and learning) initiatives is part of these innovations but its uptake, especially in the developing world faces challenges for it to be an effective innovation in teaching and learning. Blended learning effectiveness has quite a number of underlying factors that pose challenges. One big challenge is about how users can successfully use the technology and ensuring participants’ commitment given the individual learner characteristics and encounters with technology (Hofmann, 2014 ). Hofmann adds that users getting into difficulties with technology may result into abandoning the learning and eventual failure of technological applications. In a report by Oxford Group ( 2013 ), some learners (16%) had negative attitudes to blended learning while 26% were concerned that learners would not complete study in blended learning. Learners are important partners in any learning process and therefore, their backgrounds and characteristics affect their ability to effectively carry on with learning and being in blended learning, the design tools to be used may impinge on the effectiveness in their learning.

This study tackles blended learning effectiveness which has been investigated in previous studies considering grades, course completion, retention and graduation rates but no studies regarding effectiveness in view of learner characteristics/background, design features and outcomes have been done in the Ugandan university context. No studies have also been done on how the characteristics of learners and design features are predictors of outcomes in the context of a planning evaluation research (Guskey, 2000 ) to establish the effectiveness of blended learning. Guskey ( 2000 ) noted that planning evaluation fits in well since it occurs before the implementation of any innovation as well as allowing planners to determine the needs, considering participant characteristics, analyzing contextual matters and gathering baseline information. This study is done in the context of a plan to undertake innovative pedagogy involving use of a learning management system (moodle) for the first time in teaching and learning in a Ugandan university. The learner characteristics/backgrounds being investigated for blended learning effectiveness include self-regulation, computer competence, workload management, social and family support, attitude to blended learning, gender and age. We investigate the blended learning design features of learner interactions, face-to-face support, learning management system tools and technology quality while the outcomes considered include satisfaction, performance, intrinsic motivation and knowledge construction. Establishing the significant predictors of outcomes in blended learning will help to inform planners of such learning environments in order to put in place necessary groundwork preparations for designing blended learning as an innovative pedagogical approach.

Kenney and Newcombe ( 2011 ) did their comparison to establish effectiveness in view of grades and found that blended learning had higher average score than the non-blended learning environment. Garrison and Kanuka ( 2004 ) examined the transformative potential of blended learning and reported an increase in course completion rates, improved retention and increased student satisfaction. Comparisons between blended learning environments have been done to establish the disparity between academic achievement, grade dispersions and gender performance differences and no significant differences were found between the groups (Demirkol & Kazu, 2014 ).

However, blended learning effectiveness may be dependent on many other factors and among them student characteristics, design features and learning outcomes. Research shows that the failure of learners to continue their online education in some cases has been due to family support or increased workload leading to learner dropout (Park & Choi, 2009 ) as well as little time for study. Additionally, it is dependent on learner interactions with instructors since failure to continue with online learning is attributed to this. In Greer, Hudson & Paugh’s study as cited in Park and Choi ( 2009 ), family and peer support for learners is important for success in online and face-to-face learning. Support is needed for learners from all areas in web-based courses and this may be from family, friends, co-workers as well as peers in class. Greer, Hudson and Paugh further noted that peer encouragement assisted new learners in computer use and applications. The authors also show that learners need time budgeting, appropriate technology tools and support from friends and family in web-based courses. Peer support is required by learners who have no or little knowledge of technology, especially computers, to help them overcome fears. Park and Choi, ( 2009 ) showed that organizational support significantly predicts learners’ stay and success in online courses because employers at times are willing to reduce learners’ workload during study as well as supervisors showing that they are interested in job-related learning for employees to advance and improve their skills.

The study by Kintu and Zhu ( 2016 ) investigated the possibility of blended learning in a Ugandan University and examined whether student characteristics (such as self-regulation, attitudes towards blended learning, computer competence) and student background (such as family support, social support and management of workload) were significant factors in learner outcomes (such as motivation, satisfaction, knowledge construction and performance). The characteristics and background factors were studied along with blended learning design features such as technology quality, learner interactions, and Moodle with its tools and resources. The findings from that study indicated that learner attitudes towards blended learning were significant factors to learner satisfaction and motivation while workload management was a significant factor to learner satisfaction and knowledge construction. Among the blended learning design features, only learner interaction was a significant factor to learner satisfaction and knowledge construction.

The focus of the present study is on examining the effectiveness of blended learning taking into consideration learner characteristics/background, blended learning design elements and learning outcomes and how the former are significant predictors of blended learning effectiveness.

Studies like that of Morris and Lim ( 2009 ) have investigated learner and instructional factors influencing learning outcomes in blended learning. They however do not deal with such variables in the contexts of blended learning design as an aspect of innovative pedagogy involving the use of technology in education. Apart from the learner variables such as gender, age, experience, study time as tackled before, this study considers social and background aspects of the learners such as family and social support, self-regulation, attitudes towards blended learning and management of workload to find out their relationship to blended learning effectiveness. Identifying the various types of learner variables with regard to their relationship to blended learning effectiveness is important in this study as we embark on innovative pedagogy with technology in teaching and learning.

Literature review

This review presents research about blended learning effectiveness from the perspective of learner characteristics/background, design features and learning outcomes. It also gives the factors that are considered to be significant for blended learning effectiveness. The selected elements are as a result of the researcher’s experiences at a Ugandan university where student learning faces challenges with regard to learner characteristics and blended learning features in adopting the use of technology in teaching and learning. We have made use of Loukis, Georgiou, and Pazalo ( 2007 ) value flow model for evaluating an e-learning and blended learning service specifically considering the effectiveness evaluation layer. This evaluates the extent of an e-learning system usage and the educational effectiveness. In addition, studies by Leidner, Jarvenpaa, Dillon and Gunawardena as cited in Selim ( 2007 ) have noted three main factors that affect e-learning and blended learning effectiveness as instructor characteristics, technology and student characteristics. Heinich, Molenda, Russell, and Smaldino ( 2001 ) showed the need for examining learner characteristics for effective instructional technology use and showed that user characteristics do impact on behavioral intention to use technology. Research has dealt with learner characteristics that contribute to learner performance outcomes. They have dealt with emotional intelligence, resilience, personality type and success in an online learning context (Berenson, Boyles, & Weaver, 2008 ). Dealing with the characteristics identified in this study will give another dimension, especially for blended learning in learning environment designs and add to specific debate on learning using technology. Lin and Vassar, ( 2009 ) indicated that learner success is dependent on ability to cope with technical difficulty as well as technical skills in computer operations and internet navigation. This justifies our approach in dealing with the design features of blended learning in this study.

Learner characteristics/background and blended learning effectiveness

Studies indicate that student characteristics such as gender play significant roles in academic achievement (Oxford Group, 2013 ), but no study examines performance of male and female as an important factor in blended learning effectiveness. It has again been noted that the success of e- and blended learning is highly dependent on experience in internet and computer applications (Picciano & Seaman, 2007 ). Rigorous discovery of such competences can finally lead to a confirmation of high possibilities of establishing blended learning. Research agrees that the success of e-learning and blended learning can largely depend on students as well as teachers gaining confidence and capability to participate in blended learning (Hadad, 2007 ). Shraim and Khlaif ( 2010 ) note in their research that 75% of students and 72% of teachers were lacking in skills to utilize ICT based learning components due to insufficient skills and experience in computer and internet applications and this may lead to failure in e-learning and blended learning. It is therefore pertinent that since the use of blended learning applies high usage of computers, computer competence is necessary (Abubakar & Adetimirin, 2015 ) to avoid failure in applying technology in education for learning effectiveness. Rovai, ( 2003 ) noted that learners’ computer literacy and time management are crucial in distance learning contexts and concluded that such factors are meaningful in online classes. This is supported by Selim ( 2007 ) that learners need to posses time management skills and computer skills necessary for effectiveness in e- learning and blended learning. Self-regulatory skills of time management lead to better performance and learners’ ability to structure the physical learning environment leads to efficiency in e-learning and blended learning environments. Learners need to seek helpful assistance from peers and teachers through chats, email and face-to-face meetings for effectiveness (Lynch & Dembo, 2004 ). Factors such as learners’ hours of employment and family responsibilities are known to impede learners’ process of learning, blended learning inclusive (Cohen, Stage, Hammack, & Marcus, 2012 ). It was also noted that a common factor in failure and learner drop-out is the time conflict which is compounded by issues of family , employment status as well as management support (Packham, Jones, Miller, & Thomas, 2004 ). A study by Thompson ( 2004 ) shows that work, family, insufficient time and study load made learners withdraw from online courses.

Learner attitudes to blended learning can result in its effectiveness and these shape behavioral intentions which usually lead to persistence in a learning environment, blended inclusive. Selim, ( 2007 ) noted that the learners’ attitude towards e-learning and blended learning are success factors for these learning environments. Learner performance by age and gender in e-learning and blended learning has been found to indicate no significant differences between male and female learners and different age groups (i.e. young, middle-aged and old above 45 years) (Coldwell, Craig, Paterson, & Mustard, 2008 ). This implies that the potential for blended learning to be effective exists and is unhampered by gender or age differences.

Blended learning design features

The design features under study here include interactions, technology with its quality, face-to-face support and learning management system tools and resources.

Research shows that absence of learner interaction causes failure and eventual drop-out in online courses (Willging & Johnson, 2009 ) and the lack of learner connectedness was noted as an internal factor leading to learner drop-out in online courses (Zielinski, 2000 ). It was also noted that learners may not continue in e- and blended learning if they are unable to make friends thereby being disconnected and developing feelings of isolation during their blended learning experiences (Willging & Johnson, 2009). Learners’ Interactions with teachers and peers can make blended learning effective as its absence makes learners withdraw (Astleitner, 2000 ). Loukis, Georgious and Pazalo (2007) noted that learners’ measuring of a system’s quality, reliability and ease of use leads to learning efficiency and can be so in blended learning. Learner success in blended learning may substantially be affected by system functionality (Pituch & Lee, 2006 ) and may lead to failure of such learning initiatives (Shrain, 2012 ). It is therefore important to examine technology quality for ensuring learning effectiveness in blended learning. Tselios, Daskalakis, and Papadopoulou ( 2011 ) investigated learner perceptions after a learning management system use and found out that the actual system use determines the usefulness among users. It is again noted that a system with poor response time cannot be taken to be useful for e-learning and blended learning especially in cases of limited bandwidth (Anderson, 2004 ). In this study, we investigate the use of Moodle and its tools as a function of potential effectiveness of blended learning.

The quality of learning management system content for learners can be a predictor of good performance in e-and blended learning environments and can lead to learner satisfaction. On the whole, poor quality technology yields no satisfaction by users and therefore the quality of technology significantly affects satisfaction (Piccoli, Ahmad, & Ives, 2001 ). Continued navigation through a learning management system increases use and is an indicator of success in blended learning (Delone & McLean, 2003 ). The efficient use of learning management system and its tools improves learning outcomes in e-learning and blended learning environments.

It is noted that learner satisfaction with a learning management system can be an antecedent factor for blended learning effectiveness. Goyal and Tambe ( 2015 ) noted that learners showed an appreciation to Moodle’s contribution in their learning. They showed positivity with it as it improved their understanding of course material (Ahmad & Al-Khanjari, 2011 ). The study by Goyal and Tambe ( 2015 ) used descriptive statistics to indicate improved learning by use of uploaded syllabus and session plans on Moodle. Improved learning is also noted through sharing study material, submitting assignments and using the calendar. Learners in the study found Moodle to be an effective educational tool.

In blended learning set ups, face-to-face experiences form part of the blend and learner positive attitudes to such sessions could mean blended learning effectiveness. A study by Marriot, Marriot, and Selwyn ( 2004 ) showed learners expressing their preference for face-to-face due to its facilitation of social interaction and communication skills acquired from classroom environment. Their preference for the online session was only in as far as it complemented the traditional face-to-face learning. Learners in a study by Osgerby ( 2013 ) had positive perceptions of blended learning but preferred face-to-face with its step-by-stem instruction. Beard, Harper and Riley ( 2004 ) shows that some learners are successful while in a personal interaction with teachers and peers thus prefer face-to-face in the blend. Beard however dealt with a comparison between online and on-campus learning while our study combines both, singling out the face-to-face part of the blend. The advantage found by Beard is all the same relevant here because learners in blended learning express attitude to both online and face-to-face for an effective blend. Researchers indicate that teacher presence in face-to-face sessions lessens psychological distance between them and the learners and leads to greater learning. This is because there are verbal aspects like giving praise, soliciting for viewpoints, humor, etc and non-verbal expressions like eye contact, facial expressions, gestures, etc which make teachers to be closer to learners psychologically (Kelley & Gorham, 2009 ).

Learner outcomes

The outcomes under scrutiny in this study include performance, motivation, satisfaction and knowledge construction. Motivation is seen here as an outcome because, much as cognitive factors such as course grades are used in measuring learning outcomes, affective factors like intrinsic motivation may also be used to indicate outcomes of learning (Kuo, Walker, Belland, & Schroder, 2013 ). Research shows that high motivation among online learners leads to persistence in their courses (Menager-Beeley, 2004 ). Sankaran and Bui ( 2001 ) indicated that less motivated learners performed poorly in knowledge tests while those with high learning motivation demonstrate high performance in academics (Green, Nelson, Martin, & Marsh, 2006 ). Lim and Kim, ( 2003 ) indicated that learner interest as a motivation factor promotes learner involvement in learning and this could lead to learning effectiveness in blended learning.

Learner satisfaction was noted as a strong factor for effectiveness of blended and online courses (Wilging & Johnson, 2009) and dissatisfaction may result from learners’ incompetence in the use of the learning management system as an effective learning tool since, as Islam ( 2014 ) puts it, users may be dissatisfied with an information system due to ease of use. A lack of prompt feedback for learners from course instructors was found to cause dissatisfaction in an online graduate course. In addition, dissatisfaction resulted from technical difficulties as well as ambiguous course instruction Hara and Kling ( 2001 ). These factors, once addressed, can lead to learner satisfaction in e-learning and blended learning and eventual effectiveness. A study by Blocker and Tucker ( 2001 ) also showed that learners had difficulties with technology and inadequate group participation by peers leading to dissatisfaction within these design features. Student-teacher interactions are known to bring satisfaction within online courses. Study results by Swan ( 2001 ) indicated that student-teacher interaction strongly related with student satisfaction and high learner-learner interaction resulted in higher levels of course satisfaction. Descriptive results by Naaj, Nachouki, and Ankit ( 2012 ) showed that learners were satisfied with technology which was a video-conferencing component of blended learning with a mean of 3.7. The same study indicated student satisfaction with instructors at a mean of 3.8. Askar and Altun, ( 2008 ) found that learners were satisfied with face-to-face sessions of the blend with t-tests and ANOVA results indicating female scores as higher than for males in the satisfaction with face-to-face environment of the blended learning.

Studies comparing blended learning with traditional face-to-face have indicated that learners perform equally well in blended learning and their performance is unaffected by the delivery method (Kwak, Menezes, & Sherwood, 2013 ). In another study, learning experience and performance are known to improve when traditional course delivery is integrated with online learning (Stacey & Gerbic, 2007 ). Such improvement as noted may be an indicator of blended learning effectiveness. Our study however, delves into improved performance but seeks to establish the potential of blended learning effectiveness by considering grades obtained in a blended learning experiment. Score 50 and above is considered a pass in this study’s setting and learners scoring this and above will be considered to have passed. This will make our conclusions about the potential of blended learning effectiveness.

Regarding knowledge construction, it has been noted that effective learning occurs where learners are actively involved (Nurmela, Palonen, Lehtinen & Hakkarainen, 2003 , cited in Zhu, 2012 ) and this may be an indicator of learning environment effectiveness. Effective blended learning would require that learners are able to initiate, discover and accomplish the processes of knowledge construction as antecedents of blended learning effectiveness. A study by Rahman, Yasin and Jusoff ( 2011 ) indicated that learners were able to use some steps to construct meaning through an online discussion process through assignments given. In the process of giving and receiving among themselves, the authors noted that learners learned by writing what they understood. From our perspective, this can be considered to be accomplishment in the knowledge construction process. Their study further shows that learners construct meaning individually from assignments and this stage is referred to as pre-construction which for our study, is an aspect of discovery in the knowledge construction process.

Predictors of blended learning effectiveness

Researchers have dealt with success factors for online learning or those for traditional face-to-face learning but little is known about factors that predict blended learning effectiveness in view of learner characteristics and blended learning design features. This part of our study seeks to establish the learner characteristics/backgrounds and design features that predict blended learning effectiveness with regard to satisfaction, outcomes, motivation and knowledge construction. Song, Singleton, Hill, and Koh ( 2004 ) examined online learning effectiveness factors and found out that time management (a self-regulatory factor) was crucial for successful online learning. Eom, Wen, and Ashill ( 2006 ) using a survey found out that interaction, among other factors, was significant for learner satisfaction. Technical problems with regard to instructional design were a challenge to online learners thus not indicating effectiveness (Song et al., 2004 ), though the authors also indicated that descriptive statistics to a tune of 75% and time management (62%) impact on success of online learning. Arbaugh ( 2000 ) and Swan ( 2001 ) indicated that high levels of learner-instructor interaction are associated with high levels of user satisfaction and learning outcomes. A study by Naaj et al. ( 2012 ) indicated that technology and learner interactions, among other factors, influenced learner satisfaction in blended learning.

Objective and research questions of the current study

The objective of the current study is to investigate the effectiveness of blended learning in view of student satisfaction, knowledge construction, performance and intrinsic motivation and how they are related to student characteristics and blended learning design features in a blended learning environment.

Research questions

What are the student characteristics and blended learning design features for an effective blended learning environment?

Which factors (among the learner characteristics and blended learning design features) predict student satisfaction, learning outcomes, intrinsic motivation and knowledge construction?

Conceptual model of the present study

The reviewed literature clearly shows learner characteristics/background and blended learning design features play a part in blended learning effectiveness and some of them are significant predictors of effectiveness. The conceptual model for our study is depicted as follows (Fig.  1 ):

Conceptual model of the current study

Research design

This research applies a quantitative design where descriptive statistics are used for the student characteristics and design features data, t-tests for the age and gender variables to determine if they are significant in blended learning effectiveness and regression for predictors of blended learning effectiveness.

This study is based on an experiment in which learners participated during their study using face-to-face sessions and an on-line session of a blended learning design. A learning management system (Moodle) was used and learner characteristics/background and blended learning design features were measured in relation to learning effectiveness. It is therefore a planning evaluation research design as noted by Guskey ( 2000 ) since the outcomes are aimed at blended learning implementation at MMU. The plan under which the various variables were tested involved face-to-face study at the beginning of a 17 week semester which was followed by online teaching and learning in the second half of the semester. The last part of the semester was for another face-to-face to review work done during the online sessions and final semester examinations. A questionnaire with items on student characteristics, design features and learning outcomes was distributed among students from three schools and one directorate of postgraduate studies.

Participants

Cluster sampling was used to select a total of 238 learners to participate in this study. Out of the whole university population of students, three schools and one directorate were used. From these, one course unit was selected from each school and all the learners following the course unit were surveyed. In the school of Education ( n  = 70) and Business and Management Studies ( n  = 133), sophomore students were involved due to the fact that they have been introduced to ICT basics during their first year of study. Students of the third year were used from the department of technology in the School of Applied Sciences and Technology ( n  = 18) since most of the year two courses had a lot of practical aspects that could not be used for the online learning part. From the Postgraduate Directorate ( n  = 17), first and second year students were selected because learners attend a face-to-face session before they are given paper modules to study away from campus.

The study population comprised of 139 male students representing 58.4% and 99 females representing 41.6% with an average age of 24 years.

Instruments

The end of semester results were used to measure learner performance. The online self-regulated learning questionnaire (Barnard, Lan, To, Paton, & Lai, 2009 ) and the intrinsic motivation inventory (Deci & Ryan, 1982 ) were applied to measure the constructs on self regulation in the student characteristics and motivation in the learning outcome constructs. Other self-developed instruments were used for the other remaining variables of attitudes, computer competence, workload management, social and family support, satisfaction, knowledge construction, technology quality, interactions, learning management system tools and resources and face-to-face support.

Instrument reliability

Cronbach’s alpha was used to test reliability and the table below gives the results. All the scales and sub-scales had acceptable internal consistency reliabilities as shown in Table  1 below:

Data analysis

First, descriptive statistics was conducted. Shapiro-Wilk test was done to test normality of the data for it to qualify for parametric tests. The test results for normality of our data before the t- test resulted into significant levels (Male = .003, female = .000) thereby violating the normality assumption. We therefore used the skewness and curtosis results which were between −1.0 and +1.0 and assumed distribution to be sufficiently normal to qualify the data for a parametric test, (Pallant, 2010 ). An independent samples t -test was done to find out the differences in male and female performance to explain the gender characteristics in blended learning effectiveness. A one-way ANOVA between subjects was conducted to establish the differences in performance between age groups. Finally, multiple regression analysis was done between student variables and design elements with learning outcomes to determine the significant predictors for blended learning effectiveness.

Student characteristics, blended learning design features and learning outcomes ( RQ1 )

A t- test was carried out to establish the performance of male and female learners in the blended learning set up. This was aimed at finding out if male and female learners do perform equally well in blended learning given their different roles and responsibilities in society. It was found that male learners performed slightly better ( M  = 62.5) than their female counterparts ( M  = 61.1). An independent t -test revealed that the difference between the performances was not statistically significant ( t  = 1.569, df = 228, p  = 0.05, one tailed). The magnitude of the differences in the means is small with effect size ( d  = 0.18). A one way between subjects ANOVA was conducted on the performance of different age groups to establish the performance of learners of young and middle aged age groups (20–30, young & and 31–39, middle aged). This revealed a significant difference in performance (F(1,236 = 8.498, p < . 001).

Average percentages of the items making up the self regulated learning scale are used to report the findings about all the sub-scales in the learner characteristics/background scale. Results show that learner self-regulation was good enough at 72.3% in all the sub-scales of goal setting, environment structuring, task strategies, time management, help-seeking and self-evaluation among learners. The least in the scoring was task strategies at 67.7% and the highest was learner environment structuring at 76.3%. Learner attitude towards blended learning environment is at 76% in the sub-scales of learner autonomy, quality of instructional materials, course structure, course interface and interactions. The least scored here is attitude to course structure at 66% and their attitudes were high on learner autonomy and course interface both at 82%. Results on the learners’ computer competences are summarized in percentages in the table below (Table  2 ):

It can be seen that learners are skilled in word processing at 91%, email at 63.5%, spreadsheets at 68%, web browsers at 70.2% and html tools at 45.4%. They are therefore good enough in word processing and web browsing. Their computer confidence levels are reported at 75.3% and specifically feel very confident when it comes to working with a computer (85.7%). Levels of family and social support for learners during blended learning experiences are at 60.5 and 75% respectively. There is however a low score on learners being assisted by family members in situations of computer setbacks (33.2%) as 53.4% of the learners reported no assistance in this regard. A higher percentage (85.3%) is reported on learners getting support from family regarding provision of essentials for learning such as tuition. A big percentage of learners spend two hours on study while at home (35.3%) followed by one hour (28.2%) while only 9.7% spend more than three hours on study at home. Peers showed great care during the blended learning experience (81%) and their experiences were appreciated by the society (66%). Workload management by learners vis-à-vis studying is good at 60%. Learners reported that their workmates stand in for them at workplaces to enable them do their study in blended learning while 61% are encouraged by their bosses to go and improve their skills through further education and training. On the time spent on other activities not related to study, majority of the learners spend three hours (35%) while 19% spend 6 hours. Sixty percent of the learners have to answer to someone when they are not attending to other activities outside study compared to the 39.9% who do not and can therefore do study or those other activities.

The usability of the online system, tools and resources was below average as shown in the table below in percentages (Table  3 ):

However, learners became skilled at navigating around the learning management system (79%) and it was easy for them to locate course content, tools and resources needed such as course works, news, discussions and journal materials. They effectively used the communication tools (60%) and to work with peers by making posts (57%). They reported that online resources were well organized, user friendly and easy to access (71%) as well as well structured in a clear and understandable manner (72%). They therefore recommended the use of online resources for other course units in future (78%) because they were satisfied with them (64.3%). On the whole, the online resources were fine for the learners (67.2%) and useful as a learning resource (80%). The learners’ perceived usefulness/satisfaction with online system, tools, and resources was at 81% as the LMS tools helped them to communicate, work with peers and reflect on their learning (74%). They reported that using moodle helped them to learn new concepts, information and gaining skills (85.3%) as well as sharing what they knew or learned (76.4%). They enjoyed the course units (78%) and improved their skills with technology (89%).

Learner interactions were seen from three angles of cognitivism, collaborative learning and student-teacher interactions. Collaborative learning was average at 50% with low percentages in learners posting challenges to colleagues’ ideas online (34%) and posting ideas for colleagues to read online (37%). They however met oftentimes online (60%) and organized how they would work together in study during the face-to-face meetings (69%). The common form of communication medium frequently used by learners during the blended learning experience was by phone (34.5%) followed by whatsapp (21.8%), face book (21%), discussion board (11.8%) and email (10.9%). At the cognitive level, learners interacted with content at 72% by reading the posted content (81%), exchanging knowledge via the LMS (58.4%), participating in discussions on the forum (62%) and got course objectives and structure introduced during the face-to-face sessions (86%). Student-teacher interaction was reported at 71% through instructors individually working with them online (57.2%) and being well guided towards learning goals (81%). They did receive suggestions from instructors about resources to use in their learning (75.3%) and instructors provided learning input for them to come up with their own answers (71%).

The technology quality during the blended learning intervention was rated at 69% with availability of 72%, quality of the resources was at 68% with learners reporting that discussion boards gave right content necessary for study (71%) and the email exchanges containing relevant and much needed information (63.4%) as well as chats comprising of essential information to aid the learning (69%). Internet reliability was rated at 66% with a speed considered averagely good to facilitate online activities (63%). They however reported that there was intermittent breakdown during online study (67%) though they could complete their internet program during connection (63.4%). Learners eventually found it easy to download necessary materials for study in their blended learning experiences (71%).

Learner extent of use of the learning management system features was as shown in the table below in percentage (Table  4 ):

From the table, very rarely used features include the blog and wiki while very often used ones include the email, forum, chat and calendar.

The effectiveness of the LMS was rated at 79% by learners reporting that they found it useful (89%) and using it makes their learning activities much easier (75.2%). Moodle has helped learners to accomplish their learning tasks more quickly (74%) and that as a LMS, it is effective in teaching and learning (88%) with overall satisfaction levels at 68%. However, learners note challenges in the use of the LMS regarding its performance as having been problematic to them (57%) and only 8% of the learners reported navigation while 16% reported access as challenges.

Learner attitudes towards Face-to-face support were reported at 88% showing that the sessions were enjoyable experiences (89%) with high quality class discussions (86%) and therefore recommended that the sessions should continue in blended learning (89%). The frequency of the face-to-face sessions is shown in the table below as preferred by learners (Table  5 ).

Learners preferred face-to-face sessions after every month in the semester (33.6%) and at the beginning of the blended learning session only (27.7%).

Learners reported high intrinsic motivation levels with interest and enjoyment of tasks at 83.7%, perceived competence at 70.2%, effort/importance sub-scale at 80%, pressure/tension reported at 54%. The pressure percentage of 54% arises from learners feeling nervous (39.2%) and a lot of anxiety (53%) while 44% felt a lot of pressure during the blended learning experiences. Learners however reported the value/usefulness of blended learning at 91% with majority believing that studying online and face-to-face had value for them (93.3%) and were therefore willing to take part in blended learning (91.2%). They showed that it is beneficial for them (94%) and that it was an important way of studying (84.3%).

Learner satisfaction was reported at 81% especially with instructors (85%) high percentage reported on encouraging learner participation during the course of study 93%, course content (83%) with the highest being satisfaction with the good relationship between the objectives of the course units and the content (90%), technology (71%) with a high percentage on the fact that the platform was adequate for the online part of the learning (76%), interactions (75%) with participation in class at 79%, and face-to-face sessions (91%) with learner satisfaction high on face-to-face sessions being good enough for interaction and giving an overview of the courses when objectives were introduced at 92%.

Learners’ knowledge construction was reported at 78% with initiation and discovery scales scoring 84% with 88% specifically for discovering the learning points in the course units. The accomplishment scale in knowledge construction scored 71% and specifically the fact that learners were able to work together with group members to accomplish learning tasks throughout the study of the course units (79%). Learners developed reports from activities (67%), submitted solutions to discussion questions (68%) and did critique peer arguments (69%). Generally, learners performed well in blended learning in the final examination with an average pass of 62% and standard deviation of 7.5.

Significant predictors of blended learning effectiveness ( RQ 2)

A standard multiple regression analysis was done taking learner characteristics/background and design features as predictor variables and learning outcomes as criterion variables. The data was first tested to check if it met the linear regression test assumptions and results showed the correlations between the independent variables and each of the dependent variables (highest 0.62 and lowest 0.22) as not being too high, which indicated that multicollinearity was not a problem in our model. From the coefficients table, the VIF values ranged from 1.0 to 2.4, well below the cut off value of 10 and indicating no possibility of multicollinearity. The normal probability plot was seen to lie as a reasonably straight diagonal from bottom left to top right indicating normality of our data. Linearity was found suitable from the scatter plot of the standardized residuals and was rectangular in distribution. Outliers were no cause for concern in our data since we had only 1% of all cases falling outside 3.0 thus proving the data as a normally distributed sample. Our R -square values was at 0.525 meaning that the independent variables explained about 53% of the variance in overall satisfaction, motivation and knowledge construction of the learners. All the models explaining the three dependent variables of learner satisfaction, intrinsic motivation and knowledge construction were significant at the 0.000 probability level (Table  6 ).

From the table above, design features (technology quality and online tools and resources), and learner characteristics (attitudes to blended learning, self-regulation) were significant predictors of learner satisfaction in blended learning. This means that good technology with the features involved and the learner positive attitudes with capacity to do blended learning with self drive led to their satisfaction. The design features (technology quality, interactions) and learner characteristics (self regulation and social support), were found to be significant predictors of learner knowledge construction. This implies that learners’ capacity to go on their work by themselves supported by peers and high levels of interaction using the quality technology led them to construct their own ideas in blended learning. Design features (technology quality, online tools and resources as well as learner interactions) and learner characteristics (self regulation), significantly predicted the learners’ intrinsic motivation in blended learning suggesting that good technology, tools and high interaction levels with independence in learning led to learners being highly motivated. Finally, none of the independent variables considered under this study were predictors of learning outcomes (grade).

In this study we have investigated learning outcomes as dependent variables to establish if particular learner characteristics/backgrounds and design features are related to the outcomes for blended learning effectiveness and if they predict learning outcomes in blended learning. We took students from three schools out of five and one directorate of post-graduate studies at a Ugandan University. The study suggests that the characteristics and design features examined are good drivers towards an effective blended learning environment though a few of them predicted learning outcomes in blended learning.

Student characteristics/background, blended learning design features and learning outcomes

The learner characteristics, design features investigated are potentially important for an effective blended learning environment. Performance by gender shows a balance with no statistical differences between male and female. There are statistically significant differences ( p  < .005) in the performance between age groups with means of 62% for age group 20–30 and 67% for age group 31 –39. The indicators of self regulation exist as well as positive attitudes towards blended learning. Learners do well with word processing, e-mail, spreadsheets and web browsers but still lag below average in html tools. They show computer confidence at 75.3%; which gives prospects for an effective blended learning environment in regard to their computer competence and confidence. The levels of family and social support for learners stand at 61 and 75% respectively, indicating potential for blended learning to be effective. The learners’ balance between study and work is a drive factor towards blended learning effectiveness since their management of their workload vis a vis study time is at 60 and 61% of the learners are encouraged to go for study by their bosses. Learner satisfaction with the online system and its tools shows prospect for blended learning effectiveness but there are challenges in regard to locating course content and assignments, submitting their work and staying on a task during online study. Average collaborative, cognitive learning as well as learner-teacher interactions exist as important factors. Technology quality for effective blended learning is a potential for effectiveness though features like the blog and wiki are rarely used by learners. Face-to-face support is satisfactory and it should be conducted every month. There is high intrinsic motivation, satisfaction and knowledge construction as well as good performance in examinations ( M  = 62%, SD = 7.5); which indicates potentiality for blended learning effectiveness.

Significant predictors of blended learning effectiveness

Among the design features, technology quality, online tools and face-to-face support are predictors of learner satisfaction while learner characteristics of self regulation and attitudes to blended learning are predictors of satisfaction. Technology quality and interactions are the only design features predicting learner knowledge construction, while social support, among the learner backgrounds, is a predictor of knowledge construction. Self regulation as a learner characteristic is a predictor of knowledge construction. Self regulation is the only learner characteristic predicting intrinsic motivation in blended learning while technology quality, online tools and interactions are the design features predicting intrinsic motivation. However, all the independent variables are not significant predictors of learning performance in blended learning.

The high computer competences and confidence is an antecedent factor for blended learning effectiveness as noted by Hadad ( 2007 ) and this study finds learners confident and competent enough for the effectiveness of blended learning. A lack in computer skills causes failure in e-learning and blended learning as noted by Shraim and Khlaif ( 2010 ). From our study findings, this is no threat for blended learning our case as noted by our results. Contrary to Cohen et al. ( 2012 ) findings that learners’ family responsibilities and hours of employment can impede their process of learning, it is not the case here since they are drivers to the blended learning process. Time conflict, as compounded by family, employment status and management support (Packham et al., 2004 ) were noted as causes of learner failure and drop out of online courses. Our results show, on the contrary, that these factors are drivers for blended learning effectiveness because learners have a good balance between work and study and are supported by bosses to study. In agreement with Selim ( 2007 ), learner positive attitudes towards e-and blended learning environments are success factors. In line with Coldwell et al. ( 2008 ), no statistically significant differences exist between age groups. We however note that Coldwel, et al dealt with young, middle-aged and old above 45 years whereas we dealt with young and middle aged only.

Learner interactions at all levels are good enough and contrary to Astleitner, ( 2000 ) that their absence makes learners withdraw, they are a drive factor here. In line with Loukis (2007) the LMS quality, reliability and ease of use lead to learning efficiency as technology quality, online tools are predictors of learner satisfaction and intrinsic motivation. Face-to-face sessions should continue on a monthly basis as noted here and is in agreement with Marriot et al. ( 2004 ) who noted learner preference for it for facilitating social interaction and communication skills. High learner intrinsic motivation leads to persistence in online courses as noted by Menager-Beeley, ( 2004 ) and is high enough in our study. This implies a possibility of an effectiveness blended learning environment. The causes of learner dissatisfaction noted by Islam ( 2014 ) such as incompetence in the use of the LMS are contrary to our results in our study, while the one noted by Hara and Kling, ( 2001 ) as resulting from technical difficulties and ambiguous course instruction are no threat from our findings. Student-teacher interaction showed a relation with satisfaction according to Swan ( 2001 ) but is not a predictor in our study. Initiating knowledge construction by learners for blended learning effectiveness is exhibited in our findings and agrees with Rahman, Yasin and Jusof ( 2011 ). Our study has not agreed with Eom et al. ( 2006 ) who found learner interactions as predictors of learner satisfaction but agrees with Naaj et al. ( 2012 ) regarding technology as a predictor of learner satisfaction.

Conclusion and recommendations

An effective blended learning environment is necessary in undertaking innovative pedagogical approaches through the use of technology in teaching and learning. An examination of learner characteristics/background, design features and learning outcomes as factors for effectiveness can help to inform the design of effective learning environments that involve face-to-face sessions and online aspects. Most of the student characteristics and blended learning design features dealt with in this study are important factors for blended learning effectiveness. None of the independent variables were identified as significant predictors of student performance. These gaps are open for further investigation in order to understand if they can be significant predictors of blended learning effectiveness in a similar or different learning setting.

In planning to design and implement blended learning, we are mindful of the implications raised by this study which is a planning evaluation research for the design and eventual implementation of blended learning. Universities should be mindful of the interplay between the learner characteristics, design features and learning outcomes which are indicators of blended learning effectiveness. From this research, learners manifest high potential to take on blended learning more especially in regard to learner self-regulation exhibited. Blended learning is meant to increase learners’ levels of knowledge construction in order to create analytical skills in them. Learner ability to assess and critically evaluate knowledge sources is hereby established in our findings. This can go a long way in producing skilled learners who can be innovative graduates enough to satisfy employment demands through creativity and innovativeness. Technology being less of a shock to students gives potential for blended learning design. Universities and other institutions of learning should continue to emphasize blended learning approaches through installation of learning management systems along with strong internet to enable effective learning through technology especially in the developing world.

Abubakar, D. & Adetimirin. (2015). Influence of computer literacy on post-graduates’ use of e-resources in Nigerian University Libraries. Library Philosophy and Practice. From http://digitalcommons.unl.edu/libphilprac/ . Retrieved 18 Aug 2015.

Ahmad, N., & Al-Khanjari, Z. (2011). Effect of Moodle on learning: An Oman perception. International Journal of Digital Information and Wireless Communications (IJDIWC), 1 (4), 746–752.

Google Scholar  

Anderson, T. (2004). Theory and Practice of Online Learning . Canada: AU Press, Athabasca University.

Arbaugh, J. B. (2000). How classroom environment and student engagement affect learning in internet-basedMBAcourses. Business Communication Quarterly, 63 (4), 9–18.

Article   Google Scholar  

Askar, P. & Altun, A. (2008). Learner satisfaction on blended learning. E-Leader Krakow , 2008.

Astleitner, H. (2000) Dropout and distance education. A review of motivational and emotional strategies to reduce dropout in web-based distance education. In Neuwe Medien in Unterricht, Aus-und Weiterbildung Waxmann Munster, New York.

Barnard, L., Lan, W. Y., To, Y. M., Paton, V. O., & Lai, S. (2009). Measuring self regulation in online and blended learning environments’. Internet and Higher Education, 12 (1), 1–6.

Beard, L. A., Harper, C., & Riley, G. (2004). Online versus on-campus instruction: student attitudes & perceptions. TechTrends, 48 (6), 29–31.

Berenson, R., Boyles, G., & Weaver, A. (2008). Emotional intelligence as a predictor for success in online learning. International Review of Research in open & Distance Learning, 9 (2), 1–16.

Blocker, J. M., & Tucker, G. (2001). Using constructivist principles in designing and integrating online collaborative interactions. In F. Fuller & R. McBride (Eds.), Distance education. Proceedings of the Society for Information Technology & Teacher Education International Conference (pp. 32–36). ERIC Document Reproduction Service No. ED 457 822.

Cohen, K. E., Stage, F. K., Hammack, F. M., & Marcus, A. (2012). Persistence of master’s students in the United States: Developing and testing of a conceptual model . USA: PhD Dissertation, New York University.

Coldwell, J., Craig, A., Paterson, T., & Mustard, J. (2008). Online students: Relationships between participation, demographics and academic performance. The Electronic Journal of e-learning, 6 (1), 19–30.

Deci, E. L., & Ryan, R. M. (1982). Intrinsic Motivation Inventory. Available from selfdeterminationtheory.org/intrinsic-motivation-inventory/ . Accessed 2 Aug 2016.

Delone, W. H., & McLean, E. R. (2003). The Delone and McLean model of information systems success: A Ten-year update. Journal of Management Information Systems, 19 (4), 9–30.

Demirkol, M., & Kazu, I. Y. (2014). Effect of blended environment model on high school students’ academic achievement. The Turkish Online Journal of Educational Technology, 13 (1), 78–87.

Eom, S., Wen, H., & Ashill, N. (2006). The determinants of students’ perceived learning outcomes and satisfaction in university online education: an empirical investigation’. Decision Sciences Journal of Innovative Education, 4 (2), 215–235.

Garrison, D. R., & Kanuka, H. (2004). Blended learning: Uncovering its transformative potential in higher education. Internet and Higher Education, 7 (2), 95–105.

Goyal, E., & Tambe, S. (2015). Effectiveness of Moodle-enabled blended learning in private Indian Business School teaching NICHE programs. The Online Journal of New Horizons in Education, 5 (2), 14–22.

Green, J., Nelson, G., Martin, A. J., & Marsh, H. (2006). The causal ordering of self-concept and academic motivation and its effect on academic achievement. International Education Journal, 7 (4), 534–546.

Guskey, T. R. (2000). Evaluating Professional Development . Thousands Oaks: Corwin Press.

Hadad, W. (2007). ICT-in-education toolkit reference handbook . InfoDev. from http://www.infodev.org/en/Publication.301.html . Retrieved 04 Aug 2015.

Hara, N. & Kling, R. (2001). Student distress in web-based distance education. Educause Quarterly. 3 (2001).

Heinich, R., Molenda, M., Russell, J. D., & Smaldino, S. E. (2001). Instructional Media and Technologies for Learning (7th ed.). Englewood Cliffs: Prentice-Hall.

Hofmann, J. (2014). Solutions to the top 10 challenges of blended learning. Top 10 challenges of blended learning. Available on cedma-europe.org .

Islam, A. K. M. N. (2014). Sources of satisfaction and dissatisfaction with a learning management system in post-adoption stage: A critical incident technique approach. Computers in Human Behaviour, 30 , 249–261.

Kelley, D. H. & Gorham, J. (2009) Effects of immediacy on recall of information. Communication Education, 37 (3), 198–207.

Kenney, J., & Newcombe, E. (2011). Adopting a blended learning approach: Challenges, encountered and lessons learned in an action research study. Journal of Asynchronous Learning Networks, 15 (1), 45–57.

Kintu, M. J., & Zhu, C. (2016). Student characteristics and learning outcomes in a blended learning environment intervention in a Ugandan University. Electronic Journal of e-Learning, 14 (3), 181–195.

Kuo, Y., Walker, A. E., Belland, B. R., & Schroder, L. E. E. (2013). A predictive study of student satisfaction in online education programs. International Review of Research in Open and Distributed Learning, 14 (1), 16–39.

Kwak, D. W., Menezes, F. M., & Sherwood, C. (2013). Assessing the impact of blended learning on student performance. Educational Technology & Society, 15 (1), 127–136.

Lim, D. H., & Kim, H. J. (2003). Motivation and learner characteristics affecting online learning and learning application. Journal of Educational Technology Systems, 31 (4), 423–439.

Lim, D. H., & Morris, M. L. (2009). Learner and instructional factors influencing learner outcomes within a blended learning environment. Educational Technology & Society, 12 (4), 282–293.

Lin, B., & Vassar, J. A. (2009). Determinants for success in online learning communities. International Journal of Web-based Communities, 5 (3), 340–350.

Loukis, E., Georgiou, S. & Pazalo, K. (2007). A value flow model for the evaluation of an e-learning service. ECIS, 2007 Proceedings, paper 175.

Lynch, R., & Dembo, M. (2004). The relationship between self regulation and online learning in a blended learning context. The International Review of Research in Open and Distributed Learning, 5 (2), 1–16.

Marriot, N., Marriot, P., & Selwyn. (2004). Accounting undergraduates’ changing use of ICT and their views on using the internet in higher education-A Research note. Accounting Education, 13 (4), 117–130.

Menager-Beeley, R. (2004). Web-based distance learning in a community college: The influence of task values on task choice, retention and commitment. (Doctoral dissertation, University of Southern California). Dissertation Abstracts International, 64 (9-A), 3191.

Naaj, M. A., Nachouki, M., & Ankit, A. (2012). Evaluating student satisfaction with blended learning in a gender-segregated environment. Journal of Information Technology Education: Research, 11 , 185–200.

Nurmela, K., Palonen, T., Lehtinen, E. & Hakkarainen, K. (2003). Developing tools for analysing CSCL process. In Wasson, B. Ludvigsen, S. & Hoppe, V. (eds), Designing for change in networked learning environments (pp 333–342). Dordrecht, The Netherlands, Kluwer.

Osgerby, J. (2013). Students’ perceptions of the introduction of a blended learning environment: An exploratory case study. Accounting Education, 22 (1), 85–99.

Oxford Group, (2013). Blended learning-current use, challenges and best practices. From http://www.kineo.com/m/0/blended-learning-report-202013.pdf . Accessed on 17 Mar 2016.

Packham, G., Jones, P., Miller, C., & Thomas, B. (2004). E-learning and retention key factors influencing student withdrawal. Education and Training, 46 (6–7), 335–342.

Pallant, J. (2010). SPSS Survival Mannual (4th ed.). Maidenhead: OUP McGraw-Hill.

Park, J.-H., & Choi, H. J. (2009). Factors influencing adult learners’ decision to drop out or persist in online learning. Educational Technology & Society, 12 (4), 207–217.

Picciano, A., & Seaman, J. (2007). K-12 online learning: A survey of U.S. school district administrators . New York, USA: Sloan-C.

Piccoli, G., Ahmad, R., & Ives, B. (2001). Web-based virtual learning environments: a research framework and a preliminary assessment of effectiveness in basic IT skill training. MIS Quarterly, 25 (4), 401–426.

Pituch, K. A., & Lee, Y. K. (2006). The influence of system characteristics on e-learning use. Computers & Education, 47 (2), 222–244.

Rahman, S. et al, (2011). Knowledge construction process in online learning. Middle East Journal of Scientific Research, 8 (2), 488–492.

Rovai, A. P. (2003). In search of higher persistence rates in distance education online programs. Computers & Education, 6 (1), 1–16.

Sankaran, S., & Bui, T. (2001). Impact of learning strategies and motivation on performance: A study in Web-based instruction. Journal of Instructional Psychology, 28 (3), 191–198.

Selim, H. M. (2007). Critical success factors for e-learning acceptance: Confirmatory factor models. Computers & Education, 49 (2), 396–413.

Shraim, K., & Khlaif, Z. N. (2010). An e-learning approach to secondary education in Palestine: opportunities and challenges. Information Technology for Development, 16 (3), 159–173.

Shrain, K. (2012). Moving towards e-learning paradigm: Readiness of higher education instructors in Palestine. International Journal on E-Learning, 11 (4), 441–463.

Song, L., Singleton, E. S., Hill, J. R., & Koh, M. H. (2004). Improving online learning: student perceptions of useful and challenging characteristics’. Internet and Higher Education, 7 (1), 59–70.

Stacey, E., & Gerbic, P. (2007). Teaching for blended learning: research perspectives from on-campus and distance students. Education and Information Technologies, 12 , 165–174.

Swan, K. (2001). Virtual interactivity: design factors affecting student satisfaction and perceived learning in asynchronous online courses. Distance Education, 22 (2), 306–331.

Article   MathSciNet   Google Scholar  

Thompson, E. (2004). Distance education drop-out: What can we do? In R. Pospisil & L. Willcoxson (Eds.), Learning Through Teaching (Proceedings of the 6th Annual Teaching Learning Forum, pp. 324–332). Perth, Australia: Murdoch University.

Tselios, N., Daskalakis, S., & Papadopoulou, M. (2011). Assessing the acceptance of a blended learning university course. Educational Technology & Society, 14 (2), 224–235.

Willging, P. A., & Johnson, S. D. (2009). Factors that influence students’ decision to drop-out of online courses. Journal of Asynchronous Learning Networks, 13 (3), 115–127.

Zhu, C. (2012). Student satisfaction, performance and knowledge construction in online collaborative learning. Educational Technology & Society, 15 (1), 127–137.

Zielinski, D. (2000). Can you keep learners online? Training, 37 (3), 64–75.

Download references

Authors’ contribution

MJK conceived the study idea, developed the conceptual framework, collected the data, analyzed it and wrote the article. CZ gave the technical advice concerning the write-up and advised on relevant corrections to be made before final submission. EK did the proof-reading of the article as well as language editing. All authors read and approved the final manuscript.

Competing interests

The authors declare that they have no competing interests.

Author information

Authors and affiliations.

Mountains of the Moon University, P.O. Box 837, Fort Portal, Uganda

Mugenyi Justice Kintu & Edmond Kagambe

Vrije Universiteit Brussel, Pleinlaan 2, Brussels, 1050, Ixelles, Belgium

Mugenyi Justice Kintu & Chang Zhu

You can also search for this author in PubMed   Google Scholar

Corresponding author

Correspondence to Mugenyi Justice Kintu .

Rights and permissions

Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License ( http://creativecommons.org/licenses/by/4.0/ ), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made.

Reprints and permissions

About this article

Cite this article.

Kintu, M.J., Zhu, C. & Kagambe, E. Blended learning effectiveness: the relationship between student characteristics, design features and outcomes. Int J Educ Technol High Educ 14 , 7 (2017). https://doi.org/10.1186/s41239-017-0043-4

Download citation

Received : 13 July 2016

Accepted : 23 November 2016

Published : 06 February 2017

DOI : https://doi.org/10.1186/s41239-017-0043-4

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Blended learning effectiveness
• Learner characteristics
• Design features
• Learning outcomes and significant predictors

Advertisement

Trends and patterns in blended learning research (1965–2022)

• Published: 03 April 2023
• Volume 28 , pages 13987–14018, ( 2023 )

Cite this article

• Betül Tonbuloğlu   ORCID: orcid.org/0000-0003-1542-2380 1 &
• İsmail Tonbuloğlu   ORCID: orcid.org/0000-0001-7059-9125 2  

5081 Accesses

7 Citations

4 Altmetric

Explore all metrics

Blended learning (BL) applications, which are defined as a combination of online and face-to-face education processes and created with the strongest aspects of various teaching approaches, have attracted increasing attention particularly in recent years with the effect of the pandemic. Although blended learning studies, which have a wide range of content and variety of applications in the literature, have been examined by content analysis in many scientific studies, bibliometric studies that provide a comprehensive review of studies on BL and reveal a general map of scientific studies are extremely limited. The purpose of this research is to conduct a systematic analysis of BL studies around the world and to reveal general research trends by bibliometric method. In the scope of the research, 4,059 publications searched in the Scopus database between the years 1965—2022 were analyzed by VOSviewer and Leximancer software; the publications were examined from aspects such as year, subject, fund, citation, journal, country, common word, etc. An analysis of the research results reveals that studies on BL have increased in number in the literature since 2006; it has been found that the fields of social sciences, computer, medicine and engineering come to the forefront in the categorization of publications by subject, and the USA, UK, China and Australia are the most cited countries. As revealed by the findings of common word analysis, the studies mostly focus on the use of technology during the pandemic, current trends in education and technology, online learning environment and learner characteristics, teaching approaches, social media, motivation and medical education. Furthermore, it is understood that the most common terms in abstracts—keywords and titles of the studies reflect the learning process, the learner, the classroom environment, the model adopted, the system designed and medical education.

Similar content being viewed by others

Blended Learning Approach in English Teaching: Effectiveness and Challenges

Impact of Blended-Learning on Higher Education and English Language

Blended Learning as an Effective Approach to English Language Teaching at the Institutions of Higher Learning—A Case Study

Explore related subjects.

• Artificial Intelligence
• Digital Education and Educational Technology

Avoid common mistakes on your manuscript.

1 Introduction

The rapid developments in technology, historical conditions, constantly changing demands of the labor market and economic developments have effects in the field of education as well as in many areas of our lives. All these conditions have effects not only on the purpose and objectives, but also on the epistemological and methodological approaches that guide the educational programs; the recent research on learning lay an emphasis on the need to combine methodologies (Castro-Rodríguez et al., 2021 ). In this context, blended learning (BL) practices stand as an innovative pedagogical option in the scientific literature (Quitián & González, 2020 ).

It is observed that BL practices have gained momentum with the pandemic process in the recent years, while maintaining their prevalence and importance in education practices by taking advantage of technological possibilities. BL has a variety of names mixed learning, blended learning, hybrid learning (Khan, 2005 ) as well as many different definitions in the international literature (Bersin Associates, 2003 ; Driscoll, 2002 ; Eastman, 2015 ; Horn & Staker, 2017 ; Maxwell, 2016 ; Medina, 2018 ; Porter et al., 2014 ). However, it is stated that the common point of these definitions is to combine the presentation environments and teaching methods in distance and face-to-face teaching (Graham et al., 2003 ) and to bring together the strengths of the two approaches (Graham, 2006 ). BL is defined as the combination of online and face-to-face education processes (Rooney, 2003 ).

A review of recent studies reveals that blended learning practices have had positive effects on learning and performance, which is supported by many examples of practice (Blended Learning Universe, 2022 , Cambridge, 2020 ; Carpe Diem, 2012; Waterloo University, 2020 ) and scientific studies (Horn & Staker, 2017 ; Kokoulina, 2019 ; McCarthy, 2016 ; Noonoo, 2012 ). Singh ( 2021 ) suggests that BL not only offers more options in the teaching process, but is also more effective. BL environments support students' individual learning through collaborative efforts in the virtual environment, and help students and teachers connect more (Omar et al., 2021 ). It is stated that, within a pedagogically well-structured BL practice, instructors (1) center teaching and education, (2) have a student-centered pedagogical belief, (3) support change, (4) are open to experimentation, (5) can share their needs and concerns, 6) can reflect on themselves critically and (7) connect technologies to learning processes (Bruggeman et al., 2021 ). Thanks to BL, students, instructors and educational institutions, as the components of the education process, are able to achieve flexibility to distinguish the strengths and limitations of face-to-face and distance education when using them, and also to achieve flexibility in terms of speed, location, space and time with station and parallel teaching designs (Bozkurt & Sharma, 2021 ). BL practices enable students to develop their digital learning skills by using various technologies (Sahara et al., 2021 ). In the study by Limaymanta et al. ( 2021 ), it is noted that the complementation of information and communication technologies with teaching methods in approaches such as flipped education, active learning, BL, e-learning strengthens the effect of the learning process. Moreover, it is mentioned that using different types of learning activities can contribute to student participation and help students reach higher and more meaningful levels (Bliuc et al., 2007 ).

In addition to all its advantages, BL practices also have some limitations. Such limitations are customized on the basis of BL models as BL practices can be configured according to different models and in different scopes in line with pedagogical purposes and possibilities. Racheva ( 2019 ) stated that the models where education is mostly carried out in traditional classrooms and tasks such as individual assignments and additional activities are carried out online have the following limitations: rising costs, availability of fewer options for personalized learning, and teacher-centered configuration of education. The limitations shown for the models in which education is mostly provided in an asynchronous environment are stated as follows: low motivation due to social isolation, high dropout rates, and limited interaction and feedback. Whereas, the models where virtual web conferences are mainly used to deliver education address limitations such as dependence on proper functioning of the technology used and the limited number of students in the virtual classroom (Racheva, 2019 ). Moreover, the lack of ICT skills and infrastructure is among the challenges most commonly faced by teachers, students and institutions (Ashraf, et al., 2022 ). In addition, the concepts of Cultural Adaptation , and Digital Divide , which refers to the difference in information technologies of students with different socioeconomic levels, are also shown as limitations of blended learning applications (Graham, 2006 ).

Since face-to-face education had to be suspended in many institutions during the pandemic, e-learning activities in the global context were experienced within the scope of emergency remote teaching, which helped steer BL practices back to the agenda with strong emphasis. It is anticipated that the trend towards BL will continue with the return to physical classroom environments after the pandemic (Singh, 2021 ). BL has a wide scope and multidimensional structure (Khan, 2005 ). There are also numerous BL models to allow users and institutions to construct BL practices in line with their physical possibilities and pedagogic purposes (Horn &Staker, 2017 ; Prescott et al., 2018 ; TeachThought Staff, 2019). It is important that each of the planning, implementation and evaluation stages of the BL process is structured in view of those different dimensions and models. A detailed analysis and evaluation of the studies in the literature on BL is needed in order to be able to plan future configurations in the field of BL and predict what research opportunities will be like. Omar et al. ( 2021 ), also pointed out the high impact of citations from other researchers in this field on the studies on BL, and noted that comprehensive studies on BL are needed. A review on the position of BL activities in the literature from past to present, addressing research trends, citation rankings and trending topics on this subject will be significantly helpful for a holistic and comprehensive view of the literature. Furthermore, it is important to conduct comprehensive analyses of the studies in the literature so that future BL studies can be planned within this context.

2 Literature review

2.1 development of bl.

Blended learning applications have been in use with flexible learning practices in the literature since 1965. In his study on the flexible teaching laboratory in biochemistry, Winzler ( 1965 ) developed a flexible and satisfactory laboratory program to provide the most meaningful and useful laboratory possible for students with different abilities and interests, and stated that the main aspect of the program is to switch to the laboratory after the didactic part of the course is completed. In their study on the results of rigid and flexible learning, Baker et al. ( 1977 ) stated that the use of a rigid coding strategy will cause difficulties in learning materials that cannot be included in this strategy, while a flexible strategy seems to be superior in the long run, though being not advantageous in the short run. Russon ( 1983 ) on the development of secretarial education laid an emphasis on the flexibility need of education so that trainees can adapt to changing office environment and learning needs of the individual can be met; it was also noted that flexible learning can satisfy the requirements of changing technology.

The need for flexible learning environments has created a new learning model by combining the strengths of face-to-face and e-learning environments. Osguthorpe and Graham ( 2003 ) noted that the distinctions between traditional face-to-face education and distance learning environments soon became blurred with the innovative use of technology since the 1990s, and that face-to-face and distance education systems were combined in blended learning. In a study performed by Mazur ( 1997 ) for peer teaching, an electronic platform was developed for students to do their reading at home. It was also aimed in that study to reinforce knowledge through cooperative learning activities and peer interaction in the classroom. Baker ( 2016 ) shared the lecture presentations with students before the class for the first time in 1995, ensuring students to work on these notes before the class time, thus reinforcing the time in the classroom with activities and introducing this method as Flipped Classroom. The model applied in these studies reflects Flipped Learning, which is currently included in the literature as one of the blended learning models (Correa, 2015 ); these practices constitute the initial examples of blended learning. There are many models that suggest implementation of blended learning by building face-to-face and e-learning environments in different ways (such as rotation model, flexible model, a la carte model, enriched virtual model) (Horn & Staker, 2017 ), and there are many sub-dimensions that should be evaluated within the scope of blended learning (Khan, 2005 ).

2.2 BL & other learning practices

Face-to-face learning and e-learning environments have their advantages and disadvantages specific to their respective contexts. The challenges in face-to-face learning approach include, for example, instructor-centered teaching environment, difficulty of having individual feedbacks, teachers facing expectations of evaluating the teaching and process within a limited time, and difficulty of individual participation by each of the students in the learning process (Gülbahar et al., 2020 ). It is more appropriate to examine the e-learning approach by separating it as first and second generation practices in the literature. Problems were observed in the first generation e-learning practices due to poor understanding of the dynamics of e-learning. Such problems include examples such as focusing rather on the presentation of physical information, presenting face-to-face teaching content over the internet, limited availability of suitable teaching options, participation and social contact for learners (Singh, 2021 ). For the second generation e-learning practices, on the other hand, it is noted that learning designs were expanded and the methods used to convey information were developed (Singh, 2021 ). On the other hand, there is emphasis on the limitations of e-learning environments on a variety of aspects such as motivation, evaluation and ease of application, and safe and ethical teaching environment provided in face-to-face learning environments (Feng et al., 2021 ). There are numerous scientific studies containing comparisons of these two learning approaches with their own distinct features (Jasperson & Miller, 2018 ; Kumar et al., 2011 ; Lucky et al., 2019 ; Souza et al., 2018 ). However, it is seen that BL practices designed as a combination of the strong aspects of these two approaches continue to exist increasingly in the academic literature.

BL practices can also be confused with technology-enhanced learning practices as they include online and face-to-face learning environments. Whereas, BL practices involve leveraging the strengths of different teaching approaches beyond the use of technology to strengthen education. Another basic difference of BL from ‘technology-enhanced learning’ is the fact that the student has control over factors such as time, place, route and speed in the online learning part of BL (Horn & Staker, 2017 ). Technology-enhanced learning practices, however, do not necessitate such control. It is further indicated that the devices used in technology-enhanced learning are aimed to support traditional teaching, whereas the devices in BL are perceived as a tool for personalization opportunities (Maxwell, 2016 ).

2.3 Bibliometric analysis

Bibliometric analysis method is a popular research method used by researchers to review the past and future growth of scientific studies (Di-Stefano et al., 2010 ) and allowing an overview of the academic literature (Van-Nunen et al., 2018 ). Many indicators such as country of publication, subject area, journal, keywords are used to analyze trends and performance in bibliometric analyses (Chen & Ho, 2015 ). Bibliometric studies are employed for a variety of purposes, such as identifying the latest developments, trend topics, research directions (Wang et al., 2014 ), general reviews, and analyses by leading researchers, institutions and countries (Bjork et al., 2014 ) in a particular area of research. Bibliometric methods allow in-depth analysis of a myriad of researches and the utmost use of visual mapping (Li & Hale, 2016 ; Wang et al., 2014 ; Zupic & Cater, 2015 ). Bibliometric studies provide an accurate and objective method to measure the contribution of a research to the advancement of knowledge (Yang et al., 2013 ).

2.4 Systematic and bibliometric analyses of BL studies

With a wide range of content and diverse applications in the literature, BL studies have been examined with the content analysis method in many scientific studies and inferences have been made on their effectiveness. Rasheed et al ( 2020 ) performed a systematic review of the challenges in the online component of blended learning. In frame of this study, they noted that students had difficulties in self-regulation and using learning technology, teachers were reluctant to integrate technology into face-to-face teaching, and educational institutions had difficulties in providing the suitable technology. Anthony et al ( 2020 ) also conducted a theoretical and systematic review of the acceptance and application of blended learning in higher education, and carried out meta-analysis of 94 BL research articles published from 2004 to 2020. Research findings revealed that BL practices consisted of activities, information, resources, evaluation and feedback for students, and technology, pedagogy, content and knowledge for lecturers. Furthermore, it was underlined that the technology acceptance model, the unified theory of acceptance and use of technology, the information system success model, and the diffusion of innovations theories are most commonly used theories in the research on the adoption of BL (Anthony et al., 2020 ). Vallée et al ( 2020 ) explored the status of blended learning in medical education compared to traditional learning through systematic review and meta-analysis, conducting a systematic review of blended learning in medical education on MEDLINE from January 1990 to July 2019. Research findings have demonstrated that blended learning consistently shows better effects on knowledge outcomes than traditional learning in health education. Upon an analysis of blended learning practices in nursing education with a scoping review, Jowsey et al ( 2020 ) discovered that blended learning makes positive effects on student achievement when delivered purposefully. They also emphasized that active engagement, learning support, family support, teacher communication and students’ self-esteem are the keys to student success for blended learning environments.

Despite all this content analysis studies regarding blended learning, bibliometric studies that provide a comprehensive review of the literature and aim to collect data on the scientific production of studies on BL worldwide are extremely limited. In their study analyzing BL research trends with the bibliometric method, Arifin et al. ( 2021 ) carried out an analysis, via Publish or Perish software, of 200 studies published in Scopus between the years 2015–2020. The study involved a research on active researchers, countries, journals and methods in this field. In their research to assess the effect of BL on higher education, Castro-Rodriguez et al. ( 2021 ) carried out a content analysis on 119 open access publications in Scopus and WoS databases. Their conclusions included the statements that BL is applied in all scientific and professional fields, the learning model has a positive effect on motivation and learning effectiveness, and promotes student autonomy. Al Mamun et al. ( 2022 ) analyzed the use of flipped learning, a blended learning model, in engineering education with the bibliometric method, and underlined the effectiveness of the model by noting that this model has recently entered the exponential growth phase in engineering education. There are also different bibliometric studies conducted to reveal trends on flipped learning (Kushairi & Ahmi, 2021 ; Limaymanta et al., 2021 ; Yang et al., 2017 ). Yet, these studies are focused solely on the development of one model of blended learning.

2.5 Research gap and study objectives

This research is aimed at conducting an analysis of BL studies around the world with bibliometric method, and revealing general research trends. In this way, it will be possible to explore the development trends of BL, the most intriguing topics and resources, and the future aspects of BL. Studies in which the research conducted with different models of BL and in different scopes are analyzed within a wide time interval are extremely limited in the literature. The research differs from other bibliometric studies in the sense that it covers applications of different BL models of different levels, and presents an analysis of all the research conducted between 1965 and 2022 as shown in the Scopus database,. In this context, it is aimed to offer a more holistic and up-to-date view of the literature.

Within the scope of the research, answers are sought to the following questions:

As for the analyzed studies on BL;

How is the distribution of the number of reviewed publications by year, subject area and sponsoring organizations?

How are journals, countries and publications ranked by citation?

What kind of a structure is revealed by common word analysis?

Bibliometric analysis technique was used in this research to determine the main research trends in the field of blended learning. Many review techniques such as narrative review, systematic review, integrative review, meta-analysis, semi-systematic review are mentioned in the literature (Snyder, 2019 ). The reason for the selection of the bibliometric analysis technique for this research is that the technique has the potential to provide richer insights by quantitatively synthesizing research topics in any discipline through citation mapping (Zupic & Cater, 2015 ).

Bibliometric analysis method is a popular research method used by researchers to review the past and future growth of scientific studies (Di-Stefano et al., 2010 ) and allowing an overview of the academic literature (Van-Nunen et al., 2018 ). Bibliometric studies are employed for a variety of purposes, such as identifying the latest developments, research directions, main topics (Wang et al., 2014 ), general reviews, and analyses by leading researchers (Bjork et al., 2014 ) in a particular area of research. Bibliometric methods, which are employed for purposes such as general reviews of developments and trending topics in a research field, as well as for analysis of leading researchers, institutions and countries, allow in-depth analysis of a myriad of researches making the utmost use of visual mapping (Li & Hale, 2016 ; Wang et al., 2014 ; Zupic & Cater, 2015 ).

The review process under this research can be formed into a scheme in Fig.  1 :

Bibliometric mapping- review methodology

3.1 Data collection process

Scopus database was used for the selection of articles on BL within the scope of this study. Scopus was selected as the database as it is one of the largest scientific literature databases in the world (Bar-Ilan, 2008 ; Burnham, 2006 ), it provides the widest coverage of citation and abstract literature (Ahmi et al., 2019 ), and it is used as a data source to describe the dynamics of science and technology in many of the bibliometric studies in the literature (Cheng et al., 2014 ; Gupta & Dhawan, 2009 , 2020 ; Köseoğlu & Bozkurt, 2018 ; Tibaná-Herrera et al., 2018 ). Logical operators and keywords were used in the search, which was structured according to the PRISMA framework as shown in Fig.  2 (Page et al., 2020 ).

Selection of the publications included in the research by PRISMA method

The search was carried out on 22.02.2022, and the word matrix specified in Table 1 was used as keywords. The mention of the keywords in the title, keyword or abstract was determined as the criterion. 21,943 publications were found as a result of the search. The records that are not open access (n = 16,200) and not in English language (n = 269) were eliminated, and the remaining records were scanned; the publications that did not have an article format (n = 1,406) were filtered and finally, 4,059 publications were included in the analysis.

3.2 Analysis of data

The analyses showing the distribution of the reviewed publications by year, subject area and sponsoring organizations were carried out on the basis of the data provided by Scopus. Bibliometric analyses were performed via the VOSviewer software (version 1.6.2; Van Eck & Waltman, 2014 ), and the bibliographic data of 4,059 publications downloaded from Scopus was analyzed by VOSViewer and Leximancer software. A wide range of distance-based, graphic-based and time-based methods and tools are employed to visualize bibliometric networks (Van Eck & Waltman, 2014 ). The main research areas in a scientific field can be identified and the relationship and exchange between these research areas can be clarified by means of mapping and clustering (Tibaná-Herrera et al., 2018 ). Leydesdorff et al. ( 2015 ) laid emphasis on the effectiveness of VOSviewer software in their journal mapping studies, particularly in terms of visualization of node labels on the map and multidimensional scaling. Leximancer, on the other hand, is a text mining program that provides an automatic form of analysis based on the properties of texts and has been increasingly used in types of research where large amounts of qualitative data are analyzed (Cretchley et al., 2010 ; Jones & Diment, 2010 ). Leximancer software, which is used to automatically analyze textual data, enables visual display of selected data in the form of mind maps, network clouds and concept synonyms by using statistics-based algorithms (Smith & Humphreys, 2006 ). Moreover, Fisk et al. ( 2012 ) reported that computer aided content analysis is an effective method for mapping a research area.

In this context, citation analyses were performed via the VOSViewer software on basis of journal, country and publication, with rankings of journals and countries listed by number of publications, and common word analysis used. Keyword analysis of the publications included in the analysis was performed on Leximancer software. The data were examined in detail before each analysis on both of the software; ‘thesaurus files’ were created and used in the analysis for combining synonymous -similar words (e.g. e-learning and online learning) and for editing journal and country names that are expressed differently (e.g. U.S.A. instead of California).

3.3 Limitations

This study is limited to open access articles in English language on BL. Another limitation is related to the sole use of Scopus database. Only the part of the studies on BL that was searched in Scopus was included in the scope of study. Due to the large size of the data set obtained upon the search performed and the absence of irrelevant studies in the preliminary reviews, it was assumed that all of the articles covered BL. Although the small number of studies in the bibliometric analysis are not expected to affect the overall result, the fact that no individual review of each study was conducted in terms of suitability can be considered as a limitation regarding the structuring of the data set. Moreover, the bibliometric analysis method used is among the limitations of the research, in the sense that it reveals the social effects and executes the analyses by considering the metadata rather than the actual data of the studies (Bornmann, 2014 ; Mishra et al., 2021 ).

4.1 Distribution of the number of reviewed publications by year, subject area and sponsoring organizations

4.1.1 distribution of the number of publications by year.

The distribution of 4,059 publications by year, which was found upon examination of the publications included in the research, is shown in Fig.  3 . As can be seen from Fig.  3 , the highest number of publications on BL were produced in 2021. It was followed by the years 2020 and 2019, respectively. Considering the date of the search (22.02.2022), it can be suggested that the first months of 2022 are also an effective time period for BL studies in terms of their ratio, although an approximately 2-month period was analyzed for the year 2022.

Distribution of publications by year

It is considered that the increase in the number of publications as of 2019 is associated with the transition to emergency remote teaching due to the Covid-19 pandemic, and that social and economic developments around the world have affected BL studies. Also, the rapid development of technology and the increase in the rate of access to technological devices are also thought to have an effect on the described situation.

4.1.2 Distribution of the number of publications by subject area

The distribution of the publications included in the research by subject area is shown in Fig.  4 . As can be seen from Fig.  4 , the highest number of publications on BL is in the field of social sciences, which is followed by computer sciences, medicine and engineering.

Distribution of publications by subject area

There are studies on BL in 27 different fields, which reveals the interdisciplinary nature of the field and the interest it attracts. The concentration of studies on the category of social sciences is largely attributable to the fact that studies in field of education are regarded within the category of social sciences. Moreover, studies concentrating on computer sciences, medicine and engineering indicate the fact that the combined use of online and face-to-face education environments is regarded relatively more important in these fields. The figure, demonstrating the usability of the BL process for other fields as well, points out the positive aspects and contributions of this learning approach.

4.1.3 Top funding sponsors

According to the analyzed research data, the top 10 organizations that sponsor the studies are presented in Fig.  5 . As can be seen, the National Science Foundation ranked at the top of the sponsorship list with its support for 69 researches, followed by the National Natural Science Foundation of China and the National Institutes of Health.

Distribution of the top funding sponsors

The mentioned top-ranking organizations are among those that focus on basic research and education in the fields of science, engineering, natural sciences and medicine, which obviously means that more funding could be found for the BL studies carried out in these fields. The fact that the top funding sponsors are based in the United States, China, Europe, Germany, Japan, Taiwan, Korea and England demonstrates that BL policies and researches are given special importance and encouragement in these countries. When the countries of funding sponsors are examined, it is seen that the sponsors other than Taiwan are among the G20 countries, while all of the countries other than China, Taiwan and South Korea are among the G7 countries. Based on this information, it is understood that developed countries tend to have a significant status within the context of sponsorship.

4.2 Citation ranking of journals, countries and publications

4.2.1 journals with highest citation and publication count.

Table 2 presents the top 10 most cited journals listed among 1,387 journals in which the articles analyzed within the scope of the research were published. The most cited journal was International Review of Research in Open and Distance Learning with 2,282 citations, which was followed by Bmc Medical Education with 2,001 citations and Australasian Journal of Educational Technology with 1,717 citations. Journal Impact Factor (JIF), which is a measure of the importance or impact of a journal, and Journal Citation Indicator and Q indices of journals, which are normalized for different fields of research and their widely varying rates of publication and citation, and which allow easy interdisciplinary interpretation and comparison, and provide a single journal-level metric that can be easily interpreted and compared across disciplines (Journal Citation Reports, 2020).

It is seen that the subject areas of these most cited journals (in parallel with the data in Fig.  4 showing the distribution of publications by subject area) are related to social sciences, medicine and engineering, which cover the field of education.

The top 10 journals with the highest number of publication among journals in which the articles in scope of the research was published are provided in Table 3 . The journal with the highest number of articles was International Journal of Emerging Technologies in Learning with 146 publications, which was followed by Bmc Medical Education with 108 publications and Education Sciences with 87 publications.

It is noted that International Journal of Emerging Technologies in Learning, Bmc Medical Education, Australasian Journal of Educational Technology and International Review of Research in Open and Distance Learning are listed not only among the most cited but also the most productive journals; the mentioned journals are seen to have both high citation counts and high publication counts. This can lead to the conclusion that studies on BL maintain their importance on educational technologies and medicine. Apparently, there is no relation between the ranking of publication count and JIF- JCI- Q rankings of the journals.

4.2.2 Ranking of countries by the number of citations and publications

Upon a review of the publications included in the research, Fig.  6 shows the ranking of the top 10 most cited countries from 138 countries, and Fig.  7 presents the country citation network map created on VOSviewer taking into account the total link strength of countries together with other countries. As can be seen, the country with the highest number of citations with its publications on BL is the U.S.A with 15,616 citations, which is followed by UK with 7,410 citations and Australia with 5,727 citations.

Ranking of countries by citation

Citation network map of countries

When the citation network map of the countries is examined, it is observed that USA, UK and China plays a central role, building up strong citation networks with each other as well as with other countries, and that, China's total link strength is lower than the USA and the UK, and located a little further from the center. This indicates that China has lower link strength with other countries.

The top 30 countries with the highest numbers of publication that are among those included in research are shown in Fig.  8 .

Ranking of countries by number of publications

The United States, UK, China, Australia, Spain, Germany and Turkey are seen to be among the top 10 countries with the highest publication and citation count. It is worthy of note that, the U.S.A is ahead by a clear margin according to the analysis of country rankings by publication count, with its number of publications almost twice that of the UK, which comes after the U.S.A. Unlike the rankings by citation count, Fig.  8 includes Indonesia, Malaysia and India in the list of top 10 countries; however, despite the publication productivity of these countries, their citation rates are seen to be low (Indonesia ranks 15th with 1,064 citations, Malaysia ranks 12th with 1,137 citations, and India ranks 23rd with 642 citations).

4.2.3 Most cited publications

Table 4 presents the top 10 most cited articles listed among 4,059 articles included in the research.

An examination of the most cited articles reveals the intensity of studies on flipped learning, medical education and systematic review-meta-analysis method. Motivation, cognitive load, instructional design, Covid-19 pandemic, student learning and perception, feedback and learning analytics are also among the topics studied. Most of the studies belong to the years 2014–2015, which can be attributed to the fact that citation rates and therefore view rates of studies from previous years are higher. The most cited studies are mainly published in Q1 and Q2 journals, which is also an expected result.

4.3 Common word analysis

When the duplicate keywords in the examined publications were analyzed with VOSViewer, it was observed that 9,154 different keywords were used. Upon the analysis of the keywords repeated at least 20 times, 52 frequently used words were found. An examination of the common word analysis map shown in Fig.  9 reveals that 6 cluster structures are formed (yellow, turquoise, red, purple, blue, green).

Common word analysis map (keyword analysis) created by VOSViewer

Technology during the pandemic

The yellow cluster appears to focus on the Covid-19 pandemic, online learning and BL. BL, covid-19 and online learning, as centrally positioned concepts with high link strength, demonstrate that studies on online and BL particularly during the pandemic are represented in this cluster. On the other hand, it is understood that the studies related to professional development, evaluation, simulation, technology-enhanced learning and dental education in this cluster are closely related to each other. It can be suggested that this cluster mainly focuses on technology use and development trends during the pandemic.

Current trends in education and technology

The turquoise cluster contains the words deep learning, learning management systems, machine learning, team-based learning and technology. It is considered that this cluster generally focuses on current trends in education and technology, placing emphasis on pedagogically up-to-date and necessary competences and skills. The concept of learning management systems is closely positioned to the terms mooc, self-directed learning, student engagement and problem-based learning in the red cluster, which shows that these terms are often studied together.

Online learning environment and learner characteristics

The red cluster, having a large structure as compared to other clusters, contains the concepts ‘flipped learning’ and ‘higher education’ in central position with high link strength, which demonstrates that studies related to flipped learning experiences in higher education are predominant in this cluster. An examination of other commonly used keywords reveals that topics such as academic achievement, active learning, assessment, collaboration, collaborative learning, engagement, information literacy, instructional design, learning analytics, moocs, problem-based learning, self-directed and self-regulated learning are studied. In the light of this information, it can be said that this cluster focuses on the characteristics of online learning environment and of learners. Skills such as collaboration, active learning, information literacy, engagement, self-management and self-regulation are frequently studies topics under the category of learner characteristics, which ids noteworthy as it demonstrates the increasing importance of the mentioned skills. The concepts of instructional design and learning analytics also demonstrate that studies on the design and evaluation of learning environments are frequently carried out. Given the close relation of the terms flipped learning, BL, and online learning, it can be said that these concepts are often studied together.

Teaching approaches and social media

The purple cluster is seen to cover the terms remote teaching, flexible learning, mobile learning, pedagogy, social media and student engagement. In this frame, it can be suggested that there is focus on the relationship of different teaching approaches and social media with pedagogy in this cluster. Particularly the fact that the terms ‘student engagement’ and ‘pedagogy’ are intertwined with the red cluster, shows that the terms are also often studied together with the terms in the other cluster.

Educational technologies and motivation

The blue cluster is seen to contain the terms academic performance, motivation, educational innovation, educational technology, engineering education, gamification, information technologies and moodle. It is seen that the cluster generally focuses on subjects related to educational technologies, and that success and motivation elements are among the subjects studied together.

Medical education

The green cluster contains the concepts of curriculum, education, effectiveness, learning, medical education, medical students, online, students, teaching, and self-efficacy. Apparently, this cluster covers subjects related to education and health sciences education in general.

4.4 Text analysis

The titles, keywords and abstracts of the articles included in the study were subjected to text analysis on Leximancer software; the concept map created upon the analysis is given in Fig.  10 . Abstracts, titles and keywords of the articles were specifically chosen for the analysis since those parts have a high textual density and reflect the basic concepts. Leximancer allows display of basic concepts in the analyzed text and creation of concept maps showing the relationship between the concepts based on the frequency of co-occurrence of the words, and defines thematic regions shown with colored circles according to the most prominent concept in the region, depending on the relationship between the concepts (Zawacki-Richter et al., 2018 ).

Thematic concept map created with Leximancer, providing lexical analysis of keywords, titles and abstracts

As can be seen from Fig.  10 , the following eight themes were obtained from the analysis of the abstracts-keywords and titles of the analyzed studies: Student, Classroom, Model, Network, System, Learning, Education and Medical. As we find out from these themes, educational studies are often discussed in contexts such as Covid- management-development-support- digitalization—quality and e-learning, BL studies are included in the field of health, and BL is a model that places the student, learning and education at the center. Furthermore, this thematic concept map demonstrates that the BL system can be constructed with flexible or complex frameworks, and that the classroom structure is addressed in BL studies within different contexts such as performance, time and strategy.

The themes in the thematic concept maps created by Leximancer are ranked by level of importance in Fig.  11 , and the concepts that are prominent in terms of count and relevance are presented in Fig.  12 .

Levels of importance of the themes in the thematic concept map created by Leximancer

Major concepts in the thematic concept map created by Leximancer

When the findings presented in Figs.  11 and 12 are examined, it is seen that the concepts of Learning, Students, Classroom and Education stand out as the major concepts that also name the themes. This situation can be considered to indicate that mainly the pedagogical element is in the foreground in blended learning studies. The word ‘medical’ is among the most frequently repeated words, which can be interpreted to mean that blended learning is given special importance in health education and the Covid-19 process has a great impact on blended learning studies. When Fig.  12 is examined, it is seen that the model, methods, approach, design and system elements of blended learning are emphasized. This indicates that it is regarded important to perform planning of blended learning practices in the literature with a particular approach and ensure their systematic structuring, and that there is emphasis, with a systematic approach, on the addressing of this learning style as a separate model. It can be said that the concepts such as group, research, development, skills and performance in Fig.  12 point to the development and collaborative aspect of BL.

4.5 Comparative analysis of common words and text analysis

As a result of the comparative analysis of the findings obtained from the common word analysis and text analysis, it was seen that the findings overlapped with each other, which is indicated in Fig.  13 .

Comparative analysis of common word and text analysis

It can be seen from Fig.  13 that the scope of the analyzed BL studies is closely associated with elements related to education and technology, teaching approaches and the field of healthcare in general. This supports Fig.  4 , which states that publications on blended learning focus on social sciences, computer sciences, and health sciences.

5 Discussion

A review of researches conducted on BL since 1965 shows that the field has gained more presence in the literature since 2006. The increase in presence of such research continued at the same pace in the following years; a rapid increase was noted in the number of studies, particularly in the period between 2019–2021. In their research that examined studies on BL over 801 publications dated between 1997–2021, Omar et al ( 2021 ) similarly reported that the literature on BL increased rapidly after 2004, particularly with a significantly high increase observed in the number of studies on BL between 2018–2020. It is reported in another bibliometric study that the annual number and speed of publications of researches on BL increased in the period from 2000 to 2018 (Raman et al., 2021 ). In parallel to these findings, the study by Yang et al. ( 2017 ) in which they examined the researches on flipped classroom between the years 2000–2015, reveals that there was a significant increase in the 2011–2015 period. The findings of this research is supported by the finding obtained by all those researches that BL has been receiving increasingly more attention and studied more. As stated in Graham’s ( 2006 ) study, the prediction that blended learning will turn into a more dominant structure in the future and develop in a holistic way with face-to-face learning seems to be in parallel with the findings of this study in a historical sense. It is considered that the rising interest in studies on BL with the pandemic has a significant impact on the rapid increase in the number of those studies within the last three years.

Social sciences, computer sciences, medicine and engineering are the major fields in classification of publications on BL by subject. The findings of this study match with the most commonly studied top four fields regarding BL as demonstrated in the study by Omar et al. ( 2021 ). In a study on flipped classroom research, the major fields were listed as education, chemistry and medicine (Yang et al., 2017 ). Another study in which the research on e-learning is analyzed reveals that the predominant disciplines are social sciences, computer sciences and health sciences (Tibaná-Herrera et al., 2018 ). The prominence of social sciences can be associated with the fact that BL is a learning and teaching approach, and the prominence of computer sciences can be associated with its connection with technological processes. Most of the publications on open education practices (AEP) are in fields of social sciences (55.2%) and computer sciences (29.9%), which also supports this anticipation (Köseoğlu & Bozkurt, 2018 ). Whereas, the fact that medicine and engineering are the major fields in studies on BL may be due to the pandemic, cost of education and productivity. Furthermore, the fact that particularly applied trainings in medicine and engineering science are linked to BL processes increase the effect of BL publications (Nijakowski et al., 2021 ). The results of research conducted on BL in the field of health sciences also revealed positive differences compared to conventional teaching, which significantly contributes to the number of researches on BL in the field of health sciences (Arifin et al., 2021 ; Cheng et al., 2019 ; Sweileh, 2021 ). Omar et al. ( 2021 ) also pointed out the high number of studies on BL in the field of medicine, noting that such studies would be very beneficial if they focused on the competencies of medical graduates in the current digital health environment. It is stated that, apart from the major disciplines, there is a significant need for studies on BL in other disciplines (Siqueira & Alfinito, 2014 ). The high attention that the studies on BL attracted in practices for different disciplines can be considered as a sign of the anticipation that this field will remain increasingly important in the future.

The top 10 institutions that sponsor publications on BL are located in countries such as the USA, China, Europe, Germany, Japan, Taiwan, Korea and the UK. The UK and the USA are ranked among the top in terms of sponsorship in other studies on BL, which supports the findings of this study (Omar et al., 2021 ; Raman et al., 2021 ). It is stated by Yang et al. ( 2017 ) that the USA is the top funding sponsor in researches conducted on FC practices. There are also different studies in which the countries of funding sponsors in the field of educational technologies are listed as the UK, the USA, Australia, Taiwan, Netherlands and Canada (Bond et al., 2019 ; Chen et al., 2020 ). While the USA, China and Taiwan are the top ranking countries listed in the results of another bibliometric study on CALL (Goksu et al., 2020 ), there is another study on e-learning in which the USA, Spain, England and China are listed as the most productive countries (Gao et al.., 2022 ). In the view of these findings in the literature, it is observed that the research findings are in parallel with the data obtained by other studies within the context of countries listed as the top funding sponsors. It is considered that listing of China among the major countries in terms of findings is attributable to the effects of the Covid-19 pandemic on the researches conducted on BL as well as to the fact that the pandemic originated from China.

When the journals in which BL researches are published are evaluated in terms of the number of citations and productivity, it is seen that the International Journal Of Emerging Technologies in Learning, International Review Of Research in Open And Distance Learning, Bmc Medical Education and Australasian Journal of Educational Technology are ranked among the top journals in terms of both citation and publication counts. The finding that shows the International Journal Of Emerging Technologies in Learning as the most productive journal in which researches conducted on e-learning between the years 1998–2020 are published (Gao et al., 2022 ) is consistent with the complementary aspects of BL and e-learning processes. A bibliometric analysis of 12.272 publications dated between 2015 and 2020 was performed in another study on e-learning. Similarly, Computers & Education and International Journal of Emerging Technologies in Learning were listed among the most influential journals. The distribution of citation count in this study does not match with the ranking of the journals with high JIF, JCI and Q values indicating the importance of journals, which is suggestive of the fact that some of the publications in the journals stand out in terms of citation count.

It is seen that the USA, UK, China and Australia are the most cited countries in publications on BL. When the citation network map of the countries is examined, it is observed that the USA, UK and China plays a central role, building up strong citation networks with each other as well as with other countries, and that, China's total link strength is lower than that of the USA and UK, and located a little further from the center. In their study analyzing the studies on BL between the years 2012–2021, Ashraf et al. ( 2022 ) noted that most of the studies concentrated on developed countries; they stated that China, the USA and Australia have the highest number of publications. Gao et al. ( 2022 ) also indicated that the countries with the highest publication count for studies on e-learning are concentrated in Europe and the USA. It is indicated that there is need for integrated cooperation between countries to facilitate the adoption of BL in developing countries (Ashraf et al., 2022 ). The fact that some regions of Asia and Africa are less included in the studies on BL can be associated with the investment in digital learning environments. Yao et al ( 2021 ) point out an inequality between these regions and others within the context of digital learning environments. In their examination of the regional cost requirements around the world from 2021 to 2030 in an effort to eliminate such inequality, they found out that there was a higher cost requirement in South Asia (USD 14.4 billion), East Asia and Pacific (USD 9.5 billion), and Eastern and Southern Africa (USD 6,9 billion) as compared to other regions. Providing regions with equal opportunities in terms of digital learning can also have an effect on the number of studies on BL practices in other regions.

When BL studies are examined in terms of content, it is seen that the topics on the learning process and assessment have a key role in these studies. It is understood that the most common terms in abstracts—keywords and titles of the studies reflect the learning process, the learner, the classroom environment, the model adopted and the system designed. The concepts of assessment and learning analytics are among the most commonly used keywords. Feedback and learning analytics, on the other hand, are among the most cited article topics. The eight general dimensions by which BL is defined by Khan (2005) are seen to have features that match the themes created under the research within “pedagogical, technological, evaluation, management and institutional” contexts. In parallel with the research findings, there are various studies in which it is stated that e-learning, feedback, assessment and learning analytics are frequently addressed in the studies on BL (Chen et al., 2021 ) and the need for these areas is noted in respect of digital learning applications (Yao et al., 2021 ). It is worthy of note that ethics, interface design and resource support dimensions in Khan's BL framework are not among the major topics of this research. Due to the presence of face-to-face education environments as well as online environments in the studies on BL, this may be due to the fact that ethical concerns and issues related to interface design are not highlighted. Moreover, the dimension of resource support in BL may be less addressed in the studies due to the adoption of a BL model by the institution according to its own budget.

It is seen that pedagogical elements are frequently addressed in the research on BL. Teaching approaches, student participation, educational technologies and motivation element are among the major topics in the studies on BL. These elements were also addressed by Bruggeman et al. ( 2021 ) within the context of teacher’s attitude towards BL, teacher-centered pedagogical approach and technology acceptance, and they were interpreted as the factors preventing educators from adopting the BL process. The study by Vaughan ( 2020 ) discusses research trends for BL in higher education from student, faculty, and administrative perspectives. While student perspective focuses on the concepts of learning, process, engagement, and skill, lecturer perspective focuses on course design, scholarship of teaching, and professional development dimensions. Administrative perspective, on the other hand, focuses on topics such as savings, alignment with the institutional vision and mission, and focusing on the entire curriculum rather than the classes. Vaughan ( 2020 ) addresses research trends on BL from student, faculty, and administrative perspectives, which supports the themes of student, classroom, education, system, and model obtained in the text analysis of this study. While listing the costs that low- and lower-middle-income countries should allocate for 'digital solutions' with 'engagement', Yao et al. ( 2021 ) allocate the highest share under that title to teacher skills improvement, content, and engagement, which similarly highlights the importance of pedagogical elements in BL practices.

The emphasis on the online dimension of the educational process in the studies on BL is also worthy of note. It was seen that the studies examined in this research underlined the use of technology during the pandemic, current trends in education and technology, and the online learning environment, and that ‘the characteristics of online learning environment and learners’ emerged as a separate cluster structure. It is understood that the most cited articles in the publications are related to topics such as flipped learning, motivation, cognitive load, instructional design, and Covid-19. Similarly, it is emphasized that the keywords e-learning, online learning, collaborative learning and flipped classroom stand out in another study on BL (Omar et al., 2021 ). The study carried out by Chen et al. ( 2020 ) on educational technologies, in which they examined the last fifty years of research, listed technology-enhanced classroom pedagogy, BL, and online social communities as the subject areas among those mainly addressed, which are consistent with the findings of this research. “Online integration, personalization, online interaction and data practices”, which are referred to as the four key components of BL in another study by Graham et al. ( 2019 ), are consistent with the theme “the characteristics of the online learning environment and learners” as mentioned in this study. Accordingly, it can be deduced that the proper construction of online learning environments is of utmost importance for the success of BL practices. At the same time, it should be noted that the themes of student, classroom, education, system and model, which stand out in the text analysis findings of this study, are the main elements that underscore the face-to-face learning process of BL. The importance of face-to-face learning environments in BL to support higher-order processing, social interaction and engagement (Buhl-Wiggers et al., 2023 ) should also not be underestimated.

6 Conclusion

BL provides flexibility for students, instructors and educational institutions in sequential and simultaneous planning of design, configuration of time, space, speed and route. With this flexibility, the strengths of physical and online learning can be merged and its limitations can be minimized. Transforming the learning process into an efficient and effective form can be possible with the realization of a good pedagogical and technological structuring process as well as flexibility. A technologically and pedagogically well-structured BL process will make significant contributions to the reduction of inequalities in education (Bozkurt & Sharma, 2021 ). The purpose of this research is to examine the literature on BL and to identify general trends.

Considering the technological developments and innovative teaching approaches in recent years, there is a need for an up-to-date extensive evaluation of the status of BL, which has the mentioned and similar contributions to educational environments, in the literature. The results of this study, which we designed on the basis of this need, provide a holistic insight of the literature on BL, manifesting the direction of development of this field. This research shows research trends for BL, presents the thematic analysis of the most common topics of research, and allows evaluation of the studies carried out with a holistic perspective. Research findings demonstrate that BL practices have an increasing research potential, most of the research on BL is concentrated on developed countries, various applications of BL are addressed in different scientific fields, and pedagogy and technology are among the key elements in BL practices. Besides, considerations of relatively new approaches to the learning process, assessment, and learning analytics are seen to be significant in the research on BL. It is anticipated that revealing the current trend of BL practices will contribute to the identification of their future research potential.

Increasing the number of bibliometric studies in which publications in different databases will be included with a view to assess the effectiveness of the research on BL and to strengthen its position in the literature, is expected to be enriching for the field. Furthermore, it is recommended that the studies on BL in the literature be examined in different contexts such as IT leadership, privacy and cyber security, teacher competencies, and with different methods such as learning analytics, meta-analysis, social network analysis, and educational data mining.

6.1 Implications for practice or policy

Researchers aiming to implement the blended learning process effectively need to be aware of the development trend of the blended learning literature and to know about the topics that the research mainly focuses on.

Instructors are expected to have an idea about the changes in practices addressed in blended learning research, their role, impact and use on learning.

Principals of educational institutions should have an understanding of the scope of information presented by the literature regarding the scope, model, system and features of blended learning practices, and develop supportive measures at their respective institutions.

Ahmi, A., Elbardan, H., & Raja Mohd Ali, R. H. (2019). Bibliometric Analysis of Published Literature on Industry 4.0. In 2019 International Conference on Electronics, Information, and Communication (ICEIC) (pp. 1–6). IEEE. https://doi.org/10.23919/ELINFOCOM.2019.8706445

Al Mamun, M. A., Azad, M. A. K., & Boyle, M. (2022). Review of flipped learning in engineering education: Scientific mapping and research horizon. Education and Information Technologies, 27 (1), 1261–1286. https://doi.org/10.1007/s10639-021-10630-z

Article   Google Scholar  

Anthony, B., Kamaludin, A., Romli, A., Raffei, A. F. M., Phon, D. N. A., Abdullah, A., & Ming, G. L. (2020). Blended learning adoption and implementation in higher education: A theoretical and systematic review. Technology, Knowledge and Learning, 1–48. https://doi.org/10.1007/s10758-020-09477-z

Arifin, M. Z., Jalal, F., & Makmuri (2021). Bibliometric Analysis and Visualization of Blended Learning Research Trends with PoP and VOS Viewer. Turkish Journal of Computer and Mathematics Education (TURCOMAT), 12 (11). https://doi.org/10.17762/turcomat.v12i2.673

Ashraf, M. A., Yang, M., Zhang, Y., Denden, M., Tlili, A., Liu, J., Huang, R., & Burgos, D. (2022). A systematic review of systematic reviews on blended learning: Trends, gaps and future directions. Psychology Research and Behavior Management, 1525–1541. https://doi.org/10.2147/PRBM.S331741

Baker, L., Santa, J. L., & Gentry, J. M. (1977). Consequences of rigid and flexible learning. Bulletin of the Psychonomic Society, 9 , 58–60. https://doi.org/10.3758/BF03336929

Baker, J. W. (2016). The origins of “the classroom flip.”. In Proceedings of the 1st Annual Higher Education Flipped Learning Conference, Greeley, Colorado. https://digscholarship.unco.edu/cgi/viewcontent.cgi?article=1058&context=heflc#page=21 . Accessed 5 May 2022

Bar-Ilan, J. (2008). Which h-index? A comparison of WoS, Scopus and Google Scholar. Scientometrics, 74 (2), 257–271. https://doi.org/10.1007/s11192-008-0216-y

Bersin & Associates. (2003). Blended learning: What works?: An industry study of the strategy, implementation, and impact of blended learning . Bersin & Associates.

Bjork, S., Offer, A., & Söderberg, G. (2014). Time series citation data: The Nobel Prize in economics. Scientometrics, 98 (1), 185–196. https://doi.org/10.1007/s11192-013-0989-5

Blended Learning Universe - BLU (2022). www.blendedlearning.org . Accessed 1 May 2022

Bliuc, A. M., Goodyear, P., & Ellis, R. A. (2007). Research focus and methodological choices in studies into students’ experiences of blended learning in higher education. The Internet and Higher Education, 10 (4), 231–244. https://doi.org/10.1016/j.iheduc.2007.08.001

Bond, M., Zawacki-Richter, O., & Nichols, M. (2019). Revisiting five decades of educational technology research: A content and authorship analysis of the British Journal of Educational Technology. British Journal of Educational Technology, 50 , 12–63. https://doi.org/10.1111/bjet.12730

Bornmann, L. (2014). Do altmetrics point to the broader impact of research? An overview of benefits and disadvantages of altmetrics. Journal of Informetrics, 8 (4), 895–903. https://doi.org/10.1016/j.joi.2014.09.005

Bozkurt, A., & Sharma, R. C. (2021). In Pursuit of the Right Mix: Blended Learning for Augmenting, Enhancing, and Enriching Flexibility. Asian Journal of Distance Education , 16( 2). http://www.asianjde.com/ojs/index.php/AsianJDE/article/view/609

Bruggeman, B., Tondeur, J., Struyven, K., Pynoo, B., Garone, A., & Vanslambrouck, S. (2021). Experts speaking: Crucial teacher attributes for implementing blended learning in higher education. The Internet and Higher Education, 48 , 100772. https://doi.org/10.1016/j.iheduc.2020.100772

Buhl-Wiggers, J., Kjærgaard, A., & Munk, K. (2023). A scoping review of experimental evidence on face-to-face components of blended learning in higher education. Studies in Higher Education, 48 (1), 151–173. https://doi.org/10.1080/03075079.2022.2123911

Burnham, J. F. (2006). Scopus database: A review. Biomedical Digital Libraries, 3 (1), 1. https://doi.org/10.1186/1742-5581-3-1

Cambridge, (2020). Shaping up education models for the future in the UAE. https://www.cambridgeassessment.org.uk/news/shaping-up-education-models-for-the-future-in-the-uae/ . Accessed 10 June 2022.

Castro-Rodríguez, M., Marín-Suelves, D., López-Gómez, S., & Rodríguez-Rodríguez, J. (2021). Mapping of Scientific Production on Blended Learning in Higher Education. Education Sciences, 11 (9), 494. https://doi.org/10.3390/educsci11090494

Chen, H., & Ho, Y. S. (2015). Highly cited articles in biomass research: A bibliometric analysis. Renewable and Sustainable Energy Reviews, 49 , 12–20. https://doi.org/10.1016/j.rser.2015.04.060

Chen, X., Zou, D., & Xie, H. (2020). Fifty years of British Journal of Educational Technology: A topic modeling based bibliometric perspective. British Journal of Educational Technology, 51 (3), 692–708. https://doi.org/10.1111/bjet.12907

Chen, X., Zou, D., Xie, H., & Wang, F. L. (2021). Past, present, and future of smart learning: A topic-based bibliometric analysis. International Journal of Educational Technology in Higher Education, 18 (1), 1–29. https://doi.org/10.1186/s41239-020-00239-6

Cheng, B., Wang, M., Mørch, A. I., Chen, N. S., Kinshuk, & Michael, J. S. (2014). Research on e-learning in the workplace 2000–2012: A bibliometric analysis of the literature. The Journal of Educational Research Review., 11 , 56–72. https://doi.org/10.1016/j.edurev.2014.01.001

Cheng, L., Ritzhaupt, A. D., & Antonenko, P. (2019). Effects of the flipped classroom instructional strategy on students’ learning outcomes: A meta-analysis. Educational Technology Research and Development, 67 (4), 793–824. https://doi.org/10.1007/s11423-018-9633-7

Correa, M. (2015). Flipping the foreign language classroom and critical pedagogies a (new) old trend. Higher Education for the Future, 2 (2), 114–125. https://doi.org/10.1177/2347631115584122

Cretchley, J., Gallois, C., Chenery, H., & Smith, A. (2010). Conversations between Carers and People with Schizophrenia: A Qualitative Analysis Using Leximancer. Qualitative Health Research, 20 (12), 1611–1628. https://doi.org/10.1177/1049732310378297

Carpe Diem: Seize the Digital Revolution. (2012). https://homeworksforstudents.com/carpe_diem_seize_the_digital_revolution/

Di-Stefano, G., Peteraf, M., & Verona, G. (2010). Dynamic capabilities deconstructed: A bibliographic investigation into the origins, development, and future directions of the research domain. Industrial and Corporate Change, 19 (4), 1187–1204. https://doi.org/10.1093/icc/dtq027

Driscoll, M. (2002). Blended Learning: Let’s get beyond the hype. elearning, 54. https://bit.ly/3kkSQJR

Eastman, P. (2015). Blended Learning Design Guideline . Office of the State Superintendent of Education Charter School Incubator Initiative.

Google Scholar  

Feng, W., Wang, W., & Wang, F. (2021). Security and Privacy in E-learning. In International Conference on Web-Based Learning (pp. 129–133). Springer, Cham. https://doi.org/10.1007/978-3-030-90785-3_11

Fisk, K., Cherney, A., Hornsey, M., & Smith, A. (2012). Using computer -aided content analysis to map a research domain: A case study of institutional legitimacy in postconflict east timor . SAGE Open, 2(4). https://doi.org/10.1177/2158244012467788

Gao, Y., Wong, S. L., Khambari, M. N. M., & Noordin, N. (2022). A Bibliometric Analysis of the Scientific Production of e-Learning in Higher Education (1998–2020). International Journal of Information and Education Technology, 12 (5). https://doi.org/10.18178/ijiet.2022.12.5.1632

Goksu, I., Ozkaya, E., & Gunduz, A. (2020). The content analysis and bibliometric mapping of CALL journal. Computer Assisted Language Learning , 1–31. https://doi.org/10.1080/09588221.2020.1857409

Graham, C. R., Allen, S., & Ure, D. (2003). Blended learning environments: A review of the research literature. Unpublished manuscript, Provo, UT.

Graham, C. R., Borup, J., Short, C. R., & Archambault, L. (2019). K-12 Blended Teaching: A Guide to Personalized Learning and Online Integration. Ed Tech Books. Teacher Edition Version 1.0. https://drive.google.com/file/d/1P2nftugd8ALWddD50tsXxrpZpkDvucT/view

Graham, C. R. (2006). Blended learning systems: Definition, current trends, and future directions In CJ Bonk & CR Graham (Eds.), The handbook of blended learning: Global perspectives, local designs (pp. 3–21). San Francisco, CA: Pfeiffer. http://www.publicationshare.com/graham_intro.pdf

Gülbahar, Y., Kalelioğlu, F., & Afacan Adanır, G. (2020). Harmanlanmış Öğrenme [Blended Learning] . Pegem Akademi Publishing.

Book   Google Scholar  

Gupta, B. M., & Dhawan, S. M. (2020). A Scientometric Assessment of Global Publications in E-Learning Research during 2003–18. DESIDOC Journal of Library & Information Technology, 40 (6). https://doi.org/10.14429/djlit.40.06.15565

Gupta, B. M., & Dhawan, S. M. (2009). Status of India in science and technology as reflected in its publication output in the Scopus international database, 1996–2006. Scientometrics, 80 (2), 473–490. https://doi.org/10.1007/s11192-008-2083-y

Horn, M. B., & Staker, H. (2017). Blended: Using disruptive innovation to improve schools. John Wiley & Sons. https://books.google.com.tr/books?hl=tr&lr=&id=tKdyCwAAQBAJ&oi=fnd&pg=RA1-PA3&dq=blended+learning&ots=BikIIrvFfk&sig=b9EX5EgSgtRhl4RWvcpQDp45B6A&redir_esc=y#v=onepage&q=blended%20learning&f=false

Jasperson, J., & Miller, R. (2018). Speed of light vs. Speed of sound: comparing online and face-to-face classes in an undergraduate Business Law setting. In Edulearn 18. 10th International Conference on Education and New Learning Technology: (Palma, 2nd-4th of July, 2018). Conference proceedings (pp. 1685–1690). IATED Academy. https://library.iated.org/view/JASPERSON2018SPE

Jones, M., & Diment, K. (2010). The CAQDA Paradox: A Divergence between Research Method and Analytical Tool. 2nd International Workshop on Computer-Aided Qualitative Research, 82–86. Utrecht, The Netherlands: Merlien Institute, June 4–5. https://ro.uow.edu.au/commpapers/777/

Jowsey, T., Foster, G., Cooper-Ioelu, P., & Jacobs, S. (2020). Blended learning via distance in pre-registration nursing education: A scoping review. Nurse education in practice, 44 , 102775. https://doi.org/10.1016/j.nepr.2020.102775

Khan, B. H. (Ed.). (2005). Managing e-learning: Design, delivery, implementation, and evaluation . IGI Global https://doi.org/10.4018/978-1-59140-634-1

Kokoulina, 2019. What is blended learning and how can it be used? https://www.ispringsolutions.com/blog/blended-learning-a-primer . Accessed 5 May 2022

Köseoğlu, S., & Bozkurt, A. (2018). An exploratory literature review on open educational practices. Distance Education, 39 (4), 441–461. https://doi.org/10.1080/01587919.2018.1520042

Kumar, K. R., Ravi, S., & Srivatsa, S. K. (2011). Role of a Teacher in e-learning & Face-to-Face Learning Environment. International Journal of Computer Science and Network Security, 11 (7), 72–80.

Kushairi, N., & Ahmi, A. (2021). Flipped classroom in the second decade of the Millenia: A Bibliometrics analysis with Lotka’s law. Education and Information Technologies, 26 (4), 4401–4431. https://doi.org/10.1007/s10639-021-10457-8

Leydesdorff, L., Moya-Anegón, F., & Guerrero, V. (2015). Journal maps, interactive overlays, and the measurement of Interdisciplinarity on the basis of Scopus data (1996–2012). Journal of the Association for Information Science and Technology, 66 (5), 1001–1016. https://doi.org/10.1002/asi.23243

Li, J., & Hale, A. (2016). Output distributions and topic maps of safety related journals. Safety Science, 82 , 236–244. https://doi.org/10.1016/j.ssci.2015.09.004

Limaymanta, C., Apaza-Tapia, L., Vidal, E., & Gregorio-Chaviano, O. (2021). Flipped classroom in higher education: A bibliometric analysis and proposal of a framework for its implementation. International Journal of Emerging Technologies in Learning (iJET), 16 (9), 133–149. https://doi.org/10.3991/ijet.v16i09.21267

Lucky, A., Branham, M., & Atchison, R. (2019). Collection-based education by distance and face to face: Learning outcomes and academic dishonesty. Journal of Science Education and Technology, 28 (4), 414–428. https://doi.org/10.1007/s10956-019-9770-8

Maxwell, C. (2016). What blended learning is-and isn’t. BLU: Blended Learning Universe. http://www.blendedlearning.org/what-blended-learning-is-and-isnt/ . Accessed 3 May 2022

Mazur, E. (1997). Peer instruction: A user’s manual . Prentice Hall. https://doi.org/10.1063/1.881735

McCarthy, M. (2016). The Cambridge guide to blended learning for language teaching. Cambridge University Press . https://doi.org/10.1017/9781009024754

Medina, L. C. (2018). Blended learning: Deficits and prospects in higher education. Australasian Journal of Educational Technology, 34 (1), 42–56. https://doi.org/10.14742/ajet.3100

Mishra, S., Sahoo, S., & Pandey, S. (2021). Research trends in online distance learning during the COVID-19 pandemic. Distance Education, 1–26. https://doi.org/10.1080/01587919.2021.1986373

Nijakowski, K., Lehmann, A., Zdrojewski, J., Nowak, M., & Surdacka, A. (2021). The Effectiveness of the Blended Learning in Conservative Dentistry with Endodontics on the Basis of the Survey among 4th-Year Students during the COVID-19 Pandemic. International Journal of Environmental Research and Public Health, 18 (9), 4555. https://doi.org/10.3390/ijerph18094555

Noonoo, S. (2012). Flipped learning founders set the record straight. THE Journal . https://thejournal.com/articles/2012/06/20/flipped-learning-founders-q-and-a.aspx . Accessed 1 June 2022

Omar, R., Kaliappen, N., Khamis, K. A., & Sulisworo, D. (2021). Blended Learning Approach in Graduate Studies: A Bibliometric Analysis from 1997–2021. International Journal of Information and Education Technology , 11 (11). https://doi.org/10.18178/ijiet.2021.11.11.1563

Osguthorpe, R. T., & Graham, C. R. (2003). Blended learning systems: Definitions and directions. Quarterly Review of Distance Education, 4(3), 227–234. https://www.learntechlib.org/p/97576/ . Accessed 1 June 2022

Page, M.J., McKenzie, J., Bossuyt, P., Boutron, I., Hoffmann, T., Mulrow, C.D., Shamseer, L., Tetzlaff, J., Akl, E., Brennan, S.E., Chou, R., Glanville, J., Grimshaw, J., Hróbjartsson, A., Lalu, M.M., Li, T., Loder, E., Mayo-Wilson, E., McDonald, S., … Moher, D. (2020). The PRISMA 2020 statement: An updated guideline for reporting systematic reviews. Systematic reviews, 10 (1), 1–11. https://doi.org/10.31222/osf.io/v7gm2

Porter, W. W., Graham, C. R., Spring, K. A., & Welch, K. R. (2014). Computers & Education Blended learning in higher education : Institutional adoption and implementation. Computers & Education, 75 , 185–195. https://doi.org/10.1016/j.compedu.2014.02.011

Prescott, J. E., Bundschuh, K., Kazakoff, E. R., & Macaruso, P. (2018). Elementary school–wide implementation of a blended learning program for reading intervention. The Journal of Educational Research, 111 (4), 497–506. https://doi.org/10.1080/00220671.2017.1302914

Quitián, S., & González, J. (2020). Aspectos pedagógicos para ambientes Blended-Learning. HAMUT’AY, 7 , 48–59.

Racheva, V. (2019). Top three models of blended learning: Examples, pros, and cons. https://www.schoology.com/blog/top-three-models-blended-learning-examples-pros-and-cons . Accessed 10 June 2022

Raman, A., Thannimalai, R., Don, Y., & Rathakrishnan, M. (2021). A Bibliometric Analysis of Blended Learning in Higher Education: Perception, Achievement and Engagement. International Journal of Learning, Teaching and Educational Research , 20 (6). https://doi.org/10.26803/ijlter.20.6.7

Rasheed, R. A., Kamsin, A., & Abdullah, N. A. (2020). Challenges in the online component of blended learning: A systematic review. Computers & Education, 144 , 103701. https://doi.org/10.1016/j.compedu.2019.103701

Rooney, J. E. (2003). Blending learning opportunities to enhance educational programming and meetings. Association Managment, 55 (5), 26–32.

Russon, M. (1983). How secretarial studies might evolve. The Vocational Aspect of Education, 35 (90), 31–36. https://doi.org/10.1080/10408347308001851

Sahara, A., Mardji, M., Hadi, S., & Elmaunsyah, H. (2021). Blended learning management in higher education institutions in Indonesia. The International Journal of Learning in Higher Education, 28 (1), 65–73. https://doi.org/10.18848/2327-7955/CGP/v28i01/65-73

Singh, H. (2021). Building effective blended learning programs. In Challenges and Opportunities for the Global Implementation of E-Learning Frameworks (pp. 15–23). IGI Global. https://doi.org/10.4018/978-1-7998-7607-6.ch002

Siqueira, M. B., & Alfinito, S. (2014). Análise bibliométrica das TICs na Educação. SIED: EnPED-Simpósio Internacional de Educação a Distância e Encontro de Pesquisadores em Educação a Distância. http://www.sied-enped2014.ead.ufscar.br/ojs/index.php/2014/article/view/677/394 . Accessed 5 Apr 2022.

Smith, A. E., & Humphreys, M. S. (2006). Evaluation of unsupervised semantic mapping of natural language with Leximancer concept mapping. Behavior Research Methods, 38 (2), 262–279. https://doi.org/10.3758/BF03192778

Snyder, H. (2019). Literature review as a research methodology: An overview and guidelines. Journal of Business Research, 104 , 333–339. https://doi.org/10.1016/j.jbusres.2019.07.039

Souza, C. L., Mattos, L. B., Stein, A. T., Rosário, P., & Magalhães, C. R. (2018). Face-to-face and distance education modalities in the training of healthcare professionals: a quasi-experimental study. Frontiers in psychology , 1557. https://doi.org/10.3389/fpsyg.2018.01557

TeachThought Staff, (2019). 12 of the most common types of blended learning. https://www.teachthought.com/learning/12-types-of-blended-learning/ . Accessed 5 Apr 2022.

Sweileh, W. M. (2021). Global research activity on e-learning in health sciences education: A bibliometric analysis. Medical Science Educator, 31 (2), 765–775. https://doi.org/10.1007/s40670-021-01254-6

Tibaná-Herrera, G., Fernández-Bajón, M. T., & Moya-Anegón, D. (2018). Categorization of E-learning as an emerging discipline in the world publication system: A bibliometric study in SCOPUS. International Journal of Educational Technology in Higher Education, 15 (1), 1–23. https://doi.org/10.1186/s41239-018-0103-4

Vallée, A., Blacher, J., Cariou, A., & Sorbets, E. (2020). Blended learning compared to traditional learning in medical education: systematic review and meta-analysis. Journal of medical Internet research, 22 (8), e16504. https://doi.org/10.2196/16504

Van Eck, N. J., & Waltman, L. (2014). Visualizing bibliometric networks. Measuring Scholarly Impact. Springer International Publishing, 53 (2), 285–320. https://doi.org/10.1007/978-3-319-10377-8_13

Van-Nunen, K., Li, J., Reniers, G., & Ponnet, K. (2018). Bibliometric analysis of safety culture research. Safety Science, 108 , 248–258. https://doi.org/10.1016/j.ssci.2017.08.011

Vaughan, N. D. (2020). Blended Learning: Approaches, Trends, Research, and Publication Opportunities. In Innovating Education in Technology-Supported Environments (pp. 3–20). Springer, Singapore. https://doi.org/10.1007/978-981-15-6591-5_1

Wang, B., Pan, S. Y., Ke, R. Y., Wang, K., & Wei, Y. M. (2014). An overview of climate change vulnerability: A bibliometric analysis based on Web of Science database. Natural Hazards, 74 (3), 1649–1666. https://doi.org/10.1007/s11069-014-1260-y

Waterloo University (2020). Some examples of blended courses. https://uwaterloo.ca/centre-for-teaching-excellence/resources/blended-learning/some-examples-blended-courses . Accessed 1 Apr 2022.

Winzler, R. J. (1965). A flexible teaching laboratory in biochemistry. Journal of Medical Education, 40 (10), 965–971. https://doi.org/10.1097/00001888-196510000-00007

Yang, L., Chen, Z., Liu, T., Gong, Z., Yu, Y., & Wang, J. (2013). Global trends of solid waste research from 1997 to 2011 by using bibliometric analysis. Scientometrics, 96 , 133–146. https://doi.org/10.1007/s11192-012-0911-6

Yang, L., Sun, T., & Liu, Y. (2017). A bibliometric investigation of flipped classroom research during 2000–2015. International Journal of Emerging Technologies in Learning (ijet), 12 (06), 178–186. https://doi.org/10.3991/ijet.v12i06.7095

Yao, H., Brossard, M., Mizunoya, S., Nasir, B., Walugembe, P., Cooper, R., Rafique, A., & Reugei, N. (2021). How Much Does Universal Digital Learning Cost?. Policy Brief. UNICEF Office of Research-Innocenti. https://files.eric.ed.gov/fulltext/ED619426.pdf

Zawacki-Richter, O., Bozkurt, A., Alturki, U., & Aldraiweesh, A. (2018). What research says about MOOCs–An explorative content analysis. The International Review of Research in Open and Distributed Learning , 19 (1). https://doi.org/10.19173/irrodl.v19i1.3356

Zupic, I., & Cater, T. (2015). Bibliometric methods in management and organization. Organizational Research Methods, 18 (3), 429–434. https://doi.org/10.1177/1094428114562629

Download references

The authors did not receive support from any organization for the submitted work.

Author information

Authors and affiliations.

Distance Learning Research and Application Centre, Yildiz Technical University, Istanbul, Turkey

Betül Tonbuloğlu

Faculty of Education, Department of Computer Education and Instructional Technology, Yildiz Technical University, Istanbul, Turkey

İsmail Tonbuloğlu

You can also search for this author in PubMed   Google Scholar

Contributions

All authors were involved in preparing this manuscript and have provided significant intellectual input. All authors read and approved the final manuscript.

Corresponding author

Correspondence to Betül Tonbuloğlu .

Ethics declarations

Conflict of interest.

The authors have no relevant financial or non-financial interests to disclose.

Data will be made available on reasonable request.

Additional information

Publisher's note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Tonbuloğlu, B., Tonbuloğlu, İ. Trends and patterns in blended learning research (1965–2022). Educ Inf Technol 28 , 13987–14018 (2023). https://doi.org/10.1007/s10639-023-11754-0

Download citation

Received : 22 August 2022

Accepted : 22 March 2023

Published : 03 April 2023

Issue Date : November 2023

DOI : https://doi.org/10.1007/s10639-023-11754-0

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Blended learning
• Hybrid education
• Flipped learning
• Bibliometrics
• Citation analysis
• Common word analysis
• Find a journal
• Publish with us
• Track your research

ORIGINAL RESEARCH article

Evaluating blended learning effectiveness: an empirical study from undergraduates’ perspectives using structural equation modeling.

• Department of Elementary School Education, School of Primary Education, Shanghai Normal University Tianhua College, Shanghai, China

Following the global COVID-19 outbreak, blended learning (BL) has received increasing attention from educators. The purpose of this study was: (a) to develop a measurement to evaluate the effectiveness of blended learning for undergraduates; and (b) to explore the potential association between effectiveness with blended learning and student learning outcomes. This research consisted of two stages. In Stage I, a measurement for evaluating undergraduates’ blended learning perceptions was developed. In Stage II, a non-experimental, correlational design was utilized to examine whether or not there is an association between blended learning effectiveness and student learning outcomes. SPSS 26.0 and AMOS 23.0 were utilized to implement factor analysis and structured equation modeling. The results of the study demonstrated: (1) The hypothesized factors (course overview, course objectives, assessments, 1148 class activities, course resources, and technology support) were aligned as a unified system in blended learning. (2) There was a positive relationship between the effectiveness of blended learning and student learning outcomes. Additional findings, explanations, and suggestions for future research were also discussed in the study.

1. Introduction

Following the global COVID-19 outbreak, blended learning (BL) has received increasing attention from educators. BL can be defined as an approach that combines face-to-face and online learning ( Dos, 2014 ), which has become the default means of delivering educational content in the pandemic context worldwide due to its rich pedagogical practices, flexible approaches, and cost-effectiveness ( Tamim, 2018 ; Lakhal et al., 2020 ). Moreover, empirical research has demonstrated that BL improves learners’ active learning strategies, multi-technology learning processes, and learner-centered learning experiences ( Feng et al., 2018 ; Han and Ellis, 2021 ; Liu, 2021 ). Furthermore, students are increasingly requesting BL courses due to the inability to on-campus attendance ( Brown et al., 2018 ). In addition, researchers have examined the positive effects of BL on engaging students, improving their academic performance and raising student satisfaction ( Alducin-Ochoa and Vázquez-Martínez, 2016 ; Manwaring et al., 2017 ).

In China, the Ministry of Education has strongly supported educational informatization since 2012 by issuing a number of policies (the Ministry of Education, 2012). In 2016, China issued the Guiding Opinions of the Ministry of Education on Deepening the Educational and Teaching Reform of Colleges and Universities, emphasizing the promotion of the BL model in higher education. In 2017, the Ministry of Education listed BL as one of the trends in driving education reform in the New Media Alliance Horizon Report: 2017 Higher Education Edition. In 2018, Minister Chen Baosheng of the Ministry of Education proposed at the National Conference on Undergraduate Education in Colleges and Universities in the New Era to focus on promoting classroom revolution and new teaching models such as flipped classroom and BL approach. In 2020, the first batch of national BL courses was identified, which pushed the development of BL to the forefront of teaching reform. During the pandemic era in China, BL was implemented in all universities and colleges.

However, a number of researchers produced opposing results regarding the benefits of BL. Given the pre-requisites, resources, and attitudes of the students, BL model is suspected to be inapplicable to all courses, such as practicum courses ( Boyle et al., 2003 ; Naffi et al., 2020 ). Moreover, it should be noted that students, teachers, and educational institutions may lack BL experience and therefore they are not sufficiently prepared (such as technology access) to implement BL methods or focus on the efficiency of BL initiatives ( Xiao, 2016 ; Liliana, 2018 ; Adnan and Anwar, 2020 ). Another big concern is that BL practice is hard to evaluate because there are few standardized BL criteria ( Yan and Chen, 2021 ; Zhang et al., 2022 ). In addition, a number of studies have concluded there was no significant contribution of BL in terms of student performance and test scores, compared to traditional learning environments ( U.S. Department of Education, 2009 ). Therefore, it is extremely necessary to explore the essential elements of BL in higher education and examine the effect of BL on student academic achievement. This paper offers important insights for those attempting to implement BL in classroom practice to effectively support student needs in higher education.

The purpose of this research was: (1) To develop a measurement with key components to evaluate BL in undergraduates; (2) To explore the associations between perceptions of BL effectiveness and student learning outcomes (SLOs) in a higher education course using the developed measurement.

The significance of the current study was listed as follows: (1) The researchers noted that there have only a few studies have focused on the BL measurement in higher education and its effects on SLOs. Therefore, the current study results will add to the literature regarding BL measurement and its validity. (2) The Ministry of Education in China has an explicit goal the desire to update university teaching means and strategies in accordance with the demands of the twenty-first century. Therefore, the current study will contribute to the national goals of the Ministry of Education in China, enhance understanding of BL, and provide a theoretical framework and its applicability. (3) Faculty members in higher education who attempt to apply BL model in their instructions will be aware of the basic components of BL that contribute to SLOs.

2. Literature review

2.1. definitions of bl.

BL is referred to as “hybrid,” “flexible,” “mixed,” “flipped” or “inverted” learning. The BL concept was first proposed in the late 20 century against the backdrop of growing technological innovation ( Keogh et al., 2017 ). The general definition of BL is that it integrates traditional face-to-face teaching with a web-based approach.

However, this description has been hotly debated by researchers in recent years. Oliver and Trigwell (2005) posited that BL may have different attributions in relation to various theories, meaning that the concept should be revised. Others attempted to clarify the significance of BL by classifying the proportion of online learning in BL and the different models that come under the BL umbrella. Allen and Seaman (2010) proposed that BL should include 30–70% online-in person learning (otherwise, it would be considered online learning (more than 70%) or traditional face-to-face learning (less than 30%)). In The Handbook of Blended Learning that edited by Bonk and Graham (2006) set out three categories of BL: web-enhanced learning, reduced face-time learning, and transforming blends. Web-enhanced learning pertains to the addition of extra online materials and learning experiences to traditional face-to-face instruction. Reduced face-time learning means to shift part of face-to-face lecture time to computer-mediated activities. Transforming blends mixes traditional face-to-face instruction with web-based interactions, through which students are able to actively construct their knowledge.

This study views BL as an instructional approach that provides both synchronous and asynchronous modes of delivery through which students construct their own understandings and interact with others in these settings, which is widely accepted by numerous researchers ( Liliana, 2018 ; Bayyat et al., 2021 ). To phrase this in another way, this description emphasizes that learning has to be experienced by the learner.

2.2. Essential elements of BL

Previous studies, universities, and cooperation have discussed the essential components of online learning courses. Blackboard assesses online learning environments on four scales (course design, cooperation, assessment, and learner support) with 63 items. Quality Matters evaluated online learning according to the following categories: course overview, objectives, assessment, teaching resources, activities and cooperation, course technology, learner support, and practicability. Californian State Universities rated their criteria on a ten scale of 58 items, including learning evaluation, cooperation and activities, technology support, mobile technology, accessibility, and course reflection. New York State Universities evaluate BL under the following six sub scales: course overview, course design, and assignment, class activities, cooperation, and assessment. Due to the lack of criteria for BL, these standards have been considered in evaluating BL.

The present study utilized Biggs’ (1999) constructive alignment as the main theoretical framework to analyze BL courses. “Constructive means the idea that students construct meaning through relevant activities … and the alignment aspect refers to what the teachers do, which is to set up a learning environment” ( Biggs, 1999 , p. 13). Later, Biggs and Tang (2011) elaborated on the two terms — ‘constructive’ and ‘alignment’ originated from constructivist theory and curriculum theory, respectively, in the book Teaching for Quality Learning at University. Constructivism was regarded as “learners use their own activity to construct their knowledge as interpreted through their own exiting schemata.” The term “alignment” emphasized that the assessments set were relevant and conducive to the intended learning goals ( Biggs and Tang, 2011 , p. 97). According to Biggs’ statement, various critical components should be closely linked within the learning context, including learning objectives, teaching learning activities, and assessment tasks. These main components have been defined in detail:

(1) Learning objectives indicate the expected level of student understanding and performance. They tell students what they have to do, how they should do it, and how they will be assessed. Both course overview and learning objectives involve intended learning outcomes.

(2) Teaching/learning activities are a set of learning processes that the students have to complete by themselves to achieve a given course’s intended learning outcomes. In BL, activities include both online and face-to-face activities where students are able to engage in collaborations and social interactions ( Hadwin and Oshige, 2011 ; Ellis et al., 2021 ). The interactive learning activities are chosen to best support course objectives and students’ learning outcomes ( Clark and Post, 2021 ). Examples of activities in BL include: group problem-solving, discussion with peers/teachers, peer instruction, answering clicker questions or in-class polls ( Matsushita, 2017 ).

(3) Assessment tasks are tools to determine students’ achievements based on evidence. In BL, assessments can be conducted either online or in-class. Examples of assessments in BL include: online quizzes, group projects, field-work notes, individual assignments.

(4) Besides, based on the definition of BL and the integration of information technology improvement in recent years, online resources and technological support have become essential components of BL courses ( Darling-Aduana and Heinrich, 2018 ; Turvey and Pachler, 2020 ). On a similar note, Ellis and Goodyear (2016) and Laurillard (2013) emphasized the role of technical devices in BL, whilst Zawacki-Richter (2009) regarded online resources and technological support as central to achieving BL course requirements. In addition, Liu (2021) suggested that a BL model should include teaching objectives, operating procedures, teaching evaluation, and teaching resources before class, during class, and after class, respectively. With this in mind, the present study integrates both essential curriculum components in the face-to-face course and information technology into the teaching and learning aspects of the BL course.

2.3. BL effectiveness and SLOs

Many researchers have demonstrated the benefits of BL approach on SLOs because of the importance of BL in improving teaching methods and better reflecting the improvement of the learner skills, talents, and interest in learning. Garrison and Kanuka (2004) reported increased completion rates as a result of BL application. Similar results were agreed upon by other researchers. Kenney and Newcombe (2011) and Demirkol and Kazu (2014) conducted comparisons and found that students in BL environments had higher average scores than those in non-BL environments. Alsalhi et al. (2021) utilized a quasi-experimental study at Ajman University ( n  = 268) and indicated that the use of BL has a positive effect on students’ academic success in a statistics course. “BL helps to balance a classroom that contains students with different readiness, motivation, and skills to learn. Moreover, BL deviates from traditional teaching and memorizing of students” ( Alsalhi et al., 2021 , p. 253). No statistical significant difference was found among students based on the variables of the university they attended.

However, researchers also showed that BL approach may not be applicable to all learners or improve their learning outcomes. Oxford Group (2013) reported that about 16% of learners had negative attitudes toward BL, while 26% of learners chose not to complete BL. Kintu et al. (2017) examined the relationship between student characteristics, BL design, and learning outcomes and indicated that BL design is beneficial to raise student satisfaction ( n  = 238). The study also found that BL predicted learning outcomes for learners with high self-regulation skills. Similar results were reported by Siemens (2005) who indicated that students who have higher learner interactions resulted in higher satisfaction and learning outcomes. Hara (2000) identified ambiguous course design and potential technical difficulties as major barriers in BL practice, which led to dissatisfied learning outcomes. Clark and Post (2021) utilized a hybrid study in higher education to explore the effectiveness of different instructional approaches (face-to-face, eLearning, and BL) and indicated that the individual student valued active learning in both face-to-face classes and eLearning classes. Moreover, having an eLearning experience prior to face-to-face classes is beneficial for students to perform well on the assessment. However, the study noted that students who took face-to-face courses were positively associated with their final grades.

2.4. Research questions and hypotheses

To fill in the gaps, the research questions and hypotheses were raised in the present research as follows:

RQ1 : What components (among course overview, course objectives, assessments, activities, course resources, and technology support) contribute to the measurement?

RQ2 : Is there an association between BL effectiveness and SLOs in higher education?

H1 : All components (among course overview, course objectives, assessments, class activities, course resources, and technology support) contribute to the BL course model.

H2 : There is an association between BL effectiveness and SLOs.

3. Methodology

The study employed a non-experimental, correlational design and used survey responses from undergraduates to address the research questions. Specifically, a higher education institution in Shanghai with a specialization in teacher education was studied. The present study was a part of an instructional initiative project at this institution designed to identify students’ perceptions of the effectiveness of BL and explore the possible relationships between BL effectiveness and SLOs.

The present research consisted of two stages: Stage I (from March 2021 to July 2021) aimed to develop a measurement for evaluating undergraduates’ BL perceptions through a survey of undergraduates who had experienced BL courses. Stage II (from September 2021 to January 2022) aimed to use the developed measurement to examine whether or not there is an association between BL effectiveness and SLOs.

3.1. Instruments

3.1.1. effectiveness of bl scale (ebls).

In Stage I, according to Biggs’ theoretical framework and the existing literature, the measurement used in this study was composed of six sub-scales: course overview, learning objectives, assessments, course resources, teaching/learning activities, and technology support. After comparing these criteria, the instrument titled “Blended Learning Evaluation” was derived from Quality Matters Course Design Rubric Standards (QM Rubric) and revised. Following consultation with experienced teaching experts who had experience in BL design and application, the revised QM Rubric can be applied to both the online and face-to-face portions of the course. Table 1 details the modified measurement.

Table 1 . Items of measurement.

Then, a panel of two experts, two blended course design trainers, and two faculty members in the curriculum and instruction department were asked to evaluate the appropriateness and relevance of each item included in the instrument. Subsequently, a group of 10 sophomores and senior students were asked to check how the questions are read and understood and accordingly give feedback. Based on their comments and the suggestions from the panel, a few minor changes were made and content validity was again evaluated by the panel of experts prior to the administration of the instrument.

Finally, the EBLS that modified from Quality Matters Course Design Rubric Standards was determined as the initial scale that was preparing for the construct validity and reliability checks. The EBLS composed of six sub-scales (25 items in total): course overview (4 items), course objectives (4 items), assessments (4 items), course resources (5 items), in-class and online activities (4 items), and technology support (4 items). The measurement applied a 5-point Likert Scale (1-Strongly Disagree, 2-Disagree, 3-So-so, 4-Agree, and 5-Strongly Agree).

3.1.2. Student learning outcomes

In Stage II, the course marks from the Curriculum and Instruction Theorem module were used as an indicator of students’ learning outcomes. Multiple regression analysis was utilized via SPSS 25.0 to perform data analysis.

In the second stage of the study, the students’ course marks from the Curriculum and Instruction Theorem module were used as an indicator of students’ learning outcomes. This curriculum was a semester-long mandatory course for 91 sophomores which ran for 16 weeks from September 2021 to January 2022. It aimed to develop the students’ knowledge of in-depth disciplinary and academic content but also skills pertaining to cooperation, technology, inquiry, discussion, presentation, and reflection. The course was designed as a synchronous BL curriculum, in which students all had both face-to-face and technologically-mediated interactions (see Table 2 ). Each week, there were 1.5 h of face-to-face learning that combined lectures, tutorials, and fieldwork. The lectures covered teaching key concepts with examples and non-examples and connected teaching theories to practical issues. Meanwhile, the tutorials provided opportunities for students to collaborate with peers or in groups. The fieldwork offered opportunities for students to observe real classes and interview cooperative teachers or students in local elementary schools. Technologically-mediated interactions supported by the Learning Management System (LMS) provided supplementary learning resources, reading materials, relative videos, cases, assessment and other resources from the Internet. Students were required to complete online quizzes, assignments, projects, and discussions as well on LMS.

Table 2 . Weekly blended learning design mode.

The final marks of the course were derived from both formative and summative assessments. The formative assessments covered attendance and participation, individual assignments (quizzes, reflections, discussions, case studies, and class observation reports) and group projects (lesson plan analysis, mini-instruction, reports). The summative assessment was a paper-based final examination, as required by the college administrators.

3.2. Participants

In Stage I of the study, the target population was sophomore and junior undergraduates from different majors at a higher education institution in Shanghai. Detailed demographic information has been reported in the results section of this study. Notably, due to practical constraints, a convenience sample was employed in the present study. As explained by McMillan and Schumacher (2010) , although the generalizability of the results is more limited, the findings are nevertheless useful when considering BL effectiveness. Thus, care was taken to gather the demographic background information on the respondents to ensure an accurate description of the participants could be achieved.

In Stage II of the study, the participants were 91 sophomores who took the synchronous BL course, Curriculum and Instruction Theorem, in School of Primary Education in the fall of 2021 (September 2021–January 2022).

3.3. Data collection procedures, analysis and presentation

Institutional Review Board (IRB) approval was obtained prior to the collection of Stage I and Stage II. In Stage I, the informed IRB-approved Informed Consent Form included a brief introduction to the study purpose, the length of time required to complete the survey, possible risks and benefits, the researcher’s contact information, etc. It also clarified to the potential respondent that the survey was voluntary and anonymous. SurveyMonkey 1 was used to administer the survey. In Stage II, the informed IRB-approved Informed Consent Form was also provided to participants. LMS was used for data collection.

To address RQ1, the study used the four following steps:

1. An initial measurement was modified and translated from QM rubrics, and the content validity was checked by the authority.

2. Secondly, the reliability of measurement was examined.

3. Exploratory factor analysis (EFA) was conducted to test the construct validity.

4. Confirmatory factor analysis (CFA) was examined to correct for the relationships between the modeling and data. Ultimately, a revised BL measurement was developed with factor loadings and weights. In Stage I, SPSS 26.0 and AMOS 23.0 were utilized to implement factor analysis and structured equation modeling.

To address RQ2, the study followed two steps:

1. Descriptive statistics (mean, standard deviation, minimum rating, and maximum rating) were calculated on the undergraduates’ perspectives on BL effectiveness, as identified by the author.

2. SLOs were regressed on the perceived BL effectiveness. This research question examined whether the overall BL effectiveness was associated with student achievement. In Stage II, SPSS 26.0 was utilized to implement correlations and multiple regressions.

3.4. Limitations

Based on the threats to the validity of internal, external, structural, and statistical findings summarized by McMillan and Schumacher (2010) , the following limitations of this study are acknowledged. First, since data were self-reported by participants, may have been influenced and the answers they provided may not reflect their true feelings or behaviors. Second, the study used a convenience sample rather than a database consisting of all undergraduates in higher education in Shanghai; therefore, the population external validity was limited to those faculties with response characteristics. Last, although care was taken to generally phrase the research questions in terms of association rather than effects, a limitation of the study is that correlational design limits our ability to draw causal inferences. The results may be suggestive, but further research is needed in order to draw conclusions about BL impacts.

4.1. What factors (among course overview, course objectives, assessments, class activities, course resources, and technology support) contribute to the measurement?

4.1.1. demographic information in stage i.

In Stage I, a survey with 25 items in 6 sub-scales was delivered to undergraduates who had experienced BL in higher education. In total, 295 valid questionnaires were collected in Stage I (from March 2021 to July 2021). Demographic information of the participants were reported as follows: the percentage of male respondents was 27% while the percentage of female respondents was 73%. The majors of respondents included education (51%), literature (22%), computer science (11%), business (10%), arts (5%), and others (1%). All the respondents were single and aged in the range of 19–20 years old.

4.1.2. Reliability analysis

To address RQ1, reliability and EFA were conducted on the questionnaire results. Test reliability refers to “the consistency of measurement – the extent to which the results are similar over different forms of the same instrument or occasions of data collection” ( McMillan and Schumacher, 2010 , p. 179). To be precise, the study tested internal consistency (Cronbach’s Alpha), composite reliability (CR), and Average of Variance Extracted (AVE) evidence for reliability. According to Table 3 , the reliability of the measurement (25 Items) showed the internal reliability for this scale was 0.949 ( N  = 295). The alpha reliability value for each sub-scale is as follows: 0.859, 0.873, 0.877, 0.910, 0.902, and 0.881, respectively. Since the total scale’s alpha value and sub-scales’ alpha values were all greater than 0.70, the reliability of the survey was relatively high and therefore acceptable. Moreover, the AVE of each sub-scale was greater than 0.50, indicating that the reliability and convergence of this measurement were good. In addition, CR values were all greater than 0.80. This indicates that the composite reliability is high. Therefore, this blended course evaluation measurement is deemed reliable.

Table 3 . Reliability results for the measurement ( N  = 295).

4.1.3. Exploratory factor analysis

According to the research design, EFA was then carried out to determine its construct validity by using SPSS 26.0 to identify if some or all factors (among course overview, course objectives, assessments, class activities, course resources, and technology support) perform well in the context of a blended course design. According to Bryant and Arnold (1995) , to run EFA, the sample should be at least five times the number of variables. The subjects-to-variables ratio should be 5 or greater. Furthermore, every analysis should be based on “a minimum of 100 observations regardless of the subjects-to-variables ratio” (p. 100). This study included 25 variables, meaning that 300 samples were gathered. The number of samples was more than 12 times greater than the variables. Compared to the criteria proposed by Kaiser and Rice (1974) , the KMO of measurement in this study was greater than 0.70 (0.932). This result indicates the sampling is more than adequate. According to Table 4 (showing Bartlett’s Test of Sphericity), the approximate Chi-square of Bartlett’s test of Sphericity is 4124.801 ( p  = 0.000 < 0.001). This shows that the test was likely to be significant. Therefore, EFA could be used to examine the study.

Table 4 . Bartlett’s test of sphericity.

EFA refers to “how items are related to each other and how different parts of an instrument are related” ( McMillan and Schumacher, 2010 , p. 176). Factor analysis (principal component with varimax rotation) analysis was deployed to assess the degree to which 25 blended course design level questions were asked in the “Blended Course Evaluation Survey.” According to the EFA results detailed in Table 5 (Rotated Factor Matrix), the 25 items loaded on six factors with eigenvalues were greater than 1. The results of the rotated factor matrix showed the loadings were all close to or higher than 0.70 ( Comrey and Lee, 1992 ). Therefore, these six factors mapped well to the dimensions and the measurement can be seen to have relatively good construct validity. Hence, to answer RQ1, all of the factors (among course overview, course objectives, assessments, class activities, course resources, and technology support) performed well in the measurement.

Table 5 . Rotated factor matrix * .

To further address to what extent factors contribute to the measurement, the hypothesized model in the present study was examined, after which the weight of each factor was calculated for educators based on its structural equation modeling. To discern whether the hypothesized model reflects the collected data, AMOS 23.0 was utilized to carry out confirmatory factor analysis. Compared the fit indexes to the criteria in Table 6 Comparison of Fit Indexes for Alternative Models of the Structure of the Blended Course Design Measurement below, the Root Mean Square Error of Approximation (RMSEA) was 0.034, lower than our rule of thumb of 0.05, which would indicate a good model. Additionally, the results of TLI (0.979) and CFI (0.981) were above our target for a good model. Moreover, CMIN/DF was 1.284, lower than 3; GFI was 0.909, greater than 0.8; AGFI was 0.886, greater than 0.8; NFI was 0.922, greater than 0.9; IFI was 0.982, greater than 0.9; and RMR was 0.013 lower than 0.08. Based on these criteria, it appears that the initial model fits the data well. In other words, the initial model can effectively explain and evaluate a blended course design.

Table 6 . Comparison of fit indexes for alternative models of the structure of the blended learning measurement.

4.1.4. Confirmatory factor analysis

Focusing on the model itself, CFA was examined to correct the relationships between the modeling and data. Figure 1 shows that most subtests provided relatively strong measures of the appropriate ability or construct. Specifically, one factor was positively correlated to the others. For instance, course overview was positively correlated to course objectives, assessments, course resources, class activities, and technology support. The coefficients of the correlations for the respective factors are as follows: 0.72, 0.52, 0.61, 0.63, 0.55. This means that in any BL, if the course overview rises by 1 point, the other variables will rise by 0.72, 0.52, 0.61, 0.63, 0.55 points, respectively. The results match the statement that “cognitive tests and cognitive factors are positively correlated” ( Keith, 2015 , p. 335). Additionally, this study tested the discriminant validity of the measurement to ensure that each factor performed differently in the model itself. According to Fornell and Lacker’s (1981) criteria, the square root of AVE value must be greater than the correlation value between the other concepts. The results in Table 7 illustrated that the value of the variables (0.777 which was the lowest) exceeded the correlation value (0.72 which was the greatest). From this, it can be confirmed that the hypothesized model used in the present study had sufficient discriminant validity. Therefore, the hypothesized model in the present study reflected reality well.

Figure 1 . Standardized estimates for the initial blended course design six-factor model.

Table 7 . Discriminant validity.

The weight of each factor in the model was further calculated for educators based on structural equation modeling (see Figure 2 ). For example, the weight of course overview = 0.84/(0.84 + 0.79 + 0.70 + 0.73 + 0.74 + 0.70) = 0.187. Using the same way to calculate the other weighs. The relevant calculations are shown below and the results are shown in Table 8 . The total score of a blend course design is calculated as follows: the score of course overview * 0.187 + the score of course objectives * 0.176 + the score of assessment * 0.155 + the score of course resources * 0.162 + the score of class activities * 0.164 + the score of technology support * 0.156. The total grade for this measurement is 100.

Figure 2 . Weight of factors in the present model.

Table 8 . Confirmatory factor loading and weightings.

4.2. Is there an association between the effectiveness of blended learning and student learning outcomes?

4.2.1. demographic information in stage ii.

In Stage II, there were 91 respondents collected through LMS. The percentage of male respondents was 16% while the percentage of female respondents was 84%. All the respondents in Stage II took the synchronous BL course, Curriculum and Instruction Theorem, in School of Primary Education in Fall 2021 (September 2021–January 2022).

4.2.2. Descriptive statistics

In answering RQ2, the descriptive statistics were reported in Table 9 for the undergraduates’ perspectives on BL effectiveness and SLOs. Higher scores for this measure of BL effectiveness indicate undergraduates perceive BL as more effective, with responses of 1 for “Strongly Disagree” to 4 for “Strongly Agree.” The results revealed that the six elements of BL effectiveness had an overall mean of 93.65 (corresponding to an item average of 3.74, which corresponds to “agree”). The scores of each sub-scale are very similar and, again, correspond to undergraduates reporting that they “agree” with the efficacy of BL with respect to course overview, course objectives, assessment, course resources, class activities, and technology support. Table 8 also provided information about overall SLOs. Specifically, it illustrated that students’ learning outcomes (final marks composed of formative assessments and summertime assessments) in BL had an overall mean of 80.65. The maximum and minimum scores were 93.00 and 60.00, respectively.

Table 9 . Descriptive statistics for the overall scores and sub scales of the measures of blended learning effectiveness and student achievement ( N  = 91).

4.2.3. Regressions between BL effectiveness and SLOs

To further address the relationship between BL effectiveness and SLOs, SLOs was regressed on the perceived BL effectiveness. This research question examined whether the overall BL effectiveness was associated with student achievement. Additionally, Pearson correlations (shown in Table 10 ) between key variables were calculated. The results showed that the overall score of BL effectiveness was significantly correlated with student achievement ( r  = 0.716, p  < 0.01).

Table 10 . Descriptive statistics and Pearson correlations between key variables in the regression models.

Table 11 shows the results for the regression of total student academic performance on the overall BL effectiveness scores across six components (course overview, course objectives, assessment, course resources, class activities, and technology support). Notably, the full model was statistically significant. Directly addressing RQ2, undergraduates reported that regarding BL effectiveness explained 51.3% of the additional variance, F (1, 89) = 93.843, p  < 0.001, ΔR 2  = 0.508. Moreover, it was statistically significant and considered to have a large effect. Accordingly, when the perception of BL effectiveness increased by a value of one point, the student’s academic performance would increase by 0.563 ( b  = 0.563, p  < 0.001). Thus, to answer the final research question, there is a positive correlation between the effectiveness of BL and student learning achievement.

Table 11 . Summary of simultaneous multiple linear regression results predicting student achievement from perceptions of the blended learning effectiveness.

5. Conclusion and discussion

BL is a combination of face-to-face interactions and online learning, where the instructor manages students in a technological learning environment. In the post-pandemic era, BL courses are widely used and accepted by educators, students, and universities. However, the validity of BL remains controversial. The lack of an accurate BL scale was one of the big concerns. The study developed a measurement to evaluate BL for undergraduates and investigated the relationship between the effectiveness of BL and SLOs. Biggs’ 1999) constructive alignment, including factors like course overview, learning objectives, teaching/learning activities, and assessment, was utilized as the primary theoretical framework for conceptualizing the scale. Later, related literature indicated the importance of adding technology and resources as essential components. Therefore, a scale was developed with six subscales.

RQ1 explored the essential components of BL. Stage I recruited 295 undergraduates from different majors at a university in Shanghai. Hypothetical measurements that include 6 sub-scales (25 items in total) were examined. Construct validity was examined with EFA and CFA. As a result, a 6-factor 5-point Likert-type scale of BL effectiveness made up of 25 times was developed. The total variance regarding the six factors of this scale was calculated as 68.4%. The internal consistency reliability coefficient (Cronbach’s alpha) for the total scale was calculated to be 0.949. The alpha reliability values for each sub-scale were as follows: 0.859, 0.873, 0.877, 0.910, 0.902, and 0.881, respectively. The results of the study demonstrated that the hypothesized factors (course overview, course objectives, assessments, class activities, course resources, and technology support) mainly proposed by Biggs (1999) are aligned as a unified system in BL. Furthermore, the results reflect the real concerns of students as they experience BL in higher education However, the participants in the present study were selected from among students enrolled in BL at the university. The characteristics of these samples were as limited as the responders. In future research, a larger scale including undergraduates in other universities may be recruited to test the validity.

RQ2 examined the association between BL validity and SLOs. In Stage II, the study recruited 91 students who participated in a synchronous BL course at the College of Education. The results demonstrated a positive relationship between the effectiveness of BL and SLOs: the more effective that undergraduates perceived BL, the better their SLOs. It supported the results of the previous literature ( Demirkol and Kazu, 2014 ; Alsalhi et al., 2021 ). Moreover, the descriptive analysis provided additional findings for educators when designing and implementing BL for undergraduates. First, undergraduates expect a clear class overview about how to start the course, how to learn through the course, and how to evaluate their learning outcomes. A clear syllabus with detailed explanations should be prepared and distributed at the outset of BL. Second, undergraduates pay attention to curriculum objectives and continuously compare their work as they progress through the course to see if it helps them achieve those objectives; on this basis, outlining the objectives at the beginning of chapter learning and showing expected learning outcomes (such as rubrics) are recommended. Finally, undergraduates enjoy rich social interactions in both face-to-face activities and online interactions, therefore, a variety of classroom activities for different levels of students is recommended. In future study, more detailed analyses could be considered. For example, it would be valuable to explore the indirect effect of the effectiveness of BL on SLOS. Besides, qualitative research could be conducted to identify the underlying reasons why BL affects SLOs.

Data availability statement

The datasets presented in this article are not readily available because the datasets generated for this study are not publicly available due to the permissions gained from the target group. Requests to access the datasets should be directed to XH, [email protected] .

Ethics statement

The studies involving human participants were reviewed and approved by the Shanghai Normal University Tianhua College. The patients/participants provided their written informed consent to participate in this study.

Author contributions

XH: drafting the manuscript, data analysis and perform the analysis, and funding acquisition. JF: theoretical framework, and methodology. BW: supervision. YC: data collection and curation. YW: reviewing and editing. The author confirms being the sole contributor of this work and has approved it for publication.

This research was supported by the Chinese Association for Non-Government Education (grant number: CANFZG22268) and Shanghai High Education Novice Teacher Training Funding Plan (grant number: ZZ202231022).

Acknowledgments

I would like to express my appreciation to my colleagues: Prof. Jie Feng and Dr. Yinghui Chen. They both provided invaluable feedback on this manuscript.

Conflict of interest

The author declares that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Publisher’s note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

1. ^ www.surveymonkey.com

Adnan, M., and Anwar, K. (2020). Online learning amid the COVID-19 pandemic: students’ perspectives. J. Pedagog. Sociol. Psychol. 2, 45–51. doi: 10.33902/JPSP.2020261309

CrossRef Full Text | Google Scholar

Alducin-Ochoa, J. M., and Vázquez-Martínez, A. I. (2016). Academic performance in blended- learning and face-to-face university teaching. Asian Soc. Sci. 12:207. doi: 10.5539/ass.v12n3p207

Allen, I. E., and Seaman, J. (2010). Class differences: online education in the United States, 2010. The Sloan consortium. Babson Survey Research Group. Available at: http://sloanconsortium.org/publications/survey/class_differences (Accessed October 7, 2014)

Google Scholar

Alsalhi, N., Eltahir, M., Al-Qatawneh, S., Quakli, N., Antoun, H., Abdelkader, A., et al. (2021). Blended learning in higher education: a study of its impact on students’ performance. Int. J. Emerg. Technol. Learn. 16, 249–268. doi: 10.3991/ijet.v16i14.23775

Bayyat, M., Muaili, Z., and Aldabbas, L. (2021). Online component challenges of a blended learning experience: a comprehensive approach. Turk. Online J. Dist. Educ. 22, 277–294. doi: 10.17718/tojde.1002881

Biggs, J. (1999). Teaching for quality learning at university. open university press/society for research in higher education, Buckingham.

Biggs, J., and Tang, C. (2011). Teaching for quality learning at university: what the student does (4th). London: McGraw-Hill.

Bonk, C., and Graham, C. (2006). The handbook of blended learning: global perspectives, local designs . San Francisco, CA: Pfeiffer Publishing.

Boyle, T., Bradley, C., Chalk, P., Jones, R., and Pickard, P. (2003). Using blended learning to improve student success rates in learning to program. J. Educ. Media 28, 165–178. doi: 10.1080/1358165032000153160

Brown, C., Davis, N., Sotardi, V., and Vidal, W. (2018). Towards understanding of student engagement in blended learning: a conceptualization of learning without borders. Available at: http://2018conference.ascilite.org/wp-content/uploads/2018/12/ASCILITE-2018-Proceedings-Final.pdf. 318-323

Bryant, F. B., and Arnold, P. R. (1995). “Principal-components analysis and exploratory and confirmatory factor analysis” in Reading and understanding multivariate statistics . eds. L. G. Grimm and P. R. Arnold (Washington, DC: American Psychological Association), 99–136.

Clark, C., and Post, G. (2021). Preparation and synchronous participation improve student performance in a blended learning experience. Australas. J. Educ. Technol. 37, 187–199. doi: 10.14742/ajet.6811

Comrey, A. L., and Lee, H. B. (1992). A first course in factor analysis (2nd). Hillsdale, NJ: Lawrence Erlbaum Associates.

Darling-Aduana, J., and Heinrich, C. J. (2018). The role of teacher capacity and instructional practice in the integration of educational technology for emergent bilingual students. Comput. Educ. 126, 417–432. doi: 10.1016/j.compedu.2018.08.002

Demirkol, M., and Kazu, I. Y. (2014). Effect of blended environment model on high school students’ academic achievement. Turk. Online J. Educ. Technol. 13, 78–87.

Dos, B. (2014). Developing and evaluating a blended learning course. Anthropologist 17, 121–128. doi: 10.1080/09720073.2014.11891421

Ellis, R., Bliuc, A. M., and Han, F. (2021). Challenges in assessing the nature of effective collaboration in blended university courses. Australas. J. Educ. Technol. 37:1–14. doi: 10.14742/ajet.5576

Ellis, R. A., and Goodyear, P. (2016). Models of learning space: integrating research on space, place and learning in higher education. Rev. Educat. 4, 149–191. doi: 10.1002/rev3.3056

Feng, X., Wang, R., and Wu, Y. (2018). A literature review on blended learning: based on analytical framework of blended learning. J. Dist. Educ. 12, 13–24. doi: 10.15881/j.cnki.cn33-1304/g4.2018.03.002

Fornell, C., and Lacker, D. F. (1981). Evaluating structural equation models with unobservable variables and measurement error. J. Mark. Res. 18, 39–50. doi: 10.1177/002224378101800104

Garrison, D. R., and Kanuka, H. (2004). Blended learning: uncovering its transformative potential in higher education. Internet High. Educ. 7, 95–105. doi: 10.1016/j.iheduc.2004.02.001

Hadwin, A., and Oshige, M. (2011). Self-regulation, coregulation, and socially shared regulation: exploring perspectives of social in self-regulated learning theory. Teach. Coll. Rec. 113, 240–264. doi: 10.1007/s11191-010-9259-6

Han, F., and Ellis, R. (2021). Patterns of student collaborative learning in blended course designs based on their learning orientations: a student approaches to learning perspective. Int. J. Educ. Technol. High. Educ. 18:66. doi: 10.1186/s41239-021-00303-9

Hara, N. (2000). Students distress with a web-based distance education course. Information Communication and Society Communication & Society . 557–579. doi: 10.1080/13691180010002297

Kaiser, H. F., and Rice, J. (1974). Little jiffy, mark IV. Educ. Psychol. Meas. 34, 111–117. doi: 10.1177/001316447403400115

Keith, T. (2015). Multiple regression and beyond: an introduction to multiple regression and structural equation modeling (2nd). New York: Routledge.

Kenney, J., and Newcombe, E. (2011). Adopting a blended learning approach: challenges, encountered and lessons learned in an action research study. J. Asynchronous Learn. Netw. 15, 45–57. doi: 10.24059/olj.v15i1.182

Keogh, J. W., Gowthorp, L., and McLean, M. (2017). Perceptions of sport science students on the potential applications and limitations of blended learning in their education: a qualitative study. Sports Biomech. 16, 297–312. doi: 10.1080/14763141.2017.1305439

PubMed Abstract | CrossRef Full Text | Google Scholar

Kintu, M., Zhu, C., and Kagambe, E. (2017). Blended learning effectiveness: the relationship between student characteristics, design features and outcomes. Int. J. Educ. Technol. High. Educ. 14, 1–20. doi: 10.1186/s41239-017-0043-4

Lakhal, S., Mukamurera, J., Bedard, M. E., Heilporn, G., and Chauret, M. (2020). Features fostering academic and social integration in blended synchronous courses in graduate programs. Int. J. Educ. Technol. High. Educ. 17:5. doi: 10.1186/s41239-020-0180-z

Laurillard, D. (2013). Rethinking university teaching: a conversational framework for the effective use of learning technologies . London: Routledge

Liliana, C. M. (2018). Blended learning: deficits and prospects in higher education. Australas. J. Educ. Technol. 34, 42–56. doi: 10.14742/ajet.3100

Liu, Y. (2021). Blended learning of management courses based on learning behavior analysis. Int. J. Emerg. Technol. Learn. 16, 150–165. doi: 10.3991/IJET.V16I09.22741

Manwaring, K. C., Larsen, R., Graham, C. R., Henrie, C. R., and Halverson, L. R. (2017). Investigating student engagement in blended learning settings using experience sampling and structural equation modeling. Internet High. Educ. 35, 21–33. doi: 10.1016/j.iheduc.2017.06.002

Matsushita, K. (2017). Deep active learning: toward greater depth in university education . Singapore: Springer

McMillan, J. H., and Schumacher, S. (2010). Research in education: evidence-based inquiry (7th). Upper Saddle River, NJ: Pearson Education, Inc.

Naffi, N., Davidson, A.-L., Patino, A., Beatty, B., Gbetoglo, E., and Duponsel, N. (2020). Online learning during COVID-19: 8 ways universities can improve equity and access. The Conversation. Available at: https://theconversation.com/online-learning-during-covid-19-8-ways-universities-can-improve-equity-and-access-145286

Oliver, M., and Trigwell, K. (2005). Can “blended learning” be recommended? E-Learn. Digit. Media 2, 17–26. doi: 10.2304/elea.2005.2.1.17

Oxford Group, (2013). Blended learning-current use, challenges and best practices. Available at: ( http://www.kineo.com/m/0/blended-learning-report-202013.pdf )

Siemens, G. (2005). Connectivism: A learning theory for the digital age. Int. J. Educ. Technol. High. Educ. 2, 3–10. Retrieved from: http://itdl.org/Journal/Jan_05/Jan_05.pdf

Tamim, R. M. (2018). Blended learning for learner empowerment: voices from the middle east. J. Res. Technol. Educ. 50, 70–83. doi: 10.1080/15391523.2017.1405757

Turvey, K., and Pachler, N. (2020). Design principles for fostering pedagogical provenance through research in technology supported learning. Comput. Educ. 146:103736. doi: 10.1016/j.compedu.2019.103736

U.S. Department of Education, Office of planning, evaluation, and policy development, evaluation of evidence-based practices in online learning: a meta-analysis and review of online learning studies , Washington, DC U.S. Department of Education (2009).

Xiao, J. (2016). Who am I as a distance tutor? An investigation of distance tutors’ professional identity in China. Distance Educ. 37, 4–21. doi: 10.1080/01587919.2016.1158772

Yan, Y., and Chen, H. (2021). Developments and emerging trends of blended learning: a document co-citation analysis (2003-2020). Int. J. Emerg. Technol. Learn. 16, 149–164. doi: 10.3991/ijet.v16i24.25971

Zawacki-Richter, O. (2009). Research areas in distance education—a Delphi study. International review of research in open and distributed learning . 10, 1–17.

Zhang, Q., Zhang, M., and Yang, C. (2022). Situation, challenges and suggestions of college teachers’ blended teaching readiness. E-Educ. Res. 12, 46–53. doi: 10.13811/j.cnki.eer.2022.01.006

Keywords: blended learning effectiveness, measurement, undergraduates, student learning outcomes, structural equation modeling

Citation: Han X (2023) Evaluating blended learning effectiveness: an empirical study from undergraduates’ perspectives using structural equation modeling. Front. Psychol . 14:1059282. doi: 10.3389/fpsyg.2023.1059282

Received: 01 October 2022; Accepted: 02 May 2023; Published: 18 May 2023.

Reviewed by:

Copyright © 2023 Han. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY) . The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.

*Correspondence: Xiaotian Han, [email protected]

Disclaimer: All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article or claim that may be made by its manufacturer is not guaranteed or endorsed by the publisher.

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings

Preview improvements coming to the PMC website in October 2024. Learn More or Try it out now .

• Advanced Search
• Journal List
• Int J Environ Res Public Health

Effectiveness of Blended Learning in Nursing Education

María consuelo sáiz-manzanares.

1 Departamento de Ciencias de la Salud, Facultad de Ciencias de la Salud, Universidad de Burgos, C/ Comendadores s/n, 09001 Burgos, Spain; se.ubu@ralocsec

María-Camino Escolar-Llamazares

Álvar arnaiz gonzález.

2 Departamento de Ingeniería Informática, Escuela Politécnica Superior, Universidad de Burgos, Avda. Cantabria s/n, 09006 Burgos, Spain; se.ubu@garavla

Associated Data

Currently, teaching in higher education is being heavily developed by learning management systems that record the learning behaviour of both students and teachers. The use of learning management systems that include project-based learning and hypermedia resources increases safer learning, and it is proven to be effective in degrees such as nursing. In this study, we worked with 120 students in the third year of nursing degree. Two types of blended learning were applied (more interaction in learning management systems with hypermedia resources vs. none). Supervised learning techniques were applied: linear regression and k-means clustering. The results indicated that the type of blended learning in use predicted 40.4% of student learning outcomes. It also predicted 71.9% of the effective learning behaviors of students in learning management systems. It therefore appears that blended learning applied in Learning Management System (LMS) with hypermedia resources favors greater achievement of effective learning. Likewise, with this type of Blended Learning (BL) a larger number of students were found to belong to the intermediate cluster, suggesting that this environment strengthens better results in a larger number of students. BL with hypermedia resources and project-based learning increase students´ learning outcomes and interaction in learning management systems. Future research will be aimed at verifying these results in other nursing degree courses.

1. Introduction

In approximately the last decade there has been a marked interest in investigating ways of teaching other than traditional face-to-face. The incorporation of technological resources such as virtual platforms and hypermedia resources, combined with other innovative, methodological techniques such as project-based or problem-based learning, have revolutionized the teaching–learning process. The aim is to teach in the most efficient way possible and to make the most of resources while ensuring sustainability. These technological and methodological resources have been applied to different disciplines, especially in the field of health sciences (medicine, pharmacy, psychology, veterinary, etc.). However, in recent years these resources have been incorporated into nursing studies. Next, an approach will be made for the most relevant concepts of teaching in a virtual platform, which has been called blended teaching, as well as the implementation of methodological resources for project-based learning. Likewise, special importance will be given to its application for the formation of future nursing programs by analyzing the pros and cons of this form of teaching and learning in the society of the 21st century. For this reason, the most relevant concepts of these new forms of teaching and their specific application to the nursing degree will be dealt with below. The final objective of this work is to study the effectiveness of different blended learning environments in the teaching of future nurses.

Twenty-first century society requires students and graduates to develop a series of skills related to two important leitmotifs: collaborative work and operation of information and communications technology (ICT). It is increasingly necessary to possess effective and rapid problem-solving skills and to develop digital competences [ 1 ]. The use of learning management systems (LMS) is, therefore, a reference in instructional practice, especially in higher education, as is the implementation of collaborative work in these methodological settings for the resolution of tasks and problems. A good example might be the use of project-based learning (PBL) methodology [ 2 ]. Recent investigations have confirmed that if such a methodology is accompanied by the use of hypermedia resources (e.g., flipped learning experiences, quizzes, use of wikis, on-line glossaries, etc.), then acquisition of deep learning is strengthened in students [ 3 ]. Deep learning is a concept developed in the framework of the taxonomy of Bloom [ 4 ]. It corresponds to the highest level of learning competences (comprehending, analyzing, summarizing, and evaluating their own learning). One of the currents of thought in LMS learning environments suggests that learning in these environments implies deeper learning from the point of view of cognitive and metacognitive complexity, as these facilitate self-regulated learning (SRL) and meaningful learning [ 5 ].

Likewise, LMS permit a more precise analysis of interactions which are logged in records (or logs). The logs represent units of information and registration that stores precise data on the frequency of user interactions and their duration [ 6 ]. LMS also facilitates the inclusion of hypermedia resources [ 7 ]. The use of these resources is especially relevant in health science degrees (nursing, medicine, pharmacy, etc.) since it implements practical assumptions in the work, which has been proven to reduce errors in the workplace [ 8 ].

There are various stages in this instruction process that will facilitate or inhibit the efficiency and depth of the learning process. One of them is the design of learning tasks in LMS [ 9 , 10 ]. Another essential element is that the teacher plans for process-oriented feedback [ 11 ].

1.1. Teaching through Learning Management Systems

The teacher has to reflect, among other points, on the following points: (1) the aims of the subject module, (2) to whom it is addressed, (3) what previous knowledge is required for a successful approach to the subject matter, (4) the type of learning tasks that facilitate content acquisition, (5) the metacognitive skills of the students prior to the instruction, (6) the cognitive and the metacognitive skills in each task needed for its effective solution, and (7) when and where the teaching–learning process will be developed. Likewise, the teacher has to plan follow-up with both the individual student and the group behavior on the platform. As has been argued, solving problems in a collaborative way is one of the most demanding skills in 21st century society. These types of competences are key references in educational and technological areas and for entry into employment. Collaborative work facilitates the construction of deep and effective learning in the students [ 10 ]. A scheme for the preparation of pedagogic design in the LMS may be seen in Table 1 .

Preliminary elements to take into account for the design of learning activities.

Nevertheless, computational techniques are required to conduct a conclusive analysis of student behavior in LMS. As previously mentioned, at present a broad percentage of learning is done in virtual environments, in what is called blended learning teaching. A lot of data can be recorded by LMS and accessed through logs. However, educational data mining (EDM) [ 12 , 13 ] is needed to study them precisely. Machine learning techniques can be applied to EDM. Subsequently, possible applications of those techniques will be presented in the analysis of learning data in LMS environments.

1.2. Application of Artificial Intelligence Techniques to Analyze the Teaching and Learning Process

Development of the internet and information and communications technology (ICT) has expanded learner access to information, and they have changed the way that information is taught and the way it is learned [ 14 ]. A learning management system (LMS) is an interactive learning environment that facilitates both teaching and learning. In addition, these software environments record all the actions performed by the teacher and by the students, under individual and group headings. However, those logs store a lot of data and learning analytics have to be used in order to study them in a flexible and accurate manner. These techniques can be simple, such as the ones usually found in LMS (descriptive statistics). However, more complex analytical techniques can be used, such as machine learning techniques (a subset of artificial intelligence). The latter are analogous to the computational thought of the human brain and operate with what is known as artificial intelligence. Machine learning techniques of classification and clustering [ 15 ] are among the most widely applied techniques for data analysis in educational environments. The use of these techniques for the analysis of both student and teacher behaviors will provide the teacher and those responsible for educational institutions with ideas to introduce improvements into the learning environment [ 11 ].

In brief, machine learning techniques are used, as these techniques are currently considered to provide the researcher with more data in the field of cognitive psychology and learning than traditional statistical techniques [ 16 , 17 ]. In particular, machine learning techniques permit personalized learning and provide individualized information on the development of student learning. Prediction techniques facilitate early detection of at-risk students and, therefore, personalized [ 18 ] help from the teacher [ 19 , 20 ]. Machine learning techniques also provide information on the effects of predicting the independent variable over each of the dependent variables in percent effects [ 21 ].

1.3. Design of the Blended-Learning Space in Nursing Instruction

Blended teaching, increasingly present in educational scenarios, is done through a blend of face-to-face (F2F) and virtual learning on LMS, known as blended learning. However, there is no generalized agreement on the taxonomy of blended learning [ 22 ]. Nevertheless, its differences with blended learning are accepted; in the blended learning environment, the student completes 80% with LMS and 20% is F2F, hereafter referred to as Blended Learning type 1. In contrast, blended learning (80% interaction in the LMS) is a space where feedback is done 80% of the time through F2F and 20% through LMS [ 18 , 23 ], hereafter referred to as Blended Learning type 2. Recent investigations have found that the replacement blended environment accompanied by the use of active methodologies (e.g., PBL, use of hypermedia resources, flipped learning experiences, and quizzes, or all at once) improved the learning results of students [ 3 , 24 ]. These achievements are especially significant in university environments [ 25 , 26 ] because future graduates will have to develop collaborative work skills, problem-solving independence, and the use of new technologies. These skills are essential for good development of entrepreneurship.

Along these lines, recent studies have shown that [ 27 ] an educational intervention that applies blended learning methodology can easily be added into nursing curricula. This type of learning enhances learning in this field. Recent systematic research indicated that blended learning together with PBL is a methodology that ensures effective learning among nursing students [ 28 ]. This type of paradigm is more effective than traditional teaching such as face to face. The reasons are that students need to develop the knowledge and skills necessary in clinical practice. Several studies recommend nursing teachers to use multifaceted techniques (blended learning, learning based in projects, etc.) to promote effective learning beyond face-to-face teaching [ 29 ]. While these studies highlight the need to train teachers in these techniques [ 28 ], the main reason is that, traditionally, teaching has been done face to face, and an organized transfer towards the use of these methodological resources is needed. A recent systematic review showed that, since 2018, there has been a growing interest in the implementation of these experiences in nursing studies. However, an increase in these experiences and more research in this discipline of knowledge are needed [ 30 ].

Moreover, blended learning environments permit an evaluation of the whole teaching–learning process in a systematic and simple way. Thus, the suggestion is that there are different variables that influence successful learning in this line of investigation into e-evaluation models, especially the learning strategies employed by the students themselves [ 31 , 32 ], the environment in which the learning takes place [ 33 ], the teaching design that the teacher brings to the class [ 30 ], and the behavioral learning of the students in the LMS [ 23 ]. The prediction interval of these variables is situated around 56%-61% [ 34 ].

1.4. Extraction and Analysis of Information on the Teaching–Learning Process Recorded in LMS

As we have mentioned earlier, development of the teaching through LMS will facilitate the student in learning recording and follow-up behaviours [ 35 ]. Many of these learning managers use supervised machine learning techniques, techniques such as multiple regression analysis (MRA), neural network, and SVM. Those techniques help with the detection and subsequent prediction of successful and risk behaviours. Behaviours of the students in LMS that have been related to successful learning are, among others [ 23 ]:

• the time that is used in carrying out the tasks;
• student time expended on studying theoretical content;
• the results in the self-evaluation test (quiz efforts);
• the quality of forum discussions (type and length of message);
• time employed in analysing the feedback given by the teacher;
• the number and type of messages sent;
• the frequency of access to LMS;
• contribution to content creation;
• files opened; and
• delivery time of the activities.

Therefore, the frequency and systematicness of student interactions and their interactions with LMS are directly related to effective learning [ 36 ]. Along these lines, recent investigations [ 23 ] have revealed differences in predicting learning results in relation to the variable “teaching methodology” (understood in terms of the pedagogic structure of the teaching, the evaluation procedures, and feedback). The type of activities and the evaluation tests (quizzes, tests, projects, presentations…) are understood to determine the effectiveness of behavioural learning logged on the LMS.

As previously mentioned, application of machine learning techniques to study the logs will allow the teachers to analyze the behavioural learning of their students and to detect at-risk students. In these cases, early intervention will presumably improve student learning responses. Recent studies have confirmed [ 17 , 35 ] that following up with student behavioural learning in the LMS facilitates the identification of at-risk students with an explained variance of 67.2%.

In summary, the use of machine learning techniques will permit the study of behavioral learning of both students and teachers on the platform, which will facilitate the application of prediction techniques to the learning results [ 37 ]. Reviewing the investigations presented earlier, we consider it important to study the behavior of PBL in the LMS. As has been indicated, there are few studies in that field, and more information is needed that will help to improve teaching practices in these environments [ 38 ]. Project work and personalization of learning in LMS have been proven to have significant effects on the quality of learning. Particular relevance has been in Health Science degrees, such as nursing or medicine, etc., since it facilitates work on clinical cases in a collaborative way and optimizes the results applied to real learning contexts [ 39 ].

This research study was performed to analyze data of students’ online and face-to-face (F2F) activity in a blended nursing learning course. We applied two types of blended learning: Blended Learning type 1 [in which the interaction between the teacher and students is 80% in the LMS and 20% Face to Face (F2F)] and Blended Learning type 2 [in which the interaction between the teacher and students is 20% in the LMS and 80% Face to Face (F2F)].

In light of the above, the hypotheses in this study were the following:

H 1: The types of blended learning (Blended Learning type 1 vs. Blended Learning type 2) used will predict student learning outcomes;

H 2: The types of blended learning (Blended Learning type 1 vs. Blended Learning type 2) used will predict the learning behaviours logged on the LMS; and

H 3: The type of clusters will be different for each type of blended learning used (Blended Learning type 1 vs. Blended Learning type 2).

2. Materials and Methods

2.1. design.

A quasi-experimental post-treatment design with an equal control group (in terms of metacognitive skill) was used. Likewise, learning outcomes (learning outcomes in the development of project-based learning; learning outcomes in exhibition of project-based learning; learning outcomes in the test; and learning outcomes total) and behavioral learning in the LMS were the dependent variables (access to complementary information; Access to guidance to prepare PBL; Access to theoretical information; Access to teacher feedback; and mean visits per day).

2.2. Participants

A sample of 120 university students was assembled following the third year of their nursery degree in Spain (the degree has four years) during one semester (9 weeks): 63 followed the Blended Learning type 1 methodology and 57 followed the Blended Learning type 2 methodology (see Table 2 ). The students were assigned to each blended learning group (Blended Learning type 1 vs. Blended Learning type 2) by means of convenience sampling. The work was developed in the subject of “Quality management methodology of nursing services.”

Group assignment and descriptive statistics for age, a n = 60. b n = 62.

Note. M age = Mean Age; SD age = Standard Deviation.

2.3. Instruments

a. LMS UBUVirtual version 3.1 . A Moodle-based learning management system (LMS) was used that began with a constructivist approach and was developed through a modular system. It is a personalized Moodle-based LMS. An LMS is a modular learning environment that permits interaction and feedback between teacher and students, in many cases in real time, and in addition it facilitates the process of automated feedback.

b. The (ACRAr) Scales of Learning Strategies by Román & Poggioli [ 40 ]. This widely tested instrument identifies 32 strategies at different points in the information processing cycle. The reliability indicators on the scale were between α = 0.75 to α = 0.90 and the indicators of content validity were between r = 0.85 and r = 0.88. The subscale of metacognitive skills was applied in this study; this scale incorporated 17 strategies about the use of metacognitive skills into the problem solving tasks. A reliability index of α = 0.80 was obtained in this study; the reliability indicator on this subscale was α = 0.90 and the indicator of validity was r = 0.88.

c. Student learning results: the results were recorded in the different evaluation procedures . (1) Multiple-choice tests on the theoretical contents of the subject (test) were assigned a weight of 30% of the final grade. The test had 10 multiple-choice questions (four possible answers) with only one correct response. As well, five questionnaire-type quizzes were administered, one for each thematic unit. Cronbach’s Alpha reliability of the test was α = 0.81. (2) Development of PBL, with a weight of 25%, was measured with a rubric, which can be seen in Supplemental Material Table S1 . (3) Likewise, the exhibition of the PBL, with a weight of 20%, was also measured with a rubric and can be seen in Supplemental Material Table S2 . In the final mark, Cronbach’s Alpha reliability of PBL was α = 0.62. This result is lower because there was less dispersion among the scores in this type of evaluation test. Since the performance of the groups was quite uniform, this aspect can be checked in the results section and it is in accordance with the philosophy of PBL. Finally, the learning outcomes total covered the weighted scores of all the results (over 10 points). 4) The students solved five practices, and this part was 25% of the final grade. However, in this part all students had the highest qualification since the teacher reviewed the practices continuously, and if they were not correct the teacher ordered them to be repeated. Therefore, because it is not discriminate it has not been included in the analysis. Examples of the PBLs developed can be found at this link https://riubu.ubu.es/handle/10259/3753/discover .

2.4. Procedure

Convenience sampling was followed for the choice of the sample. This was due to the possibility of working with this methodology by a specialist teacher who attended to both groups, and in this way the "type of teacher" effect was avoided. Before the instructional intervention, the two groups (Blended Learning type 1 vs. Blended Learning type 2) were scored on the metacognitive skills Scale of ACRAr [ 40 ], with the aim of establishing the similarities between both groups in terms of metacognitive skills.

As stated in the introduction, Blended Learning type 1 was applied to the experimental group, a learning environment in which the interactions between teacher and student were 20% F2F and 80% LMS. Likewise, Blended Learning type 2 was applied to the control group, a learning environment in which the interactions between teacher and students were 20% LMS and 80% F2F. In the experimental Group, hypermedia resources were used such as videos, and feedback was through the LMS. In contrast, classroom interactions between teacher and students and feedback in the control group were all F2F. In both groups, PBL methodology was followed. The difference, as has been pointed out, consisted of the type of blended learning in use (Blended Learning type 1 vs. Blended Learning type 2). Project development was done in both (the control and the experimental) groups in a collaborative way. The project work was completed in small groups of students of between 2 and 5 members.

2.5. Data Analysis

The following statistical analyses were applied: (1) Analysis of asymmetry and kurtosis; (2) analysis of the variance of a fixed-effect factor (ANOVA); (3) multiple regression analysis (MRA) [appropriate Tolerance (T) values were considered close to one and, with respect to the variance inflation factor, the values were between 1–10]; (4) cluster analysis. Package for the Social Sciences (SPSS) v.24 was used to perform the different analyses [ 41 ]. Likewise, the Goodness-of-fit indices were measured by structural equation modeling (SEM) and was used to study the settings of the machine learning technique to predict the learning results. The calculations were performed with the Statistical Package for the Social Sciences (SPSS) AMOS v.24 [ 42 ]. (5) Finally, to visualize the results in a cluster analysis, RapidMiner Studio software [ 43 ] was used.

2.6. Ethical Considerations

The research project was approved by the Ethics Committee of the University of Burgos. Previously, at the start of the project, the students were informed of the objectives, and their participation was at all times on a voluntary basis. Likewise, informed consent of each participant was recorded in writing.

3.1. Previous Statistical Normalcy Analysis in the Sample

Before starting the research, the indicators of normality were studied. The results obtained from earlier statistical analyses with regard to the normality of the sample are presented below (values higher than |2.00| indicate extreme asymmetry, the lowest values indicate normality, and the values of between |8.00| and |20.00| suggest extreme kurtosis [ 44 ]). The results of metacognitive skills on the ACRAr subscale in both groups were acceptable for both indicators (see Table 3 ). Therefore, parametric statistics were used. Descriptive statistics are also shown in Table A1 and Table A2 (see Appendix A ).

Indicators of asymmetry and kurtosis in the experimental group and control group.

Note. M = Mean Age; SD = Standard Deviation; A = Asymmetry ; K = Kurtosis; ASE = Asymmetry Standard Error; SEK = Kurtosis Standard Error.

3.2. Previous Statistical Analysis of Homogeneity between the Groups before the Intervention

Significant differences between both groups (experimental and control) in their use of metacognitive strategies were anlayzed before application of the different types of blended learning (Type 1 vs. Type 2). To do so, a single-factor ANOVA with fixed-effects was performed (blended learning type) on the results. No significant differences were found between both, so they can be considered similar groups (F 1 , 119 = 0.276; p = 0.601; η 2 = 0.002) in the ACRAr subscale of metacognitive skills.

Similarly, in order to study which type of supervised learning technique would be the most appropriate, the Goodness-of-fit indices were measured in the structural equation modeling (SEM) that was used to study the settings of the machine learning technique to predict the learning results. The calculations were performed with the Statistical Package for the Social Sciences (SPSS) AMOS v.24, as may be seen in Table 4 , and no dependent relations between the observed values and the different prediction methods (LR, DT, RBFN, and kNN) were found for any of the four prediction models. Among these possibilities, the following were applied in the MRA.

Goodness-of-fit indices.

Note. df = degrees of liberty; χ 2 = Chi squared; LR = Linear Regression; DT = Decision Trees; RBFN = Radial basis function network; kNN = k-Nearest Neighbor classification; NFI = normed-fit-index; RMSEA = Root-Mean-Square Error of Approximation; SRMR = Standardized Root-Mean-Square Residual; TLI = Tucker–Lewis index; CFI = comparative fit index; AIC = Akaike Information criterion; ECVI = parsimony index.

3.3. Hypothesis 1.

MRA was performed to study the predictive value of the variable blended learning type applied to the student learning outcomes. An R 2 = 0.404 was found, which indicates that this variable explained 40.04% of the variance in the learning results. The Tolerance ( T) values were within an interval of 0.106 and 0.336 and the Variance Inflation Factor ( VIF ) between 3.491 and 9.45, so none of the variables had to be removed. Likewise, the highest partial correlation was found in the Learning Outcomes Total ( r = 0.586; p = 0.000), see Table A3 .

3.4. Hypothesis 2.

MRA yielded a figure of R 2 = 0.719 in the study of the predictive value of blended learning applied to student behaviors on the platform. This figure indicated that the blended learning type in use explained 71.19% of the variance in the learning behaviors of students on the platform. The Tolerance (T) values were situated within an interval between 0.136 and 0.539 and the Variance Inflation Factor (VIF) between 1.472 and 7.346, so that no variable had to be removed. The highest partial correlation was found in Access to Teacher Feedback ( r = 0.448), see Table A4 .

3.5. Hypothesis 3.

A k-means clustering technique was applied in each type of blended learning in use (Blended Learning type 1 vs. Blended Learning type 2), as seen in Table 5 . Three clusters are shown in Table 5 that were found in the two types of blended learning (Cluster 1, Sufficient; Cluster 2, Intermediary; and Cluster 3, Excellent. The classification of Cluster type was according to the maximum possible value in each learning outcome and number of accesses obtained). Higher values for performance were found in the Blended Learning type 1 rather than the Blended Learning type 2 in all three clusters, specifically in Learning Outcomes Total. Likewise, with regard to the learning behaviors developed by students in the type of blended learning in use (Blended Learning type 1 vs. Blended Learning type 2), as may be seen in Table 5 , a higher number of log-ons to the platform in the Blended Learning type 1 rather than the Blended Learning type 2 environment were found, except for student queries on theoretical information provided by the teacher (see Table 6 ).

Centers of final clusters for the learning results variable in Blended Learning type 1 and type 2, Blended Learning type 1: a n = 1; b n = 45; c n = 17; Blended Learning type 2: A lost value is observed. a n = 9; b n = 30; c n = 18

Note. PBLD = Project-Based Learning Development; PBLE = Project-Based Learning Exhibition.

Centers of final clusters and the variable behavioral learning on the LMS in Blended Learning type 1 and type 2. Blended Learning type 1: a lost value is observed. a n = 31; b n = 27; c n = 5; Blended Learning type 2: Two lost values were observed. a n = 36; b n = 16; c n = 6.

Note; PBL = Project-Based Learning Development.

Figure 1 shows the scores in the two groups: experimental group (red color) and control group (blue color). As can be seen, there was a greater homogeneity of higher scores in the experimental group for different types of Learning outcomes. Similarly, Figure 2 points to the distributions of LMS behavioral learning scores in different resources.

Distribution of scores in the different types of Learning outcomes. Note. Development PBL = Development Project-Based Learning outcomes; Exhibition PBL = Exhibition Project-Based Learning outcomes; Test = Test Learning outcomes; Total LO = Learning outcomes Total.

Distribution of scores in the different types of behavioral learning in the LMS. Note. CI = Access to Complementary Information scores; CPBL = Access to guidance to prepare PBL scores; TI = Access to Theoretical Information scores; F = Access to Teacher Feedback; MVD = Mean Visits per day.

4. Discussion

In the blended learning environments, the type of teaching design appears to be a predictive factor in both the learning results and the learning behaviors that the students develop in the LMS. blended learning with 80% of interactions in LMS appeared to be more effective, both with respect to the learning results of the students and the effectiveness of the learning behaviors that they develop. This type of pedagogic design includes the use of hypermedia resources that strengthen teacher feedback in real time, which furthers the development of SRL strategies [ 24 , 35 ]. This aspect is of special relevance for teachers in nursing higher education, and the implicit message is that they would be well advised to design their materials for use in a blended learning environment [ 27 , 28 , 29 , 30 ], as those environments appear to have increased the effectiveness of active methodologies, especially PBL with hypermedia resources in LMS [ 38 ]. In addition, Blended Learning type 1 (80% the interaction in the LMS) strengthens students’ use of learning-based projects that have been considered more effective in the LMS [ 2 , 23 ]. These behaviors range from access to feedback given by the teacher to tasks carried out by the student or the collaborative groups and the average number of visits per day [ 23 ]. All of this indicates that the Blended Learning type 1 design increases the interaction of the student in the LMS and that interaction also facilitates student access to feedback from the teacher, as the LMS can be consulted as many times as necessary when learning, an aspect that is less feasible with F2F instruction [ 9 , 10 , 18 ]. In this way, the teachers can structure their help and prepare specific materials for each group.

In addition, machine learning techniques have been used in this study, in view of their effective use with what is known as data mining [ 13 , 18 , 36 ]. In particular, supervised and unsupervised machine learning techniques have been used (linear regression and clustering k-means methods, respectively). Prediction and clustering studies, among others, can be conducted with these techniques, which help the teacher to gain knowledge of the learning characteristics of students and to predict at-risk students [ 23 , 34 , 38 ]. Even so, it is true that these techniques should be used throughout the whole teaching process to be able to develop personalized actions for student learning [ 35 , 36 ]. In subsequent studies, therefore, development of the learning process among students at the start, in the middle, and at the end of the study module will be analyzed with machine learning techniques [ 12 , 13 ].

5. Limitations

This study has limitations, but the results of this study should, nevertheless, be given prudent consideration. Limitations include the following: methodological intervention was in one university, the students were from a specific country, convenience sampling was applied, the knowledge area of the students was specific, and the type of design (quasi-experimental) was also specific. Although, it must be taken into consideration that there are few specific studies to test the effectiveness of this type of methodology in nursing students. Studies that have been carried out have similar characteristics that are justified from the specificity of this research [ 28 , 29 , 30 ].

Therefore, future studies will be directed at increasing the size of the sample and the diversity of the nursing degree course level. Therefore, this profession is subject to continuous theoretical and technological advances that require systematic research on how to teach better in order to learn more effectively.

6. Conclusions

This research study has identified the characteristics to design an effective LMS in the nursing degree. The use of prediction and clustering techniques is very important to facilitate personalized learning and to analyze how resources are better utilized in the blended learning space. This type of analysis can be automatically generated in LMS environments, such as Moodle, and could be integrated in modules and plugins. These tools would facilitate rapid and straightforward generation of those analyses, which would be of great utility for the teacher and would assist with the early detection of at-risk students, as well as behavioral analyses of both the individual student and the collaborative groups of students, which would foreseeably increase the teaching quality and learning outcomes. This need has been underlined in such studies as those by Peña-Ayala [ 12 ] and Romero et al. [ 17 ], and they have to be approved by university management, but they will virtually be a necessity in 21st century teaching as we move closer to personalized on-line teaching, both in F2F teaching and in virtual learning environments. In summary, this teaching design is especially significant in the nursing degree since project work is a practice that has proved very effective in the training of future professionals.

Good results have been obtained in all assessment tests in the two types of blended learning. However, the type of blended learning that applied automated feedback and hypermedia resources obtained even better results (more percentage of work in the LMS) [ 28 , 29 , 30 ]. One explanation may be that the student can access information in the LMS at any time, which is not possible for F2F interaction, and this facilitates personalization of learning and motivates the student [ 7 , 8 , 39 ]. Therefore, incorporating these forms of work in teaching in the field of health is a very effective option.

The results obtained are in line with those found in the research of Oh & Lee [ 28 ]. The use of PBL methodology in blended learning environments empowers nursing students to acquire practical skills that are of great help for nursing work in real intervention environments [ 28 ]. This form of teaching is flexible [ 30 ] because it facilitates development and tests hypotheses in the resolution of tasks similar to those they will encounter in a working environment, and in addition, group work facilitates the acquisition of collaborative work skills, which they will also encounter in such working environments. All this increases the self-efficacy and critical thinking skills of these professionals. Recent studies recommend the application of this methodology within the nursing degree curricula [ 27 ].

In sum, it can be concluded that this way of teaching seems to be effective for nursing students. Although, more studies are needed in this field aimed at studying the effectiveness of blended learning in teaching in the nursing degree.

Acknowledgments

Thanks to all the students who participated in this study and the Committee of Bioethics of University of Burgos (Spain).

Supplementary Materials

The following are available online at https://www.mdpi.com/1660-4601/17/5/1589/s1 , Table S1: Rubric to evaluate development of Project-Based Learning, Table S2: Rubric to evaluate Exhibition of Project-Based Learning.

Descriptive statistics for the Learning outcomes.

Note: M = Mean; SD = Standard Deviation; PBL = Project-Based Learning; α = Reliability index of Cronbach’s Alpha.

Descriptive statistics for Behavioral learning in the LMS.

Note: M = Mean; SD = Standard Deviation

Coefficients in the prediction of learning outcomes with variable types of Blended Learning.

Note: Dependent variable: Blended Learning type; PBL = Project-Based Learning; VIF = Variance Inflation Factor.

Coefficients in the prediction of Behavioral learning with variable types of Blended Learning.

Note: Dependent variable: Blended Learning type; VIF = Variance Inflation Factor.

Author Contributions

Conceptualization, M.C.S.-M., and M.-C.E.-L.; methodology, M.C.S.-M.; software, Á.A.G.; validation, M.C.S.-M., and Á.A.G.; formal analysis, M.C.S.-M.; investigation, M.C.S.-M., and M.-C.E.-L.; resources, M.C.S.-M.; data curation, M.C.S.-M.; writing—original draft preparation, M.C.S.-M., and M.-C.E.-L. and Á.A.G.; writing—review and editing, M.C.S.-M., and M.-C.E.-L. and Á.A.G.; visualization, Á.A.G.; supervision, M.C.S.-M., and M.-C.E.-L.; project administration, M.C.S.-M.; funding acquisition, M.C.S.-M., and M.-C.E.-L. All authors have read and agreed to the published version of the manuscript.

This research was funded by the of Consejería de Educación de la Junta de Castilla y León (Spain) (Department of Education of the Junta de Castilla y León), Grant number BU032G19, and grants from the University of Burgos for the dissemination and the improvement of teaching innovation experiences of the Vice-Rectorate of Teaching and Research Staff, the Vice-Rectorate for Research and Knowledge Transfer, 2020, at the University of Burgos (Spain).

Conflicts of Interest

The authors declare no conflicts of interest.