website design research paper

The Design and Implementation of Responsive Web Page Based on HTML5 and CSS3

Ieee account.

• Change Username/Password
• Update Address

Purchase Details

• Payment Options
• Order History
• View Purchased Documents

Profile Information

• Communications Preferences
• Profession and Education
• Technical Interests
• US & Canada: +1 800 678 4333
• Worldwide: +1 732 981 0060
• Contact & Support
• About IEEE Xplore
• Accessibility
• Terms of Use
• Nondiscrimination Policy
• Privacy & Opting Out of Cookies

A not-for-profit organization, IEEE is the world's largest technical professional organization dedicated to advancing technology for the benefit of humanity. © Copyright 2024 IEEE - All rights reserved. Use of this web site signifies your agreement to the terms and conditions.

• Web Design Research Paper: What Tips You Should Use

When learning web design in college or university, you won't avoid a task every student deals with: It's a research paper writing. While it sounds challenging for a web designer to write papers, it's necessary for those willing to systemize their knowledge and get a diploma.

In this article, you'll reveal the nature of research papers in web design and get actionable tips on how to write them like a pro. Please note this type of writing is more in-depth than a blog post and a standard essay. So, read carefully — and feel free to ask professionals for writing help when you need clarification on whether you do everything right.

But first, the basics:

What Is a Research Paper?

According to professional academic writers from https://essayshark.com/ , a research paper is an in-depth and evidence-based analytical study on a given topic.

It reminds a standard analytical essay you write in school or college. The difference is that a research paper requires original research from the author. Before writing, you need to analyze pre-existing research, conduct yours, and present the results, emphasizing the use of statistical data and contributing your input to the debate.

The purpose is to:

Check your writing skills.

Assess your ability in scholarly research.

Estimate your knowledge of the subject.

See your original thesis and evidence. (What does it contribute to the scientific community?)

Understand if your knowledge and skills are enough to complete a course and get a certificate or a diploma.

Why Do You Need to Write It on Web Design?

When you get a profession in design , you'll need to write a research paper at the end of a corresponding course. It's not only about impressing your educator and getting a high grade. As a web designer, you might want to innovate the field:

Researching web design will reveal something new for your colleagues and build you a reputation as an expert. It's your chance to stand out and bring trends, not just repeat existing tactics and techniques others use when web designing.

More than that, researching your interests can help you improve your knowledge, systemize it, and understand the tiniest details of your profession.

How to Write a Web Design Research Paper: 10 Tips to Use

Choose a topic.

Get ready for precise research.

Focus on a thesis statement.

Outline it before writing.

Organize the evidence beforehand.

Ensure you use the correct structure for a research paper.

Consider the language.

Use a proper citation style.

Edit your paper.

Proofread the draft several times.

And now, to practice.

Below are the practical tips for writing a web design research paper.

1. Choose a topic

When writing a research paper, it's you who chooses its topic. A professor doesn't assign any particular themes, but they set the specific and strict rules you'll need to follow. That is why ensure you read the assignment carefully and understand all the requirements.

What are those must-follow requirements?

They are about your research paper's length, citation style, and formatting. (Read the rules below.) Also, consider whether you must craft a cover page and present a paper's abstract beforehand. Check the deadline, the required submission method, and the overall goal of your future web design paper.

Once you understand your assignment task sheet inside and out, it will be easier to decide on the topic and craft the whole paper.

In a review on essayshark , web design students appreciate their assistance with choosing a topic for their research papers. Feel free to ask professionals if you need help with a web design topic worth researching and sharing with the audience.

The topic is essential for your paper's overall success:

Choose those with pre-existing research and evidence — you'll need content for your paper. 

Consider topics with enough data to discuss.

Avoid generic topics: They won't bring you enough ideas for interpretation, and they don't have any value to the community.

Think of something engaging and interesting to you. It's easier to research and write on the topic of your interest.

More tips on how to choose a compelling topic for your web design research paper:

Try brainstorming and talking to your fellows or your professor. They may have some good ideas you can use for your paper.

Prompt AI tools like ChatGPT to help you with topics. While most of their ideas are superficial, they can inspire or point you in the right direction.

Check other research papers in the niche. Their discussion sections often include ideas for further examination: See if there are any you could use for your paper.

2. Get ready for precise research

Once you have a topic, it's time for the preliminary research. Remember that not only will you need to discuss the existing knowledge, but you should also add original insights to the paper. With that in mind, do the following:

When reading the research on your topic, try to extract an issue to focus on in your paper. Are there any areas you could develop in your work in more detail? Do you have ideas for new research in this sub-topic? Does it miss anything? What didn't other researchers cover?

When researching, finish the sentence:

"I want to know why/how/what..."

Come up with research questions that will guide you at this stage. What and how are you going to cover in your web design paper? What will be your unique take on the topic?

What resources to use for the research? Consider books, reputable websites, scientific journals, etc. Use only those with the proven data and evidence you can refer to in your future document.

3. Focus on a thesis statement

The central argument of your research paper has the name of a thesis statement. Please do your best to craft it before writing because it will determine the content of your whole work.

A thesis should answer your research question. It's concise (1-2 sentences) and contentious, establishing the paper's purpose, position, and showing what reasoning you'll use to support it.

Your thesis statement appears in the introductory paragraph of your web design research paper. Thus, you let the readers know what your work will be about. If it's challenging for you to summarize your research in 1-2 sentences, here is the tip:

Make your topic a question, and then answer it.

4. Outline it before writing

An outline is the structure of your research paper. Think of it as a detailed plan where you prescribe every text block and make some notes on what you'll include there. It's super helpful to craft outlines before writing: With a paper's plan at hand, you'll reduce stress, save time, and be sure you get everything while drafting.

In the above picture, you can see a standard outline for research papers. It reveals all the parts to include in your document, their approximate length, and the details to cover:

The introduction is 2-3 paragraphs with a hook to grab readers' attention, context, and a thesis statement. Body paragraphs explain the problem's extent, repercussions, and possible solutions you offer. A research paper's conclusion should contain your thesis's restatement, arguments' summary, and a call to action for the audience.

Before writing your web design paper, take short notes on each block in the outline. Thus, you'll know what argument, evidence, and details to place in each section.

5. Organize the evidence beforehand

Remember point two? It was the tip about preliminary and precise research to find facts, data, statistics, and other evidence for your future paper. Now, it's time to organize them and decide which one to refer to in each section.

So, re-check all the sources you collected earlier and find the specific information you will include in every block. Add corresponding notes in your outline: When writing, it will help you see where to place this or that evidence. You won't miss anything, and you'll ensure you mention everything you want to say.

Plus, it will save you time. Knowing where and what to write, you will stay focused and create your paper's draft faster.

6. Ensure you use the correct structure for a research paper

Research papers have a more complex structure than academic papers like essays, reviews, or personal statements. The standard blocks are:

Introduction: Here, you'll introduce a topic and its background, provide the context, and write a thesis statement.

Paper body: Here, you'll have about eight paragraphs with information like the problem's extent, its cause and effect, your original research, and the solution you offer to the scientific community.

Concluding paragraph: Here, you summarize your points, restate the thesis, and leave the readers with the food for thought (a call to action, ideas for further research, etc.)

Besides, the following elements should be present in your web design paper:

The title (heading)

Abstract (reveals the purpose of your research)

Literature review (tell about the existing research on your topic)

Your methods (what methodology you used to conduct your research)

Results (what data you received from your research)

Discussion (your interpretation of the research and its results)

References (the list of resources you used as evidence in your paper)

7. Consider the language

Web design isn't about data visualization only. The text also matters, and its role becomes even more critical when writing a research paper.

Try using formal language, like the one from academic essays or scientific journals. This language is clear and specific , with related terms, transitional phrases between paragraphs, and logical flow between sections.

Ensure your grammar and punctuation are correct. Avoid filler words; don't write your paper in the first person ("I"). Past tense and passive voice are okay when speaking of experiments for your research. Graphs, charts, diagrams, and other visual content are also okay to support your point or illustrate your argument/research results.

8. Use a proper citation style

Remember the first tip? Read the task carefully: There will be guidelines concerning a citation style you should use when formatting your paper.

The two most common citation styles for research papers are MLA (Modern Language Association) and APA (American Psychological Association). Each is about specific formatting rules: How to structure references, what punctuation to use, whether to italicize names, what to write from a capital letter, etc.

There are also citation styles like Chicago (The Chicago Manual of Style) and AMA (American Medical Association). Depending on the subject or the guidelines your professor (or your educational institution) prescribes for research papers, they may change.

Learn the formatting rules for the citation style you need (all the manuals are available online) — and do your best to craft your paper accordingly.

9. Edit your paper

When your web design research paper is ready, please don't hurry to submit it for review. It's critical to edit it and ensure you have organized everything right.

Re-read the paper carefully and check if:

Your thesis statement is concise and clear.

The paper is structured correctly, and all necessary elements flow logically.

You use details, facts, and evidence, not generic statements and superficial information.

You use a proper citation style and format everything accordingly.

You've avoided repetitions, plagiarism, and biased information.

Below is the table showcasing the standard formatting rules for research papers. If nothing else is prescribed in your guidelines, feel free to use it to organize your document.

10. Proofread the draft several times

Last but not least:

Proofread your web design research paper to prevent spelling, grammar, and punctuation mistakes. Ensure you used all the names, dates, and facts correctly.

Read your paper out loud, line by line: It will help you "hear" if it sounds reasonable and logical. Feel free to use editing resources like grammar and readability checkers.

Final Words

Web design research papers are your instrument to prove your knowledge of the subject and offer new insights and techniques to the community. This paper is a must for those learning web design in college, and it serves as a document to complete the course and get your diploma.

Now that you have actionable, step-by-step tips on writing your research paper like a boss, please do not ignore them — and your web design copy will sound A-worthy and professional.

Free Design Stuff Ad

Popular Topics

Featured post

What's the best ecommerce platform.

Most Read Articles

Blog Archive

• ►  April (9)
• ►  March (28)
• ►  February (18)
• Take a Peek Inside Unique and Stylish NYC House In...
• Crafting Successful Business Apps: All You Need to...
• Remodeling Your Basement: 9 Design Tips to Follow
• A Few Updates to Refresh Your Bedroom
• Essential Tips for Developing an Effective Data Di...
• Mastering E-Commerce: Designing Success with Exper...
• 7 Essential Tips for Successful Ads Management Cam...
• Looking To Upgrade Your Kitchen’s Style? Consider ...
• ►  December (9)
• ►  November (17)
• ►  October (20)
• ►  September (7)
• ►  August (14)
• ►  July (11)
• ►  June (11)
• ►  May (12)
• ►  March (17)
• ►  February (8)
• ►  January (9)
• ►  December (7)
• ►  November (12)
• ►  October (30)
• ►  September (22)
• ►  August (16)
• ►  July (4)
• ►  June (8)
• ►  May (2)
• ►  April (7)
• ►  March (3)
• ►  February (5)
• ►  January (2)
• ►  December (3)
• ►  November (6)
• ►  October (4)
• ►  September (4)
• ►  August (7)
• ►  July (10)
• ►  June (4)
• ►  May (10)
• ►  April (8)
• ►  February (9)
• ►  January (4)
• ►  November (7)
• ►  October (5)
• ►  September (3)
• ►  August (3)
• ►  July (1)
• ►  January (1)
• ►  November (3)
• ►  September (1)
• ►  August (1)
• ►  June (3)
• ►  March (11)
• ►  November (1)
• ►  October (1)
• ►  August (2)
• ►  June (2)
• ►  April (3)
• ►  March (1)
• ►  February (4)
• ►  January (3)
• ►  December (2)
• ►  November (4)
• ►  June (1)
• ►  February (1)
• ►  December (1)
• ►  August (9)
• ►  July (2)
• ►  May (11)
• ►  April (5)
• ►  March (36)
• ►  February (31)
• ►  January (18)
• ►  November (8)
• ►  March (35)
• ►  October (3)
• ►  September (2)

web development Recently Published Documents

Total documents.

• Latest Documents
• Most Cited Documents
• Contributed Authors
• Related Sources
• Related Keywords

Website Developmemt Technologies: A Review

Abstract: Service Science is that the basis of knowledge system and net services that judge to the provider/client model. This paper developments a technique which will be utilized in the event of net services like websites, net applications and eCommerce. The goal is to development a technique that may add structure to a extremely unstructured drawback to help within the development and success of net services. The new methodology projected are going to be referred to as {the net|the online|the net} Development Life Cycle (WDLC) and tailored from existing methodologies and applied to the context of web development. This paper can define well the projected phases of the WDLC. Keywords: Web Development, Application Development, Technologies, eCommerce.

Analysis of Russian Segment of the Web Development Market Operating Online on Upwork

The Russian segment of the web services market in the online environment, on the platform of the Upwork freelance exchange, is considered, its key characteristics, the composition of participants, development trends are highlighted, and the market structure is identified. It is found that despite the low barriers to entry, the web development market is very stable, since the composition of entrenched firms that have been operating for more than six years remains. The pricing policy of most Russian companies indicates that they work in the middle price segment and have low budgets, which is due to the specifics of the foreign market and high competition.

Farming Assistant Web Services: Agricultor

Abstract: Our farming assistant web services provides assistance to new as well as establish farmers to get the solutions to dayto-day problems faced in the field. A farmer gets to connect with other farmers throughout India to get more information about a particular crop which is popular in other states. Keywords: Farmers, Assistance, Web Development

Tradução de ementas e histórico escolar para o inglês: contribuição para participação de discentes do curso técnico em informática para internet integrado ao ensino médio em programas de mobilidade acadêmica / Translation of summary and school records into english: contribution to the participation of high school with associate technical degree on web development students in academic mobility programs

Coded websites vs wordpress websites.

This document gives multiple instructions related to web developers using older as well as newer technology. Websites are being created using newer technologies like wordpress whereas on the other hand many people prefer making websites using the traditional way. This document will clear the doubt whether an individual should use wordpress websites or coded websites according to the users convenience. The Responsiveness of the websites, the use of CMS nowadays, more and more up gradation of technologies with SEO, themes, templates, etc. make things like web development much much easier. The aesthetics, the culture, the expressions, the features all together add up in order make the designing and development a lot more efficient and effective. Digital Marketing has a tremendous growth over the last two years and yet shows no signs of stopping, is closely related with the web development environment. Nowadays all businesses are going online due to which the impact of web development has become such that it has become an integral part of any online business.

Cognitive disabilities and web accessibility: a survey into the Brazilian web development community

Cognitive disabilities include a diversity of conditions related to cognitive functions, such as reading, understanding, learning, solving problems, memorization and speaking. They differ largely from each other, making them a heterogeneous complex set of disabilities. Although the awareness about cognitive disabilities has been increasing in the last few years, it is still less than necessary compared to other disabilities. The need for an investigation about this issue is part of the agenda of the Challenge 2 (Accessibility and Digital Inclusion) from GranDIHC-Br. This paper describes the results of an online exploratory survey conducted with 105 web development professionals from different sectors to understand their knowledge and barriers regarding accessibility for people with cognitive disabilities. The results evidenced three biases that potentially prevent those professionals from approaching cogni-tive disabilities: strong organizational barriers; difficulty to understand user needs related to cognitive disabilities; a knowledge gap about web accessibility principles and guidelines. Our results confirmed that web development professionals are unaware about cognitive disabilities mostly by a lack of knowledge about them, even if they understand web accessibility in a technical level. Therefore, we suggest that applied research studies focus on how to fill this knowledge gap before providing tools, artifacts or frameworks.

PERANCANGAN WEB RESPONSIVE UNTUK SISTEM INFORMASI OBAT-OBATAN

A good information system must not only be neat, effective, and resilient, but also must be user friendly and up to date. In a sense, it is able to be applied to various types of electronic devices, easily accessible at any whereand time (real time), and can be modified according to user needs in a relatively easy and simple way. Information systems are now needed by various parties, especially in the field of administration and sale of medicines for Cut Nyak Dhien Hospital. During this time, recording in books has been very ineffective and caused many problems, such as difficulty in accessing old data, asa well as the information obtained was not real time. To solve it, this research raises the theme of the appropriate information system design for the hospital concerned, by utilizing CSS Bootstrap framework and research methodology for web development, namely Web Development Life Cycle. This research resulted in a responsive system by providing easy access through desktop computers, tablets, and smartphones so that it would help the hospital in the data processing process in real time.

Web Development and performance comparison of Web Development Technologies in Node.js and Python

“tom had us all doing front-end web development”: a nostalgic (re)imagining of myspace, assessment of site classifications according to layout type in web development, export citation format, share document.

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings

Preview improvements coming to the PMC website in October 2024. Learn More or Try it out now .

• Advanced Search
• Journal List
• Comput Intell Neurosci
• v.2022; 2022

Interactive Design of Personalized Website Search Interface Based on Visual Communication

1 College of Art,The Tourism College of Changchun University, Changchun 130607, Jilin, China

2 The Language Set, Changchun Experimental Middle School, Changchun 130117, Jilin, China

Associated Data

The dataset can be accessed upon request.

Aiming at the problems of low user satisfaction and long search time in the traditional interactive design method of the personalized website search interface, a personalized website search interface interactive design method based on visual communication is proposed. Under the analysis of personalized website users' search behavior, the interactive personalized website search interface is designed through a navigation module, search module, link module, interactive layout module, and visual rendering module; in the visual rendering module, the advanced texture mapping method is used to render the personalized website search interface; on the personalized website search interface, the disturbance function is imported along the normal vector, the simplified new normal vector is intelligently calculated through the concave convex texture mapping algorithm, the normal vector is solved to generate the intersection point of the high-precision interface, the illumination brightness value of each pixel of the interface is intelligently calculated, the visual communication rendering model is constructed, and the visual communication effect of the interface is improved. The simulation results show that the website interface search time of this method is within 4.9 s and the user satisfaction is up to 100%, indicating that the interaction effect of the personalized website search interface designed by this method is good.

1. Introduction

Since entering the 21st century, the information industry has developed rapidly. The Internet has been applied in all aspects of people's life and occupies a very important position. At present, with the rapid development of information technology, for most people, the main source of external information is browsing the web. Therefore, designing a personalized website search interface with a good user experience has become particularly important. At the same time, it has attracted more and more attention from Internet enterprises. Good user experience, easy-to-use and interesting interaction effects, and comfortable and easy-to-read visual interface are important factors to win user loyalty for the website. In people's life, the demand for Internet products is growing and the number of websites is also increasing rapidly. However, the quality of websites is also mixed [ 1 ]. In order to increase the visits of websites, attract users, and bring economic benefits, major websites have tried every means to make various distinctive personalized websites. In addition, in recent years, web technologies such as H5 and CSS3 have developed rapidly and more special effects have been applied in the design of personalized website search interface. The expression forms of personalized website search interface are more rich and diverse, which is the result of the progress of technology and the blooming of products [ 2 ]. At the same time, in the design of a personalized website search interface, some people deliberately use a large number of dynamic effects, exquisite pictures, and many technical special effects in order to excessively pursue the interface effect of the website but ignore too much in the use quality of the product itself, such as the human-computer operation interaction of the website and the friendliness of the operation experience of the website; thus, there are many so-called “vase” Internet products on the market. In this case, there will be many problems when users use the website. For example, the loading time when users open the website is too long due to the excessive use of materials on the website; when the user uses the corresponding module functions, the interactive interface is too complex or too cool, which makes the user unexpected or even surprised when operating, contrary to the user's own expectations when using the product; in addition, there are many problems, such as complex operation steps of personalized website functions, illogical use process, and so on [ 3 ]. Such problems occur when users make complaints about the use of Tucao or even quit the website, which leads to the loss of users of personalized websites [ 4 ]. This kind of problem occurs in personalized website design mainly because designers do not pay enough attention to the interface interaction design in website design when designing personalized websites, resulting in incomplete consideration in designing website products, so as to design bad websites and bring bad user experience to users; at the same time, the personalized website products designed and developed are in a dilemma and cannot be revitalized until they are eliminated by the market. Internet products are in this situation due to the failure of design. How to design a search interface with more reasonable interaction and a good experience in personalized website design is particularly important [ 5 ].

Literature [ 6 ] proposed a multisensory visual interaction interface design method based on a fuzzy median filter. Based on the description of the main contents of the multisensory visual interaction interface, the edge of the page image is extracted, the smooth image signal is obtained by calculating the smoothing function, the page image is processed by wavelet transform, and the maximum value of the page image after wavelet transform is detected. The edge points of the image are determined according to the information obtained after the wavelet transform, the gradient vector of the image is calculated, the local maximum of the gradient vector modulus is used to determine the edge of the page image, the threshold is calculated according to the noise variance and the number of pixels of the page image, and the wavelet threshold is used to denoise the page image to improve the definition of the page image; finally, the generation of multisensory visual interaction interface under unconscious behavior is realized. Document [ 7 ] proposes an interface interaction design method based on big data processing technology, constructs the database of an interface interaction system, uses the hierarchical structure design method of process constraints for interface information interaction and big data fusion, and uses the fuzzy clustering method for information clustering of interface retrieval database. The program scheduling and cross compilation of the interface are carried out under the control of the Linux kernel source code. The interface interaction design system mainly includes process management, program control, and internal file management modules. Combined with big data processing technology, the optimization design of the interface interaction system is realized. The test results show that the designed interface interaction system has good big data information processing and scheduling ability, and the recall of data is good. However, the user satisfaction with the above-given two methods is low. Document [ 8 ] proposed the design method of an HTML5 mobile interactive interface system based on Yipai 360. According to the overall framework of the Yipai 360 system hardware platform, the hardware structure is designed. According to the animation control, interactive setting, social application, and data application modules of Yipai 360pc, the mobile terminal configuration is adjusted in time through the setting of the panel and the Bluetooth/RS-485 gateway module is designed; we realize the bidirectional conversion between the signal and RS-485 signal, analyze the PLC data acquisition status in the protocol data, send the data packet to Yipai 360pc, control the digital or analog input and output according to the programmable logic controller, and complete the interface interaction design with the support of the execution and evaluation of the connection between Yipai 360 and HTML5 mobile interactive interface. Literature [ 9 ] proposed an information interface interaction design method based on unconscious cognition, analyzed unconscious cognitive behavior and information interaction efficiency by means of literature, investigation, and research, guided information interface interaction design by constructing an unconscious cognitive behavior model, and provided a theoretical basis for information interface design; information interface design based on user unconscious cognition is an important method to optimize the user experience and improve the usability and interaction efficiency of information interface, which provides a new idea and method for the study of information interface design. However, the above two methods take a long time to search the interface, resulting in low interface search efficiency.

Aiming at the problems of low user satisfaction and long search time in traditional methods, this paper proposes an interactive design method of personalized website search interface based on visual communication, uses an advanced texture mapping method to render the personalized website search interface, solves the normal vector to generate the intersection of a high-precision interface, and obtains the illumination brightness value of each pixel of the generated interface; we build a visual communication rendering model to improve the visual communication effect of the interface. Under this method, the website interface search time is within 4.9 s and the user satisfaction is up to 100%, which not only solves the problems existing in the traditional methods but also lays a foundation for improving the effect of the website user experience.

2. Analysis of Search Behavior of Personalized Website Users

Before the interactive design of the personalized website search interface, firstly, the user search behavior of a personalized website is analyzed. Users usually get the use mode of something from their daily use experience, and they will continue to use it instead of going deep into the specific principle. The great success of search engines has changed the behavior mode of users [ 10 ]. In the Internet experiment, when users are allowed to find ways to solve problems on the web page where they know at will, they will go to a search engine website in 85% of the cases. Users look for answers through search engines rather than good websites. This change in their behavior makes the website focus on building “high-viscosity” websites instead of improving the optimization of personalized websites and improving the ranking in search engines. The change in users' behavior of using personalized website search interfaces has promoted the change in personalized website function, especially the personalized website directly facing users [ 11 ]. Now, search has become an essential and important function of personalized websites. The commodity search method of Internet users when shopping is shown in Figure 1 .

Commodity search method when Internet users shop.

The proportion of users using search engines to search for goods and on-site search for goods is 27.1% and 20.6%, respectively, further highlighting the importance of search in the selection of online shopping goods [ 12 ].

In interface development and design, designers design products according to their own understanding of products (i.e., design psychological model), users use products according to their own understanding of products (i.e., user psychological model), and the platform for designers to communicate with users is the system. The user mode determines users' understanding of products, and the design mode determines whether product operation methods are easy to learn and use [ 13 ]. When designing products, designers must consider the user's psychological mode and design the products from the perspective that users can understand, so that the balance between the design mode and user mode is finally reflected in the product interface [ 14 ]. The design pattern, user pattern, and system representation are shown in Figure 2 .

Design pattern, user pattern, and system representation.

Users have a strong monopoly on the use of personalized websites. They will not blindly and passively accept information but actively obtain information, which also determines that users interact more autonomously and frequently when using personalized websites. It is found that most users browse the website interface with an “F” shaped path, that is, users first browse horizontally at the top of the website interface, and then the horizontal browsing distance will be shortened as the user's line of sight moves down. Finally, users quickly browse the vertical area on the left side of the interface, as shown in Figure 3 [ 15 ]. Of course, this “F” mode does not represent the browsing behavior of all users. If the information and picture content of interest to users appear below, “F” mode will also become “e,” as shown in Figure 4 .

“F” browsing mode.

“e” browsing mode.

Therefore, when arranging the information priority of the website interface of colleges and universities, we can follow the visual browsing rule of users from left to right and from top to bottom and put the main information and key interaction on the upper left of the website interface, so as to meet the information needs of users in time and bring users a good information interaction experience [ 16 ].

3. Interactive Design of Personalized Website Search Interface Based on Visual Communication

3.1. design of the overall architecture.

Based on the analysis of personalized website user search behavior, this paper designs the personalized website search interface interactively through the navigation module, search module, link module, interactive layout module, and visual rendering module [ 17 ]. The overall framework is shown in Figure 5 .

Overall design architecture.

3.2. Navigation Module Design

Navigation is the directory of a personalized website search interface. It helps users understand their position in the personalized website and the overall structure of the website. It also guides users on where to go in the personalized website, so that users can quickly find the content and information they need. At the same time, it helps users walk freely through the site to find the content and functional elements they need. The design of navigation directly affects whether the information content of the personalized website search interface can be searched by users and browsed effectively [ 18 ]. For each kind of information, the more paths to find, the more likely it is to be read. This is also a good way to improve the browsing volume of a personalized website search interface. Therefore, the optimization of personalized website search interface navigation design can significantly promote the usability of the website. This paper uses the method of global navigation design to design personalized website search interface navigation. Global navigation mainly refers to mastering the path of the whole website and having a unified main navigation to control, as shown in Figure 6 . Generally, there is global navigation on every page in the website [ 19 ]. No matter which page the user is on, accessing any other page can be realized through global navigation.

Global navigation.

3.3. Search Module Design

The success of search engines has changed users' habits of using web pages. Search has become an important behavior of users in the website. Unless the website is really small and well organized, each page should have a search view or a link to a search page. The design of the search should be concise and follow the general formula of search: an input box, a button, and the word “search.” We should avoid using too fancy design and words. At the same time, for novice users, descriptive text can be added to the search bar to inform users of the keyword content that can be input. It is necessary to avoid indicative descriptive text similar to “input keyword” because even novice users who use the website for the first time know the function of the search bar [ 20 ]. In addition, if there is a possibility of confusing the search scope and content, you need to write it out in the search bar. The design of a personalized website search bar is shown in Figure 7 .

Personalized website search bar design.

3.4. Link Module Design

The interaction between users and personalized websites is mainly completed through links. According to the user's usage habits, the buttons and links that can be clicked at the obvious signs can improve the usability of the web page. The buttons in the web page are clickable, which is obtained from the user's experience [ 21 ]. Therefore, the marking of text links is a place to pay attention to in the principle of ease of use. For the navigation of personalized websites or all links in websites, you can remove the underline design and just change the color or add an underline when the mouse moves to the link text. For the mixed arrangement of ordinary text and hyperlink text, it is necessary to clearly distinguish the difference between the hypertext link and ordinary text [ 22 ]. In addition, you can design text links as button icons.

3.5. Interactive Layout Module Design

As an expressive visual language, personalized website search interface design pays special attention to the interactive layout of the interface. The layout of the web interface directly affects the convenience of users using the interface information. A reasonable interactive layout will enable users to quickly find the core content and services. On the contrary, they do not know how to obtain the required information, or how to browse to get the corresponding service, and then the user will choose to leave [ 23 ].

Although the personalized website search interface layout does not attract the user's visual attention to a certain location or object as other elements such as color and graphics, a good interface layout often becomes a prerequisite for attracting the user's attention and trying to choose and the user does not tend to be in a chaotic state. It takes time and energy to pay attention to an element or content in a layout without a sense of stability. Therefore, a good interactive layout first ensures that the user's visual attention is stimulated. To define whether an interactive layout is chaotic, it is first necessary to ensure that the user's visual weight on the layout reaches a certain balance; when the elements in the interface are gathered together, the visual weight is formed. The visual weight is virtual and obtained through the user's visual perception. Generally speaking, in order to achieve the balance of the interface layout, these visual weights must be offset by a weight with equal and opposite weight; otherwise, the layout will show an unstable state. A balanced layout can make the shift of sight operate in an orderly manner within a reasonable range without psychological burden and pressure [ 24 ]. A balanced layout design makes users feel stable and simple, which is often more attractive. The interface layout between the Mint Wheels website and the Dallas Baptist University website is shown in Figure 8 .

Interface layout of (a) Mint Wheels website and (b) Dallas Baptist University website.

As shown in Figure 8 , in the Mint Wheels web interface, the logo is placed on the central axis of the interface and on the top to help establish a symmetrical balance. Moreover, the visual weight on both sides of the central axis is the same and the elements on both sides also form a corresponding relationship [ 25 ]. This makes the web interface layout delicate and concise and complements its content. However, such a symmetrical visual weight balance will inevitably appear rigid and lifeless, so the concept of balance cannot be achieved only by symmetry. When the elements on both sides are asymmetric, they can also achieve the balance of visual weight. As shown in Figure 8 , the Dallas Baptist University web interface is particularly asymmetric compared with the Mint Wheels web interface, but visually, the logo balances the search box, the loose large content area on the right side of the interface balances the tight small content area on the right side, but the asymmetric arrangement of elements achieves the balance of weight, which will greatly enhance the possibility of users' visual selective attention [ 26 ].

3.6. Design of the Visual Rendering Module

The personalized website search interface includes two elements: graphics and text, and these two visual elements have one thing in common, that is, these two elements are the media to convey page information to users through visual color [ 27 ]. Color is a highly stimulating and powerful design element. Color can convey a kind of information to the browser of the interface, so as to reflect the user's psychological feelings and stimulate the user's psychological activities. Therefore, color is often easier to attract the user's attention and attract the user's attention than other elements.

Because the perception of the light wave by human eyes has a length range and different colors have different light wave lengths, people's perceptions of different colors will be different. In general, the length of human perception of light wave ranges from 400 to 700  μ m; when the color wavelength is perceived by the human eye at both ends of the light wave range, i.e., 400  μ m (red) or 700  μ m (purple), the brightness of the color will weaken. Too strong or too weak brightness of the color will cause people's visual fatigue, so when selecting the color, one must try to choose the light wave length suitable for people's vision to create a comfortable interactive environment. If people look at a position for a long time, they will feel that their sight is becoming more and more blurred. This phenomenon is called visual residue [ 28 ]. Therefore, in order to improve the visual quality of the personalized website search interface and improve the visual communication effect of the interface, when designing the personalized website search interface, the personalized website search interface is rendered through the advanced texture mapping method, which includes concave-convex and normal texture mapping algorithms:

The flow of the bump texture mapping algorithm is as follows.

Suppose that the web page crawling function P of the personalized website search interface is as follows:

where ( x , y ) represents texture coordinates [ 29 ].

Calculate the partial derivative of formula ( 1 ) to obtain P x and P y ; then, the normal vector E of point 1 on the personalized website search interface is shown in the following formula:

On the personalized website search interface, each point imports a small disturbance function along the normal vector, as shown in the following formula:

Here, the surface equation of the new personalized website search interface is described by P ′ and the disturbance function is described by S ( x , y ).

Calculate the partial derivative of P ′ in x , y direction, as shown in the following formulas:

After simplification, the new normal vector of each point in the personalized website search interface is obtained, as shown in the following formula:

where the disturbance factor is described by S X ( E × P Y ) and S X ( P Y × E ). Under the action of light, the surface of a personalized website search interface will produce an uneven rendering effect [ 30 ].

In order to improve the effect of the bump texture mapping algorithm, it is improved by normal texture mapping, as shown in Figure 9 .

Normal texture mapping structure.

As can be seen from Figure 9 , in order to solve the normal vector, emit rays from the low to high interface, generate the intersection of a high-precision interface, generate the illumination brightness value of each pixel of the interface, and construct the visual communication rendering model. The expression is

Therefore, the rendering effect of the personalized website search interface is improved.

4. Simulation Experiment Analysis

4.1. experimental design.

In order to verify the effectiveness of the interactive design method of personalized website search interface based on visual communication in practical application, a simulation experiment is carried out. The experiment adopts the general configuration of the current mainstream PC, and the compilation and running environment adopt common tools. Specific parameters are shown in Table 1 .

Experimental environment parameters.

Under the above-given experimental environment, this paper selects an automobile website as the experimental object for the experimental test. The experimental object is shown in Figure 10 .

Experimental object.

4.2. Experimental Analysis

4.2.1. website interface search time.

The interactive design method of the personalized website search interface based on visual communication proposed in this paper, the multisensory visual interactive interface design method based on the fuzzy median filter proposed in the literature [ 6 ], and the interface interactive design method based on big data processing technology proposed in interface literature [ 7 ] are used to search the automobile website interface and test the time consumed by the three search methods. The test results are shown in Table 2 .

Website interface search time of three methods.

According to Table 2 , when the number of experiments is 50, the website interface search time of the method in [ 6 ] is 11.7 s, the website interface search time of the method in [ 7 ] is 18.6 s, and the website interface search time of our method is 3.7 s; when the number of experiments is 100, the website interface search time of the method in [ 6 ] is 14.5 s, the website interface search time of the method in [ 7 ] is 24.6 s, and the website interface search time of our method is 4.9 s; the interactive design method of the personalized website search interface based on visual communication proposed in this paper takes 4.9 s to search the personalized website interface, which is shorter than the multisensory visual interactive interface design method based on fuzzy median filter proposed in the literature [ 6 ] and the interface interactive design method based on big data processing technology proposed in the literature [ 7 ]. This is because this method solves the normal vector by emitting rays from the low to high interface, generates the intersection point of the high-precision interface, obtains the illumination brightness value of each pixel of the generated interface, constructs a visual communication rendering model, improves the rendering efficiency of the personalized website search interface, and effectively improves the website interface search efficiency of this method.

4.2.2. User Satisfaction

In order to further verify the effectiveness of this method, the personalized website search interface interactive design method based on visual communication proposed in this paper, the multisensory visual interactive interface design method based on fuzzy median filter proposed in [ 6 ], and the interface interactive design method based on large data processing technology proposed in [ 7 ] are used for personalized website interface search. The satisfaction of users is tested, and the test results are shown in Figure 11 .

Comparison results of user satisfaction.

According to Figure 11 , when the number of experiments is 20, the user satisfaction of the method in [ 6 ] is 80%, the user satisfaction of the method in [ 7 ] is 68%, and the user satisfaction of the method in this paper is as high as 92%; when the number of experiments is 60, the user satisfaction of the method in [ 6 ] is 81%, that of the method in [ 7 ] is 69%, and that of our method is as high as 99%; after applying the interactive design method of personalized website search interface based on visual communication proposed in this paper, the user satisfaction is up to 100%, which shows that the interactive effect of personalized website search interface designed by our method is good. This is because this method uses the advanced texture mapping method to render the personalized website search interface, improve the visual communication effect of the interface, and improve the user satisfaction.

5. Conclusion

This paper proposes an interactive design method of a personalized website search interface based on visual communication and uses an advanced texture mapping method to render the personalized website search interface; through the concave-convex texture mapping algorithm, we intelligently calculate the simplified new normal vector, solve the normal vector to generate the intersection of the high-precision interface, intelligently calculate and generate the illumination brightness value of each pixel of the interface, build the visual communication rendering model, and improve the visual communication effect of the interface. The following conclusions are drawn through experiments:

• The interactive design method of personalized website search interface based on visual communication proposed in this paper takes 4.9 s to search the personalized website interface, and the personalized website interface search time of this method is short
• When the number of experiments is 60, the user satisfaction of this method is as high as 99%; after applying the interactive design method of personalized website search interface based on visual communication proposed in this paper, the user satisfaction is up to 100%, which shows that the interactive effect of personalized website search interface designed by this method is good

This paper has achieved high user satisfaction, but the search time still needs to be improved, and further research is needed in the future.

Acknowledgments

This work was the phased research result of the 2021 Jilin Province Vocational Education Scientific Research Project “The Innovative Practice of” Curriculum Ideology and Politics in Art Majors of Vocational Colleges,” under project number 2021XHY217.

Data Availability

Conflicts of interest.

The authors declare that there are no conflicts of interest.

Have a language expert improve your writing

Run a free plagiarism check in 10 minutes, generate accurate citations for free.

• Knowledge Base

Methodology

• What Is a Research Design | Types, Guide & Examples

What Is a Research Design | Types, Guide & Examples

Published on June 7, 2021 by Shona McCombes . Revised on November 20, 2023 by Pritha Bhandari.

A research design is a strategy for answering your   research question  using empirical data. Creating a research design means making decisions about:

• Your overall research objectives and approach
• Whether you’ll rely on primary research or secondary research
• Your sampling methods or criteria for selecting subjects
• Your data collection methods
• The procedures you’ll follow to collect data
• Your data analysis methods

A well-planned research design helps ensure that your methods match your research objectives and that you use the right kind of analysis for your data.

Table of contents

Step 1: consider your aims and approach, step 2: choose a type of research design, step 3: identify your population and sampling method, step 4: choose your data collection methods, step 5: plan your data collection procedures, step 6: decide on your data analysis strategies, other interesting articles, frequently asked questions about research design.

• Introduction

Before you can start designing your research, you should already have a clear idea of the research question you want to investigate.

There are many different ways you could go about answering this question. Your research design choices should be driven by your aims and priorities—start by thinking carefully about what you want to achieve.

The first choice you need to make is whether you’ll take a qualitative or quantitative approach.

Qualitative research designs tend to be more flexible and inductive , allowing you to adjust your approach based on what you find throughout the research process.

Quantitative research designs tend to be more fixed and deductive , with variables and hypotheses clearly defined in advance of data collection.

It’s also possible to use a mixed-methods design that integrates aspects of both approaches. By combining qualitative and quantitative insights, you can gain a more complete picture of the problem you’re studying and strengthen the credibility of your conclusions.

Practical and ethical considerations when designing research

As well as scientific considerations, you need to think practically when designing your research. If your research involves people or animals, you also need to consider research ethics .

• How much time do you have to collect data and write up the research?
• Will you be able to gain access to the data you need (e.g., by travelling to a specific location or contacting specific people)?
• Do you have the necessary research skills (e.g., statistical analysis or interview techniques)?
• Will you need ethical approval ?

At each stage of the research design process, make sure that your choices are practically feasible.

Here's why students love Scribbr's proofreading services

Discover proofreading & editing

Within both qualitative and quantitative approaches, there are several types of research design to choose from. Each type provides a framework for the overall shape of your research.

Types of quantitative research designs

Quantitative designs can be split into four main types.

• Experimental and   quasi-experimental designs allow you to test cause-and-effect relationships
• Descriptive and correlational designs allow you to measure variables and describe relationships between them.

With descriptive and correlational designs, you can get a clear picture of characteristics, trends and relationships as they exist in the real world. However, you can’t draw conclusions about cause and effect (because correlation doesn’t imply causation ).

Experiments are the strongest way to test cause-and-effect relationships without the risk of other variables influencing the results. However, their controlled conditions may not always reflect how things work in the real world. They’re often also more difficult and expensive to implement.

Types of qualitative research designs

Qualitative designs are less strictly defined. This approach is about gaining a rich, detailed understanding of a specific context or phenomenon, and you can often be more creative and flexible in designing your research.

The table below shows some common types of qualitative design. They often have similar approaches in terms of data collection, but focus on different aspects when analyzing the data.

Your research design should clearly define who or what your research will focus on, and how you’ll go about choosing your participants or subjects.

In research, a population is the entire group that you want to draw conclusions about, while a sample is the smaller group of individuals you’ll actually collect data from.

Defining the population

A population can be made up of anything you want to study—plants, animals, organizations, texts, countries, etc. In the social sciences, it most often refers to a group of people.

For example, will you focus on people from a specific demographic, region or background? Are you interested in people with a certain job or medical condition, or users of a particular product?

The more precisely you define your population, the easier it will be to gather a representative sample.

• Sampling methods

Even with a narrowly defined population, it’s rarely possible to collect data from every individual. Instead, you’ll collect data from a sample.

To select a sample, there are two main approaches: probability sampling and non-probability sampling . The sampling method you use affects how confidently you can generalize your results to the population as a whole.

Probability sampling is the most statistically valid option, but it’s often difficult to achieve unless you’re dealing with a very small and accessible population.

For practical reasons, many studies use non-probability sampling, but it’s important to be aware of the limitations and carefully consider potential biases. You should always make an effort to gather a sample that’s as representative as possible of the population.

Case selection in qualitative research

In some types of qualitative designs, sampling may not be relevant.

For example, in an ethnography or a case study , your aim is to deeply understand a specific context, not to generalize to a population. Instead of sampling, you may simply aim to collect as much data as possible about the context you are studying.

In these types of design, you still have to carefully consider your choice of case or community. You should have a clear rationale for why this particular case is suitable for answering your research question .

For example, you might choose a case study that reveals an unusual or neglected aspect of your research problem, or you might choose several very similar or very different cases in order to compare them.

Data collection methods are ways of directly measuring variables and gathering information. They allow you to gain first-hand knowledge and original insights into your research problem.

You can choose just one data collection method, or use several methods in the same study.

Survey methods

Surveys allow you to collect data about opinions, behaviors, experiences, and characteristics by asking people directly. There are two main survey methods to choose from: questionnaires and interviews .

Observation methods

Observational studies allow you to collect data unobtrusively, observing characteristics, behaviors or social interactions without relying on self-reporting.

Observations may be conducted in real time, taking notes as you observe, or you might make audiovisual recordings for later analysis. They can be qualitative or quantitative.

Other methods of data collection

There are many other ways you might collect data depending on your field and topic.

If you’re not sure which methods will work best for your research design, try reading some papers in your field to see what kinds of data collection methods they used.

Secondary data

If you don’t have the time or resources to collect data from the population you’re interested in, you can also choose to use secondary data that other researchers already collected—for example, datasets from government surveys or previous studies on your topic.

With this raw data, you can do your own analysis to answer new research questions that weren’t addressed by the original study.

Using secondary data can expand the scope of your research, as you may be able to access much larger and more varied samples than you could collect yourself.

However, it also means you don’t have any control over which variables to measure or how to measure them, so the conclusions you can draw may be limited.

As well as deciding on your methods, you need to plan exactly how you’ll use these methods to collect data that’s consistent, accurate, and unbiased.

Planning systematic procedures is especially important in quantitative research, where you need to precisely define your variables and ensure your measurements are high in reliability and validity.

Operationalization

Some variables, like height or age, are easily measured. But often you’ll be dealing with more abstract concepts, like satisfaction, anxiety, or competence. Operationalization means turning these fuzzy ideas into measurable indicators.

If you’re using observations , which events or actions will you count?

If you’re using surveys , which questions will you ask and what range of responses will be offered?

You may also choose to use or adapt existing materials designed to measure the concept you’re interested in—for example, questionnaires or inventories whose reliability and validity has already been established.

Reliability and validity

Reliability means your results can be consistently reproduced, while validity means that you’re actually measuring the concept you’re interested in.

For valid and reliable results, your measurement materials should be thoroughly researched and carefully designed. Plan your procedures to make sure you carry out the same steps in the same way for each participant.

If you’re developing a new questionnaire or other instrument to measure a specific concept, running a pilot study allows you to check its validity and reliability in advance.

Sampling procedures

As well as choosing an appropriate sampling method , you need a concrete plan for how you’ll actually contact and recruit your selected sample.

That means making decisions about things like:

• How many participants do you need for an adequate sample size?
• What inclusion and exclusion criteria will you use to identify eligible participants?
• How will you contact your sample—by mail, online, by phone, or in person?

If you’re using a probability sampling method , it’s important that everyone who is randomly selected actually participates in the study. How will you ensure a high response rate?

If you’re using a non-probability method , how will you avoid research bias and ensure a representative sample?

Data management

It’s also important to create a data management plan for organizing and storing your data.

Will you need to transcribe interviews or perform data entry for observations? You should anonymize and safeguard any sensitive data, and make sure it’s backed up regularly.

Keeping your data well-organized will save time when it comes to analyzing it. It can also help other researchers validate and add to your findings (high replicability ).

On its own, raw data can’t answer your research question. The last step of designing your research is planning how you’ll analyze the data.

Quantitative data analysis

In quantitative research, you’ll most likely use some form of statistical analysis . With statistics, you can summarize your sample data, make estimates, and test hypotheses.

Using descriptive statistics , you can summarize your sample data in terms of:

• The distribution of the data (e.g., the frequency of each score on a test)
• The central tendency of the data (e.g., the mean to describe the average score)
• The variability of the data (e.g., the standard deviation to describe how spread out the scores are)

The specific calculations you can do depend on the level of measurement of your variables.

Using inferential statistics , you can:

• Make estimates about the population based on your sample data.
• Test hypotheses about a relationship between variables.

Regression and correlation tests look for associations between two or more variables, while comparison tests (such as t tests and ANOVAs ) look for differences in the outcomes of different groups.

Your choice of statistical test depends on various aspects of your research design, including the types of variables you’re dealing with and the distribution of your data.

Qualitative data analysis

In qualitative research, your data will usually be very dense with information and ideas. Instead of summing it up in numbers, you’ll need to comb through the data in detail, interpret its meanings, identify patterns, and extract the parts that are most relevant to your research question.

Two of the most common approaches to doing this are thematic analysis and discourse analysis .

There are many other ways of analyzing qualitative data depending on the aims of your research. To get a sense of potential approaches, try reading some qualitative research papers in your field.

If you want to know more about the research process , methodology , research bias , or statistics , make sure to check out some of our other articles with explanations and examples.

• Simple random sampling
• Stratified sampling
• Cluster sampling
• Likert scales
• Reproducibility

 Statistics

• Null hypothesis
• Statistical power
• Probability distribution
• Effect size
• Poisson distribution

Research bias

• Optimism bias
• Cognitive bias
• Implicit bias
• Hawthorne effect
• Anchoring bias
• Explicit bias

A research design is a strategy for answering your   research question . It defines your overall approach and determines how you will collect and analyze data.

A well-planned research design helps ensure that your methods match your research aims, that you collect high-quality data, and that you use the right kind of analysis to answer your questions, utilizing credible sources . This allows you to draw valid , trustworthy conclusions.

Quantitative research designs can be divided into two main categories:

• Correlational and descriptive designs are used to investigate characteristics, averages, trends, and associations between variables.
• Experimental and quasi-experimental designs are used to test causal relationships .

Qualitative research designs tend to be more flexible. Common types of qualitative design include case study , ethnography , and grounded theory designs.

The priorities of a research design can vary depending on the field, but you usually have to specify:

• Your research questions and/or hypotheses
• Your overall approach (e.g., qualitative or quantitative )
• The type of design you’re using (e.g., a survey , experiment , or case study )
• Your data collection methods (e.g., questionnaires , observations)
• Your data collection procedures (e.g., operationalization , timing and data management)
• Your data analysis methods (e.g., statistical tests  or thematic analysis )

A sample is a subset of individuals from a larger population . Sampling means selecting the group that you will actually collect data from in your research. For example, if you are researching the opinions of students in your university, you could survey a sample of 100 students.

In statistics, sampling allows you to test a hypothesis about the characteristics of a population.

Operationalization means turning abstract conceptual ideas into measurable observations.

For example, the concept of social anxiety isn’t directly observable, but it can be operationally defined in terms of self-rating scores, behavioral avoidance of crowded places, or physical anxiety symptoms in social situations.

Before collecting data , it’s important to consider how you will operationalize the variables that you want to measure.

A research project is an academic, scientific, or professional undertaking to answer a research question . Research projects can take many forms, such as qualitative or quantitative , descriptive , longitudinal , experimental , or correlational . What kind of research approach you choose will depend on your topic.

Cite this Scribbr article

If you want to cite this source, you can copy and paste the citation or click the “Cite this Scribbr article” button to automatically add the citation to our free Citation Generator.

McCombes, S. (2023, November 20). What Is a Research Design | Types, Guide & Examples. Scribbr. Retrieved April 4, 2024, from https://www.scribbr.com/methodology/research-design/

Is this article helpful?

Shona McCombes

Other students also liked, guide to experimental design | overview, steps, & examples, how to write a research proposal | examples & templates, ethical considerations in research | types & examples, what is your plagiarism score.

Main Navigation

• Contact NeurIPS
• Code of Ethics
• Code of Conduct
• Create Profile
• Journal To Conference Track
• Diversity & Inclusion
• Proceedings
• Future Meetings
• Exhibitor Information
• Privacy Policy

NeurIPS 2024, the Thirty-eighth Annual Conference on Neural Information Processing Systems, will be held at the Vancouver Convention Center

Monday Dec 9 through Sunday Dec 15. Monday is an industry expo.

Registration

Pricing » Registration 2024 Registration Cancellation Policy » . Certificate of Attendance

Our Hotel Reservation page is currently under construction and will be released shortly. NeurIPS has contracted Hotel guest rooms for the Conference at group pricing, requiring reservations only through this page. Please do not make room reservations through any other channel, as it only impedes us from putting on the best Conference for you. We thank you for your assistance in helping us protect the NeurIPS conference.

Announcements

• The Call For Papers has been released
• See the Visa Information page for changes to the visa process for 2024.

Latest NeurIPS Blog Entries [ All Entries ]

Important dates.

If you have questions about supporting the conference, please contact us .

View NeurIPS 2024 exhibitors » Become an 2024 Exhibitor Exhibitor Info »

Organizing Committee

General chair, program chair, workshop chair, workshop chair assistant, tutorial chair, competition chair, data and benchmark chair, diversity, inclusion and accessibility chair, affinity chair, ethics review chair, communication chair, social chair, journal chair, creative ai chair, workflow manager, logistics and it, mission statement.

The Neural Information Processing Systems Foundation is a non-profit corporation whose purpose is to foster the exchange of research advances in Artificial Intelligence and Machine Learning, principally by hosting an annual interdisciplinary academic conference with the highest ethical standards for a diverse and inclusive community.

About the Conference

The conference was founded in 1987 and is now a multi-track interdisciplinary annual meeting that includes invited talks, demonstrations, symposia, and oral and poster presentations of refereed papers. Along with the conference is a professional exposition focusing on machine learning in practice, a series of tutorials, and topical workshops that provide a less formal setting for the exchange of ideas.

More about the Neural Information Processing Systems foundation »

tableau.com is not available in your region.

Accessibility engineering in web evaluation process: a systematic literature review

• Review Paper
• Open access
• Published: 27 January 2023

Cite this article

You have full access to this open access article

• Jinat Ara 1 ,
• Cecilia Sik-Lanyi 1 &
• Arpad Kelemen 2  

3799 Accesses

3 Citations

3 Altmetric

Explore all metrics

Several works of literature contributed to the web evaluation process in recent years to promote digital inclusion by addressing several accessibility guidelines, methods, processes, and techniques. Researchers have investigated how the web evaluation process could be facilitated by including accessibility issues to obtain an inclusive and accessible solution to improve the user experience and increase user satisfaction. Three systematic literature reviews (SLRs) have been conducted in the context of past research, considering such research focuses. This paper presents a new SLR approach concerning accessibility in the web evaluation process, considering the period from 2010 to 2021. The review of 92 primary studies showed the contribution of publications on different phases of the web evaluation process mainly by highlighting the significant studies in the framework design and testing process. To the best of our knowledge, this is the first study focused on the web accessibility literature reporting the engineering assets for evaluation of new accessible and inclusive web-based solutions (e.g., websites). Besides, in this study, we aim to provide a new direction to the web designers and developers with an updated view of process, methods, techniques, tools, and other crucial aspects to contribute to the accessible process enrichment, as well as depict the gaps and challenges that may be worthy to be investigated in the future. The findings of this SLR introduce a new dimension in web accessibility research on determining and mitigating the research gap of web accessibility issues for web designers, developers, and other practitioners.

Similar content being viewed by others

Toward a Combined Method for Evaluation of Web Accessibility

Automated Evaluation Tools for Web and Mobile Accessibility: A Systematic Literature Review

Comparison of Accessibility Tools for the Evaluation of E-Commerce Websites

Avoid common mistakes on your manuscript.

1 Introduction

In recent years, various aspects have motivated researchers to conduct studies about digital accessibility. The extension and increased availability of the web for multiple purposes (e.g., information search), the representation of the content (e.g., video, audio), and the emergence of new platforms (e.g., Internet of Things) and technologies (e.g., mobile, computer, tablets) are significant aspects to reinforce the investigation of the digital information platform. In particular, from the very beginning of the digital revolution, digital resources become the fundamental source for citizens to access information such as education, health care, government, news, and other information such as entertainment and sports [ 1 , 2 ].

According to the World Wide Web Consortium (W3C) and the Web Accessibility Initiative (WAI) report, accessibility is a broad and extensible term associated with people who have disabilities, incompetent skills, or situational-induced impairment [ 3 ]. This initiative's objective is to ensure accessibility which means people with special needs should be able to access, navigate, interact, and contribute to the information that is available on the Web/Internet, electronic resources/materials, and computer. The current mission of the WAI initiative is to coordinate international, technical, and human efforts to improve web accessibility [ 4 ]. With this mission in mind, WAI launched a set of accessibility guidelines called Web Content Accessibility Guidelines (WCAGs) [ 5 , 6 ]. A detailed description of WCAG is given in Sect.  2 .

The scientific research community has recognized that web design and development must inspect the assorted number of requirements of citizens across the population, including special needs users and elderly citizens. Earlier researchers considered accessibility checking as a supplementary requirement in the evaluation phase of any application development. However, in recent years, researchers suggested that accessibility requirements should be followed from the very beginning of the application design and development. Lack of consideration of accessibility issues during the design and development might introduce violations of accessibility guidelines and consequently basic rights of people with disabilities. A great volume of literature exists addressing accessibility guidelines in the design and development of web platforms [ 7 , 8 ]. More recently, a few studies highlighted the importance and emerging need of considering accessibility throughout the web development life cycle [ 9 , 10 ].

Few studies discussed the importance of systematic literature review (SLR) approaches to present the true insights of a particular topic for highlighting future improvement directions [ 11 , 12 , 13 ]. Campoverde-Molina et al. [ 14 ] mentioned that SLR is a synthesis process of past studies that have been published in different scientific databases focusing on a particular issue. SLR aims to review past literature on a specific domain to determine its effectiveness and find the research gap and new research areas. It helps to identify the way of knowledge improvement, promotes new theories for development, and reveals the new investigated area that needs to focus. Therefore, an SLR focusing on web accessibility engineering assets is essential to determine a way to promote an accessible web platform according to WCAG standards.

Emphasizing the necessity of the SLR approach in the web accessibility context, Akram and Sulaiman [ 14 ] and Campoverde‑Molina et al. [ 15 ] have conducted SLRs to analyze the accessibility of educational institute websites within a specific period. The first SLR performed the analysis regarding the period between 2009 and 2017. The second one conducted the investigation considering the period of 2009 to 2020. In 2021, Campoverde‑Molina et al. [ 16 ] extended their previous work intending to update the result of the past SLR and extended the period from 2002 to 2020. In general, SLR refers to the aggregation of knowledge about a particular domain of research with a set of research questions and solutions. Thus, the SLR process should be as unbiased as possible [ 17 ]. The selected SLRs are auditable and have significant effects. However, the focus on engineering assets such as processes, development techniques, and technologies is limited, which is a drawback of SLRs.

This paper presents an extensive SLR in the context of accessibility in the web evaluation process to identify several engineering processes to improve the accessibility of web platforms. This study will help a wide array of people (developers, designers, inventors, leaders, researchers, and users) and facilitate the accessible web design and evaluation process. The paper is organized as follows: In Sect.  2 , accessibility concepts, importance, and related works are presented. Section 3  describes the details of conducting the SLR. Section  4 represents the result of conducted SLR and discusses the main findings through a broad discussion. In Sect.  5 , we conclude the paper.

2 Background and related work

Digital accessibility is a process to ensure the availability of online tools or content to the users [ 13 ]. The prime objective of digital accessibility is to make an accessible, operable, and interactable online platform to provide equal information accessing opportunities for people with disabilities [ 18 , 19 ]. Several aspects might initiate barriers to implementing and ensuring digital accessible platforms or tools or content, such as limited accessibility knowledge and its guidelines. Sometimes organizational barriers and parameters such as organization size, capital, and cost influence accessibility issues. Addressing these issues, the governments and organizations of several countries declared various guidelines, standards, and conformance levels for the stakeholders [ 20 ]. Following these guidelines, associate authorities might overcome critical issues and ensure digital accessibility.

2.1 Accessibility standards

To develop an accessible solution (e.g., application, websites, software, etc.), several accessibility guidelines have been introduced by the government of several countries and various public and private institutes such as WCAG, Section 508, EN 301 549, YD/T 1761–2012, WAI-ARIA, BITV, ISO 9241 and ATAG are prominent. Web Content Accessibility Guideline (WCAG) was introduced by the Web Accessibility Initiative of the World Wide Web Consortium with several success criteria under 13 guidelines. Section 508 is accessibility requirements rules published by the US Government for digital resources to make the resources accessible. EN 301 549 is a European accessibility requirement that is suitable for public procurement of ICT products and services in Europe. YD/T 1761–2012 refers to the Chinese Technical requirements standards for web accessibility that primarily focus on ensuring accessibility in the digital platform. Besides, the WAI-ARIA standard was published by W3C to define a set of guidelines for HTML attributes to improve semantic accessibility. BITV is a German standard that is issued focusing on WCAG 2.0 to make the website and application accessible for people with disabilities by ensuring perceivable, operable, understandable, and robust guidelines. Similarly, ISO 9241 provides requirements for accessible developments throughout the application development life cycle. It concerns both hardware and software components for interactive design and development. Authoring Tool Accessibility Guidelines (ATAG) is WCAG and User Agent Accessibility guidelines-based instruction for accessible web content design and development.

Among these guidelines, WCAG is the most widely used accessibility standard. WCAG is a documented guideline that explains all the accessibility criteria and step-by-step recommendations about implementation, improvement, and measurement of accessibility to provide a better user experience, especially for people with disabilities. W3C-based WAI first developed the WCAG standards to make the web accessible [ 3 ]. As of July 2022, WAI has published five versions of the WCAG standard, including WCAG 1.0, WCAG 2.0, WCAG 2.1, WCAG 2.2, and WCAG 3.0 (draft version). The WCAG 3.0 is the most sophisticated standard, currently available as a working draft for web developers (front and back end) and designers to develop accessible and usable web content [ 21 ].

In 1999, the first version of WCAG 1.0 was released by W3C with three priorities, 14 guidelines, and 65 checkpoints [ 22 ]. In 2008, W3C released the second version of standards/guidelines, including 61 success criteria and 12 guidelines under four principles: perceivable, operable, understandable, and robust, concerning three conformance levels: Level A, Level AA, and Level AAA [ 23 ]. Furthermore, in 2018, the W3C published an updated version of WCAG 2.0 principles, namely the WCAG 2.1 standard [ 6 ]. It has all the principles, guidelines, success criteria, and conformance levels similar to WCAG 2.0 but they added one new guideline and 17 new success criteria. Therefore, completion of the WCAG 2.1 standard ensures the fulfillment of WCAG 2.0 and is followed with more accessibility concerns. The significant update in WCAG 2.1 is the ‘Operable’ principle. In this principle, a new guideline with six success criteria has been added.

In 2021, W3C extended the WCAG 2.1 guideline and released the WCAG 2.2, an updated version [ 24 ]. In this version, in the Operable principle under guideline (2.4), three new success criteria have been added. In December 2021, the last modified version of WCAG (3.0, working draft) was published, now in progress, waiting for the final draft of guidelines [ 21 ]. Figure  1 shows the WCAG standard with its principles, success criteria, and conformance levels. For the details about success criteria and conformance level, the author refers the reader to [ 24 ]. In addition, all the versions of WCAG followed three conformance levels of A, AA, and AAA to classify web content. By following the WCAG standard, developers and designers can make digital content accessible for a wide range of people with disabilities, including blindness, low vision or vision impairments, deafness and hearing loss, limited movement, dexterity, speech disabilities, sensory disorders, cognitive and learning disabilities, photo-sensitivity and combinations of these [ 25 ]. Nowadays, ensuring an accessible web and improving user experience is crucial for web engineers, researchers, and developers. According to the researchers' opinions, more research needs to be carried out in the next years to improve the accessibility of digital platforms [ 26 ]. Therefore, to understand web accessibility in-depth, a detailed and updated SLR approach is important.

Overview of web content accessibility guidelines (WCAG) version 2.0, 2.1 and 2.2

Our investigation found seven SLRs between 2010 and 2021 related to the area of digital accessibility (two), web accessibility (three), and web-based image and games accessibility (two). The main focus of these seven SLRs is to make digital content accessible for people with disabilities, which is also a prime objective of the digital accessibility consortium. A detailed discussion of the three SLRs concerning web accessibility has been described in the following subsection (2.2) and a comparison of our SLR with the seven earlier SLR studies is conducted in the discussion section.

2.2 Related SLR studies

In the web accessibility context, the first selected SLR was carried out by Akram et al. [ 14 ] to identify the issues with web accessibility of the Saudi Arabian university webpages from the web engineering point of view. To conduct this SLR, they followed three research questions: (1) what are the main principles of Web Content Accessibility Guideline 2.0 (WCAG-2.0) proposed by the W3C to improve web accessibility, (2) what is the compliance level of university and government websites with WCAG-2.0 globally, and (3) what is the compliance level of Saudi Arabian university and government websites with WCAG-2.0. To search past literature, they considered ten scientific databases: Google Scholar, Google search engine, EBSCO host, IEEE Explorer, Science Direct, The Elsevier, Springer Link, ACM Digital Library, Wiley, and Emerald, and found 15 pieces of literature from 2009 to 2017. Their systemic literature review concluded that 87% of the past research employed automatic accessibility testing tools to evaluate university websites. Their SLR also revealed that the most experimented automatic accessibility tools are Bobby, AChecker, eXaminator, TAW, Total Validator, EvalAccess, Cynthia Says, Magenta, Site Analyzer, MAUVE, FAE, WAVE, Valet, and W3C validator service. In addition, they incorporated the manual evaluation process (e.g., interview, questionnaire-based assessment). The manual investigation illustrated that in past research the majority of the work emphasized the improvement of a few accessibility issues such as navigation errors, orientation issues, timing errors, text equivalent to graphics, content, the validity of hypertext markup language (HTML), and cascading style sheets (CSS), use of HTML5, interface design, content, and scripting. However, they conclude that in Saudi Arabia, most universities do not follow World Wide Web Consortium guidelines.

The work proposed by Akram et al. is important in representing the insights of accessibility considering several aspects. However, to validate their represented statistics of implemented automatic accessibility testing with the experimented tools and to identify other possible techniques to validate the accessibility, Campoverde‑Molina et al. [ 15 ] carried out the second SLR and present the empirical results of the accessibility evaluation of educational websites. They have considered 25 past studies from 2009 to 2019 to answer ten research questions. This SLR investigated the selected papers focusing on the bibliometric analysis context and literature review. The SLR determined that 80% of past studies focused on automatic analysis through automatic accessibility evaluation tools, 8% through user incorporation, and 12% through hybrid approaches such as expert invitation, user involvement, and automated tools consideration. This SLR concluded that selected websites did not satisfy any version of the WCAG standard and their conformance levels that introduce the necessity of correction of errors by adopting automated tools and manual observation during website construction.

Following their first SLR, Campoverde‑Molina et al. [ 16 ] extended their previous SLR considering the period from 2002 to 2020 to investigate more research works to represent the accessibility insights in depth. This recent SLR aimed to analyze past literature that focused on the accessibility analysis of university websites. They performed an investigation of 42 selected papers obtained from three scientific databases (Web of Science, Scopus, and IEEE Xplore), focusing on the accessibility standards and accessibility evaluation methodologies. In 42 papers, they found that 38,416 university webpages have been experimented with in the past years. Their SLR result illustrates that all the experimented websites were from Asia. Most of the existing research has experimented with university homepages. All the past literature followed two standards: ISO/IEC 40,500:2012, and Sect. 508, to analyze the accessibility of web pages. They also concluded that past studies considered automatic evaluation tools to validate university web pages, which is around 90.47%. The most frequently used accessibility testing tools are AChecker, WAVE, Bobby, and TAW. However, the inspection result of this SLR is that most of the past investigated university websites showed violations of accessibility guidelines, most commonly adaptability, compatible, distinguishable, input assistance, keyboard accessible, navigable, predictable, readable, and text alternatives that show important accessibility issues.

The selected three systematic literature reviews represent the current insights of the web in detail, considering the term of accessibility context. Despite the importance of these SLR approaches, they have a poor concern about past research domains and lack consideration of engineering approaches, methods, etc. The lack of engineering methods shows the shortcoming of the past SLR that initiate the importance of a detailed future of SLR. In this paper, our presented SLR is unlike the other three systematic literature reviews. We consider a wide range of existing literature intending to determine the engineering approach to initiate future research to mitigate the current research gap.

3 Research methodology

This study aims to conduct a systematic literature review by following the SLR process guidelines and Kitchenham’s guidelines from Kitchenham and Charters [ 27 ]. This research considers three steps to facilitate the SLR approach: (i) planning the SLR process, (ii) conducting the SLR approach, and (iii) reporting the review findings. Figure  2 represents the flowchart of our SLR process.

Flowchart of the proposed systematic literature review (SLR)

3.1 Planning the SLR process

The main sub-activities related to planning the SLR are (i) research question specification, (ii) search string formulation, and (iii) database selection. All these sub-activities are described below.

3.1.1 Research questions

The first step of a literature review is to develop the research questions. Therefore, we developed the research questions according to our research focus. The two research questions are the following:

Research Question-1: What are the available methods, techniques, processes, and approaches to support the evaluation of accessible web?

Research Question-2: What are the current engineering assets (tools, technologies, etc.) to support the evaluation of accessible web?

3.1.2 Search string

To select the appropriate search strings, we defined a set of keywords according to our research questions concerning the accessibility and website domain. We tested the developed set of keywords in different scientific databases by searching manually and refined it based on the relevancy of the output with the research objective. The finally selected set of keywords represented using Boolean operation is the following:

{(Web engineering) or (Website accessibility) or (Web page accessibility) or (Universal accessibility design) or (Accessibility evaluation) or (Accessibility framework) or (Web accessibility methods and algorithms) or (Accessibility measuring software) or (Current accessibility violations)}.

3.1.3 Database selection

For the most relevant and updated literature identification, database selection is crucial. Several scientific databases are available, so appropriate database selection is critical. Herein considering the opinions of other researchers, we selected seven popular databases that provide quality literature and scientific publications. These databases used advanced search algorithms to extract the most related literature according to the user's interest. Seven databases used in this SLR are Scopus, Web of Science, Science Direct, ACM digital library, Google Scholar, IEEE Xplore, and PubMed.

3.2 Conducting the systematic literature review

This phase aims to describe review activities through the specification of (i) database searching and literature extraction, (ii) inclusion and exclusion implication, and (iii) data extraction and quality assessment. These sub-activities are described in detail in the following subsections. Figure  3 shows the flowchart of the review overview.

Flowchart of the review overview

3.2.1 Database searching and literature extraction

We tested the search strings in seven databases to extract past literature. These databases are accepted by scientific committees for scientific publishing. Most of the literature is open access. These databases have advanced search algorithms and semantic technology to retrieve the appropriate literature according to the search strings.

In total, 152 papers were found in the period from 2010 to November 2021 (Scopus: 30, Web of Science: 28, IEEE Xplore: 8, PubMed: 5, Science directory: 20, ACM digital library: 16 and Google Scholar: 45). Five studies were found from other source and were included in the preliminary screening process. These five papers were found in Research Gate (platform of scientific work) based on the suggestion of digital accessibility expertise (3 papers) and other colleagues’ recommendations (2 papers). These works were not available in the seven databases that we have used in this work. The considered five papers have potential contributions to web accessibility and significant observation that addressed the importance of consideration in this systematic literature review. Figure  4 shows the search result considering the number of papers selected in each database through the search query. However, Scopus, Web of Science, and Google scholar have a wider array of literature than other databases. Among 157 papers, we have selected the most related papers required for this review through inclusion and exclusion criteria (described in the next section).

The number of selected literature per database

3.2.2 Inclusion and exclusion criteria

The extracted literature has been evaluated to include the most relevant studies in this research. We excluded the literature that did not meet the inclusion criteria for the review. The inclusion criteria were the following: written in English, papers published in peer-reviewed journals or conferences (i.e., not books), publication period between 2010 and 2021, and describe accessibility improvement, development, or related to accessibility assessment.

The exclusion process was performed to eliminate papers from this review. The exclusion criteria were the following: duplicate papers, non-English papers, not directly related or irrelevant papers, papers that are not freely accessible, and those that are not research papers such as posters, letters, thesis, and editorials. After applying inclusion and exclusion criteria to 157 papers, the following observation was made: 12 papers were duplicates, 11 papers were not in English, 29 papers were not directly related to our research focus, and 7 papers were not research papers. In total, we excluded 59 papers by primary screening. After eliminating these papers, we conducted the proposed SLR process considering the selected 98 papers (including 6 past literature reviews). The entire literature selection process has been performed through the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) technique. The PRISMA flow diagram of the study selection is shown in Fig.  5 .

Study selection through PRISMA approach

3.2.3 Data extraction and quality analysis

In this study, our research was conducted based on the search results during 10–15 January 2022, returning 157 papers. To identify a high-quality paper, data extraction and quality assessment are essential. Several earlier literature reviews followed this technique for the primary evaluation of the selected studies. Therefore, we followed assessment guidelines to identify quality papers, complete paper reading, and answer our research questions. Table 1 shows the assessment criteria for the evaluation of selected studies.

For each question, we set the score to 0 or 1. For each positive answer, a paper gets a score of 1. If not relevant to the assessment questions, the score is 0. For the Q1 indexed journal, the additional points are + 0.50. Similarly, the extra points for the Q2, Q3, and Q4 indexed journal is + 0.40, + 0.30, and + 0.20, respectively. We incorporated Equation-1 and Equation-2 to calculate the final and normalization score to estimate the quality of each selected paper. After conducting the quality analysis, we consider only those studies that passed at least four quality assessment questions with α  ≥ 0.4 normalized scores. However, among 98 selected studies, six were excluded from this SLR (as shown in Fig.  5 , PRISMA diagram) based on the result of the quality assessment criteria. Table 2 shows the quality assessment result of the qualified 92 papers for this review.

3.3 Reporting the findings

In general, selected papers were related to web development, web accessibility, and information and communications technology (ICT) tools. The statistics of past research showed that existing SLRs focused on a few criteria, but other aspects also need to be considered. However, this study focused on previous SLR results and added new findings from our investigation results that were not highlighted in the earlier SLRs. Earlier SLRs considered accessibility requirements, standards, frequent violations, and improvement suggestions. However, accessible development criteria, evaluation tools development and their engineering methods, and updated validation and testing procedure need to highlight to identify the new research area. According to Durdu and Yerlikaya [ 28 ], before ensuring accessible web, web developers and designers should consider the standard guidelines and the requirements of people with disabilities. Also, Bradbard and Peters [ 29 ] shared the same observation. They highlighted that the majority of developers and designers have no adequate knowledge about accessibility requirements for people with disability and also lack knowledge about accessible web application development. Thus, in recent days, accessibility specialists have suggested checking accessibility criteria during the development and testing process through automatic accessibility testing tools and user and expert testing. Past works introduced various aspects of developing an effective webpage, but recent studies revealed that accessibility issues completely align with user satisfaction or usability. Therefore, the government of different countries and public and private organizations initiated a few guidelines concerning accessibility and usability criteria [ 30 ] that directed a new research area to make the development easier and barrier-free. In the Following, we would like to describe our findings and analysis results of the selected literature in the context of two research questions.

RQ-1: What are the available methods, techniques, processes, and approaches to support the evaluation of accessible web?

To answer the first research question, we analyzed 92 selected studies. The selected papers were classified into seven groups/processes: (i) accessibility requirements (AR), (ii) challenges (C), (iii) improvement directions (ID), (iv) framework design (FD), (v) framework implementations (FI), (vi) testing (T), and (vii) evaluation (E). All these phases are described in detail in the following subsections. Figure  6 presents the seven processes with an accounted number of papers for each process. Furthermore, nineteen studies emphasized two activities as presented in the Venn diagram of Fig.  7 , which is: {2 (AR & E) + 1 (AR & T) + 2 (AR & FI) + 1 (C & ID) + 1 (C & FD) + 4 (ID & T) + 1 (ID & E) + 1 (ID & FD) + 2 (FD & E) + 4 (T & E)}. The Venn diagram represents the number of papers that have multiple focuses instead of a particular focus or objective. In total 19 unique papers have been found that have multiple focuses. Figure  7 shows the number of papers with their associated activities through the blue arrow. For example, considering ‘accessibility requirements,’ 2 papers focused on accessibility requirements and evaluation process, 2 papers focused on accessibility requirements and implementation, and 1 paper focused on accessibility requirements and testing. Figure  7 shows the complete view of the number of papers with their multi-focused area. Moreover, results depict that past research mostly emphasized the technical processes, especially improvement direction, testing, and evaluation.

Percentage of studies considering each process related to web evaluation and accessible web applications

Venn diagram representing the number of studies for certain activities and multiple activities

3.3.1 Accessibility requirements (AR)

This section describes the accessibility and usability requirements with new methods for imposing the accessibility and usability requirements on the current web. Among 92 papers, nine (9) were related to accessibility requirements (representing 9.7% of the total literature) that emphasized ensuring the accessibility guidelines. These studies could be grouped into three main topics of interest, as presented in Table 3 .

In the context of accessibility requirements, Bai [ 31 ] and Henry et al. [ 32 ] described the importance of accessibility and usability criteria in web and mobile software applications. They added that improving web accessibility is essential for users with disabilities and non-disabled users. They indicate a significant gap between the needed strategies and the developed solutions for people with disabilities, including auditory, cognitive, neurological, physical, speech, and visual impairments. Therefore, the requirements of people with disabilities should be acknowledged during development as accessible technology is essential for equal access and interaction in today's digital world. Also, Riley-Huff [ 34 ] pointed out that the first step to developing an accessible website is following the web accessibility guidelines/standards. To ensure higher accessibility standards, a possible way is to improve accessibility and usability [ 35 ]. Thus, automatic accessibility testing tools are essential. To ensure usability, they mentioned a few existing models that are prominent to analyze. Another study by Wu et al. [ 33 ] investigated data visuality (chart type, chart embellishment, and data continuity) for people with intellectual and developmental disabilities. They emphasized that people with intellectual and developmental disabilities perform information processing differently. But the actual scenario is quite challenging that complicates the process of data visuality for these people. Thereby they suggested considering all the potential requirements with disabilities during development to improve data visuality and accessibility.

Sauer et al. [ 36 ] identified three criteria: accessibility, usability, and user experience. These are essential for making the internet platform accessible and convenient for people with and without disabilities. They suggested several methods to ensure accessibility (checklists, cognitive barrier walkthroughs, automatic checking), usability (user testing, observation, questionnaires, interviews, focus groups, heuristic evaluation, cognitive walkthrough, and data logging), and user experience. They suggested that accessibility and usability could be imposed during development to improve the user experience. Vu et al. [ 37 ] addressed that low-quality web designs often lead to user frustration that might cause abandonment of undesirable sites. They highlighted several potential and usable web design and evaluation components/methods to improve website usability, which results in a better user experience.

Furthermore, Almeida and Baranauskas [ 38 ] pointed out that web accessibility requirements for people with disabilities are crucial. The difficulty of understanding accessibility guidelines is the prime cause of inaccessible design and development. They also added that developers and designers are not experts and have limited knowledge of accessibility requirements. Therefore, they proposed an inclusive web-based collaborative tool to evaluate and modify the guidelines according to the universal and accessible design and development guidelines. It helps to represent the accessibility guidelines more skillfully. Furthermore, Gaggi and Pederiva [ 39 ] developed a tool for designers and developers concerning the same issue. They assisted the importance of accessibility measurement with a complete direction about guidelines that need attention during the web design and development phase.

3.3.2 Challenges (C)

This section describes the accessibility challenges that are generally liable for the current inaccessible web platform. Among 92 papers, four (4) studies were related to accessibility challenges (representing 4.3% of the total literature). These investigated studies could be grouped into three main topics of interest, as presented in Table 4 .

Researchers are trying to ensure an accessible web for more than a decade, including digital content, websites, user-machine interface, software, etc. Acosta-Vargas et al. [ 40 ] pointed out that to implement an accessible web, web researchers have found several challenges. They specified that accessible web page development required adequate knowledge that demands financial investment such as manufacturing and maintaining costs, testing costs, and quality assurance costs. These deliberations are crucial to improving the accessibility of the developed system. However, these deliberations rely on the organization's size, capital, opportunities, etc. Thus, ensuring these necessities is comparatively challenging. Inal et al. [ 41 ] showed their effort by conducting a user survey about digital accessibility practices to identify the challenges of creating an accessible system. They invited user experience (UX) professionals to find the most common challenges. The challenges were associated with time constraints, lack of training cost constraints, work overload, not being a requirement for the organization, not being a customer requirement, and people with disabilities or special needs not included as target users. Inal et al. highlighted that such challenges act as barriers to considering accessibility requirements seriously, which is responsible for the current inaccessible web.

Another study by Brajnik and Vigo [ 42 ] addressed some crucial challenges that need to consider for introducing an accessible web. The most pressing ones are validity, reliability, sensitivity, and adequacy of user-tailored metrics. Challenges with validity are associated with different validation systems of metrics. For example, there are no specific/gold standards to produce the output for the validation process. The reliance on tools and their limited coverage, completeness, and correctness are heterogonous issues that arise as challenges during metrics result in validation. The reliability of several evaluation performance metrics (human judgment, automatic evaluation, etc.) depends on the evaluation metric transparency and their reproducible and comparable results. Brajnik and Vigo depict that the actual cause for low reliability is the adopted sampling method to evaluate the pages, such as accessibility violation criteria, identified data, formulae, or methods to compute the final score. Sensitivity and adequacy are related to the meaningfulness and suitability of the generated scores through metrics. User-tailored metrics depend on the user's ability as all users have different needs. Accessibility barriers affect different ability users in various manners. Thus, such aspects addressed by Brajnik and Vigo need to be considered in future research.

Furthermore, Palaskar et al. [ 43 ] revealed that existing automated accessibility testing tools consider around 50% of Web Content Accessibility Guidelines. Though most of the rules are easy to understand, sometimes it is pretty challenging to implement all the natural language rules in an automatic system. They also claimed that some rules are unacceptable for ensuring accessibility, and others are inappropriate, for example, rules for color schemes and image captions accessibility checking. To develop an accessible website, consideration of some specific aspects is insufficient. Sometimes, accessibility checking requires more than the considered rules. Thus, validating the appropriate rules and incorporating all the guidelines is the major challenge for current web-based accessibility research.

3.3.3 Improvement directions (ID)

This section describes directions for future improvement of accessible development. Among 92 papers, ten (10) studies were related to improvement directions (representing 10.8% of the total literature). These investigated studies could be grouped into two main topics of interest, as presented in Table 5 .

In the context of improvement direction, few studies focused on the technological aspects of accessible development. Edelberg and Verhulsdonck [ 44 ] addressed that web developers and associated authorities choose the colors and font based on the choice of organization identity. However, through this process, it is not always possible to address accessibility issues such as inaccessible color and contrast, and fonts, which makes a difference in design and development for people with disabilities such as a person with low vision. They suggested that colors, fonts, and supporting elements should be perceived correctly during the development phase. Development should be encoded according to the content management system (CMS) to enhance the user experience of a wider audience. Brajnik and Vigo [ 42 ] pointed out the significant progress for accessibility metrics in the last decades. However, immaturity is still present in modern development. Thus, future research for further improvements is indicated. Based on their observation, they added a few improvement directions. For example, the implementation should follow Agile, an iterative development model to keep track of accessibility issues. In addition, following the hybrid approach like human judgments through different levels of expertise and users, such as disability type or user, might improve the accessibility of the development. Miesenberger et al. [ 45 ] presented some accessibility challenges related to cognitive disability with associated improvement direction. For instance, individual user-centered and personal services-based design and development should ensure. Accessibility requirements should be tested in development cycles with several testing tools (keyboard/mouse logging, eye tracking, etc.). To better usability design, an advanced development framework or platform for R&D should incorporate, and the development should follow the process model (e.g., Waterfall, Iterative, Spiral, Agile, etc.). In addition, Alismail and Chipidza [ 46 ] recommended following the WCAG 2.0 and 2.1 guidelines to develop accessible websites by addressing potential accessibility issues. Also, they emphasized user testing by involving people with disabilities, integrating assistive technologies during web accessibility evaluation, incorporating accessibility requirements during design, development, and maintenance phases, and arranging training for web developers and designers to spread accessibility awareness.

In the accessible prototype design context, Bhagat and Joshi [ 47 ] presented a few technical recommendations to overcome accessibility challenges. They also mentioned that all the accessibility requirements should be checked and validated by the website's quality assurance (QA) team. Ojha et al. [ 48 ] provided a few guidelines based on their detailed study on improving website accessibility with readability. Readability improvement suggestions are related to website structural components such as hyperlinks and image alt-text. These functions should ensure by incorporating the variable weight-based approach for different elements of web pages. Also, website dynamism should be considered in readability score computation to improve the readability in terms of the accessibility of the website. Furthermore, Morris et al. [ 52 ] emphasized ensuring alt text of visual content for screen reader users. They have articulated design guidelines for the representation of visual content with prototype design requirements, especially for people with vision impairments, to facilitate and improve visual content accessibility.

3.3.4 Framework design (FD)

This section describes several frameworks designed to contribute to the web evaluation process to facilitate web platform accessibility. Among 92 papers, seventeen (17) were related to framework design (representing 18.4% of the total literature). These investigated studies could be grouped into three main topics of interest, as presented in Table 6 .

To contribute to accessible user-centric design, Alahmadi [ 53 ] proposed a state-of-the-art framework for web accessibility evaluation to facilitate accessibility measurement and identify accessibility standards errors. The proposed model ensures user-centered design (UCD) based on usability and accessibility guidelines for deaf, visually impaired, and deaf-blindness people. Kaur and Gupta [ 54 ] proposed a quality index evaluation framework to evaluate website design to ensure the quality of web design and development. Hassouna et al. [ 55 ] addressed some significant issues for users with visual impairment. Concerning the accessibility requirements for users with vision impairment, they designed an accessible web page prototype. Few studies focused on ontology design. For example, Sapna and Mohanty [ 56 ] proposed a large-scale test scenario management process using ontology modeling with the help of Web Ontology Language (OWL) to facilitate the software and web development, and testing process by providing faster and more reliable services. Kourtiche et al. [ 57 ] designed an ontology of user-profiles considering user disability context to understand various user requirements during accessible web development. Another study proposed by Fayzrahmanov et al. [ 58 ] developed a user interface to improve web navigability considering the user requirements with visual impairment.

In the context of web accessibility evaluation, Li et al. [ 59 ] designed an interactive web accessibility evaluation system based on the Chinese government guidelines. This framework incorporates automatic tools and human inspection to make evaluation feasible for large web pages. Alsaeedi [ 60 ] proposed a novel framework for evaluating the performance of two accessibility testing protocols in webpage evaluation. Song et al. [ 61 ] designed a crowdsourcing-based web accessibility evaluation framework to validate against WCAG. It generates the automatic accessibility score of each evaluated webpage according to the weight of each checkpoint. Giovanna et al. [ 62 ] developed an open support accessibility evaluation tool to improve automatic accessible support following accessibility conformance testing (ACT) rules. Sanchez-Gordon and Luján-Mora [ 64 ] proposed an agile environment-based accessibility evaluation framework to improve evaluation results based on automated tools, simulators, and expert and user-based testing. In further evaluation, Song et al. [ 65 ] addressed the complexity of accessibility evaluation methods and the shortage of experts in this field. These aspects make the accessibility evaluation process difficult and reduce their significance. Thus, they proposed a crowdsourcing-based web accessibility evaluation system that uses decision strategies such as the golden set strategy and time-based golden set strategy. Palaskar et al. [ 43 ] claimed that most existing Americans with Disabilities (ADA) tools detect only 50% to 60% of accessibility violations because the rules are not understandable. They developed an API to test websites according to the WCAG 2.0 guidelines and A, AA, and AAA conformance level. Additionally, Acosta-Vargas et al. [ 66 ] designed a heuristic method to enable accessibility measurement of websites to ensure an accessible and inclusive web platform.

Additionally, Won [ 67 ] developed a color tool to understand website color meaning for accessible design practice. The proposed approach can evaluate webpage HTML design prototypes and provide a clear understanding of product-specific colors, cross-cultural color meanings, and color preference. It assists designers in making better color decisions during the design and development phase.

3.3.5 Framework implementations (FI)

This section describes several studies that implemented different approaches to contribute to evaluating an accessible web platform. Among 92 papers, seventeen (17) were related to implementation purposes (representing 18.4% of the total literature). These investigated studies could be grouped into three main topics of interest, as presented in Table 7 .

Concerning web accessibility evaluation, many research studies proposed decision support systems, evaluation tools, algorithms, frameworks, models, and interfaces. Mohamad et al. [ 68 ] developed a decision support system for large-scale compliance assessment against web accessibility recommendations and legislation. This architecture aims to provide scalable, interoperable, and integrated web accessibility assessment in the context of user-centric design to develop accessible web and mobile applications. Li et al. [ 69 ] proposed an EDBA decision support system for website accessibility evaluation at a lower cost. Among other scientific studies, Žuliček et al. [ 70 ] developed an accessibility evaluation tool to evaluate the whole webpage, including subpages, to provide a detailed analysis and simplified code refinement. Oliveira et al. [ 71 ] developed an accessibility assessment tool to analyze the strength and weaknesses of the website following the Web Content Accessibility Guidelines. Rashida et al. [ 72 ] developed an automated web-based tool to identify the quality of academic websites by considering websites' content of information, loading time, and overall performance metrics. Lim et al. [ 73 ] proposed an open-source customized automated accessibility testing tool based on the existing Axe accessibility testing engine to scale up the accessibility testing process. Gaggi and Pederiva [ 39 ] developed an automatic tool to assist designers and developers in understanding development aspects that should be considered during the development process to introduce an accessible website. In addition, Duarte et al. [ 74 ] developed an algorithm to automatically identify the semantic similarity between web content and its textual description in the context of web accessibility evaluation guidelines or rules. Wu et al. [ 75 ] developed a semi-supervised regression algorithm involving manual evaluation (webpage sampling) and automatic accessibility testing to generate the overall evaluation result of the website. Almeida and Baranauskas [ 38 ] developed a framework following universal design (UD) accessibility guidelines to help designers overcome accessibility barriers in a web-based system. Morato et al. [ 76 ] proposed a framework for automatic website accessibility checking in the context of readability through a linguistic characteristics analyzer to identify the best linguistic feature to detect text readability.

In the context of accessibility evaluation for visually impaired people, Michailidou et al. [ 78 ] implemented an open-source web accessibility prediction model to predict and visualize the complexity of web pages in the form of a pixelated heat map. Another work proposed by Bonacin et al. [ 79 ] developed an adaptive interface focusing on the requirements of Color Vision Deficiency (CVD) people considering automatic recoloring facilities to facilitate the interaction of CVD people with the web.

Additionally, for accessible prototype development, Matošević et al. [ 82 ] developed a machine learning algorithms-based expert knowledge system to classify web pages or parts of web pages to improve search engine optimization (SEO) guidelines.

3.3.6 Testing (T)

This section describes the studies associated with the testing purpose for accessibility validation of web platforms. Among 92 papers, thirty (30) were related to accessibility testing (32.6% of the total literature). These investigated studies could be grouped into five main topics of interest, as presented in Table 8 .

In the context of testing, many research studies focused on accessibility testing tools to validate website accessibility considering several disabilities. Few studies tested accessibility issues by incorporating automated accessibility testing tools. Martins et al. [ 83 ] tested eHealth websites using a single accessibility testing tool to identify the accessibility issues. Addressing the effectiveness of multiple automatic testing tools, Padure and Pribeanu [ 84 ] applied six accessibility evaluation tools to evaluate their selected websites. They suggested that a single testing tool is not enough to identify all the accessibility issues of a website. In other studies, Marino and Alfonzo [ 35 ] claimed that automatic tools are inadequate to clarify all the accessibility issues of websites. Thus, further manual observation is required. Therefore, Hassouna et al. [ 85 ] initiated a semi-automated evaluation process utilizing an automatic tool and human observation to evaluate design prototypes of websites. The considered evaluation tool is effective as it identifies problems in the design stage. For example, if it detects any error, it redirects to the design stage to show the problem and repair the design problems without modifying the original code. Also, Bhagat and Joshi [ 47 ] observed that a lack of awareness regarding assistive technologies and global accessibility standards is responsible for less inclusive and less accessible website design and development. Thus, they conducted the experimental procedure following automatic and user testing to help service providers, government divisions, and ministries ensure maximum accessibility of online platforms. Rysavy and Michalak [ 92 ] evaluated the library tools and services in terms of accessibility and usability with open-source tools that emphasized the involvement of blind student workers to validate the resulting transparency.

In contrast, few studies evaluated websites considering several tools and techniques to measure the performance of accessibility, usability, readability, and quality. Akgül [ 93 ] evaluated website accessibility, usability, quality, and readability using several tools and techniques. The author employed online open-source tools for accessibility testing and visual and manual inspection for usability testing considering several design standards and Google search results. For quality performance, Akgül incorporated webpage monitoring software considering download time, page size, and objects per website. Finally, evaluated readability, considering text alignment, webpage language, and all-caps text. Grant et al. [ 97 ] examined web accessibility and user experience using the hidden code optimization technique. They aim to motivate better web development practices and improve the overall holistic user experience. Ajuji et al. [ 96 ] and Kumar et al. [ 98 ] proposed two scientific studies. They depict that though the increasing web interactivity is significantly visible, still people with disabilities are finding it difficult to access. They highlighted that website has non-compliant issues against the W3C guidelines. Thus, to evaluate the websites' conformance to the WCAG, Ajuji et al. implemented an automatic accessibility testing tool to evaluate the websites in terms of Perceivable, Operable, Understandable, and Robust. Kumar et al. considered a simulator to visualize the accessibility issues for different types of disabilities. Burkard et al. [ 99 ] counteracted that the importance and awareness of digital accessibility are often not recognized during web development. Due to the complexity of the guidelines, people are not motivated to follow them. Therefore, automatic accessibility barrier checking, identifying, and fixing is an important issue. They considered several accessibility monitoring systems to validate the websites and compare tools in the context of completeness and correctness. Also, Alshamari [ 51 ] evaluated the accessibility of E-commerce websites through multiple accessibility evaluation tools to generate evaluation reports, locate potential errors, and direct warnings to help in accessible website design and development. Furthermore, Król et al. [ 105 ] evaluated the quality of the websites through automatic testing tools considering website performance, SEO quality, website availability, and mobile friendliness.

In the context of better user experience, Bai [ 31 ] emphasized accessibility and usability observation as website accessibility and usability are highly correlated. Bai choose the most frequently used automatic conformance testing tools and several usability testing models. Another study proposed by Yi [ 106 ] claimed that most websites are not accessible to people with visual impairment, even not readable by the screen reader. This problem happens as websites have too many menus, multiple frames, and a lack of alternative text. Thus, Yi proposed the web accessibility evaluation process using questionnaire-based user testing incorporating people with visual impairment. All the users tested websites using assistive technologies such as screen readers to share their opinions by answering questions about the websites’ accessibility.

3.3.7 Evaluation

This section describes several accessibility evaluation methods and techniques. Among 92 papers, nineteen (19) were related to accessibility evaluation (representing 20.6% of the total literature). These investigated studies could be grouped into three main topics of interest, as presented in Table 9 .

Here, we focus on the studies performed in the context of evaluation purposes. For accessibility evaluation, few studies focused on the questionnaire and expert-based evaluation. Hassouna et al. [ 55 ] and Moreno et al. [ 107 ] argued that the web is less accessible for people with vision impairments. They utilized questionnaire-based evaluation for accessibility prototypes with the participation of people with visual impairment. For descriptive analysis of the questionnaire result, they used statistical techniques to observe the relationship between the questionnaire items and the dependent variables. Another study by Hadadi [ 110 ] stated that designers are not careful about considering the requirements of disabilities such as color blindness. Thus, they overlooked the accessibility criteria to integrate into the design tools. This work evaluated the accessibility of widely used design tools through user feedback. The aim was to increase accessibility awareness and encourage product designers to design and develop an accessible solution. Alcaraz Martínez et al. [ 114 ] addressed that several statistical charts on websites are valuable for representing the information. Unfortunately, charts on websites are not accessible for people with low vision and CVD. Thus, they performed a heuristic accessibility evaluation of statistical charts focusing on the needs of people with low vision and CVD to find the usability problems in user interface design. In another study, Giovanna et al. [ 62 ] conducted a quantitative and qualitative analysis of user feedback regarding task completion time and computing success rate metric. In addition, some existing literature focused on automatic testing validator performance assessment and effeteness. Krawiec and Dudycz [ 108 ] evaluated the performance of automatic accessibility testing validator considering standards, the number of page validation ability, user interface interactivity, software update, free/commercial, etc. This assessment system helps to understand the most effective tool according to the specific requirements. Kous et al. [ 102 ] reinforced that several statistical methods using the quantitative data analysis concept are valuable for validating automatic web accessibility testing results. Grantham et al. [ 109 ] claimed that low literacy and numeracy skills sometimes affect user access and understanding of the website's content. Following accessibility guidelines and incorporating advanced assessment criteria against international legal accessibility requirements should be considered to ensure an accessible web.

Considering readability, Kimmons [ 111 ] claimed that most websites have accessibility issues with content understanding (readability) and structural elements. These issues introduce serious accessibility problems and act as a leading cause of reducing accessibility. Another work is conducted by Sun et al. [ 113 ] to assess e-textbooks’ accessibility. They investigated accessibility considering reading time and accuracy to content-related questions. They evaluated experiment results through composite, average, and weighted average scores to examine user experience and performance. However, Ojha et al. [ 48 ] addressed a wide array of accessibility and readability evaluation metrics for online content based on machine learning and statistical language modeling techniques.

In the context of usability evaluation, Radcliffe et al. [ 112 ] conducted m-Health app evaluation considering accessibility and usability concerns. The evaluation was performed through rapid user-testing and quantifying usability feedback. The user testing result and usability feedback were validated through several standardized evaluation methods for inclusive design requirements specification. Wu et al. [ 33 ] addressed that web designers and developers should focus on usability criteria instead of user experience as ensuring usability improves accessibility. Thus, their paper presents several methods and techniques for usability and accessibility evaluation of web design, such as naturalistic observation, participatory evaluation, web-based methods, prototyping, usability inspections, and usability laboratory testing. Giraud et al. [ 116 ] indicated that filtering redundant and irrelevant information is crucial for people with visual impairments similar to sighted users to improve the accessibility of the web. Therefore, to improve website usability, some specific needs of users with visual impairment are emerging to consider. They conducted experiments with users with vision impairment to determine the accessibility of web content in terms of filtered or not irrelevant and redundant information. Also, cognitive load, performance, and participants' satisfaction were investigated through the dual-task paradigm.

Figure  8 represents the number of papers on each topic of interest according to the seven processes. This figure depicts that the number of proposed approaches for accessibility testing to identify accessibility issues is more frequent than other approaches such as development, implementation, and evaluation. The observation result of research question 1 concludes that the number of the proposed approach for the development and implementation of the accessible web evaluation approach was relatively lower, which addresses a further concern of the web researcher.

Number of studies on each topic of research interest according to the seven processes/phase

RQ-2: What are the current engineering assets (tools, technologies, etc.) to support the evaluation of accessible web?

We analyzed the selected papers and identified several groups of interest considering our seven processes. Table 10 summarizes the 22 groups of topics of interest related to the seven processes, including verities of methods, tools, and techniques to answer our second research question.

Asset description

From the 7 process groups and 22 topics of interest (Table 10 ), we aimed to highlight the main assets related to the engineering aspects to support the technical process we have found in our SLR offered by past researchers. These findings will help developers, web engineers, accessibility researchers, and associated authorities to support the accessible design and development process. The addressed assets are listed and described below.

Assets of accessibility requirements (AR)

(AR 1 .) Assets for the importance of accessibility and usability guidelines: (1)  explanation  of higher accessibility standards in website evaluation [ 31 ]; (2)  explanation  of the importance of accessibility guidelines and user requirements for people with disabilities [ 32 ]; (3) sets of usability requirements  for conventional visualization elements design for cognitive barriers people [ 33 ]; (4)  explanation  of web usability and accessibility requirements of WCAG 2.0 and ISO 9241 standards [ 34 ].

Therefore, in this group, we identified two subgroups of assets: s-group-1: 3 studies for the explanation, and s-group-2: 1 study for requirements.

(AR 2 .) Assets for accessibility, usability, and user experience improvement methods: (1)  methods  to improve accessibility, usability, and user experience [ 36 ]; (2)  methods  to understand user perception to improve usability and user experience [ 37 ].

Therefore, in this group, we identified one subgroup of assets: s-group-1: 2 studies for methods.

(AR 3 .) Assets for accessibility requirements specification: (1) Faware is a framework for accessibility requirements representation and implementation in visualization elements design and development [ 38 ]; (2) WCAG4All is a tool for understanding accessibility requirements following standards guidelines [ 39 ];

Therefore, in this group, we identified two subgroups of assets: s-group-1: 1 study for framework and s-group-2: 1 study for tools.

Assets of challenges (C)

(C 1 .) Assets of limited resource adequacy: (1)  cost  for maintaining, testing, and quality assurance is challenging that depends on organization size, capital, and opportunities [ 40 ]; (2)  opportunities for the training program, learning materials, etc. are not enough for accessibility knowledge improvement [ 56 ]; 3) practical experience and advanced knowledge  of UX professionals from different countries are limited [ 41 ].

Therefore, in this group, we identified three subgroups of assets: s-group-1: 1 study for cost, s-group-2: 1 study for opportunities, and s-group-3: 1 study for experience and knowledge.

(C 2 .) Assets of success criteria validation: (1)  metrics  for accessibility evaluation concerning validity, reliability, sensitivity, and adequacy are challenging to ensure [ 42 ].

Therefore, in this group, one subgroup of assets was found: s-group-1: 1 study for metrics.

(C 3 .) Assets of rules optimization : (1)  guidelines  are not enough or appropriate, even difficult to incorporate in automated systems or web development processes [ 43 ].

Therefore, we found one subgroup of assets in this group: s-group-1: 1 study for guidelines.

Assets of improvement directions (ID)

(ID 1 .) Assets for technological aspects: (1) guidelines for accessible and functional prototype design and development [ 44 ]; (2) directions for accessible development [ 42 ]; (3) directions for cognitive disabilities and their particular accessibility barriers in recent development [ 45 ]; (4) suggestions for development that would be facilitated and tested during the design and development phase [ 46 ].

Therefore, in this group, we identified three subgroups of assets: s-group-1: 1 study for guidelines, s-group-2: 2 studies for directions, and s-group-3: 1 study for suggestions.

(ID 2 .) Assets for accessible prototype design: (1)  directions  for accessible prototype design and development [ 47 ]; (2)  guidelines  for improving website readability by ensuring proper structural components and website dynamism [ 48 ]; (3)  suggestions  for spreading awareness, organizing training, and focusing on the accessible prototype design to make the websites accessible to all, including people with special needs [ 49 ]; (4)  suggestions  for accessible prototype design to ensure advanced multimedia components [ 50 ]; (5)  suggestions  for some potential features that should be taken into consideration during feature development [ 51 ]; (6)  guidelines  for visual content representation and accessible prototype design for screen reader users [ 52 ].

Therefore, in this group, we identified three subgroups of assets: s-group-1: 2 studies for guidelines, s-group-2: 1 study for directions, and s-group-3: 3 studies for suggestions.

Assets of framework design (FD)

(FD 1 .) Assets for accessible user-centric design practice: (1) state-of-the-art  framework for university website accessibility evaluation for students with hearing and visual impairment [ 53 ]; (2)  evaluation  of web page prototype design considering blind user requirement [ 55 ]; (3) OUPIP is a user profile-based ontological  model  for designers and developers to develop applications, and devices considering user’s needs, disability type and dynamic context [ 57 ]; (4) multi-axial serialization  framework  for the users with visual impairment to understand and find the required information in the webpage [ 58 ]; (5)  Ontology  for test management process to provide detailed knowledge about the specific domain and captured requirements for testing [ 56 ]; (6) tool for quantitative measurement by evaluating website HTML code to identify the quality of the website design [ 54 ].

Therefore, in this group, we identified five subgroups of assets: s-group-1: 2 studies for framework, s-group-2: 1 study for evaluation, s-group-3: 1 study for models, s-group-4: 1 study for tool, and s-group-5: 1 study for ontology.

(FD 2 .) Assets for web accessibility evaluation: (1) design a cost-effective crowdsourcing  framework  for web accessibility evaluation considering 25 checkpoints and 5 conformance levels [ 59 ]; (2) proposed a  framework  in order to evaluate the well-known automatic accessibility tools in terms of webpage accessibility through their proposed measurement metrics [ 60 ]; (3) a crowdsourcing  framework  for web accessibility evaluation against web accessibility content guidelines checkpoints [ 61 ]; (4) an open and flexible accessibility testing  tool  to support single and multi-page validation [ 62 ]; (5) WUAM is a framework  for websites usability and accessibility evaluation to improve website performance [ 63 ]; (6) proposed a framework  for web accessibility improvement following ISTQB in agile contexts [ 64 ]; (7) proposed a crowdsourcing  framework  for website accessibility evaluation to identify the accessibility barriers and determine the overall accessibility level [ 65 ]; (8) proposed an API  based website accessibility testing tool following ADA guidelines to identify the potential errors and violations, even without prior knowledge [ 43 ]; (9) proposed a framework  for website accessibility barrier measurement according to several variable magnitude techniques [ 50 ]; 10) proposed a heuristic  method  to determine the level of accessibility of high ranked websites [ 66 ].

Therefore, in this group, we identified four subgroups of assets: s-group-1: 7 studies for framework, s-group-2: 1 study for method, s-group-3: 1 study for tool, and s-group-4:1 study for API.

(FD 3 .) Assets for accessible color design: (1) an accessible color suggestions  tool  for designers to improve their color judgment ability and increase their inspiration for accessible design practice [ 67 ].

Therefore, in this group, we identified one subgroup of assets: s-group-1: 1 study for tools.

Assets of framework implementation (FI)

(FI 1 .) Assets for web accessibility evaluation system: (1) user-centric holistic decision support environment  system  for web and mobile application accessibility evaluation [ 68 ]; (2) a cost-effective task assignment-based decision support  system  for web accessibility evaluation [ 69 ]; (3) a module  for automatically analyzing, identifying and solving the accessibility issues [ 70 ]; (4) an automated website readability assessment  model to improve the accessibility and readability of the website [ 76 ]; (5) ShoppingForAll is a tool  for evaluating and identifying the strength and weaknesses of the website in terms of user satisfaction and accessibility criteria [ 71 ]; (6) an  algorithm  for semantic similarity improvement of website content from the web accessibility perspective [ 74 ]; (7) a  tool  for quality assessment of the university websites by assessing website source code [ 72 ]; (8) FAware is a  tool  to provide accessibility issues and available suggestions [ 38 ]; (9) a semi-supervised  model  to evaluate and predict website accessibility [ 75 ]; (10) an open-source, industry-standard  tool  to addresses the shortcomings of current accessibility testing tools for the local government context [ 73 ]; (11) WCAG4All is a  tool  for consulting web designers and developers about accessibility guidelines [ 39 ]; (12) WAccess is a browser extension open-source accessibility testing  tool  to evaluate websites against WCAG guidelines [ 77 ].

Therefore, in this group, we identified five subgroups of assets: s-group-1: 2 studies for systems, s-group-2: 1 study for modules, s-group-3: 2 studies for models, s-group-4: 6 studies for tools, and s-group-5: 1 study for algorithms.

(FI 2 .) Assets for web accessibility evaluation for visually impaired users: (1) ViCRAM is a tool  to predict the visual complexity of the web pages associated with accessibility issues for people with visually impaired or low vision people [ 78 ]; (2) FAIBOUD is a framework  to facilitate the interaction of CVD people with the web [ 79 ]; (3) proposed an automatic  system  for identifying website drop-down menu widgets [ 80 ].

Therefore, in this group, we identified three subgroups of assets: s-group-1: 1 study for tool, s-group-2: 1 study for framework, and s-group-3: 1 study for the system.

(FI 3 .) Assets for accessible prototype improvements : (1) proposed a method  to improve accessibility issues by modifying faulty code into correct code to make content management system-based websites more accessible [ 81 ]; (2) An expert knowledge  system  to detect web page SEO quality [ 82 ].

Therefore, in this group, we identified two subgroups of assets: s-group-1: 1 study for method and s-group-2: 1 study for the expert system.

Assets of testing (T)

(T 1 .) Assets for automatic detection of accessibility issues : (1) ACCESSWEB is an automated validator  for accessibility evaluation considering different accessibility guidelines [ 83 ]; (2) TAW is an automated validator  for web pages evaluation against the web content standards [ 35 ]; (3) Total Validator is an automated validator  to validate accessibility against standards guidelines [ 86 ]; (4) A semi-automated  process  is to evaluate website design prototypes and repair without modifying the original page code [ 85 ]; (5) AChecker and TAW  automated validators are to validate the accessibility of the website and identify the associated issues that violated accessibility guidelines [ 87 ]; (6) automatic testing by AChecker, Total Validator, WAVE, and HTML/CSS/ARIA  automated validators  for evaluation of higher educational institute websites [ 88 ]; (7) hybrid accessibility testing  process  with AChecker, WAVE, and aXe automatic accessibility testing tools and JAWS and Non-Visual Desktop Access, two open-source screen reader applications [ 47 ]; (8) WAVE is an automated validator  to indicate accessibility issues and related accessibility features [ 89 ]; (9) AChecker, Cynthia Says, Mauve, TAW, Total Validator, and Wave are automated validators to identify the accessibility issues and compare their result to understand the effectiveness of the system [ 84 ]; (10) AChecker, WAVE, and SortSite are automated validators  to identify the shortcoming of websites [ 46 ]; (11) AChecker, Cynthia Says, EIII Checker, MAUVE, SortSite, TAW, Tenon, and WAVE are automated validators  to identify the effectiveness of result considering coverage completeness, correctness, specificity, inter-reliability and intra-reliability, validity, efficiency, and capacity [ 90 ]; (12) multi-tool accessibility assessment through  automated validators  such as AChecker, Cynthia Says, Tenon, WAVE, Mauve, and Hera to perform a comparative analysis of websites to identify the effective testing tool [ 49 ].

Therefore, in this group, we identified two subgroups of assets: s-group-1: 10 studies for the automated validator, and s-group-2: 2 studies for the process.

(T 2 .) Assets for content evaluation for osteoarthritis: (1) SMOG and FOG are two  automated validators  to determine webpage content readability considering informative images and relevant video [ 91 ].

Therefore, we identified one subgroup of assets in this group: s-group-1: 1 study for the automated validator.

(T 3 .) Assets for accessibility evaluation for blind users: (1) WAVE is an online  automated validator  for accessibility issues identification of library tools and services for blind users [ 92 ].

Therefore, this group identified one subgroup of assets: s-group-1: 1 study for the automated validator.

(T 4 .) Assets for accessibility evaluation: (1) propose a hybrid  evaluation  approach for improving user experience [ 97 ]; (2 ) A hybrid  evaluation  process for accessibility, usability, quality and readability testing [ 93 ]; (3) A semi-automated evaluation process incorporating AChecker, Total Validator, WAVE and expert opinion to examine the webpage code [ 94 ]; (4) AChecker is an automated validator  to analyze education cooperative websites to determine its accessibility considering disabilities [ 95 ]; (5) TAW is an automated validator  to validate websites against the conformance of WCAG 2.0 [ 96 ]; (6) A simulator  for visual, hearing and mobility impairment to visualize the accessibility issue associated with the particular disability [ 98 ]; (7) A semi-automated process considering axe Monitor, Pope Tech, Siteimprove, ARC with user feedback to validate websites accessibility [ 99 ]; (8) WAVE is an automated validator  to validate the accessibility of COVID-19 vaccine registration portals [ 100 ]; (9) accessibility evaluation through comparative  analysis  using automatic accessibility testing protocols and statistical observation [ 101 ]; (10) AChecker is an automated validator  to evaluate website accessibility [ 102 ]; (11) AChecker, Cynthia Says, and TAW are automated validators  to validate website e-accessibility [ 103 ]; (12) A comparative analysis using Webaccessibility automated accessibility validator and statistical technique to validate the websites against WCAG 2.1 conformance guidelines [ 104 ]; (13) Google PageSpeed Insights, Blink Audit Tool, Backlink Checker, WAVE and Bulk are automated validator  to assess and evaluate website quality [ 105 ]; (14) Achecker, TAW, Eval Access, MAUVE and FAE are automated validators  to identify the accessibility issues of the selected websites [ 51 ].

Therefore, in this group, we identified four subgroups of assets: s-group-1: 2 studies for evaluation, s-group-2: 7 studies for the automated validator, s-group-3: 1 study for the simulator, s-group-4: 2 studies for analysis, and s-group-5: 2 studies for the process.

(T 5 .) Assets for better user experience: (1) FAE, Nielsen’s10-item metric, and Baker’s six-dimension are automated validators  for accessibility and usability testing for better user experience [ 31 ]; (2) questionnaire-based user  assessment  to identify the accessibility incompatibility with screen reader application [ 106 ].

Therefore, in this group, we identified two subgroups of assets: s-group-1: 1 study for automated validator and s-group-2: 1 study for assessment.

Assets of evaluation (E)

(E 1 .) Assets for accessibility evaluation methods: (1) manual  assessment  through assistive technology with users and experts in this field [ 83 ]; (2) questionnaire-based  assessment  for people with visual impairment through several data analysis techniques [ 85 ]; (3) questionnaire-based  evaluation  for discovering the navigation strategies of low vision people that cause to experience accessibility barriers [ 107 ]; (4) automatic  assessment  system to identify the most effective validator for accessibility testing [ 108 ]; (5) statistical data  analysis  to validate the reliability of the questionnaire result [ 55 ]; (6) quantitative data  analysis  using statistical analysis methods [ 102 ]; (7) manual  assessment  criteria for accessibility assessment of Australian private and governmental websites against DDA standards [ 109 ]; (8) user  evaluation  of Adobe online design platforms tool with the help of mix panel data analysis [ 110 ]; (9) statistical  evaluation  for quality analysis of the websites [ 105 ].

Therefore, in this group, we identified three subgroups of assets: s-group-1: 4 studies for assessment, s-group-2: 3 studies for evaluation, and s-group-3: 2 studies for analysis.

(E 2 .) Assets for readability evaluation tools and techniques : (1)  metrics / tools  for website content readability measurement to make website content universally accessible [ 48 ]; (2) descriptive  evaluation  of university homepage to validate the readability [ 111 ].

Therefore, in this group, we identified two subgroups of assets: s-group-1: 1 study for metrics/tool and s-group-2: 1 study for evaluation.

(E 3 .) Assets for usability evaluation methods: (1) statistical  techniques  for usability testing of m-Health application [ 112 ]; (2) questionnaire-based  evaluation  for user experience testing [ 113 ]; (3) quantitative and qualitative  analysis  considering the user performance, computing task completion time, and correct task completion ratio [ 62 ]; (4) quantitative and qualitative  analysis  with statistical measurements to evaluate user perceptions [ 115 ]; (5) statistical  analysis  to determine the relationship between web accessibility and usability [ 31 ]; (6) user  evaluation  to improve website accessibility and interface usability by reducing the cognitive load of people with blindness [ 116 ]; (7) hybrid  evaluation  process to identify the effectiveness of usability and interface design [ 33 ]; (8) quantitative and qualitative  analysis  to evaluate user perceptions for interactive user interface design [ 114 ].

Therefore, in this group, we identified three subgroups of assets: s-group-1: 1 study for technique, s-group-2: 3 studies for evaluation, and s-group-3: 4 studies for analysis.

Figure  9 shows the graphical representation of assets obtained in this SLR. The observation result of research question 2 (as shown in Fig.  9 ) illustrates that automated validators, tools, and frameworks are the main research assets in the investigated area. It demonstrated that most past researchers and the scientific community contributed to accessibility research using the existing automated validators. Recently researchers focused on developing accessibility testing tools and designing frameworks to contribute to accessibility practice, though the number of developed tools and frameworks is limited. In addition, a small group of researchers has conducted studies on other aspects in the accessibility context.

Identified assets of the research outcome

4 Discussion

4.1 research context’s investigation results.

This section highlights the context we focused on in our investigation to reveal in this SLR. The first context of the discussion is the invested domain of past studies. Figure  10 shows the number of papers in each research area found in this SLR. Most studies focused on education, such as government and higher education institute websites. However, few studies focused on other areas such as libraries, health care, electronic materials (e.g., eBooks, visual charts, etc.), tourism, and E-commerce. Accessibility research has a significant contribution to national and international legislation to develop accessible software or web in different domains. However, more investigation for accessibility measurement should be carried out considering other areas to present accessible systems within a broad scope of future research. Besides, during the COVID-19 pandemic, accessible healthcare websites were significantly valuable and were a crucial requirement for the world community [ 117 ]. However, the observation result depicts that the number of proposed studies focusing on the healthcare domain is not adequate, which is the present research gap in this particular domain. This finding exposes the necessity of devoting continued effort to investigating the healthcare domain in future research.

Number of studies of each area of research considering accessibility domain

Figure  11 shows that according to the investigated platforms, most of the selected studies focused on web systems (75 studies), four (4) studies focused on tools and applications, and six (6) studies presented platform-independent approaches.

Number of investigated studies of each platform

Regarding guidelines, most of the selected studies followed WCAG standards to evaluate and develop the web or software application. Figure  12 depicts that WCAG is the dominant and accepted standard for referencing primary accessibility guidelines for the accessible solution and prototype design or user-centric design issues. WCAG is also extensively used as a referencing guideline in accessibility assessment or testing tool development. However, as WCAG is incorporated widely; a few deliberations are laborious to solve by imposing this standard alone. Thus, a wide variety of supporting resources and other guidelines or standards is crucial help for web developers and designers to improve accessibility issues and overcome the current accessibility limitation.

Number of studies according to the focused guideline

Regarding programming language, the frequently used programming language to implement the proposed methods, tools, and frameworks were JavaScript (object-oriented), Python (high-level programming language), HTML (markup language), CSS/SCSS (style description language), PHP (scripting language), C+ + (case sensitive language), OWL (knowledge representation language) and SWRL (logical inference engine). The most frequently marked engineering tools were Apache and MySQL webserver, Oracle database, JavaScript (React), FontAwesome, Axe, Chrome, and HTML Code Sniffer accessibility evaluation libraries. Frequently applied Application Programming Interfaces (APIs) are Clarifai (for image and video), Indico (for semantic matching), Swoogle, AATT, and REST API for Windows and Linux Operating systems. Most tested websites followed content management systems such as WordPress, Joomla, and Drupal. The tested report represents in extensible markup language (XML), enhanced address recognition logic (EARL), and portable document format (PDF). However, Selenium Web Browser Automation and ChromeDriver Tools with Webdriver and MutationObserver API are effective among other web engineering tools.

Generally, the effectiveness and performance of the web concerning accessibility issues have been assessed through automatic testing (accessibility and usability) and human observation. Figure  13 shows that frequently implemented testing tools are WAVE, AChecker, and TAW. However, earlier studies also addressed other accessibility and usability testing tools such as Mauve, Cynthia Says, Total Validator, aXe Monitor, Tenon, Siteimprove, SortSite, etc. Among several automatic testing tools, some specific tools have been implemented frequently in the past literature. Despite the availability of a wide array of accessibility testing tools (approximately 75 according to W3C), most tools are underrated, and even web designers and developers have no idea about these tools and their effectiveness [ 118 ]. In the investigated works of literature, only three pieces of literature compared multiple automatic accessibility testing tools to evaluate their effectiveness. This limited number of comparative analyses is not sufficient to show the usefulness of the existing automated tools. Thus, it is crucial to devote continued effort to perform further comparative analysis considering the benefits of automatic testing tools in future accessibility research.

Number of studies considering implemented testing tools

Concerning the accessibility and usability evaluation and validation results, SPSS, Microsoft Excel, and STATISTICA were the most used statistical analysis tools. Frequently used statistical standards are standard deviation (SD), Pearson’s correlation analysis, one-way ANOVA, System Usability Scale (SUS), Tierney’s 7-min accessibility assessment and app rating system, z-score calculation, Kolmogorov–Smirnov test, Shapiro–Wilk test, Wilcoxon signed-rank test, arithmetic mean, median, coefficient of variation, minimum and maximum value computation. According to past literature, these statistical techniques are effective in accessibility evaluation and validation practice.

Concerning the publication frequency, the observation result shows that between 2010 and 2021, seven (7) studies were published per year on average. Figure  14 displays that the observed number of published studies was low until 2017. Since then, the number of published works has grown. Between 2020 and 2021, the number of publications has shown tremendous growth. This significant growing number of publications depicts that nowadays, web researchers are concerned about the importance of accessible web and ensuring accessibility of the digital platform.

Number of publications per study year for the SLR

Considering our seven processes, we classified the selected papers into three periods: 2010–2013, 2014–2017, and 2018–2021. As shown in Fig.  15 , the number of publications between 2018 and 2021 was much higher in each of the 7 processes compared to the earlier periods. This increase was greatest in testing. The rise of articles in the implementation, evaluation, and design areas is also remarkable. These statistics indicate that concern about digital accessibility has increased in recent years. Compared with other processes, accessibility requirements, challenges, and improvement directions are underrated topics in accessibility research. In addition, the number of papers for development methods (development and implementation) is also limited. This observation directs the importance of devoting continued efforts to conducting future research concerning accessibility requirements, challenges, improvement directions, and development methods.

Number of publications in seven processes of the SLR considering three time periods

As the prime objective of accessibility research is to ensure online platforms are accessible to people with disabilities, thus, in this SLR study, we classified the past studies according to their focused disability type. Almost one-third of the selected studies did not focus on any group of disabilities (see Fig.  16 ). A prominent number of studies focused on issues with every disability. The number of studies focused on visual impairment is also noticeable. However, compared to these three criteria (AI (area independent), AD (all types of disabilities), and VD (visual impairment)), a few studies considered the cognitive, sensory impairment, and physical disabilities issues. Apart from the invested disability types, it is crucial to show the continued effort for other exceptional cases, such as hearing disabilities, moving disabilities, special children, and autism.

Publications with focused disabilities group

Despite the importance of applications to support during the web development process to ensure accessible application development, studies related to application development for accessibility direction are still limited compared to studies on web accessibility evaluation. This result shows the importance of putting effort into methods, tools, and assets to support the development of accessible web and web applications, considering the engineering feature of this platform.

4.2 Web accessibility in past studies

In our search for past studies, we found seven SLRs addressing web accessibility. Najadat et al. [ 119 ] indicated that research on web accessibility has grown since 2007. However, the development of accessibility evaluation tools, metrics, and standards was addressed poorly by past literature. They showed the most common web metrics regarding design, speed, size, diagnosis tools, and metrics for better provision of services. Following this, an SLR carried out by Muniandy and Sulaiman [ 120 ] depicts that for years, accessible computer application design, including mobile applications, computer applications, and online web applications for visually impaired people, has gained immense popularity. Research conducted by Baldwin and Ching [ 121 ] identified that user-centric web prototype design would be helpful to improve accessibility in upcoming development for people with disabilities.

Addressing these issues, an SLR carried out by Akram and Sulaiman [ 14 ] indicated that many studies published between 2009 to 2017 devoted to automated tools development to validate the technical aspects against the accessibility conformance or guidelines. Despite the importance of automatic accessibility testing tools, the lack of advanced techniques to develop these tools required human observation to interact with people with disabilities with interactive systems. With the same focus, an SLR carried out by Campoverde-Molina et al. [ 15 ] stated that a synthesis study is crucial to determine the web accessibility standards and the evaluation methods. They also indicated that the testing process remains the main focus of the current web research. In another SLR, Campoverde-Molina et al. [ 16 ] added that the majority of the experimented websites have potential accessibility issues that address further investigation and more research in this field.

In our findings, we identified a few studies related to the accessible design pattern of rich internet application (RIA), accessibility guidelines visualization, and user interface designs. Compared with the previous SLR studies proposed by Akram and Sulaiman and Campoverde-Molina et al., our proposed study also identifies the importance and growth of accessibility requirements elicitation. They added that research on accessible development and evaluation techniques, user-centric design, and user requirements with disabilities should consider.

Further, an SLR conducted by Oh et al. [ 122 ] indicated that web accessibility research in the area of web image analysis and web-based gamification or game development has increased. They added that understanding visual information (e.g., images) is a critical challenge for people with low vision. Another SLR proposed by Salvador-Ullauri et al. [ 123 ] depicted that web-based games are helpful for teaching and learning for people with disabilities. Web and game developers and designers are fascinated by implementing accessible features as accessibility guidelines are not limited to a particular domain of people. However, from the comparative analysis of previous SLRs, we can observe that (Table 11 ) most of the past SLR studies have lacked consideration of development and implementation approaches for web evaluation that are necessary to include in our SLR process.

4.3 Observation of research

In the investigated studies of this research, among the considered seven processes, challenges, and accessibility requirements experienced with less literature. The primary reason might be aligned with the current research focus. The majority of the research focused on the development of evaluation and testing methods, though addressing accessibility challenges during web development and enhancing the importance of ensuring accessibility guidelines is also important [ 124 ]. Without demonstrating the challenges that might be raised during the development process and their associated solutions, it is barely possible to ensure accessibility for digital sources (e.g., websites, software, etc.). To improve these issues, more attention should be given to the current research focus to identify the major challenges associated with the development of the accessible solution and demonstrate the accessibility guidelines with its advancements. Besides, the literature for framework design and development/implementation is not significant compared to the other processes (e.g., testing). Also, there was limited investigation for evaluation metrics to evaluate the correlation between experimented results and user (e.g., people with disabilities) perceptions, which introduces an urgent need to investigate accessibility result validation systems. In addition, our SLR result illustrates that most of the research focused on automatic accessibility testing tools to investigate the accessibility of the web platform. The articles found considered automatic accessibility testing tools while largely neglecting engineering asset development. Therefore, our proposed SLR depicts the importance of future research for updated methods, techniques, processes, and approaches to support the ensurement of an accessible web.

However, a positive finding observed in this SLR was the rapid growth of the number of studies in the accessibility context. Improving accessibility means developing accessible applications and solutions to help users with various disabilities. This perspective emphasized that developed systems should focus on user requirements (especially for special needs users) to ensure user-centric design, considering user involvement and global accessibility design guidelines for digital inclusion. To enable accessible development tendencies in companies and governmental organizations, several governments have proposed rules to improve the accessibility of digital services; for instance, the United Kingdom, the European Union, the Chinese government, and other public and private organizations. Despite several new digital content accessibility guidelines, investigating new processes, tools and techniques is a significant challenge that directs the importance of future investigations or state-of-the-art research.

5 Conclusion

A systematic literature review is presented in this paper, considering accessibility in the context of web evaluation processes. In this paper, we attempted to take a small step toward contributing to this research by pointing to a new direction for future goals and considerations.

This study showed automatic accessibility testing and evaluation of the focused area of research in the last decade for ensuring the inclusion of accessible web content. There was a great increase in the number of published works after 2017 compared to the previous years.

In the past, most of the literature focused on visual impairment, and very few papers discussed other disabilities, such as hearing, physical, and cognitive disabilities. In this SLR, we found requirements, challenges, engineering techniques, ontology, frameworks, API, algorithms, and testing tools for different levels of satisfaction associated with disabilities, but especially for visual impairment. Therefore, we identified and reported a research gap regarding other disabilities.

Unfortunately, there are few reference architectures for referring to accessible web design, development, and evaluation processes. For example, a framework for accessibility improvement of people with color vision deficiency [ 79 ], an approach for automatically identifying widgets [ 80 ], and an accessibility testing and refinement tool for the early design phase [ 110 ]. It would be beneficial to develop other reference architecture focusing on other contributing areas to solving three problems: (i) framework for the developer to identify and implement accessibility features to improve the accessibility issues, (ii) easy methods to understand and ensure accessibility requirements concerning every type of disabilities during the development phase, and (iii) updated automatic accessibility testing protocols incorporating the latest WCAG standards rules. To overcome these problems, we can note that developing new methods and tools could be a research topic in the upcoming years.

Considering the accessibility of current web platforms, in general, currently available web resources (websites, web-based games, web/mobile applications, etc.) are not accessible. Recently, the governments of many countries-imposed accessibility-related laws (i.e., WCAG) to ensure accessibility requirements. Furthermore, the methods and tools to solve the accessibility problems have limitations that direct future research concerning the development of engineering approaches.

For current accessibility research, there are many challenges to incorporating updated WCAG. Regarding automatic accessibility testing protocol, several studies focused on the limited number of guidelines and disability requirements. Studies for the design and development of accessibility testing protocols are limited. Thus, automatic accessibility testing protocol development concerning different disabilities and elderly user requirements could be a research area in the upcoming years.

Finally, consideration of several methodologies and open-source developments for ensuring accessibility is significantly important. Recently, several researchers and companies have been developing web-based solutions by adopting accessibility requirements. They develop open-source software that has an essential role for end-users and corporations. Accessibility is a crucial technological aspect of developing a new solution for any domain.

Abbreviations

Accessibility conformance testing

Americans with disabilities

Application programming interface

Cascading style sheets

Color vision deficiency

Disabilities discrimination act

Enhanced address recognition logic

Evaluator-decision-based assignment

Hypertext markup language

Information and communications technology

International software testing qualifications board

Web ontology language

Portable document format

Preferred reporting items for systematic reviews and meta-analyses

Rich internet application

Syntactically awesome style sheet

Search engine optimization

• Systematic literature review

Semantic web rule language

User-centric design

Universal design

Web accessibility initiative

Web content accessibility guidelines

Extensible markup language

Nagano, Y., Suginome, K., Yoshimoto, K., Tsuchiya, Y.: Activities for improving web accessibility. Fujitsu Sci. Tech. J. 45 (2), 239–246 (2009)

Google Scholar  

Basdekis, I., Klironomos, I., Metaxas, I., Stephanidis, C.: An overview of web accessibility in Greece: a comparative study 2004–2008. Univ. Access Inf. Soc. 9 (2), 185–190 (2010)

Article   Google Scholar  

World Wide Web Consortium.: 2022a: W3C—Web Accessibility Initiative (WAI), Making the Web Accessible. http://www.w3.org/WAI/

World Wide Web Consortium.: 2022b: About w3C, Groups. https://www.w3.org/Consortium/activities

World Wide Web Consortium.: 2022c. Web content accessibility guidelines (WCAG). https://www.w3.org/standards/

World Wide Web Consortium.: 2018. Web content accessibility guidelines (WCAG) 2.1. https://www.w3.org/TR/WCAG22/

Koga, T.: Policy issues regarding electronic government and Web accessibility in Japan (2006)

Wentzel, J., Velleman, E., Geest, T.V.D.: Developing accessibility design guidelines for wearables: Accessibility standards for multimodal wearable devices. In: International Conference on Universal Access in Human-Computer Interaction, pp. 109–119. Springer, Cham (2016)

Reichling, M., Cherfi, S.S.S.: Integrating accessibility as a quality property in web developments. In: IEEE 7th International Conference on Research Challenges in Information Science (RCIS), pp. 1–12). IEEE (2013)

Horton, S., Sloan, D.: Accessibility in practice: a process-driven approach to accessibility. In: Inclusive Designing, pp. 105–115. Springer, Cham (2014)

Moreno, L., Martínez, P.: A review of accessibility requirements in elderly users’ interactions with web applications. In: Proceedings of the 13th International Conference on Interacción Persona-Ordenador, pp. 1–2 (2012)

Estrada-Martínez, F.J., Hilera, J.R., Otón, S., Aguado-Delgado, J.: Semantic web technologies applied to software accessibility evaluation: a systematic literature review. Universal Access in the Information Society, 1–25 (2020)

Chadli, F.E., Gretete, D., Moumen, A.: Digital accessibility: a systematic Literature Review. In: SHS Web of Conferences, Vol. 119. EDP Sciences (2021)

Akram, M., Sulaiman, R.B.: A systematic literature review to determine the web accessibility issues in Saudi Arabian university and government websites for disable people. Int. J. Adv. Comput. Sci. Appl. 8 (6) (2017)

Campoverde-Molina, M., Lujan-Mora, S., Garcia, L.V.: Empirical studies on web accessibility of educational websites: a systematic literature review. IEEE Access 8 , 91676–91700 (2020)

Campoverde-Molina, M., Luján-Mora, S., Valverde, L.: Accessibility of university websites worldwide: a systematic literature review. Universal Access in the Information Society, 1–36 (2021)

Stapic, Z., López, E.G., Cabot, A.G., de Marcos Ortega, L., Strahonja, V.: Performing systematic literature review in software engineering. In: Central European Conference on Information and Intelligent Systems, p. 441. Faculty of Organization and Informatics Varazdin (2012)

Bai, A., Fuglerud, K.S., Skjerve, R.A., Halbach, T.: Categorization and comparison of accessibility testing methods for software development (2018)

Chiang, M.F., Cole, R.G., Gupta, S., Kaiser, G.E., Starren, J.B.: Computer and world wide web accessibility by visually disabled patients: problems and solutions. Surv. Ophthalmol. 50 (4), 394–405 (2005)

Caldwell, R.: Web-accessibility, e-learning, and academic libraries. Int. J. Public Inf. Syst. 2 (1) (2006)

World Wide Web Consortium.: 2022d. Web content accessibility guidelines (WCAG) 3.0. https://www.w3.org/TR/wcag-3.0/

World Wide Web Consortium.: 1999. Web content accessibility guidelines (WCAG) 1.0. https://www.w3.org/TR/WAI-WEBCONTENT/

World Wide Web Consortium.: 2008. Web content accessibility guidelines (WCAG) 1.0. https://www.w3.org/TR/WCAG20/

World Wide Web Consortium.: 2021. Web content accessibility guidelines (WCAG) 2.2. https://www.w3.org/TR/WCAG22/

Dror, A.A., Morozov, N.G., Layous, E., Mizrachi, M., Daoud, A., Eisenbach, N., Sela, E.: United by hope, divided by access: country mapping of COVID-19 information accessibility and its consequences on pandemic eradication. Front. Med. 7 , 1133 (2021)

Kim, N.W., Joyner, S.C., Riegelhuth, A., Kim, Y.: Accessible visualization: design space, opportunities, and challenges. In: Computer Graphics Forum, Vol. 40, No. 3, pp. 173–188 (2021)

Kitchenham, B., Charters, S.: Guidelines for performing systematic literature reviews in software engineering (2007)

Durdu, P.O., Yerlikaya, Z.: The perception of website accessibility: A survey of Turkish software professionals. AJIT-e: Bilişim Teknolojileri Online Dergisi 11 (41), 42–71 (2020)

Bradbard, D.A., Peters, C.: Web accessibility theory and practice: an introduction for university faculty (2010)

Yu, S.Y.: A review of the accessibility of ACT COVID-19 information portals. Technol. Soc. 64 , 101467 (2021)

Bai, Y.: The relationship between Website accessibility and usability: an examination of US county government online portals. Electron. J. e-Gov. 17 (1), 47–62 (2019)

Henry, S.L., Abou-Zahra, S., Brewer, J.: The role of accessibility in a universal web. In: Proceedings of the 11th Web for all Conference, pp. 1–4 (2014)

Wu, K., Petersen, E., Ahmad, T., Burlinson, D., Tanis, S., Szafir, D.A.: Understanding data accessibility for people with intellectual and developmental disabilities. In: Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems, pp. 1–16 (2021)

Riley-Huff, D.A.: Web accessibility and universal design. Libr. Technol. Rep. 48 (7), 29–35 (2012)

Marino, S.I., Alfonzo, P.L.: A web accessibility empirical analysis, a case study (2021)

Sauer, J., Sonderegger, A., Schmutz, S.: Usability, user experience and accessibility: towards an integrative model. Ergonomics 63 (10), 1207–1220 (2020)

Vu, K.P.L., Proctor, R.W., Hung, Y.H.: Website Design and Evaluation. Handbook of human factors and ergonomics, pp. 1016–1036 (2021)

Almeida, L.D.A., Baranauskas, M.C.C.: Universal design principles combined with web accessibility guidelines: a case study. In: Proceedings of the IX Symposium on Human Factors in Computing Systems, pp. 169–178 (2010)

Gaggi, O., Pederiva, V.: WCAG4All, a tool for making web accessibility rules accessible. In: 2021 IEEE 18th Annual Consumer Communications & Networking Conference (CCNC), pp. 1–6. IEEE (2021)

Acosta-Vargas, P., Acosta, T., Lujan-Mora, S.: Challenges to assess accessibility in higher education websites: a comparative study of Latin America universities. IEEE Access 6 , 36500–36508 (2018)

Inal, Y., Guribye, F., Rajanen, D., Rajanen, M., Rost, M.: Perspectives and practices of digital accessibility: a survey of user experience professionals in uwait countries. In: Proceedings of the 11th Nordic Conference on Human-Computer Interaction: Shaping Experiences, Shaping Society, pp. 1–11 (2020)

Brajnik, G., Vigo, M.: Automatic Web Accessibility Metrics. In: Web Accessibility, pp. 505–521. Springer, London (2019)

Palaskar, A., Gondkar, S.R., Gondkar, R.R.: Web Accessibility test approach on developed websites. Available at SSRN 3835104 (2021)

Edelberg, J., Verhulsdonck, G.: Addressing accessibility as advocacy: special needs users in industry web development processes. In: The 39th ACM International Conference on Design of Communication, pp. 343–346 (2021)

Miesenberger, K., Edler, C., Heumader, P., Petz, A.: Tools and applications for cognitive accessibility. In: Web Accessibility, pp. 523–546. Springer, London (2019)

Alismail, S., Chipidza, W.: Accessibility evaluation of COVID-19 vaccine registration websites across the United States. J. Am. Med. Inform. Assoc. 28 (9), 1990–1995 (2021)

Bhagat, S., Joshi, P.: Evaluation of accessibility and accessibility audit methods for e-governance portals. In: Proceedings of the 12th International Conference on Theory and Practice of Electronic Governance, pp. 220–226 (2019)

Ojha, P.K., Ismail, A., Srinivasan, K.K.: Perusal of readability with focus on web content understandability. J. King Saud Univ. Comput. Inf. Sci. 33 (1), 1–10 (2021)

Ismail, A., Kuppusamy, K.S., Nengroo, A.S.: Multi-tool accessibility assessment of government department websites: a case-study with JKGAD. Disabil. Rehabil. Assist. Technol. 13 (6), 504–516 (2018)

Kuppusamy, K.S., Balaji, V.: Evaluating web accessibility of educational institutions websites using a variable magnitude approach. Universal Access in the Information Society, pp. 1–10 (2021)

Alshamari, M.: Accessibility evaluation of Arabic e-commerce web sites using automated tools. J. Softw. Eng. Appl. 9 (9), 439–451 (2016)

Morris, M.R., Johnson, J., Bennett, C.L., Cutrell, E.: Rich representations of visual content for screen reader users. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems, pp. 1–11 (2018)

Alahmadi, T.: A multi-method evaluation of university website accessibility: foregrounding user-centred design, mining source code and using a quantitative metric. In: Proceedings of the 14th International Web for All Conference, pp. 1–2 (2017)

Kaur, S., Gupta, S.K.: A fuzzy-based framework for evaluation of website design quality index. Int. J. Digit. Libr. 22 (1), 15–47 (2021)

Hassouna, M.S., Sahari, N., Ismail, A.: University website accessibility for totally blind users. J. Inf. Commun. Technol. 16 (1), 63–80 (2017)

Sapna, P.G., Mohanty, H.: An ontology based approach for test scenario management. In: International Conference on Information Intelligence, Systems, Technology and Management, pp. 91–100. Springer, Berlin, Heidelberg (2011)

Kourtiche, A., Mohamed Benslimane, S., Hacene, S.B.: OUPIP: ontology based user profile for impairment person in dynamic situation aware social networks. Int. J. Knowl.-Based Org. (IJKBO) 10 (2), 12–34 (2020)

Fayzrahmanov, R.R., Göbel, M.C., Holzinger, W., Krüpl, B., Baumgartner, R.: A unified ontology-based web page model for improving accessibility. In: Proceedings of the 19th International Conference on World Wide Web, pp. 1087–1088 (2010)

Li, L.C., Bu, J.J., Zhi, Y., Wei, W.A., Can, W.A.: An overview of web accessibility evaluation of government websites in China. In: International Conference on Social Sciences and Development (ICSSD), pp. 2–5 (2016)

Alsaeedi, A.: Comparing web accessibility evaluation tools and evaluating the accessibility of webpages: proposed frameworks. Information 11 (1), 40 (2020)

Song, S., Bu, J., Artmeier, A., Shi, K., Wang, Y., Yu, Z., Wang, C.: Crowdsourcing-based Web accessibility evaluation with golden maximum likelihood inference. In: Proceedings of the ACM on Human-Computer Interaction, 2(CSCW), pp. 1–21 (2018)

Giovanna, B., Manca, M., Paternó, F., Pulina, F.: Flexible automatic support for web accessibility validation. In: Proceedings of the ACM on Human–Computer Interaction, p. 4 (2020)

Zeleke, Y.: Usability and accessibility model for e-government websites in Ethiopia (2020)

Sanchez-Gordon, S., Luján-Mora, S.: A method for accessibility testing of web applications in agile environments. In: Proceedings of the 7th World Congress for Software Quality (WCSQ). En proceso de publicación.(citado en la página 13, 15, 85) (2017)

Song, S., Bu, J., Wang, Y., Yu, Z., Artmeier, A., Dai, L., Wang, C.: Web accessibility evaluation in a crowdsourcing-based system with expertise-based decision strategy. In: Proceedings of the 15th International Web for All Conference, pp. 1–4 (2018)

Acosta-Vargas, P., Salvador-Ullauri, L.A., Luján-Mora, S.: A heuristic method to evaluate web accessibility for users with low vision. IEEE Access 7 , 125634–125648 (2019)

Won, S.: Conveying colour research to design practice: design and evaluation of a web-based colour tool. Color. Res. Appl. 46 (2), 350–361 (2021)

Mohamad, Y., Velasco, C.A., Kaklanis, N., Tzovaras, D., Paternò, F.: A holistic decision support environment for web accessibility. In: International Conference on Computers Helping People with Special Needs, pp. 3–7. Springer, Cham (2018)

Li, L., Wang, C., Song, S., Yu, Z., Zhou, F., Bu, J.: A task assignment strategy for crowdsourcing-based web accessibility evaluation system. In: Proceedings of the 14th International Web for All Conference, pp. 1–4 (2017)

Žuliček, L., Tomić, S., Bosnić, I.: Adapting modularized web applications to web accessibility standards. In: 2021 44th International Convention on Information, Communication and Electronic Technology (MIPRO), pp. 470–475. IEEE (2017)

Oliveira, R.P., Afonso, A.P., Pinto, A.S.: E-commerce website accessibility evaluation: case study in Portugal. Şubat 10 , 2021 (2020)

Rashida, M., Islam, K., Kayes, A.S.M., Hammoudeh, M., Arefin, M.S., Habib, M.A.: Towards developing a framework to analyze the qualities of the university websites. Computers 10 (5), 57 (2021)

Lim, Z.Y., Chua, J.M., Yang, K., Tan, W.S., Chai, Y.: Web accessibility testing for uwaitie government e-services. In: Proceedings of the 17th International Web for All Conference, pp. 1–5 (2020)

Duarte, C., Matos, I., Carriço, L.: Semantic content analysis supporting web accessibility evaluation. In: Proceedings of the 15th International Web for All Conference, pp. 1–4 (2018)

Wu, Y., Yu, Z., Li, L., Wang, W., Bu, J., Zhang, Y., Dai, L.: Using semi-supervised group sparse regression to improve web accessibility evaluation. In: International Conference on Computers Helping People with Special Needs, pp. 52–56. Springer, Cham (2018)

Morato, J., Iglesias, A., Campillo, A., Sanchez-Cuadrado, S.: Automated readability assessment for uwaiti e-government information. J. Inf. Syst. Eng. Manag. 6 (2), em0137 (2021)

Boyalakuntla, K., Sri Manasa Venigalla, A., Chimalakonda, S.: Waccess—a web accessibility tool based on the latest WCAG 2.2 guidelines. arXiv e-prints, arXiv-2107 (2021)

Michailidou, E., Eraslan, S., Yesilada, Y., Harper, S.: Automated prediction of visual complexity of web pages: tools and evaluations. Int. J. Hum. Comput. Stud. 145 , 102523 (2021)

Bonacin, R., Reis, J.C.D., de Araujo, R.J.: An ontology-based framework for improving color vision deficiency accessibility. Universal Access in the Information Society, pp. 1–26 (2021)

Antonelli, H.L., Igawa, R.A., Fortes, R.P.D.M., Rizo, E.H., Watanabe, W.M.: Drop-down menu widget identification using HTML structure changes classification. ACM Trans. Access. Comput. (TACCESS) 11 (2), 1–23 (2018)

Csontos, B., Heckl, I.: Improving accessibility of CMS-based websites using automated methods. Universal Access in the Information Society, pp. 1–15 (2020)

Matošević, G., Dobša, J., Mladenić, D.: Using machine learning for web page classification in search engine optimization. Future Internet 13 (1), 9 (2021)

Martins, J., Gonçalves, R., Branco, F.: A full scope web accessibility evaluation procedure proposal based on Iberian eHealth accessibility compliance. Comput. Hum. Behav. 73 , 676–684 (2017)

Padure, M., Pribeanu, C.: Comparing six free accessibility evaluation tools. Informatica Economica 24 (1), 15–25 (2020)

Hassouna, M.S., Sahari, N., Ismail, A.: Accessibility factors for totally blind users of Palestinian universities websites (2015)

Pribeanu, C., Fogarassy-Neszly, P.: A review of municipal web sites for accessibility: a computer-aided evaluation approach. Stud. Inf. Control 20 (3), 265–272 (2011)

Verkijika, S.F., De Wet, L.: Accessibility of South African university websites. Univ. Access Inf. Soc. 19 (1), 201–210 (2020)

AlMeraj, Z., Boujarwah, F., Alhuwail, D., Qadri, R.: Evaluating the accessibility of higher education institution websites in the State of Kuwait: empirical evidence. Univ. Access Inf. Soc. 20 (1), 121–138 (2021)

Sharma, A.: Web Accessibility of Indian University library website: an evaluation with WAVE website evaluation tool (2021)

Abduganiev, S.G.: Towards automated web accessibility evaluation: a comparative study. Int. J. Inf. Technol. Comput. Sci. 9 (9), 18–44 (2017)

Chapman, L., Brooks, C., Lawson, J., Russell, C., Adams, J.: Accessibility of online self-management support websites for people with osteoarthritis: a text content analysis. Chronic Illn. 15 (1), 27–40 (2019)

Rysavy, M.D., Michalak, R.: Assessing the accessibility of library tools & services when you aren’t an accessibility expert: Part 1. J. Libr. Adm. 60 (1), 71–79 (2020)

Akgül, Y.: Accessibility, usability, quality performance, and readability evaluation of university websites of Turkey: a comparative study of state and private universities. Univ. Access Inf. Soc. 20 (1), 157–170 (2021)

Doush, I., AlMeraj, Z.: Evaluating the accessibility of uwaiti e-government websites. Jordanian J. Comput. Inf. Technol. (JJCIT) 5 (03) (2019)

Baule, S.M.: Evaluating the accessibility of special education cooperative websites for individuals with disabilities. TechTrends 64 (1), 50–56 (2020)

Ajuji, M., Abubakar, A., Adam, Y.A., Emmanuel, D.U.: Evaluation of accessibility of randomly selected websites. ATBU J. Sci. Technol. Educ. 9 (3), 50–57 (2021)

Grant, J., Sumilang, G., Providenti, M., Cho, S., Reichler, D.: Accountability for the hidden codes toward a better user experience: case study of Hrsimple communication design for web accessibility and SEO. In: The 39th ACM International Conference on Design of Communication, pp. 347–350 (2021)

Kumar, S., Shree, D.V.J., Biswas, P.: Comparing ten WCAG tools for accessibility evaluation of websites. Technology and Disability, (Preprint), pp. 1–23 (2021)

Burkard, A., Zimmermann, G., Schwarzer, B.: Monitoring systems for checking websites on accessibility. Front. Comput. Sci. 2 (2021)

Jo, G., Habib, D., Varadaraj, V., Smith, J., Epstein, S., Zhu, J., Swenor, B.K.: COVID-19 Vaccine Website Accessibility Dashboard. Disabil. Health J. 101325 (2022)

Eusébio, C., Teixeira, L., Teixeira, P., Caneiro, M.J., Lemos, D., Silveiro, A.: The State of Web Accessibility for tourists with disabilities: a comparative study between different tourism supply agents. Disabil. Rehabil. Assistive Technol. 1–13 (2021)

Kous, K., Kuhar, S., Pavlinek, M., Heričko, M., Pušnik, M.: Web accessibility investigation of Slovenian municipalities’ websites before and after the adoption of European Standard EN 301 549. Univ. Access Inf. Soc. 20 (3), 595–615 (2021)

Ali, L.: Accessible websites for everyone—a case of UAE universities websites. Int. J. Inf. Educ. Technol. 11 (4) (2021)

Zare, S., Rahmatizadeh, S., Valizadeh-Haghi, S.: Academic medical libraries and accessibility challenges: the conformance of the websites with the WCAG2. 1. DESIDOC J. Lib. Inf. Technol. 41 (2) (2021)

Król, K., Zdonek, D.: The quality of infectious disease hospital websites in Poland in light of the COVID-19 pandemic. Int. J. Environ. Res. Public Health 18 (2), 642 (2021)

Yi, Y.J.: Web accessibility of healthcare Web sites of Korean government and public agencies: a user test for persons with visual impairment. Univ. Access Inf. Soc. 19 (1), 41–56 (2020)

Moreno, L., Valencia, X., Pérez, J.E., Arrue, M.: An exploratory study of web adaptation techniques for people with low vision. Univ. Access Inf. Soc. 20 (2), 223–237 (2021)

Krawiec, Ł., Dudycz, H.: The approach to the assessment of validators for studying web-accessibility. Inf. Syst. Manag. 6 (2017)

Grantham, J., Grantham, E., Powers, D.: Website accessibility: an Australian view. In: Proceedings of the Thirteenth Australasian User Interface Conference-Volume 126, pp. 21–28 (2012)

Hadadi, S.: Adee: bringing accessibility right inside design tools. In: The 23rd International ACM SIGACCESS Conference on Computers and Accessibility, pp. 1–4 (2021)

Kimmons, R.: Open to all? Nationwide evaluation of high-priority web accessibility considerations among higher education websites. J. Comput. High. Educ. 29 (3), 434–450 (2017)

Radcliffe, E., Lippincott, B., Anderson, R., Jones, M.: A pilot evaluation of mHealth app accessibility for three top-rated weight management apps by people with disabilities. Int. J. Environ. Res. Public Health 18 (7), 3669 (2021)

Sun, Y.T., Vu, K.P.L., Strybel, T.Z.: A validation test of an accessibility evaluation method. In International Conference on Applied Human Factors and Ergonomics, pp. 625–633. Springer, Cham 2017

Alcaraz Martínez, R., Turró, M.R., Granollers Saltiveri, T.: Methodology for heuristic evaluation of the accessibility of statistical charts for people with low vision and color vision deficiency. Universal access in the information society, pp. 1–32 (2021)

Cao, S., Loiacono, E.: Perceptions of web accessibility guidelines by student website and app developers. Behav. Inf. Technol. 1–19 (2021)

Giraud, S., Thérouanne, P., Steiner, D.D.: Web accessibility: Filtering redundant and irrelevant information improves website usability for blind users. Int. J. Hum. Comput. Stud. 111 , 23–35 (2018)

Ara, J., Sik-Lanyi, C.: Investigation of COVID-19 vaccine information websites across Europe and Asia using automated accessibility protocols. Int. J. Environ. Res. Public Health 19 (5), 2867 (2022)

Abascal, J., Arrue, M., Valencia, X.: Tools for web accessibility evaluation. In: Yesilada, Y., Harper, S. (eds) Web Accessibility. Human–Computer Interaction Series. Springer, London (2019). https://doi.org/10.1007/978-1-4471-7440-0_26

Najadat, H., Al-Badarneh, A., Alodibat, S.: A review of website evaluation using web diagnostic tools and data envelopment analysis. Bull. Electr. Eng. Inf. 10 (1), 258–265 (2021)

Muniandy, M., Sulaiman, S.: The role of multimedia design elements in computer accessibility for visually impaired: a complete review. J. Theor. Appl. Inf. Technol. 95 (20) (2017)

Baldwin, S.J., Ching, Y.H.: Accessibility in online courses: a review of national and statewide evaluation instruments. TechTrends 65 (5), 731–742 (2021)

Oh, U., Joh, H., Lee, Y.: Image accessibility for screen reader users: a systematic review and a road map. Electronics 10 (8), 953 (2021)

Salvador-Ullauri, L., Acosta-Vargas, P., Luján-Mora, S.: Web-based serious games and accessibility: a systematic literature review. Appl. Sci. 10 (21), 7859 (2020)

Araa, J., Cecilia, S.L.: Artificial intelligence in web accessibility: potentials and possible challenges. In: Proceedings of IAC 2022 in Vienna, p. 173 (2022)

Download references

Open access funding provided by University of Pannonia. The authors declare that this article is their research and was not financially supported.

Author information

Authors and affiliations.

Department of Electrical Engineering and Information Systems, University of Pannonia, Egyetem u. 10, Veszprem, 8200, Hungary

Jinat Ara & Cecilia Sik-Lanyi

Department of Organizational Systems and Adult Health, University of Maryland Baltimore, 655 W. Lombard St #455B, Baltimore, MD, 21201, USA

Arpad Kelemen

You can also search for this author in PubMed   Google Scholar

Contributions

JA and CS-L conceptualized the study. JA undertook the investigation and statistical analyses and managed the data with support from CS-L and AK and wrote the first draft of the paper. All authors critically inputted into the draft and reviewed it and agreed on the final version. All authors have read and agreed to the published version of the manuscript.

Corresponding author

Correspondence to Jinat Ara .

Ethics declarations

Conflict of interest.

The authors declare that there is no conflict of interest.

Additional information

Publisher's note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ .

Reprints and permissions

About this article

Ara, J., Sik-Lanyi, C. & Kelemen, A. Accessibility engineering in web evaluation process: a systematic literature review. Univ Access Inf Soc (2023). https://doi.org/10.1007/s10209-023-00967-2

Download citation

Accepted : 10 January 2023

Published : 27 January 2023

DOI : https://doi.org/10.1007/s10209-023-00967-2

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Universal inclusion
• Digital accessibility
• Web accessibility
• Engineering assets
• Accessibility insights
• Find a journal
• Publish with us
• Track your research

Baltimore bridge collapse: a bridge engineer explains what happened, and what needs to change

Associate Professor, Civil Engineering, Monash University

Disclosure statement

Colin Caprani receives funding from the Department of Transport (Victoria) and the Level Crossing Removal Project. He is also Chair of the Confidential Reporting Scheme for Safer Structures - Australasia, Chair of the Australian Regional Group of the Institution of Structural Engineers, and Australian National Delegate for the International Association for Bridge and Structural Engineering.

Monash University provides funding as a founding partner of The Conversation AU.

View all partners

When the container ship MV Dali, 300 metres long and massing around 100,000 tonnes, lost power and slammed into one of the support piers of the Francis Scott Key Bridge in Baltimore, the bridge collapsed in moments . Six people are presumed dead, several others injured, and the city and region are expecting a months-long logistical nightmare in the absence of a crucial transport link.

It was a shocking event, not only for the public but for bridge engineers like me. We work very hard to ensure bridges are safe, and overall the probability of being injured or worse in a bridge collapse remains even lower than the chance of being struck by lightning.

However, the images from Baltimore are a reminder that safety can’t be taken for granted. We need to remain vigilant.

So why did this bridge collapse? And, just as importantly, how might we make other bridges more safe against such collapse?

A 20th century bridge meets a 21st century ship

The Francis Scott Key Bridge was built through the mid 1970s and opened in 1977. The main structure over the navigation channel is a “continuous truss bridge” in three sections or spans.

The bridge rests on four supports, two of which sit each side of the navigable waterway. It is these two piers that are critical to protect against ship impacts.

And indeed, there were two layers of protection: a so-called “dolphin” structure made from concrete, and a fender. The dolphins are in the water about 100 metres upstream and downstream of the piers. They are intended to be sacrificed in the event of a wayward ship, absorbing its energy and being deformed in the process but keeping the ship from hitting the bridge itself.

The fender is the last layer of protection. It is a structure made of timber and reinforced concrete placed around the main piers. Again, it is intended to absorb the energy of any impact.

Fenders are not intended to absorb impacts from very large vessels . And so when the MV Dali, weighing more than 100,000 tonnes, made it past the protective dolphins, it was simply far too massive for the fender to withstand.

Read more: I've captained ships into tight ports like Baltimore, and this is how captains like me work with harbor pilots to avoid deadly collisions

Video recordings show a cloud of dust appearing just before the bridge collapsed, which may well have been the fender disintegrating as it was crushed by the ship.

Once the massive ship had made it past both the dolphin and the fender, the pier – one of the bridge’s four main supports – was simply incapable of resisting the impact. Given the size of the vessel and its likely speed of around 8 knots (15 kilometres per hour), the impact force would have been around 20,000 tonnes .

Bridges are getting safer

This was not the first time a ship hit the Francis Scott Bridge. There was another collision in 1980 , damaging a fender badly enough that it had to be replaced.

Around the world, 35 major bridge collapses resulting in fatalities were caused by collisions between 1960 and 2015, according to a 2018 report from the World Association for Waterborne Transport Infrastructure. Collisions between ships and bridges in the 1970s and early 1980s led to a significant improvement in the design rules for protecting bridges from impact.

Further impacts in the 1970s and early 1980s instigated significant improvements in the design rules for impact.

The International Association for Bridge and Structural Engineering’s Ship Collision with Bridges guide, published in 1993, and the American Association of State Highway and Transporation Officials’ Guide Specification and Commentary for Vessel Collision Design of Highway Bridges (1991) changed how bridges were designed.

In Australia, the Australian Standard for Bridge Design (published in 2017) requires designers to think about the biggest vessel likely to come along in the next 100 years, and what would happen if it were heading for any bridge pier at full speed. Designers need to consider the result of both head-on collisions and side-on, glancing blows. As a result, many newer bridges protect their piers with entire human-made islands.

Of course, these improvements came too late to influence the design of the Francis Scott Key Bridge itself.

Lessons from disaster

So what are the lessons apparent at this early stage?

First, it’s clear the protection measures in place for this bridge were not enough to handle this ship impact. Today’s cargo ships are much bigger than those of the 1970s, and it seems likely the Francis Scott Key Bridge was not designed with a collision like this in mind.

So one lesson is that we need to consider how the vessels near our bridges are changing. This means we cannot just accept the structure as it was built, but ensure the protection measures around our bridges are evolving alongside the ships around them.

Second, and more generally, we must remain vigilant in managing our bridges. I’ve written previously about the current level of safety of Australian bridges, but also about how we can do better.

This tragic event only emphasises the need to spend more on maintaining our ageing infrastructure. This is the only way to ensure it remains safe and functional for the demands we put on it today.

• Engineering
• Infrastructure
• Urban infrastructure
• container ships
• Baltimore bridge collapse

School of Social Sciences – Public Policy and International Relations opportunities

School of Social Sciences – Human Geography opportunities

School of Social Sciences – Criminology opportunities

School of Social Sciences – Academic appointment opportunities

Communications and Events Officer

Serial Season 4

“Serial” returns with a history of Guantánamo told by people who lived through key moments in Guantánamo’s evolution, who know things the rest of us don’t about what it’s like to be caught inside an improvised justice system.

Published March 21, 2024

Listen to and follow Serial Season 4

• Apple Podcasts

Listen to the trailer for Serial Season 4: Guantánamo.

Your browser doesn't support HTML5 audio. Try listening here.

“So, it felt very like college-like… Without it being… Obviously the next day wasn’t classes, it was Gitmo… ”

Poor baby raul, episode 1 — poor baby raul.

“I wanted to create a persona, a thing that was not human.”

The special project, episode 2 — the special project.

“Am I one of them? I mean, is he innocent? I don’t know!”

Ahmad the iguana feeder, episode 3 — ahmad the iguana feeder.

• Episode 4 arrives April 11

About Serial Season 4

Right after Sept. 11, the United States created a brand-new criminal justice system at Guantánamo Bay. It was a prison and a court designed to deal with the people we had captured whom we suspected of being members of the Taliban or al Qaeda.

But to do what we wanted to do at Guantánamo — to interrogate detainees the way we wanted, to hold them indefinitely without charging them with a crime — we had to push aside the old, time-tested rules for detaining prisoners of war. And the consequences of that fell on ordinary people: thousands of military personnel, hundreds of prisoners, everybody scrambling through the same experiment.

There has been great journalism about the legal maneuvering to justify Guantánamo, and about the detainee abuse and the politics and policy. But “Serial” reporters Sarah Koenig and Dana Chivvis were after the inside stories, a picture of Guantánamo you could get only from the people who went through it. For years, though, all the best stories they heard about Guantánamo were off the record. But they stuck with it, figuring maybe once enough people were back in civilian life they’d be willing to tell those stories on the record. A couple of years ago, the “Serial” team started contacting people again: guards, interrogators, commanders, lawyers, chaplains, translators and former prisoners. More than a hundred people. And a remarkable number of them said: Okay, I’m ready. Here’s what happened.

“Serial” Season 4 is a history of Guantánamo told by people who lived through key moments in its evolution, who know things the rest of us don’t about what it’s like to be caught inside an improvised justice system.

• Sofia degli Alessandri is an Italian composer based in London. Her music combines field recordings, synths, chamber instruments and electronic beats. She composes for film, television, dance and other media.
• Hosts Sarah Koenig and Dana Chivvis Producer Jessica Weisberg Editor Julie Snyder Additional Reporting Cora Currier Fact Checking and Research Ben Phelan Additional Fact Checking Jessica Suriano Music supervision, sound design, and mixing Phoebe Wang Original score Sofia degli Alessandri Additional Editing David Kestenbaum, Jen Guerra, Alvin Melathe, Ellen Weiss and Ira Glass Contributing Editors Rozina Ali and Carol Rosenberg Assistant Producer Emma Grillo Translators and Interpreters Raza Sayibzada, Nael Hijjo, Atiq Rahin, Dana El-Issa, Bachar Alhalabi, and Omama Osman Art Direction Pablo Delcan Art Max Guther Standards Editor Susan Wessling Legal Review Al-Amyn Sumar and Maya Gandhi Reporting and Research Amir Khafagy and Sami Yousafazai Additional Production Daniel Guillemette and Katie Mingle Executive Assistant Mack Miller Supervising Producer Ndeye Thioubou Deputy Managing Editor Sam Dolnick
• At the New York Times, thanks to Elizabeth Davis-Moorer, Nina Lassam, Susan Beachy, Kitty Bennett, Alain Delaquérière, Sheelagh McNeill, Kirsten Noyes, Jack Begg, Jeffrey Miranda, Colleen Wormsley, Peter Rentz, John-Michael Murphy, Jordan Cohen, Zoe Murphy, Pierre-Antoine Louis, Mahima Chablani, Kelly Doe, Anisha Muni, Kimmy Tsai, Victoria Kim, Ashka Gami, Brad Fisher, Maddy Masiello, Daniel Powell, Marion Lozano, Tug Wilson and Aisha Khan
• Special thanks Katie Mingle, Jenelle Pifer, Alissa Shipp, Nadia Reiman, Anita Badejo, Katie Fuchs, Alison Beckman at the Center for Victims of Torture, Clive Stafford Smith, Alisa Dogramadzieva, Shuaib Almosawa, Mohamed Elfaki, Freshta Taeb, Edgar August, Esther Whitfield, Lauren Myerscough-Mueller, Mark Fallon, Pardiss Kebriaei, Steve Vladeck, Charlie Savage, Michelle Shephard, Bastian Berbner, John Goetz, Sarah Mirk and everyone involved in “Guantánamo Voices,” Peter Jan Honigsberg, Tim Golden, John Ryan, Stuart Couch, Shayana Kadidal, Ray Rivera, Steven Kleinman, Steve Wood and Lee Riffaterre

Serial is a podcast from the creators of This American Life, hosted by Sarah Koenig. Serial tells one story — a true story — over the course of a season.

A high-school senior named Hae Min Lee disappeared one day after school in 1999, in Baltimore County, Maryland. A month later, her body was found in a city park. She'd been strangled. Her 17-year-old ex-boyfriend, Adnan Syed, was arrested for the crime, and within a year, he was sentenced to life in prison.

In May 2014, a U.S. Special Operations team in a Black Hawk helicopter landed in the hills of Afghanistan. Waiting for them were more than a dozen Taliban fighters and a tall American, who looked pale and out of sorts: Bowe Bergdahl. Bergdahl, a U.S. soldier, had been a prisoner of the Taliban for nearly five years, and now he was going home. Learn more

“Serial” is heading back to court. This time, in Cleveland. Not for one extraordinary case; instead, Serial wanted to tackle the whole criminal justice system. To do that we figured we’d need to look at something different: ordinary cases. So we did. Inside these ordinary cases we found the troubling machinery of the criminal justice system on full display. Learn more

Serial Productions makes narrative podcasts that have transformed the medium. From the powerful forces shaping our public schools to a mystery at the heart of a scandal that rocked Britain, Serial expands the boundaries of audio investigative storytelling. Learn more

Further Reading From The Times

The guantánamo docket, a closer look at what the u.s. lets you see of its war court at guantánamo bay, conditions at guantánamo are cruel and inhuman, u.n. investigation finds.

• Share full article

Podcasts From The New York Times

The Daily:  The biggest stories, five days a week, including the indictment of Donald Trump , the implosion of Silicon Valley Bank  and the threat to abortion pills .

The Run-Up:  Astead W. Herndon grapples with the big ideas already animating the 2024 election by reporting from inside the political establishment .

Hard Fork:  Kevin Roose and Casey Newton make sense of the world of tech, including Google's Place in the A.I. Arms Race  and the arrival of GPT-4 .

Modern Love: Anna Martin unpacks the complicated love lives of real people, exploring topics like getting ghosted  and how to stop searching for the perfect partner .

Popcast:  A weekly conversation from Jon Caramanica and The Times’s music team, with roundtables on artists like Ice Spice , Taylor Swift  and  SZA .

From Opinion

The Ezra Klein Show:  Real conversations on the forces shaping the world, such as the increasing pace of A.I. development  and China's global influence .

First Person:  Lulu Garcia-Navarro explores intimate conversations about big ideas, like the experiences of a war reporter  and treating obesity as a disease.

Advertisement