Beyond Descriptive Statistics: Analysis of Time Series of Student Interactions in Engineering Courses Through the Lens of Modern Mathematical Methods

2025-08-21

Evaluation of LLMs and Other Machine Learning Methods in the Analysis of Qualitative Survey Responses for Accessible Engineering Education Research

2024-08-04 · 2 citations

This research paper provides insights and guidance for selecting appropriate analytical tools in engineering educational research.Currently, educators and researchers face difficulties in gaining insights effectively from free-response survey data.We evaluate the effectiveness and accuracy of Large Language Models (LLMs), in addition to the existing methods that employ topic modeling, document clustering coupled with Support Vector Machine (SVM) and Random Forest (RF) approaches, and the unsupervised Latent Dirichlet Allocation (LDA) method.Free responses to open-ended questions from student surveys in multiple courses at University of Illinois Urbana-Champaign were previously collected by engineering education accessibility researchers.The data (N=129 with seven free response questions per student) were previously analyzed to assess the effectiveness, satisfaction, and quality of adding accessible digital notes to multiple engineering courses and the students' perceived belongingness, and self-efficacy.Manual codings for the seven open-ended questions were generated for qualitative tasks of sentiment analysis, topic modeling, and summarization and were used in this study as a gold standard to evaluate automated text analytic approaches.Raw text from open-ended questions was converted into numerical vectors using text vectorization and word embeddings and an unsupervised analysis using document clustering and topic modeling was performed using LDA and BERT methods.In addition to conventional machine learning models, multiple pre-trained open-sourced local LLMs were evaluated (BART and LLaMA) for summarization.The remote online ChatGPT closed-model services by OpenAI (ChatGPT-3.5 and ChatGPT-4) were excluded due to subject data privacy concerns.By comparing the accuracy, recall, and depth of thematic insights derived, we evaluated how effectively the method based on each model categorized and summarized students' responses across educational research interests of effectiveness, satisfaction, and quality of education materials.The paper will present these results and discuss the implications of our findings and conclusions.

Evaluating the Low-Stakes Assessment Performance: Student-Perceived Accessibility, Belongingness, and Self-Efficacy in Connection to the Use of Digital Notes in Engineering and Computing Courses

2024-02-07 · 1 citations

Course content plays a critical role in student success. Among all college students, students with disabilities (SWD) face numerous additional challenges when digital content is inaccessible or difficult to use. Digital note-taking has been increasingly implemented and studied in higher education for its potential to further develop universal design for learning (UDL) techniques to benefit all students, especially SWD. This study combines low-stakes assessment results with surveys about accessibility, belongingness, self-efficacy and perceived learning in connection to the use of digital notes. The multi-discipline, multi-site, and multi-timepoint research design includes two junior-level undergraduate probability and statistics courses in the disciplines of Computer Science and Industrial Engineering at the University of Illinois Urbana Champaign (UIUC). The study investigated the effect of digital notes introduced at week 10, produced using ClassTranscribe developed at UIUC. The digital notes allow for transcripts, screenshots, and mathematical equations taken directly from video recordings. The digital notes were provided to the students of the two classes in both .pdf and .epub formats with included links to the video. Additionally, we considered Students With Accessibility Needs (SWAN) if they reported conditions that prevented them from attending class at some point while not having an officially recognized disability by the university.

Understanding the needs of students with and without disabilities for inclusive UDL-based design of Engineering courses through learning management systems

2024 · 3 citations

• Computer Science
• Computer Science
• Mathematics education

As increasingly many classes are transitioning part or all of their content to online platforms, it becomes crucial to identify which aspects of learning management systems (LMS) are sufficiently accommodating the learning of students from different backgrounds.We conducted a survey across undergraduate STEM students at the University of Illinois regarding their perceptions on LMS, including how the students with disabilities (SWDs) perceive the quality and features of LMS, because online access to course materials has been identified as the number one feature needed for SWDs and SWDs preferred to use their institute's LMS [5].We designed questions using four major constructs such as education equity, system quality, information quality, and performance impact of the LMS.Beyond demographic and disability disclosure, our survey also included the students' usage and view about the quality of specific features such as the calendar, message board, file uploading, assessments, LMS discussion boards, Gradescope integration, recordings posted in various capacities and accessibility of the features.In addition, we investigated the effect of students' mode of learning such as "in person", "hybrid", or "online" on their perceptions.Preliminary results from 131 responses (including 37 SWDs) that span 69 different courses predominantly show that students prefer to have a single website or LMS for their course, and SWDs have significantly stronger preference for such design.In the in-person learning mode, SWDs are significantly less pleased about the message posting on the course website.About the four constructs for LMS quality, the majority of students are giving high evaluations.We found there is a group of students (n=13) who did not have a particular officially accommodated disability but faced conditions that prevented them from attending class at some point.Including such a group of students with SWD together and naming them SWD-like, we found that SWD-like were significantly less pleased than students without disabilities (SWODs) with the way course materials were being posted to the existing course website.Furthermore, students with SWD-like status rated the efficacy of lecture video transcripts to be higher, and were generally less satisfied with how Canvas was being used as an LMS.From these results, we can identify valuable knowledge and opportunities to develop more inclusive LMS-based courses that serve all students, especially SWDs fairly following Universal Design Principles (UDL) [4].

A Digital Book Based Pedagogy to Improve Course Content Accessibility for Students with and without Disabilities in Engineering or other STEMcourses (WIP)

2024 · 3 citations

• Computer Science
• Computer Science
• Mathematics education

Accessibility of course content plays a critical role in student success. Among all college students, students with disabilities (SWD) face numerous additional challenges when digital content is inaccessible or difficult to use. The main contributions of this paper are firstly we examined different course delivery modalities to identify components that engage both SWD and students without disabilities (SWOD) and secondly, present new accessibility features for digital book creation that were implemented at the University of Illinois. An equitable, inclusive design for everyone is our ultimate goal, but we also wanted to understand the needs, and preferences of, SWDs in particular. A national survey instrument by the Collegiate Student Assessment of Textbooks (CSAT) to study student preferences was adapted and supplemented with questions that allowed students to optionally identify as SWD with a physical, mental and/or emotional disability. This survey enabled an analysis of the textbook preferences for engineering students with and without disabilities. Results from 50 SWD and 48 SWOD indicated that SWD and SWODs prioritized similar features. The main significant difference between SWD and SWOD were in the responses about how the instructor used the textbook and the importance of graphs in the textbook. The SWDs cared strongly about the accessibility features of a textbook. The top-five desired-features among all students were: 1) The book is low-cost or free, 2) There is a search feature for the book, 3) The book is up to date, 4) The examples used in the book matched the definitions, 5) The examples used in the book are relevant. Faculty were also surveyed using the same questions, providing insight into areas of alignment in preferences between students and faculty. Data from 10 faculty revealed similar textbook preferences: 1) The book is up to date, 2) Examples are representative of the definitions provided, 3) The book is low-cost or free, 4) The book is available online and as print copy, 5) The examples are relevant. Among the textbook features, all students were least interested in being called on during class with questions from the book. Additionally, SWDs did not value features pertaining to how the instructor used the textbook. By understanding the needs of SWD and SWOD, a faculty member can be informed about techniques to increase content accessibility. Secondly, we identified and designed new accessibility functionalities, including a visual table of contents, accessibility tags, and conditional publishing on students and instructors with a focus on meeting the needs of SWD. Recommendations and techniques are offered for instructors wishing to develop digital books to provide more accessible content delivery.

Opportunities and Barriers to UDL-Based Course Designs for Inclusive Learning in Undergraduate Engineering and other STEM Courses

2024-02-07 · 2 citations

Abstract Understanding the needs of students with disabilities (SWD) is crucial for inclusive learning in college education. Universal Design for Learning (UDL)-based methodologies are recognized as a best practice to improve the accessibility for SWD. Although we see increased technology usage and changes in course design after COVID-19, it is still unclear if UDL-based practices will continue to be adopted and persist in their use, and where the future opportunities are for more inclusive engineering education for all students, and for SWD in particular. In this study, we surveyed faculty and undergraduate students (predominantly in engineering) about their experiences with 16 UDL-based course design practices and their perceived usefulness at the University of Illinois Urbana-Champaign (UIUC). These practices span the three UDL guidelines (representation, engagement, action-expression) and essential accessibility features. We also surveyed instructors about their knowledge of UDL and the barriers to implementing these practices in engineering/STEM courses. We also identified the differences in responses between underrepresented groups in engineering and the general student population, as well as differences between students and faculty. The survey design allowed students with physical, mental, and/or emotional disabilities to self-report as a SWD. Additionally, we identify students as Students with Access Challenges and Accommodation Needs (SACAN) if they faced conditions that prevented them from attending class at some point while not having an officially recognized disability by the university or if they are a SWD. Preference regarding feedback methods, class formats, and demographic questions such as gender and major were also included. Our survey results from 148 students (including 50 SWD, 77 SACAN) of more than 20 majors showed that: 1) The five most useful UDL practices among all students are: searchable recorded lectures, flexible deadlines, transcripts for videos, official discussion platform, and alternative learning formats to lectures (i.e: textbooks, and slides); 2) Female students experience UDL practices in their courses significantly less (False Discovery Rate (FDR) controlled p < 0.03 for 13 practices); 3) Female students and SWD are significantly more uncomfortable giving direct feedback to instructors (chisq p < 0.006); 4) SWD experience recorded lectures, onboarding forms for accessibility, and alternative learning formats significantly (FDR controlled p < 0.015) less than students without disabilities (SWOD); 5) All groups of students rank and experience the UDL guidelines in the following order: representation, action-expression, and engagement; 6) Although students rate the usefulness of all the UDL features higher than their frequency (except "autograders"), some features show large differences between students perceived usefulness and how often they are actually experienced. For example, "flexible deadlines" is considered useful (ranked 2nd in usefulness) while experienced very infrequently (ranked 7th in frequency). The 25 faculty responses on rankings of the UDL practices differ with that of the students; notably, "frequent low-stake tests" is valued among the faculty, but was not in the top five of the students' responses. Faculty responses suggest that the major barriers in implementing UDL practices are the lack of awareness of accessibility features and lack of training for technologies such as creating recordings and transcripts. Our study provides empirical findings about opportunities and barriers for implementing UDL practices for all stakeholders in developing more inclusive courses.

Accessibility Nuggets, Video Vignettes, and Other Instructor Development Approaches to Foster UDL Adoption and Inclusive Engineering Education

2024-08-03

Students with disabilities need accessible courses.Universal Design for Learning (UDL) has become a well-known approach to creating inclusive and accessible education.However, despite general interest around UDL among instructors, previous studies have revealed the lack of resources to adequately educate instructors on UDL, accessibility principles, and best practices.This study details multiple methodologies at the University of Illinois Urbana Champaign used to educate and inspire instructors about UDL with the goal of creating more accessible engineering courses.We implemented multiple approaches to increase interest in, and utilization of, UDL by instructors: 1) Developed a Learning Management System (LMS) based training course which gave instructors "plug-and-play" practical examples of UDL design practices; 2) Created short tips ("Accessibility Nuggets") to inspire interest and show the starting points; 3) Provided handson help for non-UDL based engineering courses to make them more accessible; 4) Surveyed students and instructors about UDL practices

SEAM methodology for context-rich player matchup evaluations

arXiv (Cornell University) · 2020-05-15

We develop the SEAM (synthetic estimated average matchup) method for describing batter versus pitcher matchups in baseball, both numerically and visually. We first estimate the distribution of balls put into play by a batter facing a pitcher, called the spray chart distribution. This distribution is conditional on batter and pitcher characteristics. These characteristics are a better expression of talent than any conventional statistics. Many individual matchups have a sample size that is too small to be reliable. Synthetic versions of the batter and pitcher under consideration are constructed in order to alleviate these concerns. Weights governing how much influence these synthetic players have on the overall spray chart distribution are constructed to minimize expected mean square error. We then provide novel performance metrics that are calculated as expectations taken with respect to the spray chart distribution. These performance metrics provide a context rich approach to player evaluation. Our main contribution is a Shiny app that allows users to evaluate any batter-pitcher matchup that has occurred or could have occurred in the last five years. One can access this app at \url{this https URL}. This interactive tool has utility for anyone interested in baseball as well as team executives and players.

SEAM methodology for context-rich player matchup evaluations

arXiv (Cornell University) · 2020-05-15 · 1 citations

We develop the SEAM (synthetic estimated average matchup) method for describing batter versus pitcher matchups in baseball. We first estimate the distribution of balls put into play by a batter facing a pitcher, called the empirical spray chart distribution. Many individual matchups have a sample size that is too small to be reliable for use in predicting future outcomes. Synthetic versions of the batter and pitcher under consideration are constructed in order to alleviate these concerns. Weights governing how much influence these synthetic players have on the overall estimated spray chart distribution are constructed to minimize expected mean square error. We provide a Shiny web application that allows users to visualize and evaluate any batter-pitcher matchup that has occurred or could have occurred during the Statcast era (specifically 2017-present). This methodology and web application could be used to determine defensive alignments, lineup construction, or pitcher selection through estimation of spray densities based on any input matchup. One can access this web application at https://seam.stat.illinois.edu/. The computational speed with which the method calculates the spray densities allows the app to display the visualizations for any input almost instantly. Therefore, SEAM offers distributional interpretations of dependent matchup data which is computationally fast.

Additional file 6 of Statistical inference for time course RNA-Seq data using a negative binomial mixed-effect model

Figshare · 2016-01-01

Functional Annotation Clustering for genes specifically found by maSigPro in comparison with NBMM. (XLSX 11 kb)