Learner-as-Principal Investigator: An Innovative Research Educational Program

NSF-CREST Center for Enviro Nano: Empowering Scientists, Fostering Inclusive Community Science

Influence of personalization classes on students' attitude and motivation at a HBCU

	Videos such as movies or TV episodes usually need to divide the long storyline into cohesive units, i.e., scenes, to facilitate the understanding of video semantics. The key challenge lies in finding the boundaries of scenes by comprehensively considering the complex temporal structure and semantic information. To this end, we introduce a novel Context-Aware Transformer (CAT) with a self-supervised learning framework to learn high-quality shot representations, for generating well-bounded scenes. More specifically, we design the CAT with local-global self-attentions, which can effectively consider both the long-term and short-term context to improve the shot encoding. For training the CAT, we adopt the self-supervised learning schema. Firstly, we leverage shot-to-scene level pretext tasks to facilitate the pre-training with pseudo boundary, which guides CAT to learn the discriminative shot representations that maximize intra-scene similarity and inter-scene discrimination in an unsupervised manner. Then, we transfer contextual representations for fine-tuning the CAT with supervised data, which encourages CAT to accurately detect the boundary for scene segmentation. As a result, CAT is able to learn the context-aware shot representations and provides global guidance for scene segmentation. Our empirical analyses show that CAT can achieve state-of-the-art performance when conducting the scene segmentation task on the MovieNet~\cite{RaoXXXHZL20} dataset, e.g., offering 2.15 improvements on AP.

Towards Global Video Scene Segmentation with Context-Aware Transformer

Smartphone based labs are a cornerstone of our efforts to reform the introductory labs in both our calculus-based and algebra-based classes. Students use smartphones to collect data in the lab (Bluetooth paired with sensors) and at home (using the native sensors in the phone). In this talk, we will give a brief overview of the project, then emphasize the at-home labs. These labs have allowed us to substantially reduce crowding in the lab, emphasize data analysis, and give students an opportunity to be more exploratory in their approach to lab. We will highlight examples of our at-home labs and discuss lessons learned during the first year of the project.

Smartphones in the introductory mechanics lab

For the second year of the pandemic, we developed new labs for our life-science stream intro physics courses. Despite the return to on-campus learning, our students engaged in unscheduled, remote experiments. The goal of our labs was to connect life science topics (fluids, optics, spectroscopy) to an everyday context through hands-on experiments. We created ‘at-home’ lab kits for students, combining common items (rulers, string, battery) reinforced with standardized items for consistency (e.g. known masses) and specialized materials (capillaries, LEDs, diffraction grating). We employed smart phone apps for measurement, embedded Geogebra for data analysis, and complemented activities with the video- and simulation-elements from virtual labs of the previous year. Some of highlights of our experience over the year include: a citizen-science lab, insights into the workflow of students, and their preference regarding ‘scheduled & on-campus’ vs ‘asynchronous & remote’.

Lab Kits for ‘At-Home’ Experiments in Physics for Life Sciences

In this work we study the association between the song lyrics and \textit{mood} associations with the song, and conduct a data driven analysis to compare how lyrics and acoustics each contribute to the understanding of a song's mood. We conduct our study on a data set containing nearly one million songs, with song-mood associations derived from user playlists on a streaming platform. We take advantage of  state-of-the-art natural language processing models based on transformers to learn the association between the lyrics and moods. We find that a pretrained transformer-based model in zero-shot setting -- i.e., out of the box with no further training on our data -- is powerful for capturing song-mood associations. Moreover, we illustrate that training on song-mood associations results in a highly accurate model that predicts these associations for unseen songs. Furthermore, by comparing the prediction of a model using lyrics with one using acoustics, we observe that the relative importance of lyrics in comparison with acoustics depends on the specific mood. Finally, we verify if the models are capturing the same information about lyrics and acoustics as humans by conducting a manual annotation task on a hundred songs where we obtain human judgments of mood-song relevance based on lyrics and acoustics in isolation.

The Contribution of Lyrics and Acoustics to Collaborative Understanding of Mood

Advances in Natural Language Processing (NLP) have revolutionized the way researchers and practitioners address crucial societal problems. Large language models are now the standard to develop state-of-the-art solutions for text detection and classification tasks. However, the development of advanced computational techniques and resources is disproportionately focused on the English language, sidelining a majority of the languages spoken globally. While existing research has developed better multilingual and monolingual language models to bridge this language disparity between English and non-English languages, we explore the promise of incorporating the information contained in images via multimodal machine learning. Our comparative analyses on three detection tasks focusing on crisis information, fake news, and emotion recognition, as well as five high-resource non-English languages, demonstrate that: (a) detection frameworks based on pre-trained large language models like BERT and multilingual-BERT systematically perform better on the English language compared against non-English languages, and (b) including images via multimodal learning bridges this performance gap. We situate our findings with respect to existing work on the pitfalls of large language models, and discuss their theoretical and practical implications.

Overcoming Language Disparity in Online Content Classification with Multimodal Learning

During the COVID-19 pandemic, local news organizations have played an integral role in keeping communities informed about the spread and impact of the virus. We explore how political, social media, and economic factors impacted the way local media have reported on COVID-19 developments at a national scale. We construct and make available a dataset of over 10,000 local news organizations across the U.S. and their social media handles. We use social media data to model the population reach of outlets, and the underlying content relationships between them. Building on this data, we analyze how local and national media covered four key COVID-19 news topics: Statistics and Case Counts, Vaccines and Testing, Public Health Guidelines, and Economic Effects. Our results show that news outlets with higher population reach have reported proportionally more on COVID-19 than more local outlets. When broken out by topic, we observe that the Statistics and Case Counts topic is covered proportionally more by outlets that have a lower population reach and are in more Republican-leaning areas, an effect which is re- versed for the Economic Effects topic. Our analysis further shows that while social media audiences generally engage less with outlets with a smaller population reach, they tend to engage more with these outlets on posts about COVID-19. People also use relatively more emotive reactions on the posts of low population reach outlets. Finally, we demonstrate that COVID-19 posts in Republican-leaning counties generally receive more comments and fewer likes than in Democratic counties.

Understanding Local News Social Coverage and Engagement at Scale During the COVID-19 Pandemic

Creating computer generated human animations without the use of motion capture technology is a tedious and time consuming activity. Although there are several publications regarding animation synthesis using data driven methods, not many are dedicated towards the task of inbetweening, which consists of generating transition movements between \textit{frames}.
In this work, we are interested on short-term transitions and different kinds of movement, such as martial arts and Indian dance using the least amount of input \textit{frames} as possible.

Underline digital video library

Learner-as-Principal Investigator: An Innovative Research Educational Program

NSF-CREST Center for Enviro Nano: Empowering Scientists, Fostering Inclusive Community Science

Influence of personalization classes on students' attitude and motivation at a HBCU

Towards Global Video Scene Segmentation with Context-Aware Transformer

Smartphones in the introductory mechanics lab

Lab Kits for ‘At-Home’ Experiments in Physics for Life Sciences

The Contribution of Lyrics and Acoustics to Collaborative Understanding of Mood

Overcoming Language Disparity in Online Content Classification with Multimodal Learning

Understanding Local News Social Coverage and Engagement at Scale During the COVID-19 Pandemic

Short-term Inbetweening of 3D Human Motions

Using podcasts in storytelling and increasing research visibility - Joy Owango, Adelle Onyango

OWSD Visions: Telling science stories through film

Stay up to date with the latest Underline news!

PRESENTATIONS

CONFERENCES

COMPANY

RESOURCES