Sounding source localization is a challenging cross-modal task due to the difficulty of cross-modal alignment. 
Although supervised cross-modal methods achieve encouraging performance, heavy manual annotations are expensive and inefficient. Thus it is valuable and meaningful to develop unsupervised solutions. 
In this paper, we propose an **U**nsupervised **S**ounding **P**ixel **L**earning (USPL) approach which enables a pixel-level sounding source localization in unsupervised paradigm. 
We first design a mask augmentation based multi-instance contrastive learning to realize unsupervised cross-modal coarse localization, which aligns audio-visual features to obtain coarse sounding maps. 
Secondly, we present an *Unsupervised Sounding Map Refinement (SMR)* module which employs the visual semantic affinity learning to explore inter-pixel relations of adjacent coordinate features. It contributes to recovering the boundary of coarse sounding maps and obtaining fine sounding maps. 
Finally, a *Sounding Pixel Segmentation (SPS)* module is presented to realize audio-supervised semantic segmentation. 
Extensive experiments are performed on the AVSBench-S4 and VGGSound datasets, exhibiting encouraging results compared with previous SOTA methods.

Unsupervised Sounding Pixel Learning

In multimodal understanding tasks, visual and linguistic ambiguities can arise. Visual ambiguity can occur when visual objects require a model to ground a referring expression in a video without strong supervision, while linguistic ambiguity can occur from changes in entities in action flows. As an example from the cooking domain, "oil" mixed with "salt" and "pepper" could later be referred to as a "mixture". Without a clear visual-linguistic alignment, we cannot know which among several objects shown is referred to by the language expression “mixture”, and without resolved antecedents, we cannot pinpoint what the mixture is. We define this chicken-and-egg problem as Visual-linguistic Ambiguity. In this paper, we present Find2Find, a joint anaphora resolution and object localization dataset targeting the problem of \emph{visual-linguistic ambiguity}, consisting of 500 anaphora-annotated recipes with corresponding videos. We present experimental results of a novel end-to-end joint multitask learning framework for Find2Find that fuses visual and textual information and shows improvements both for anaphora resolution and object localization with one joint model in multitask learning, as compared to a strong single-task baseline.

Find-2-Find: Multitask Learning for Anaphora Resolution and Object Localization

When used with deep learning, the symbolic music modality is often coupled with language model architectures. To do so, the music needs to be tokenized, i.e. converted into a sequence of discrete tokens. This can be achieved by different approaches, as music can be composed of simultaneous tracks, of simultaneous notes with several attributes. Until now, the proposed tokenizations rely on small vocabularies of tokens describing the note attributes and time events, resulting in fairly long token sequences, and a sub-optimal use of the embedding space of language models. Recent research has put efforts on reducing the overall sequence length by merging embeddings or combining tokens. In this paper, we show that Byte Pair Encoding, a compression technique widely used for natural language, significantly decreases the sequence length while increasing the vocabulary size. By doing so, we leverage the embedding capabilities of such models with more expressive tokens, resulting in both better results and faster inference in generation and classification tasks. The [source code is shared on Github](https://github.com/Natooz/bpe-symbolic-music), along with a [companion website](https://Natooz.github.io/BPE-Symbolic-Music). Finally, BPE is directly implemented in [MidiTok](https://github.com/Natooz/MidiTok), allowing the reader to easily benefit from this method.

Byte Pair Encoding for Symbolic Music

Autoregressive language modeling (ALM) have been successfully used in self-supervised pre-training in Natural language processing (NLP). However, this paradigm has not achieved comparable results with other self-supervised approach in computer vision (e.g., contrastive learning, mask image modeling). In this paper, we try to find the reason why autoregressive modeling does not work well on vision tasks. To tackle this problem, we fully analyze the limitation of visual autoregressive methods and proposed a novel stochastic autoregressive image modeling (named SAIM) by the two simple designs. First, we employ stochastic permutation strategy to generate effective and robust image context which is critical for vision tasks. Second, we create a parallel encoder-decoder training process in which the encoder serves a similar role to the standard vision transformer focus on learning the whole contextual information, and meanwhile the decoder predicts the content of the current position, so that the encoder and decoder can reinforce each other. By introducing stochastic prediction and the parallel encoder-decoder, SAIM significantly improve the performance of autoregressive image modeling. Our method achieves the best accuracy (83.9\%) on the vanilla ViT-Base model among methods using only ImageNet-1K data. Transfer performance in downstream tasks also show that our model achieves competitive performance.

Exploring Stochastic Autoregressive Image Modeling for Visual Representation

Over the last decade, there has been a resurgence of interest in the climatic and tectonic mechanisms that drove the Messinian salinity crisis (MSC) and the associated deposition of thick evaporites. The MSC represents an unprecedented palaeoceanographic change that led to a very short (c. 630 kyr) ecological and environmental crisis (Andreetto et al., 2021).  However, across the Levantine offshore basin, the sedimentological nature of the top evaporitic units and the mechanisms that controlled the transition from a hypersaline evaporitic unit to brackish deposits (final MSC stage 3) are still disputed. Here,  we re-evaluate the deposits associated with the terminal phase of the MSC (ca. later stage of  3.2 by Meilijson et al.,2019) ,  named in offshore Lebanon as the Nahr Menashe deposits (NM, Madof et al.,2019). We describe the NM seismic facies, characterize and map its internal seismic stratigraphy, and provide a new interpretation of its depositional environment, which persisted during the late Messinian and then evolved through a regional reflooding event. The base of the NM and some of its internal units overlies semi-circular depressions, randomly distributed linear marks and surface collapse features, which are indicative of a period of intense evaporite dissolution. The NMU seismic facies observed from the slope to the deep part of the basin support the interpretation of a layered salt-evaporite-sand depositional system subject to complex reworking, dissolution, deposition, and final erosion. A drainage network of valleys and complex tributary channels incising into the top NM shows marked erosional characteristics, which indicate a dominant southwards sediment transfer following deposition of the NM. This erosional drainage network formed due to the base-level fall associated with the last phase of the MSC. The base of the channel/valley network does not cut below the bottom of the Narh Menashe dissolution surfaces. The channel and valley network was subsequently infilled by layered sediments interpreted here to represent the post-MSC marine sediments deposited during reflooding.  Our analysis point out that the NM may represent a mixed evaporite-siliciclastic affected by karstic processes, deposited or re deposited in a shallow marine or lacustrine during the tilting of the offshore Lebanese basin .  Only subsequently did the NM experience a rapid erosional event followed by swift burial of  transgressive/high-stand sediments.

Formation, dissolution and fluvial incision of the top evaporite seismic unit (Nahr Menashe) recording the end of the  Messinian salinity Crisis , NE offshore Levant Basin, Eastern Mediterranean

Based on a user-study with experts in motion planning algorithms, we propose a taxonomy of explanations, including kinds of questions/explanations/methods related to motion planning, and a set of design considerations for explainable motion planning.

How experts explain motion planner output: a preliminary user-study to inform the design of explainable planners

Nurses have to carry out a myriad of tasks, which is one of the main reasons for burnout. In this paper, a robotic Multitasking Intelligent Nurse Aid (MINA) is proposed to assist nurses with everyday tasks and tackle nurse burnout. MINA uses Simultaneous Localization And Mapping (SLAM) to map shelves in a storage room and pointcloud using an RGB-Depth camera. The barcode of the item is first detected using zbar library, and then the pointcloud is generated in ROS and saved accordingly. Preliminary experiments with respect to the creation of a pointcloud map with barcode detection are presented. The preliminary results reveal that the system shows a 100% barcode detection rate, and with each barcode detection a corresponding pointcloud map is generated and saved automatically without any intervention.

Underline digital video library

Unsupervised Sounding Pixel Learning

Find-2-Find: Multitask Learning for Anaphora Resolution and Object Localization

Byte Pair Encoding for Symbolic Music

Exploring Stochastic Autoregressive Image Modeling for Visual Representation

Formation, dissolution and fluvial incision of the top evaporite seismic unit (Nahr Menashe) recording the end of the Messinian salinity Crisis , NE offshore Levant Basin, Eastern Mediterranean

How experts explain motion planner output: a preliminary user-study to inform the design of explainable planners

MINA: A Multitasking Intelligent Nurse Aid Robot

Stay up to date with the latest Underline news!

PRESENTATIONS

CONFERENCES

COMPANY

RESOURCES