Multi-view camera-based 3D object detection has become popular due to its low cost, but accurately inferring 3D geometry solely from camera data remains challenging and may lead to inferior performance. Although distilling precise 3D geometry knowledge from LiDAR data could help tackle this challenge, the benefits of LiDAR information could be greatly hindered by the significant modality gap between different sensory modalities. To address this issue, we propose a Simulated multi-modal Distillation (SimDistill) method by carefully crafting the model architecture and distillation strategy. Specifically, we devise multi-modal architectures for both teacher and student models, including a LiDAR-camera fusion-based teacher and a simulated fusion-based student. Owing to the ``identical'' architecture design, the student can mimic the teacher to generate multi-modal features with merely multi-view images as input, where a geometry compensation module is introduced to bridge the modality gap. Furthermore, we propose a comprehensive multi-modal distillation scheme that supports intra-modal, cross-modal, and multi-modal fusion distillation simultaneously in the Bird's-eye-view space. Incorporating them together, our SimDistill can learn better feature representations for 3D object detection while maintaining a cost-effective camera-only deployment. Extensive experiments validate the effectiveness and superiority of SimDistill over state-of-the-art methods, achieving an improvement of 4.8% mAP and 4.1% NDS over the baseline detector. The source code will be released at https://github.com/ViTAE-Transformer/SimDistill.

SimDistill: Simulated Multi-Modal Distillation for BEV 3D Object Detection

In the realm of Zero-Shot Learning (ZSL), we address biases in Generalized Zero-Shot Learning (GZSL) models, which favor seen data. To counter this, we introduce an end-to-end generative GZSL framework called D$^3$GZSL. This framework respects seen and synthesized unseen data as in-distribution and out-of-distribution data, respectively, for a more balanced model. D$^3$GZSL comprises two core modules: in-distribution dual space distillation (ID$^2$SD) and out-of-distribution batch distillation (O$^2$DBD). ID$^2$SD aligns teacher-student outcomes in embedding and label spaces, enhancing learning coherence. O$^2$DBD introduces low-dimensional out-of-distribution representations per batch sample, capturing shared structures between seen and unseen categories. Our approach demonstrates its effectiveness across established GZSL benchmarks, seamlessly integrating into mainstream generative frameworks. Extensive experiments consistently showcase that D$^3$GZSL elevates the performance of existing generative GZSL methods, underscoring its potential to refine zero-shot learning practices.The code is available at: https://github.com/PJBQ/D3GZSL.git

Data Distribution Distilled Generative Model for Generalized Zero-Shot Recognition | VIDEO

Background: Endoscopy is commonly used to diagnose upper gastrointestinal (GI) system disorders. However, excess mucus can significantly impair visualization, and reducing mucus intra-procedurally can increase procedure time. Several studies have evaluated the efficacy of pre-medication with mucolytic agents prior to gastroscopy to improve the diagnostic yield of endoscopy and reduce procedural time. Here, we systematically review recent evidence assessing the efficacy of mucolytic agents, including N-acetyl cysteine (NAC), in improving visibility in upper GI endoscopy.
Methods: Studies were identified by interrogating the PubMed database using the search terms "endoscopy" and "N-acetyl cysteine" for the period of 2016-2023. Only randomized control trials were included in the search. Records written in languages other than English were excluded. Ten (10) records matching these criteria were retrieved. Title and abstract screening resulted in five (5) records, which were subsequently extracted.
Results: The studies were conducted in the United Kingdom, Chile, Portugal, Thailand, and the Czech Republic. Four (4) studies consisted of randomized, double-blinded, placebo-controlled trials, while the remaining study was a randomized, non-blinded controlled trial. Four (4) of the studies showed that pre-administration of NAC (600 mg) plus simethicone (100 mg) significantly improved the visibility of the upper GI tract (esophagus, stomach, and duodenum) when compared to placebo control. The only study that failed to show an improvement in visibility scores was attributed to sub-threshold doses of NAC (400 mg) and simethicone (20 mg). Although none of the studies included an arm consisting of NAC alone, one study comparing simethicone +/- NAC showed superior visibility when NAC was administered at a high dose (1000 mg), while another study comparing simethicone +/- NAC failed to show improvement when NAC was administered at a moderate dose (600 mg). Lastly, one study found that the procedure time and volume of mucosal cleansing solution required were significantly reduced when NAC and simethicone were administered together.


Conclusion: Our survey of recent randomized control trials supports that pre-medication with NAC (600-1000 mg) and simethicone (100 mg) significantly improves the visibility of the upper GI mucosal surfaces during upper endoscopic procedures and, therefore, may improve diagnostic yield in settings where visibility is impaired. This approach may be particularly useful in lower-resource settings where less sophisticated endoscopic tools are utilized.

Efficacy of N-Acetyl Cysteine in the Prevention of Post-Operative Atrial Fibrillation and Acute Kidney Injury Post-Cardiac Surgery: A Systematic Review

Background
Lung cancer is the leading cause of cancer deaths worldwide. The increasing use of minimally invasive surgical approaches has expanded the scope of treatment options for patients with lung cancer. However, these operations are not risk-free as there have been several reported cases of intraoperative catastrophes, with major hemorrhage from the pulmonary artery being the most common. The objective of this study was to develop a computer vision-based approach to reliably identify the pulmonary artery during robotic right lower lobectomies. 

Methods
Twelve patients with biopsy-proven Stage I non-small-cell lung cancer of the right lower lobe underwent robotic lobectomy with mediastinal lymph node dissection. Complete videos of each operation were obtained, and video fragments of the pulmonary artery were identified by two board-certified thoracic surgeons. Annotation masks of the pulmonary artery were then created using Computer Vision Annotation Tool, an open-source web-based annotation tool. The labeled data was next used to train a state-of-the-art instance segmentation algorithm called Mask Regional-Convolutional Neural Network (Mask R-CNN) using an 80:20 random sample split from ten cases for training and validation, respectively. Two cases were utilized for generalization testing. Three custom performance metrics commonly utilized in deep learning-based instance segmentation were used to evaluate our approach: Intersection over Union (IoU), Average Precision at IoU threshold of 50% (AP50) and at 75% (AP75). 

Results
Annotation masks of the pulmonary artery were created in 4,139 images across twelve cases of robotic right lower lobectomy. 2,893 and 684 annotated images across ten (83.3%) cases were used for training and validation, respectively; and 261 and 301 annotated images from two (16.7%) cases were used for generalization testing. A mean IoU of 87.7% was achieved in the validation dataset. The AP50 attained by our model was 96.7% and AP75 was 82.3%. In generalizability testing, when the model was tested on data never exposed to it, it was able to partially generalize with a mean IoU of 86.0%. 

Conclusion
Our study shows that our Mask R-CNN model was able to identify surgical anatomy during robotic lobectomy with high accuracy in the validation dataset. While still at an early stage, more variable labeled data and collaborative efforts are needed to improve the generalizability of our model. We envision such a computer vision system to be valuable in early surgeon training and incorporate future real-time augmentation of video feedback to improve outcomes.






Using Artificial Intelligence for Automatic Detection of the Pulmonary Artery During Robotic Right Lower Lobectomies

Androids that strongly but imperfectly resemble humans in shape can elicit negative emotions in people, a phenomenon known as the "uncanny valley," which has been replicated in laboratory experiments. Recently, the accuracy of face recognition utilizing machine learning has increased, raising the question of whether machine learning can replicate the uncanny valley. Using FaceNet as a representative face recognition algorithm, we examined the similarity of face recognition to human evaluation and its replication of the uncanny valley. The results revealed a strong correlation between machine learning and human evaluation of human-like shapes. However, it is evident that only certain aspects of the uncanny valley were replicated. Furthermore, visualization of the activation maps suggests that localized regions, such as the mouth and chin, acted as the basis for judgment. These findings support the idea that human and machine learning have distinct areas of attention.

Does Machine Learning Replicate the Uncanny Valley? An Example using FaceNet

Human physical scene understanding requires more than simply localizing and recognizing objects — we can quickly adapt our predictions about how a scene will unfold by incorporating objects' latent physics properties, such as the masses of the objects in the scene. What are the underlying computational mechanisms that allow humans to infer these physical properties and adapt their physical predictions so efficiently from visual inputs? One hypothesis is that general intuitive physics knowledge can be learned from enough raw data, instantiated as computational models that predict future video frames in large datasets of complex scenes. To test this hypothesis, we evaluate existing state-of-the-art video models. We measured both model and human performance on Physion++, a novel dataset and benchmark that rigorously evaluates visual physical prediction in humans and machines, under circumstances where accurate physical prediction relies on accurate estimates of the latent physical properties of objects in the scene. Specifically, we tested scenarios where accurate prediction relied on accurate estimates of objects' mechanical properties, including masses, friction, elasticity and deformability, and the values of these mechanical properties could only be inferred by observing how these objects moved and interacted with other objects and/or fluids. We found that models that encode objectness and physical states tend to perform better, yet there is still a huge gap compared to human performance. We also found most models' predictions correlate poorly with that made by humans. These results show that current deep learning models that succeed in some settings nevertheless fail to achieve human-level physical prediction in other cases, especially those where latent property inference is required.

Physion++: Evaluating Physical Scene Understanding with Objects Consisting of Different Physical Attributes in Humans and Machines

The ability to discriminate between large and small quantities is a core aspect of basic numerical competence in both humans and animals. In this work, we examine the extent to which the state-of-the-art neural networks designed for vision exhibit this basic ability. Motivated by studies in animal and infant numerical cognition, we use the numerical bisection procedure to test number discrimination in different families of neural architectures. Our results suggest that vision-specific inductive biases are helpful in numerosity discrimination, as models with such biases have lowest test errors on the task, and often have psychometric curves that qualitatively resemble those of humans and animals performing the task. However, even the strongest models, as measured on standard metrics of performance, fail to discriminate quantities in transfer experiments with differing training and testing conditions, indicating that such inductive biases might not be sufficient.

Evaluating Visual Number Discrimination in Deep Neural Networks

Recent work in computer vision and cognitive reasoning has given rise to an increasing adoption of the Violation-of-Expectation (VoE) paradigm in synthetic datasets. Inspired by infant psychology, researchers are now evaluating a model’s ability to label scenes as either expected or surprising with knowledge of only expected scenes. However, existing VoE-based 3D datasets in physical reasoning provide mainly vision data with little to no heuristics or inductive biases. Cognitive models of physical reasoning reveal infants create high-level abstract representations of objects and interactions. Capitalizing on this knowledge, we established a benchmark to study physical reasoning by curating a novel large-scale synthetic 3D VoE dataset armed with ground-truth heuristic labels of causally relevant features and rules. To validate our dataset in five event categories of physical reasoning, we benchmarked and analyzed human performance. We also proposed the Object File Physical Reasoning Network (OFPR-Net) which exploits the dataset's novel heuristics to outperform our baseline and ablation models. The OFPR-Net is also flexible in learning an alternate physical reality, showcasing its ability to learn universal causal relationships in physical reasoning to create systems with better interpretability.

A Benchmark for Modeling Violation-of-Expectation in Physical Reasoning Across Event Categories

We present a novel method for image saliency prediction that leverages a cognitive model of visual attention as an inductive bias.
This is in stark contrast to recent purely data-driven
models that have achieved performance improvements mainly by increased model capacity, resulting in high computational costs and the need for large scale, domain specific training data. 
We demonstrate that by leveraging a cognitive model of visual attention, our method achieves competitive performance to the state-of-the-art across several natural image datasets while only requiring a fraction of the parameters.
Furthermore, we set the new state of the art for saliency prediction on information visualizations, demonstrating the effectiveness of our approach for cross-domain generalization.
We further provide large-scale cognitively plausible synthetic gaze data on corresponding images in the full MSCOCO and FigureQA datasets, which we used for pre-training.
These results are highly promising and underline the significant potential of bridging between first-principle cognitive and data-driven models for computer vision tasks, potentially also beyond saliency prediction, and even visual attention.

\bf Keywords: neural networks, computational models of cognition, computer vision, saliency prediction, human visual attention

Improving neural saliency prediction with a cognitive model of human visual attention

As acquiring manual labels on data could be costly, unsupervised domain adaptation (UDA), which transfers knowledge learned from a rich-label dataset to the unlabeled target dataset, is gaining increasing popularity. While extensive studies have been devoted to improving the model accuracy on target domain, an important issue of model robustness is neglected. To make things worse, conventional adversarial training (AT) methods for improving model robustness are inapplicable under UDA scenario since they train models on adversarial examples that are generated by supervised loss function. In this paper, we present a new meta self-training pipeline, named SRoUDA, for improving adversarial robustness of UDA models. Based on self-training paradigm, SRoUDA starts with pre-training a source model by applying UDA baseline on source labeled data and taraget unlabeled data with a developed random masked augmentation (RMA), and then alternates between adversarial target model training on pseudo-labeled target data and fine-tuning source model by a meta step. While self-training allows the direct incorporation of AT in UDA, the meta step in SRoUDA further helps in mitigating error propagation from noisy pseudo labels. Extensive experiments on various benchmark datasets demonstrate the state-of-the-art performance of SRoUDA where it achieves significant model robustness improvement without harming clean accuracy.

SRoUDA: Meta Self-training for Robust Unsupervised Domain Adaptation

In image denoising networks, feature scaling is widely used to enlarge the receptive field size and reduce computational costs. This practice, however, also leads to the loss of high-frequency information and fails to consider within-scale characteristics. Recently, dynamic convolution has exhibited powerful capabilities in processing high-frequency information (e.g., edges, corners, textures), but previous works lack sufficient spatial contextual information in filter generation. To alleviate these issues, we propose to employ dynamic convolution to improve the learning of high-frequency and multi-scale features. Specifically, we design a spatially enhanced kernel generation (SEKG) module to improve dynamic convolution, enabling the learning of spatial context information with a very low computational complexity. Based on the SEKG module, we propose a dynamic convolution block (DCB) and a multi-scale dynamic convolution block (MDCB). The former enhances the high-frequency information via dynamic convolution and preserves low-frequency information via skip connections. The latter utilizes shared adaptive dynamic kernels and the idea of dilated convolution to achieve efficient multi-scale feature extraction. The proposed multi-dimension feature integration (MFI) mechanism further fuses the multi-scale features, providing precise and contextually enriched feature representations. Finally, we build an efficient denoising network with the proposed DCB and MDCB, named ADFNet. It achieves better performance with low computational complexity on real-world and synthetic Gaussian noisy datasets. The source code is available at https://github.com/it-hao/ADFNet.

Adaptive Dynamic Filtering Network for Image Denoising

Images captured in low-light environments have problems of insufficient brightness and low contrast, which will affect subsequent image processing tasks. Although most current enhancement methods can obtain high-contrast images, they still suffer from noise amplification and color distortion. To address these issues, this paper proposes a low-light image enhancement network based on multi-scale feature complementation (LIEN-MFC), which is a U-shaped encoder-decoder network supervised by multiple images of different scales. In the encoder, four feature extraction branches are constructed to extract features of low-light images at different scales. In the decoder, to ensure the integrity of the learned features at each scale, a feature supplementary fusion module (FSFM) is proposed to complement and integrate features from different branches of the encoder and decoder. In addition, a feature restoration module (FRM) and an image reconstruction module (IRM) are built in each branch to reconstruct the restored features and output enhanced images. To better train the network, a joint loss function is defined, in which a discriminative loss term is designed to ensure that the enhanced results better meet the visual properties of the human eye. Extensive experiments on benchmark datasets show that the proposed method outperforms some state-of-the-art methods subjectively and objectively.

Underline digital video library

SimDistill: Simulated Multi-Modal Distillation for BEV 3D Object Detection

Data Distribution Distilled Generative Model for Generalized Zero-Shot Recognition | VIDEO

Efficacy of N-Acetyl Cysteine in the Prevention of Post-Operative Atrial Fibrillation and Acute Kidney Injury Post-Cardiac Surgery: A Systematic Review

Using Artificial Intelligence for Automatic Detection of the Pulmonary Artery During Robotic Right Lower Lobectomies

Does Machine Learning Replicate the Uncanny Valley? An Example using FaceNet

Physion++: Evaluating Physical Scene Understanding with Objects Consisting of Different Physical Attributes in Humans and Machines

Evaluating Visual Number Discrimination in Deep Neural Networks

A Benchmark for Modeling Violation-of-Expectation in Physical Reasoning Across Event Categories

Improving neural saliency prediction with a cognitive model of human visual attention

SRoUDA: Meta Self-training for Robust Unsupervised Domain Adaptation

Adaptive Dynamic Filtering Network for Image Denoising

Low-Light Image Enhancement Network Based on Multi-scale Feature Complementation

Stay up to date with the latest Underline news!

PRESENTATIONS

CONFERENCES

COMPANY

RESOURCES