Italy

Human activity recognition plays a central role in the development of intelligent systems for video surveillance, public security, health care and home monitoring, where detection and recognition of activities can improve the quality of life and security of humans. Typically, automated, intuitive and real-time systems are required to recognize human activities and identify accurately unusual behaviors in order to prevent dangerous situations. In this work, we explore the combination of three modalities (RGB, depth and skeleton data) to design a robust multi-modal framework for vision-based human activity recognition. Especially, spatial information, body shape/posture and temporal evolution of actions are highlighted using illustrative representations obtained from a combination of dynamic RGB images, dynamic depth images and skeleton data representations. Therefore, each video is represented with three images that summarize the ongoing action. Our framework takes advantage of transfer learning from pre-trained models to extract significant features from these newly created images. Next, we fuse extracted features using Canonical Correlation Analysis and train a Long Short-Term Memory network to classify actions from visual descriptive images. Experimental results demonstrated the reliability of our feature-fusion framework that allows us to capture highly significant features and enables us to achieve the state-of-the-art performance on the public UTD-MHAD and NTU RGB+D datasets.

ICPR 2020

Vision-Based Multi-Modal Framework for Action Recognition

transfert learning

canonical correlation analysis

multi modal data fusion

human activity recognition

ICPR 2020 is the premier international conference on Pattern Recognition, gathering leading researchers from around the world to present their latest achievements in Pattern Analysis and Recognition, applied Artificial Intelligence and Machine Learning. The topic of the year is " Putting Artificial intelligence to work on Patterns". 
For the first time, ICPR has employed a two-round review process similar to journal submissions, with new papers allowed to be submitted in either the first or second round and papers submitted in the first round and not accepted allowed to be revised and re-submitted for second round review. 

In this edition of the **ICPR 2020** Conference, there are 1407 Oral/Poster presentations organized in five Tracks: 
- Artificial Intelligence, Machine Learning for Pattern Analysis; 
- Biometrics, Human Analysis and Behavior Understanding; Computer Vision, Robotics and Intelligent Systems; 
- Document and Media Analysis; and 
- Image and Signal Processing. 

The **ICPR 2020** Main Conference program also includes 7 Keynotes from prominent world-renowned scientists, 8 Tutorials, 28 Demos, 8 Challenges, 2 Panels, as well as an Industry/Sponsor Exhibition. In addition, 41 Workshops on a range of important current and emerging research topics complete the ICPR 2020 program. 


poster

Most existing works regarding facial demographic estimation are focused on still image datasets, although nowadays the need to analyze video content in real applications is increasing. We propose to tackle gender, age and ethnicity estimation in the context of video scenarios. Our main contribution is to use an attribute-specific quality assessment procedure to select most relevant frames from a video sequence for each of the three demographic modalities. Selected frames are classified with fine-tuned MobileNet models and a final video prediction is obtained with a majority voting strategy. Our validation on three different datasets and our comparison with state-of-the art models, show the effectiveness of the proposed demographic classifiers and the quality pipeline, which allows to reduce both: the number of frames to be classified and the processing time in practical applications; and improves the soft biometrics prediction accuracy.

Attribute-based quality assessment for demographic estimation in face videos

Convolutional neural networks are shown to be vulnerable to changes in the background. The proposed method is an end-to-end method that augments the training set by introducing new backgrounds during the training process. These backgrounds are created by a generative network that is trained as an adversary to the model. A case study is explored based on overhead power line insulators detection using a drone – a training set is prepared from photographs taken inside a
laboratory and then evaluated using photographs that are harder to collect from outside the laboratory. The proposed method improves performance by over 20% for this case study.

Background Invariance by Adversarial Learning




Recognizing human activities from multi-channel time series data collected from wearable sensors has become an important practical application of machine learning. A serious challenge comes from the presence of coherent activities or body movements, such as movements of the head while walking or sitting, since signals representing these movements are mixed and interfere with each other. Basic multi-label classification typically assumes independence within the multiple activities. This is over-simplified and reduces modeling power even when using state-of-the-art deep learning methods. In this paper, we investigate this new problem, which we name ``Coherent Human Activity Recognition (Co-HAR)'', that keeps complete conditional dependency between the multiple labels. Additionally, we treat Co-HAR as a dense labeling problem that classifies each sample on a time step with multiple coherent labels to provide high-fidelity and duration-sensitive support to high-precision applications. To explicitly model conditional dependency, a novel condition-aware deep architecture ``Conditional-UNet'' is developed to allow for multiple dense labeling for Co-HAR. We also contribute a first-of-its-kind Co-HAR dataset for head gesture recognition associated with a user's activity, walking or sitting, to the research community. Extensive experiments on this dataset show that our model outperforms state-of-the-art deep learning methods and achieves up to 92% accuracy on context-based head gesture classification.

Conditional-UNet: A Condition-aware Deep Model for Coherent Human Activity Recognition From Wearables

This article proposes a novel attention-based body-pose encoding for human activity recognition. Most of the existing human activity recognition approaches based on 3D pose data often enrich the input data using additional handcrafted representations such as velocity, super-normal vectors, pairwise relations, and so on. The enriched data complements the 3Dbody joint position data and improves model performance. In this paper, we propose a novel approach that learns enhanced feature representations from a given sequence of 3D body joints.To achieve this encoding, the approach exploits two body pose streams: 1) a spatial stream which encodes the spatial relationship between various body joints at each time point to learn spatial structure involving the spatial distribution of different body joints 2) a temporal stream that learns the temporal variation of individual body joints over the entire sequence duration to present a temporally enhanced representation. Afterwards,these two pose streams are fused with a multi-head attention mechanism. We also capture the contextual information from the RGB video stream using a deep Convolutional NeuralNetwork (CNN) model combined with a multi-head attention and a bidirectional Long Short-Term Memory (LSTM) network.Finally, the RGB video stream is combined with the fused body-pose stream to give a novel end-to-end deep model for effective human activity recognition.

Attention-Driven Body Pose Encoding for Human Activity Recognition


Accurate and robust detection of objects from
monocular images is a fundamental vision task. This paper
describes a novel approach of holistic scene understanding that
can simultaneously achieve multiple tasks of scene reconstruction
and object detection from a single monocular image. Rather than
pursuing an independent solution for each individual task as
most existing work does, we seek a globally optimal solution that
holistically resolves the multiple perception and reasoning tasks
in an effective manner. The approach explores the complementary
properties of multimodal RGB images and depth data to improve
scene perception tasks. It uniquely combines the techniques
of canonical correlation analysis and deep learning to learn
the most correlated features to maximize the modal crosscorrelation
for improving performance and robustness of object
detection in complex environments. Extensive experiments have
been conducted to evaluate and demonstrate the performances
of the proposed approach.

Object Detection on Monocular Images with Two-Dimensional Canonical Correlation Analysis


Representing brain activities by networks is very crucial to understand various cognitive states. This study proposes a novel method to estimate static and dynamic brain networks using Kulback-Leibler divergence. The suggested brain networks are based on the probability distributions of voxel intensity values measured by functional Magnetic Resonance Images (fMRI) recorded while the subjects perform a predefined cognitive task, called complex problem solving. We investigate the validity of the estimated brain networks by modeling and analyzing the different phases of complex problem solving process of human brain, namely planning and execution phases. The suggested computational network model is tested by a classification schema using Support Vector Machines. We observe that the network models can successfully discriminate the planning and execution phases of complex problem solving process with more than 90% accuracy, when the estimated dynamic networks, extracted from the fMRI data, are classified by Support Vector Machines.

Estimating Static and Dynamic Brain Networks by Kulback-Leibler Divergence from fMRI Data

This work describes an intelligent price suggestion system for online second-hand listings. In contrast to conventional pricing strategies which are employed to a large number of identical products, or to non-identical but similar products such as homes on Airbnb, the proposed system provides price suggestions for online second-hand items which are non-identical and fall into numerous different categories. Moreover, simplifying the item listing process for users is taken into consideration when designing the price suggestion system. Specifically, we design a truncate loss to train a vision-based price suggestion module which mainly takes some vision-based features as input to first classify whether an uploaded item image is qualified for price suggestion, and then offer price suggestions for items with qualified images. For the items with unqualified images, we encourage users to input some text descriptions of the items, and with the text descriptions, we design a multimodal item retrieval module to offer price suggestions. Extensive experiments demonstrate the effectiveness of the proposed system.

Price Suggestion for Online Second-hand Items


This paper provides a comprehensive comparative study of traditional, hybrid, and deep learning (DL) methods for detecting marine species in echograms. Acoustic backscatter data obtained from multi-frequency echosounders is visualized as echograms and typically interpreted by marine biologists via manual or semi-automatic methods, which are time-consuming. Challenges related to automatic echogram interpretation are the variable size and acoustic properties of the biological targets (marine life), along with significant inter-class similarities. Our
study explores and compares three types of approaches that cover the entire range of machine learning methods. Based on our experimental results, we conclude that an end-to-end DL-based framework, that can be readily scaled to accommodate new species, is overall preferable to other learning approaches for echogram interpretation, even when only a limited number of annotated training samples is available.

Detecting Marine Species in Echograms via Traditional, Hybrid, and Deep Learning Frameworks


Layout analysis of historical handwritten documents
is a key pre-processing step in document image analysis that, by
segmenting the image into its homogeneous regions facilitates
subsequent procedures such as optical character recognition and
automatic transcription. Learning-based approaches have shown
promising performances in layout analysis, however, the majority
of them require tedious pixel-wise labelled training data to
achieve generalisation capabilities, this limitation preventing their
application due to the lack of large labelled datasets. This paper
proposes a novel unsupervised representation learning method
for documents’ layout analysis that reduces the need for labelled
data: a sparse autoencoder is first trained in an unsupervised
the manner on a historical text document’s image; representation
of image patches, computed by the sparse encoder, is then used
to classify pixels into various region categories of the document
using a feed-forward neural network. A new training method,
inspired by the ISTA algorithm, is also introduced here to train
the sparse encoder. Experimental results on DIVA-HisDB dataset
demonstrate that the proposed method outperforms previous
approaches based on unsupervised representation learning while
achieving performances comparable to the state-of-the-art fully
supervised methods.

Ancient Document Layout Analysis: Autoencoders meet Sparse Coding


In recent years, deep neural networks have achieved
state-of-the-art performance in a variety of recognition and
segmentation tasks in medical imaging including brain tumor
segmentation. We investigate that segmenting a brain tumor is
facing to the imbalanced data problem where the number of pixels
belonging to the background class (non tumor pixel) is much
larger than the number of pixels belonging to the foreground
class (tumor pixel). To address this problem, we propose a multitask
network which is formed as a cascaded structure. Our model
consists of two targets, i.e., (i) effectively differentiate the brain
tumor regions and (ii) estimate the brain tumor mask. The first
objective is performed by our proposed contextual brain tumor
detection network, which plays a role of an attention gate and
focuses on the region around brain tumor only while ignoring the
far neighbor background which is less correlated to the tumor.
Different from other existing object detection networks which
process every pixel, our contextual brain tumor detection network
only processes contextual regions around ground-truth instances
and this strategy aims at producing meaningful regions proposals.
The second objective is built upon a 3D atrous residual network
and under an encode-decode network in order to effectively segment
both large and small objects (brain tumor). Our 3D atrous
residual network is designed with a skip connection to enables the
gradient from the deep layers to be directly propagated to shallow
layers, thus, features of different depths are preserved and used
for refining each other. In order to incorporate larger contextual
information from volume MRI data, our network utilizes the
3D atrous convolution with various kernel sizes, which enlarges
the receptive field of filters. Our proposed network has been
evaluated on various datasets including BRATS2015, BRATS2017
and BRATS2018 datasets with both validation set and testing
set. Our performance has been benchmarked by both regionbased
metrics and surface-based metrics. We also have conducted
comparisons against state-of-the-art approaches

Premium content

Vision-Based Multi-Modal Framework for Action Recognition

Downloads

Next from ICPR 2020

Attribute-based quality assessment for demographic estimation in face videos

Similar lecture

Attention-Driven Body Pose Encoding for Human Activity Recognition

Stay up to date with the latest Underline news!

PRESENTATIONS

CONFERENCES

COMPANY

RESOURCES