Continual Adaptation of Semantic Segmentation using Complementary 2D-3D Data Representations

Semantic segmentation networks are usually pre-trained once and not updated during deployment. As a consequence, misclassifications commonly occur if the distribution of the training data deviates from the one encountered during the robot's operation. We propose to mitigate this problem by adap...

Full description

Saved in:
Bibliographic Details
Published in:IEEE robotics and automation letters 2022, Vol.7 (4), p.1-8
Main Authors: Frey, Jonas, Blum, Hermann, Milano, Francesco, Siegwart, Roland, Cadena, Cesar
Format: Article
Language:English
Subjects:
Adaptation
Cameras
Continual Learning
Datasets
Deep Learning for Visual Perception
Neural networks
Representations
Robots
Semantic Scene Understanding
Semantic segmentation
Semantics
Task analysis
Three-dimensional displays
Training
Citations: Items that this one cites
Items that cite this one
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:Semantic segmentation networks are usually pre-trained once and not updated during deployment. As a consequence, misclassifications commonly occur if the distribution of the training data deviates from the one encountered during the robot's operation. We propose to mitigate this problem by adapting the neural network to the robot's environment during deployment, without any need for external supervision. Leveraging complementary data representations, we generate a supervision signal, by probabilistically accumulating consecutive 2D semantic predictions in a volumetric 3D map. We then train the network on renderings of the accumulated semantic map, effectively resolving ambiguities and enforcing multi-view consistency through the 3D representation. In contrast to scene adaptation methods, we aim to retain the previously-learned knowledge, and therefore employ a continual learning experience replay strategy to adapt the network. Through extensive experimental evaluation, we show successful adaptation to real-world indoor scenes both on the ScanNet dataset and on in-house data recorded with an RGB-D sensor. Our method increases the segmentation accuracy on average by 9.9% compared to the fixed pre-trained neural network, while retaining knowledge from the pre-training dataset.
ISSN:2377-3766
2377-3766
DOI:10.1109/LRA.2022.3203812