DM-Fusion: Deep Model-Driven Network for Heterogeneous Image Fusion

Heterogeneous image fusion (HIF) is an enhancement technique for highlighting the discriminative information and textural detail from heterogeneous source images. Although various deep neural network-based HIF methods have been proposed, the most widely used single data-driven manner of the convolut...

Full description

Saved in:
Bibliographic Details
Published in:IEEE transaction on neural networks and learning systems 2024-07, Vol.35 (7), p.10071-10085
Main Authors: Xu, Guoxia, He, Chunming, Wang, Hao, Zhu, Hu, Ding, Weiping
Format: Article
Language:English
Subjects:
Alternating direction method of multiplier (ADMM)
Artificial neural networks
Computer vision
Data integration
Deep learning
deep model-driven
Feature extraction
Image enhancement
Image fusion
Image reconstruction
Learning systems
Machine learning
Neural networks
Objective function
Source code
Task analysis
task-driven
Training
Citations: Items that this one cites
Items that cite this one
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:Heterogeneous image fusion (HIF) is an enhancement technique for highlighting the discriminative information and textural detail from heterogeneous source images. Although various deep neural network-based HIF methods have been proposed, the most widely used single data-driven manner of the convolutional neural network always fails to give a guaranteed theoretical architecture and optimal convergence for the HIF problem. In this article, a deep model-driven neural network is designed for this HIF problem, which adaptively integrates the merits of model-based techniques for interpretability and deep learning-based methods for generalizability. Unlike the general network architecture as a black box, the proposed objective function is tailored to several domain knowledge network modules to model the compact and explainable deep model-driven HIF network termed DM-fusion. The proposed deep model-driven neural network shows the feasibility and effectiveness of three parts, the specific HIF model, an iterative parameter learning scheme, and data-driven network architecture. Furthermore, the task-driven loss function strategy is proposed to achieve feature enhancement and preservation. Numerous experiments on four fusion tasks and downstream applications illustrate the advancement of DM-fusion compared with the state-of-the-art (SOTA) methods both in fusion quality and efficiency. The source code will be available soon.
ISSN:2162-237X
2162-2388
2162-2388
DOI:10.1109/TNNLS.2023.3238511