qlty: Handling large tensors in scientific imaging deep-learning workflows

In scientific imaging, deep learning has become a pivotal tool for image analytics. However, handling large volumetric datasets, which often exceed the memory capacity of standard GPUs, require special attention when subjected to deep learning efforts. This paper introduces qlty, a toolkit designed...

Full description

Saved in:
Bibliographic Details
Published in:Software impacts 2024-09, Vol.21 (C), p.100696, Article 100696
Main Author: Zwart, Petrus H.
Format: Article
Language:English
Subjects:
Deep learning
Denoising
Machine learning
Segmentation
Citations: Items that this one cites
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:In scientific imaging, deep learning has become a pivotal tool for image analytics. However, handling large volumetric datasets, which often exceed the memory capacity of standard GPUs, require special attention when subjected to deep learning efforts. This paper introduces qlty, a toolkit designed to address these challenges through tensor management techniques. qlty offers robust methods for subsampling, cleaning, and stitching of large-scale spatial data, enabling effective training and inference even in resource-limited environments. •Efficient tensor management for large-scale spatial data on limited hardware.•Out-of-core processing for handling datasets exceeding available GPU memory.•Successful applications in X-ray diffraction and tomography segmentation.
ISSN:2665-9638
2665-9638
DOI:10.1016/j.simpa.2024.100696