TensorLightning: A Traffic-Efficient Distributed Deep Learning on Commodity Spark Clusters

With the recent success of deep learning, the amount of data and computation continues to grow daily. Hence a distributed deep learning system that shares the training workload has been researched extensively. Although a scale-out distributed environment using commodity servers is widely used, not o...

Full description

Saved in:
Bibliographic Details
Published in:IEEE access 2018-01, Vol.6, p.27671-27680
Main Authors: Lee, Seil, Kim, Hanjoo, Park, Jaehong, Jang, Jaehee, Jeong, Chang-Sung, Yoon, Sungroh
Format: Article
Language:English
Subjects:
Algorithms
Apache Spark
Artificial neural networks
Clustering algorithms
Clusters
Commodities
commodity servers
Communications traffic
Computational modeling
Deep learning
distributed system
Government
Libraries
Machine learning
Neural networks
Parameters
Recurrent neural networks
Redesign
Servers
Sparks
TensorLightning
Training
Citations: Items that this one cites
Items that cite this one
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:With the recent success of deep learning, the amount of data and computation continues to grow daily. Hence a distributed deep learning system that shares the training workload has been researched extensively. Although a scale-out distributed environment using commodity servers is widely used, not only is there a limit due to synchronous operation and communication traffic but also combining deep neural network (DNN) training with existing clusters often demands additional hardware and migration between different cluster frameworks or libraries, which is highly inefficient. Therefore, we propose TensorLightning which integrates the widely used data pipeline of Apache Spark with powerful deep learning libraries, Caffe and TensorFlow. TensorLightning embraces a brand-new parameter aggregation algorithm and parallel asynchronous parameter managing schemes to relieve communication discrepancies and overhead. We redesign the elastic averaging stochastic gradient descent algorithm with pruned and sparse form parameters. Our approach provides the fast and flexible DNN training with high accessibility. We evaluated our proposed framework with convolutional neural network and recurrent neural network models; the framework reduces network traffic by 67% with faster convergence.
ISSN:2169-3536
2169-3536
DOI:10.1109/ACCESS.2018.2842103