Audio-To-Symbolic Arrangement Via Cross-Modal Music Representation Learning

Could we automatically derive the score of a piano accompaniment based on the audio of a pop song? This is the audio-to-symbolic arrangement problem we tackle in this paper. A good arrangement model should not only consider the audio content but also have prior knowledge of piano composition (so tha...

Full description

Saved in:
Bibliographic Details
Main Authors: Wang, Ziyu, Xu, Dejing, Xia, Gus, Shan, Ying
Format: Conference Proceeding
Language:English
Subjects:
automatic arrangement
Conferences
cross-modal representation
Data mining
disentangled representation
Gaussian distribution
Representation learning
Signal processing
Task analysis
Training
Online Access:Request full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:Could we automatically derive the score of a piano accompaniment based on the audio of a pop song? This is the audio-to-symbolic arrangement problem we tackle in this paper. A good arrangement model should not only consider the audio content but also have prior knowledge of piano composition (so that the generation "sounds like" the audio and meanwhile maintains musicality). To this end, we contribute a cross-modal representation-learning model, which 1) extracts chord and melodic information from the audio, and 2) learns texture representation from both audio and a corrupted ground truth arrangement. We further introduce a tailored training strategy that gradually shifts the source of texture information from corrupted score to audio. In the end, the score-based texture posterior is reduced to a standard normal distribution, and only audio is needed for inference. Experiments show that our model captures major audio information and outperforms baselines in generation quality. 1
ISSN:2379-190X
DOI:10.1109/ICASSP43922.2022.9747884