Ghost in the Encoder
Decodable Artist Identity Representations in Lyrics-to-Song Generation
ISMIR 2026
Text-to-song generation models can be prompted to imitate specific artists or regurgitate entire songs from their training data. Although these phenomena have been documented behaviorally on small datasets, little is known about the internal representations that may give rise to them. Prior interpretability work on generative audio has focused on locating semantic concepts such as genre or time signature within model activations. In this work, we show that a trained model can be probed for linearly decodable representations of artist identity from song lyrics alone, without any additional identifiers. Through a controlled case study of ACE-Step 1.5 spanning 2,000 songs across 100 artists, we demonstrate that the artist associated with a given set of lyrics can be identified within the model's internal activations, and that this conditioning signal propagates from the lyric encoder to the diffusion backbone during inference. These findings indicate that lyrics constitute an artist-level conditioning channel not addressed by prompt-side replication safeguards. More broadly, our work highlights how latent space analysis can be used to audit what generative music models have implicitly learned from their training data.
For each song, we show the original recording alongside audio generated by ACE-Step 1.5 under no-CoT and with-CoT conditioning. We present 2 songs ranked by CLEWS from each genre in our dataset.
Original recordings remain the property of their respective rights holders.
The website template is taken from Custom Diffusion (which was built on DreamFusion's project page).