The Unbearable Weight of Massive Latent Space