MelodyFlow

Melody Flow is a music generation and editing model developed by Meta's Reality Labs XRTech Core AI team. Based on a diffusion transformer architecture, Melody Flow can generate and edit high-fidelity stereo music using simple text prompts. It is designed to produce music samples with rich quality and flexible durations, leveraging a 48 kHz stereo variational autoencoder to avoid the information loss typical of other models.

This model supports both text-guided music generation and music editing, allowing users to generate original compositions or edit existing music samples. Melody Flow introduces an innovative latent inversion method, which significantly improves the model’s ability to edit music in a zero-shot, test-time setting, outperforming previous techniques like denoising diffusion models.

Key Features:

• High-Fidelity Music Generation: Generates diverse, high-quality stereo samples from text prompts.

• Text-Guided Editing: Allows users to edit existing music samples based on text descriptions.

• Advanced Diffusion Architecture: Trained with flow-matching objectives, ensuring efficient and accurate music generation.

• Stereo Sound at 48 kHz: Ensures music quality with continuous latent representations, eliminating data loss.

• Open-Source: Fully open-source, with code available under MIT and model weights under CC-BY-NC 4.0 license.

Intended Use: Melody Flow is ideal for researchers and developers interested in AI-based music generation and editing, and provides a platform for further exploration of generative audio models. It can generate and edit instrumental music with text descriptions, though it is limited in generating realistic vocals. Its use should focus on non-commercial applications, with attention to its biases and limitations in representing all music cultures equally.

Limitations:

• No Vocal Generation: Melody Flow does not generate realistic vocals and performs best with instrumental tracks.

• Language Limitations: Optimized for English descriptions, and may not perform as well with prompts in other languages.

• Biases in Music Genres: The model may not equally represent all musical genres or cultures due to the nature of its training data.

Melody Flow offers an exciting tool for AI-driven music creation, combining text-based interaction with music composition and editing, pushing forward the boundaries of creative possibilities in sound design and AI music.