HKUST Releases Audio-Omni, a Unified Audio Model

Researchers from the Hong Kong University of Science and Technology (HKUST) have released Audio-Omni, a new model that aims to unify a wide range of audio generation tasks. Unlike specialized models designed for a single purpose, Audio-Omni is an "any-to-any" system, capable of handling diverse audio inputs and outputs.

The model is built on a diffusion-based architecture, which allows it to generate high-fidelity audio by progressively refining noise into a coherent signal. This single framework is designed to understand and process various audio modalities, from human speech to complex musical compositions and environmental sounds, treating them all as interchangeable data types.

A Generalist Approach to Audio

Audio-Omni's versatility allows it to perform a broad set of tasks that would typically require multiple different models. As detailed on its Hugging Face repository, its key capabilities include:

• Conversion: Transforming speech to music, music to speech, or one style of music to another.
• Generation: Creating music or speech from text prompts.
• Editing: Modifying existing audio, such as separating stems or in-painting missing sections.
• Continuation: Extending an existing audio clip in a consistent style.

This release represents another step toward building more generalized foundation models for audio. By consolidating disparate tasks into one model, Audio-Omni points to a future where audio generation is less fragmented and more universally accessible. The model is available for research and non-commercial use under a CC BY-NC 4.0 license.