Meta, the tech giant formerly known as Facebook, has revealed its latest breakthrough in the realm of artificial intelligence: Voicebox AI. This groundbreaking tool is designed to generate spoken speech based on textual cues, heralding a new era for voice assistants and speech synthesis.
Voicebox AI operates on a similar generative model foundation as ChatGPT and DALL-E but diverges in its ability to generate spoken language instead of text or images. This innovative system has been trained on a colossal dataset encompassing 50,000 hours of unfiltered audio content. This corpus comprises transcripts of publicly available audiobooks, recorded in a range of languages, including English, French, Spanish, German, Polish, and Portuguese.
The diversity of this dataset empowers Voicebox AI to produce what Meta refers to as “more conversational speech.” This adaptability is invaluable, as it enables the system to generate human-like spoken language irrespective of the languages spoken by the conversational participants.
Meta asserts that the synthetic speech generated by Voicebox is nearly on par with real speech, particularly in terms of speech recognition models. The company claims that these models trained on Voicebox-generated synthetic speech exhibit commendable performance, nearly matching that of models trained on genuine human speech. Notably, Voicebox surpasses Microsoft’s VALL-E in text-to-language conversion regarding both intelligibility (5.9% word error rate vs. 1.9%) and audio similarity (0.580% vs. 0.681%). Moreover, it accomplishes this feat at a remarkable speed, boasting a 20-fold improvement in efficiency.
Voicebox AI offers an array of functionalities that extend beyond voice generation. It has the capability to edit audio, eliminate background noise, and rectify mispronunciations. The system’s versatility empowers users to pinpoint distorted audio segments caused by extraneous noises (e.g., a dog barking), trim them, and instruct the model to rectify these segments.
The training approach behind Voicebox AI, referred to as Flow Matching, represents a novel method for constructing speech synthesis capabilities from scratch. However, as of now, Meta has not released the Voicebox program or its source code to the public. The company cites concerns about the potential misuse of the technology as the reason behind this decision.
Researchers envision a multitude of applications for Voicebox AI in various domains. Its potential extends to prosthetics for individuals with impaired vocal cords, enhancing the capabilities of gaming non-playable characters (NPCs), and optimizing digital assistants.
It’s worth noting that while Meta has made strides in making certain AI models open-source, such as its LLaMA AI language model, the company has faced challenges with maintaining control over its proprietary technologies. In January, Meta released LLaMA as an open-source package for the AI community, only to witness it being leaked and distributed through unauthorized channels.
Meta’s dedication to advancing AI technologies continues to drive innovation across a broad spectrum, from language models to image segmentation and animation. As Voicebox AI joins its portfolio of cutting-edge solutions, Meta remains at the forefront of the AI landscape, shaping the future of artificial intelligence in meaningful and transformative ways.