Building a Human-like TTS | Lily Clifford (Founder & CEO, Rime Labs)

In The Future of Voice AI series of interviews, I ask three questions to my guests:

- What problems do you currently see in Enterprise Voice AI?
- How does your company solve these problems?
- What solutions do you envision in the next 5 years?

This episode’s guest is Lily Clifford, Founder & CEO at Rime Labs.

Lily is the CEO and co-founder of Rime Labs, which develops enterprise-ready speech synthesis technologies for compelling voice experiences.

Rime Labs focuses on developing speech synthesis technology. Rime’s platform, named Mist, is powered by the first Text-to-speech model capable of reproducing genre-specific characteristics of voice, delivering them through an API with sub-200ms latencies. Rime offers over 200 distinct voices for use in various applications and their solutions are designed to meet the needs of enterprise customers with lightning-fast response times.

Summary

• Innovations and Challenges in Text-to-Speech Technology: Lily discusses the evolution and current state of TTS tech. Starting from the basics of how TTS systems traditionally worked using extensive datasets to capture the nuances of a single voice, Lily highlights the shift towards more sophisticated, neural network-based models. These advanced models can handle multiple languages and speakers, allowing for the creation of new, fictitious voices by interpolating within a vast, multi-dimensional space of linguistic variation.

• Rime’s Unique Approach: Unlike traditional TTS models that often rely on audiobook data, Rime collects its data through real conversations, aiming to produce more naturalistic and relatable voices. This approach stems from Lily's background in sociolinguistics, emphasizing the importance of capturing unselfconscious speech for more compelling and human-like voice synthesis.

• Applications and Future of TTS: The interview covers the significant impact of TTS in areas such as enterprise calling and customer support, where the quality and relatability of the voice can greatly enhance user experience. Lily envisions a future where TTS technology operates independently of text, leveraging direct speech-to-speech interactions to overcome the limitations of current speech recognition systems. This advancement could lead to more accurate and efficient communication between humans and AI systems, potentially transforming the way we interact with technology.

• Market Growth: Lily notes the rapid expansion of the TTS market, driven by the demand for synthetic media and the increasing quality of voice synthesis. She points out emerging use cases like dubbing and localization, underscoring the potential of TTS to revolutionize content creation and accessibility. The discussion also touches on the importance of emotional and human-like voices in making automated systems more engaging and effective.

• Challenges and Future: Despite significant progress, TTS is far from being a solved problem. The conversation delves into the ongoing challenges in the field, such as data quality, model size, and the trade-offs between voice quality and system latency. Looking ahead, Lily emphasizes the need for more sophisticated models that can fully replicate human conversational dynamics without relying on text, suggesting a future where speech-to-speech models could offer a more intuitive and seamless interface for human-computer interaction.

Takeaways

• Evolution of TTS Systems: The transition from simple, parametric models to complex, neural network-based systems that can handle multiple languages and speakers, enabling the creation of fictitious voices through the interpolation of vast linguistic datasets.

• Importance of Human-Like Quality: The critical role of voice quality and human likeness in enhancing user engagement, particularly in applications like enterprise calling and customer support, where the voice's compelling nature can significantly impact the outcome.

• Emerging Use Cases: The rapid expansion of the TTS market, driven by the demand for synthetic media, dubbing, localization, and more natural, engaging customer interaction platforms. TTS technology is considered foundational in the evolution of generative AI, with potential to revolutionize various sectors.

• Future Directions: The conversation points towards a future where TTS may operate independently of text, enabling more accurate and efficient speech-to-speech interactions and potentially obviating the need for text-based natural language understanding (NLU) systems.