OpenAI has introduced some exciting new features to its speech-to-text large language model, gpt-realtime. These updates are designed to help businesses create smarter, more autonomous voice assistants that can do more than just listen and respond. With support for remote tool access, phone system integration, and improved understanding, companies can now build voice agents that are more versatile and capable.

Connecting Voice Agents to External Tools with MCP Support

One of the big updates is the addition of remote model context protocol (MCP) server support. This feature is now generally available through OpenAI’s API. It allows developers to connect their voice-based agents to external tools and capabilities hosted on different servers or on the internet. Charlie Dai, VP at Forrester, explains that this makes it easier to extend what a voice agent can do without having to build everything from scratch.

To enable this, companies can include the URL of a remote MCP server in their API session. Once connected, the API automatically manages calling these external tools whenever needed. This means developers don’t have to manually set up complex integrations, saving time and making it easier to add new features to voice agents. For example, a customer service bot could access a remote database for personalized information or connect to a weather service to give real-time updates.

SIP Support Brings Voice Agents Closer to Phone Systems

Another new feature is support for SIP, which stands for Session Initiation Protocol. SIP is a standard used for starting and managing voice calls over IP networks. By supporting SIP, OpenAI makes it possible for voice agents to connect directly with existing phone systems like PBX, which many businesses rely on for their internal and customer calls.

Dai notes that this can open up many use cases. For instance, companies could automate call handling, schedule appointments, or support multiple languages in customer service centers. This integration can streamline communication workflows, reduce wait times, and improve overall service quality.

Enhanced Multimodal Capabilities and Smarter Conversations

OpenAI has also added the ability for the model to interpret images in sessions. Now, users can upload pictures, screenshots, or other visuals alongside voice or text inputs. This allows the AI to understand and respond based on what’s shown in the images. For example, a user could ask, “What do you see?” or “Can you read this text?” and the model will analyze the picture to provide an answer.

Experts see this as a major step forward in multimodal AI, where models can handle multiple types of data simultaneously. Dai points out that competitors like Google with Project Astra are also working on similar capabilities. This makes the technology more versatile, especially for enterprises needing visual recognition alongside voice or text.

In addition to visual support, OpenAI has improved the core model’s understanding and memory. The new gpt-realtime can follow more complex instructions, call tools more accurately, and produce more natural and expressive speech. These enhancements are vital for real-time applications like medical transcription, virtual booking assistants, and customer support in banking, insurance, and telecom sectors.

Finally, enterprises using the API can choose from two new voice options, Cedar and Marin. Microsoft, OpenAI’s largest investor, has also announced two new text-to-speech models aimed at expanding enterprise use cases. Overall, these updates make the API more powerful and flexible, giving businesses the tools to build advanced, natural-sounding voice agents that can handle a wide variety of tasks seamlessly.

Inspired by

• https://www.infoworld.com/article/4048375/openai-adds-mcp-and-sip-support-to-gpt-realtime-for-smarter-voice-based-agents.html

Sources

• gmpg.org
• forrester.com
• podigee.com
• podigee.com