‘Hey Copilot’: Microsoft gives Windows 11 a voice, vision and action with new AI features

As Microsoft bids adieu to Windows 10 and the wait for Windows 12 reached its pinnacle, the Redmond based tech giant has given its current operating system a major AI push. Micosoft is more deeply integrating its Copilot AI assistant at the center of Windows experience that the company claims that every Windows 11 PC will become an AI PC.

The new voice feature for Copilot is opt-in, meaning users would need to enable it in the settings of the Copilot app before they start talking to the AI assistant. After summoning Copilot with the hotword, users will see a Copilot microphone appear on their screen along with a chime that indicates that Copilot is listening.

The unique feature that Microsoft is bringing here is the the use of “Goodbye” hotword to end a conversation. Users can also end the conversation by tapping on the ‘X’ icon or Copilot will terminate it on its own if their is no interaction for a few seconds.

Microsoft says using voice with Copilot leads to users engaging twice as much as when use text while leading to deeper engagement with the chatbot.

Copilo now has ‘vision’:

Apart from adding voice to Copilot, Microsoft is also giving it ‘vision’ which will be available in all markets where the AI chatbot is already present. With the new Coplilot Vision feature, users will be able to share their desktop or applictions and Copilot can help analyze content, provide insights and answer their questions.

Users can also go beyond just asking questions about the task on their screen and ask the AI assistant to ‘show’ them how to perform a specific task. Copilot can also now give tips while playing games, viewing photos, or reviewing travel intenary.

Copilot will also be able to see beyond what's available on the users' screen when sharing Word, Excel or PowerPoint files. For instance when users share a PowerPoint deck with Copilot, it can analyze the entire presentation witheeding to flip through each slide.

Microsoft says it is also planning to bring text in text out feature for Windows Insiders where users will be able to converse with Copilot Vision using text and get answers the same way.

Copilot in Windows taskbar:

Microsoft is bringing a new Ask Copilot option on the taskbar for Windows Insiders. The new option can help users find apps, files and settings right from the taskbar while having one click access to Copilot Voice and Vision.

“This experience leverages existing Windows APIs to return your apps, files and settings, like the Windows Search experience and does not grant Copilot access to your content.” said Yusuf Mehdi – Executive Vice President, Consumer Chief Marketing Officer at Microsoft.

“While this is happening, you can choose to focus on other tasks. At any time, you can take over the task or check in on the progress of the action, including reviewing what actions have been taken” Mehdi says

The new feature can help users accomplish tasks like resizing photos, extracting information from PDF and performing bulk edits on documents.