Google is bringing a new live view feature to its Gemini apps on iOS and Android, allowing users to live stream their surroundings to Gemini to get feedback on what they're seeing. Integration is rolling out starting today.
In a demo, Gemini identified objects that the user was looking at, corrected wrong assumptions about what an object was, and provided context by answering questions. Gemini's live view can identify objects in real-time, provide assistance with DIY or home improvement projects, help organize spaces, assist with shopping, and more.
You can also share your iPhone or iPad screen with Gemini to get feedback on something that you see online.
Gemini's Live option could be helpful for those who are blind or who have low vision, as it can provide a live vocal feed of a person's surroundings. Live is built on Project Astra, which Google previewed at I/O last year, and it previously rolled out to Pixel devices.
This summer, Google also plans to add personal context to Gemini for features like Gmail's smart reply. Gemini will be able to use relevant information across Google apps to provide a more personalized experience. For example, the AI reply feature in Gmail can sound like you, rather than having a generic voice. It is able to scan past emails to match tone, style, and word choice, plus it can look up notes, documents from Google Drive, and more to add context to emails.
