Confirmed AI Features in Windows 11 Version 24H2

• Microsoft is set to introduce a variety of new on-device AI functionalities in Windows 11 24H2.
• Highlighted features include Windows Recall, Windows Studio Effects, Live Captions, Auto Super Resolution, Voice Clarity, Voice Focus, Click to Do, Cocreator in the Paint app, and Restyle Image in the Photos app.
• Additionally, there will be enhancements made to Copilot for Windows 11.

Since October 1, Windows 11 24H2 (2024 Update) has been rolled out for eligible devices, though it initially lacks the anticipated AI features. Microsoft is dedicating time to ensure these features are fully prepared for deployment. Nevertheless, details of the initial AI offerings have been revealed, beginning with the integration of the Windows Copilot Runtime.

This revamped version of the OS will introduce several exclusive AI features designed for Copilot+ PCs, including Windows Recall, Windows Studio Effects, Live Captions (including real-time translations), Auto Super Resolution, Voice Clarity, Click to Do, Voice Focus, Cocreator in the Paint app, and Restyle Image within the Photos app.

These innovations stem from the new “Windows Copilot Runtime,” a system component that integrates over 40 AI models operating directly on the device, providing the necessary infrastructure for their functions.

This also facilitates developers to embed AI capabilities into their applications through the new APIs available in the Windows Copilot Library.

AI Features in Windows 11 version 24H2

Here’s a summary of the AI features anticipated for Windows in 2024 and beyond.

Windows Recall

Windows Recall functions as a background photographic memory, automatically capturing snapshots of your activities on the computer. Utilizing several AI models running locally, this feature makes all content—whether text or images—searchable using natural language.

This innovative Recall feature is exclusively available on Copilot+ PCs, which must be equipped with compatible hardware, including an NPU-enabled processor, 16GB of RAM, and 256GB of SSD storage.

During the initial out-of-box experience (OOBE), users can activate or deactivate this feature, with additional control available through the Settings app or Group Policy.

Windows Studio Effects

Windows Studio Effects encompasses a suite of functionalities that leverage AI to enhance video call and audio quality through Neural Processing Units (NPUs).

Currently, there are five main effects, including Automatic Framing, Background Blur, Eye Contact, Voice Focus, and Creative Filters.

• Automatic Framing: This keeps you centered in the camera view even while moving.
• Background Blur: Blurs your background during video calls for added privacy or to conceal clutter, featuring both “Standard Blur” and “Portrait Blur.”
• Eye Contact: Adjusts your appearance to ensure eye contact is perceived, with “Standard” and “Teleprompter” modes available.
• Voice Focus: Minimizes background noise for clearer audio during calls. There are two variants, one utilizing NPU and another relying on software alone.

While Studio Effects have existed in Windows 11 for some time, Microsoft intends to broaden their availability with version 24H2 specifically for Copilot+ PCs.

Live Captions with Translation

The Live Captions feature, which has been present in Windows 11, is getting an upgrade to include live translations. This tool is designed to aid everyone, particularly those with hearing impairments, in understanding audio content via automatic transcription.

The upgraded feature translates audio from video or voice calls into your native language in real-time, regardless of the source—whether from YouTube or another platform. It is capable of translating more than 40 languages into English without needing an internet connection, provided it is supported by an NPU.

It’s essential to note that this feature has shifted from simple transcription to real-time audio translation.

Voice Clarity

The Voice Clarity feature uses AI to eliminate background noise, cancel echoes, and reduce reverberation during video or voice calls as well as audio recordings.

This functionality also benefits online gaming by providing clearer communication. Applications can enable “Deep Noise Suppression”for optimal audio quality. Voice Clarity operates automatically on supported apps with compatible hardware equipped with an NPU, without the need for manual activation.

Click to Do

If you’re familiar with “Circle to Search”on Android or “Google Lens”in Chrome, then “Click to Do”will feel similar. It employs AI to recognize and interpret screen content, offering various actionable tasks.

To use it, press and hold the “Windows”key while clicking on the screen to open an overlay menu presenting different action options. Also, a search box is available at the top, integrating with Windows Recall.

Enhanced Windows Search

While not officially named as such, the Windows Search feature will be revamped with AI capabilities. This enhancement allows users to perform natural language searches for apps, files, and settings directly in File Explorer, the Settings app, and the Taskbar, rather than merely relying on keywords.

Auto Super Resolution

Auto Super Resolution (SR) utilizes AI technology to enhance game resolution and visual quality, contributing to a smoother and more efficient gameplay experience while maximizing frame rates. Essentially, it employs AI to upscale games, resulting in improved graphics and performance.

This feature bears similarities to Nvidia’s DLSS, AMD’s FidelityFX Super Resolution (FSR), and Intel’s Xe Super Sampling (XeSS), all of which leverage AI for enhanced gaming visuals.

As part of this initiative, Microsoft has developed the “DirectSR” API in collaboration with Nvidia, AMD, and Intel to streamline SR into future gaming titles.

However, this capability will exclusively cater to Copilot+ PCs featuring the latest Qualcomm Snapdragon X Elite processors and will be limited to a select number of games.

Cocreator and Generative Fill and Erase in Paint

Microsoft is introducing a new Cocreator feature in the Paint app that scans your drawings instead of generating an AI image. This feature will assist you in refining, editing, and developing your artistic concepts.

This feature employs a diffusion-based algorithm to yield a high-quality image with minimal effort, and offers settings to customize whether the creation is more literal or more expressive.

It’s crucial to clarify that Cocreator is exclusive to Copilot+ PCs, and should not be confused with “Image Creator,”an AI function utilizing cloud resources to convert text prompts into visuals.

Generative Fill

Within Paint, the “Fill”feature allows users to select a specific area of an image and generate new elements to enhance that section based on a text prompt. This feature will initially be accessible on Copilot+ PCs with the latest Qualcomm processors.

Generative Erase

The Paint application will also introduce the “Erase”feature, enabling users to select and eliminate sections of an image using AI—akin to features available on mobile devices.

Expect this capability to be rolled out across all devices, including those lacking AI support.

Restyle Image, Image Creator, and Super Resolution in Photos

The Photos app will incorporate a new “Restyle Image” function aimed at altering the style of a photo through AI and the device’s NPU, utilizing advanced image generation and editing techniques.

This feature allows users to select an image to transform its style, much like applying filters on mobile apps. Users can also input a text prompt to create a new background.

Image Creator

The Photos application will feature the same “Image Creator” tool as found in Paint, allowing users to create AI-generated images based on a simple text prompt. Unlike the Paint version, this feature is not exclusive to Copilot+ PCs since it operates in the cloud using the DALL-E language model.

Super Resolution

The Super Resolution functionality will enable users to upscale images through AI, exclusively available on Copilot+ PCs. It will utilize the NPU to enhance and sharpen images up to eight times their original resolution.

Microsoft is also promising that the Super Resolution feature in Photos can restore old, low-quality images back to life.

Updates to Copilot for Windows 11

Although Copilot has been a feature of Windows 11 for a while, version 24H2 will introduce updates. The app will function as a standard application that can be pinned or removed from the Taskbar, allowing users to resize and move it just like any other app (it is also available on Windows 11 23H2).

The application is actually a web version installed as a web app on Windows 11. While its basic functionalities remain the same, it will no longer be able to change system settings. Additionally, the interface has been refreshed to resemble the OpenAI ChatGPT web experience.

Furthermore, Copilot will now be integrated into the Settings app. Instead of an explicit button to summon the chatbot, users will find “Copilot suggestions” within the settings. For instance, in the System section, users may see helpful suggestions for addressing audio issues, activating energy-saving settings, or streaming to a wireless display.

In File Explorer, Microsoft aims to streamline user tasks with Copilot’s assistance, allowing actions applicable to images when hovering over them and clicking the Copilot button.

Moreover, Copilot will also become part of the Notifications feature. When receiving a toast notification, it will present suggestions related to the specific scenario. For example, if an email includes a PowerPoint attachment, Copilot might recommend summarizing its content.

This integration will allow users to share their screen with Copilot for assistance during various tasks.

Developers will also begin incorporating AI features into their applications, indicating a broader integration of AI capabilities in future operating system updates.

The recent changes to Copilot were first introduced during a recent Surface launch event, but since then, further details about Copilot suggestions have yet to be disclosed.

Conclusion

The 2024 Update for Windows 11 is currently available, packed with new functionalities and enhancements, though the AI features are yet to be included. Microsoft plans to begin testing these AI capabilities, including Windows Recall with enhanced security features, sometime in October for Copilot+ PCs via the Windows Insider Program, with a broader rollout expected in November 2024.

Copilot+ PCs are a new category of smart computers designed for AI applications. The initial lineup features Qualcomm Snapdragon X series processors along with integrated NPUs that support extensive AI functionalities while ensuring all-day battery performance. These machines are equipped with at least 16GB of RAM and 256GB of storage. New offerings from Intel and AMD are also expected to include processors with NPUs to facilitate these advanced AI experiences.

Update October 8, 2024: This article has been revised to provide accurate and up-to-date content.

Source