Microsoft reveals Windows 11’s AI roadmap: Smart search, upscaling, more

What’s next for Windows? Microsoft may have just shipped the Windows 11 2024 Update (24H2), but the company is already disclosing its plans for next-gen Windows apps — and there’s some very interesting AI-powered features due by the holidays.

Microsoft disclosed that it is working on several AI additions to Windows and Windows apps: improved Windows search using natural descriptive language, super resolution in Photos, generative fill and erase in Paint, as well as the debut of Recall. All (save Recall) will appear in October as part of the Windows Insider program, with an expected launch in November.

Other breaking Microsoft news you need to know:

Windows 11’s big 2024 Update: How and when you’ll get it

Copilot’s AI will be able to ‘see’ and talk to you, Microsoft says

Windows 11’s 2024 Update: 5 big changes I really like (and more)

All of these features will depend on the NPU inside of Copilot+ PCs, which will now include PCs that feature Qualcomm Snapdragon X Elite processors as well as AMD’s Ryzen AI 300 and Intel’s Lunar Lake. Microsoft also plans several more Copilot features that will run in the cloud, which will include Copilot Voice and Copilot Vision, similar to the innovations used in competing AI services. The timing of these new features will vary by platform, however, as Snapdragon X PCs have shipped for several months now; Microsoft will enable the Copilot+ capable PCs from AMD and Intel with their own updates.

Microsoft has already disclosed more details of its Recall revamp, which the company now says can be bypassed when setting up a new PC or removed later. Windows Recall takes snapshots of your screen every so often, extracting the data and then storing it in case you need it later. The feature came under fire because of claims that it invaded user privacy and was insecure. Now, Microsoft says that it’s storing the screenshots used in Recall and the extracted data inside an encrypted enclave. Security researchers have previously said that the data was stored unencrypted.




This appears to be improved semantic search in conjunction with Recall, where the user is searching for one thing and the search results are interpreting what the user is looking for.



This appears to be improved semantic search in conjunction with Recall, where the user is searching for one thing and the search results are interpreting what the user is looking for.Microsoft

This appears to be improved semantic search in conjunction with Recall, where the user is searching for one thing and the search results are interpreting what the user is looking for.Microsoft


Microsoft

Insiders who own Snapdragon X PCs will be able to experience Recall via the Windows Insider program beginning in October. If you own a Copilot+ PC with an AMD or Intel chip inside, the Insider trial will begin in November instead. When will Microsoft release Recall to the public? For now, that’s still a mystery.

New AI features coming for Copilot+ PCs

Microsoft said that it plans to improve search on your PC by using more natural language when searching for files on your PC. You may have already seen this in apps like Microsoft Photos, or Google’s Photos app, too; if you search for “beach,” for example, the apps will use AI to identify beach scenes. Microsoft will apply the same techniques in File Explorer, though it’s not clear what folders or files they’ll apply to.

The improved Windows search does seem to do a better job at identifying scenes than it did before: “BBQ party” was listed as an example search term in a demonstration, below.








Microsoft

Microsoft


Microsoft

“You no longer need to remember file names, settings locations, or even worry about spelling — just type what’s in your mind to find it on a Copilot+ PC,” Microsoft says. It seems unlikely, however, that you’ll be able to find a particular.ini file in the depths of your user folder with the same ease that you’ll find a photo of your aunt’s wedding.

Improved Search will begin with File Explorer, then expand to Windows Search and Settings “in the coming months.”




An example of Microsoft’s improved search in Windows Settings. Normally, this specific search term would elicit a shrug from Search.



An example of Microsoft’s improved search in Windows Settings. Normally, this specific search term would elicit a shrug from Search.Microsoft

An example of Microsoft’s improved search in Windows Settings. Normally, this specific search term would elicit a shrug from Search.Microsoft


Microsoft

Super Resolution within Photos may be my favorite potential app, if only because a.) I have many old photos from older, crappy digital cameras, and b.) journalists often are supplied with low-resolution photos that need to be expanded or upscaled for publishing. In either case, the new Super Resolution feature promises to solve those problems.

Microsoft announced Automatic Super Resolution as a way to improve games, but Super Resolution for Photos appears even more practical. There are numerous sites and applications that promise upscaling, and it’s not clear whether this new app will outperform those. Still, Super Resolution for Photos will be free. Tapping the Copilot+ PC’s AI TOPS, you’ll be able to increase the resolution up to eight times, Microsoft says. Super Resolution will be part of Photos, which can already automatically adjust lighting and tint, remove a background, add generative elements, and more.




Super Resolution uses AI to fill in details that aren’t there — here, within the Windows Photos app.



Super Resolution uses AI to fill in details that aren’t there — here, within the Windows Photos app.Microsoft

Super Resolution uses AI to fill in details that aren’t there — here, within the Windows Photos app.Microsoft


Microsoft

Speaking of generative AI, you knew that Microsoft had more up its sleeve. In Paint, Microsoft already added Image Creator and Cocreator, which allows for AI-generated images that can be added via layers — also part of Paint. In Photos, generative erase is also a current feature. Photos previously had spot erase, which was wonderful for digitally removing dust that evaded my cleaning cloth when taking a photo of a laptop or docking station. Generative erase, also already in Photos, allows you to define an entire region — such as a power cord, say — and remove that, too.

(You know what’s weird? Generative erase in Photos doesn’t demand an NPU to work. But Microsoft is saying that generative erase in Paint, will.)

Generative Fill basically allows you to define an area within Paint and then create a piece of generative art to fill that region, intelligently working its way into the background. You could sort of already do this in Paint, by creating a second layer, then a piece of AI art, then combining the two. This new feature adds smarts: It instructs the AI to accommodate both your defined region as well as the background when creating the art, making a seamless whole.




Generative Fill in action, within Microsoft Paint. The added lighthouse fits thematically into the scene.



Generative Fill in action, within Microsoft Paint. The added lighthouse fits thematically into the scene.Microsoft

Generative Fill in action, within Microsoft Paint. The added lighthouse fits thematically into the scene.Microsoft


Microsoft

The final feature, called Click to Do, is a bit of a puzzle. Normally you can hover your mouse cursor over any part of a web page, right click, and receive any number of options. Now, Microsoft wants you to click the Copilot button, then (left) click somewhere on the screen to receive a list of options that include Copy, Save, Blur background, and more.

“At your request, Click to Do seamlessly connects you to tools and actions to accomplish tasks, like suggesting help with Visual Search with Bing, background blur or erasing objects in Photos, removing backgrounds with Paint, and more,” Microsoft explained. “It also assists with text-related actions such as rewrite, summarize or explain text in line, opening in a text editor, sending an email (with a recognized email address), web searches, and opening websites (with a recognized URL).” Additional actions will be added in the coming months, Microsoft said.




Click to Do looks useful, but why couldn’t this be in the existing right-click menu?



Click to Do looks useful, but why couldn’t this be in the existing right-click menu?Microsoft

Click to Do looks useful, but why couldn’t this be in the existing right-click menu?Microsoft


Microsoft

Microsoft demonstrated Click to Do “analyzing” a web page when a user used the appropriate click command. What’s unclear is whether users will remember to actually use the “Copilot click” to do things, or just fall into old habits.