Everything we know about Apple Intelligence

Apple’s latest iPhones support a new breed of Apple AI called Apple Intelligence, a collection of artificial intelligence (AI) tools that will be made available across the company’s platforms starting in October with the release of iOS 18.1, iPadOS 18.1, and macOS Sequoia 15.1.

Apple Intelligence supplements Apple’s existing machine-learning tools and relies on generative AI (genAI) technology similar to that used by OpenAI’s ChatGPT. Apple’s version to a great extent runs on its own self-trained genAI models, which are built to be integrated across platforms, capable of using a user’s personal information, and private by the design.

Announced at this year’s Worldwide Developer’s Conference in June, Apple Intelligence is designed “to make your most personal products even more useful and delightful.” (That’s how Apple CEO Tim Cook described it.)

 
 

Essentially, the company has moved to build an AI ecosystem that is personal, private, and powerful, what Apple calls “AI for the rest of us.”

Here’s a look at what’s coming and how Apple got to this point.

Why Apple Intelligence matters

Apple has worked with AI since its earliest days (more about this below), but in in the last couple of years — since the arrival of ChatGPT and others — the company has been perceived as falling behind its competitors. There are many reasons for that, not least that Apple’s innate secrecy was a turn-off to researchers at the cutting edge of AI. Internal squabbles over precious R&D resources may also have slowed development.

But one moment that might have changed the scene took place over the winter holidays in late 2023, when Apple Senior Vice President for Software Craig Federighi tested GitHub Copilot code completion. He was reportedly blown away — and redirected Apple’s software development team to begin to apply Large Language Models (LLMs, a basic part of genAI tools) across Apple products. The company now sees this work as foundational to future product innovation and has diverted vast quantities of resources to bringing its own genAI technologies to its devices.

Analysts note that with Apple Intelligence soon to be available across the newer Macs, iPhones, and iPads, the company could quickly become one of the most widely used AI ecosystems in the world. (Wedbush Securities analyst Daniel Ives predicts Apple’s devices will be running 25% of global AI soon.) This matters, since AI smartphones and PCs will drive sales in both markets across the coming months, and Apple now has a viable product family to tout.

Apple

How Apple approaches Apple Intelligence

To deliver AI on its devices, Apple has refused to dilute its longstanding commitment to user privacy. With that in mind, it has developed a three-point approach to handling queries using Apple Intelligence:

On device

Some Apple Intelligence features will work natively on the device. This has the advantage of working faster while preserving privacy. Edge-based processing also reduces energy requirements, because no cloud communication or server-side processing is required. (More complex tasks must still be handled in the cloud.)

In the cloud

Apple is deploying what it calls Private Cloud Compute. This is a cloud intelligence system designed specifically for private AI processing and capable of handling complex tasks using massive LLMs.

The idea behind this system is that it provides the ability to flex and scale computational capacity between on-device processing and larger, server-based models. The servers used for these tasks are made by Apple, use Apple Silicon processors, and run a hardened operating system that aims to protect user data when tasks are transacted in the cloud. The advantage here is you can handle more complex tasks while maintaining privacy.

Externally

Apple has an agreement with OpenAI to use ChatGPT to process AI tasks its own systems can’t handle. Under the deal, ChatGPT is not permitted to gather some user data. But there are risks to using third-party services, and Apple ensures that users are aware if their requests need to be handled by a third-party service. 

The company says it has designed its system so when you use Private Cloud Compute, no user data is stored or shared, IP addresses are obscured, and OpenAI won’t store requests that go to ChatGPT. The focus throughout is to provide customers with the convenience of AI, while building strong walls around personal privacy.

Apple

What Apple Intelligence features exist?

Apple has announced a range of initial features it intends making available within its Apple Intelligence fleet. The first new tools will appear with iOS 18.1, which is expected to appear when new Apple Silicon Macs and iPads are introduced later this fall.

Additional services will be introduced in a staggered rollout in subsequent releases. While not every announced feature is expected to be available this year, all should be in place by early 2025. In the background, Apple is not resting on its laurels; its teams are thought to be exploring additional ways Apple Intelligence can provide useful services to customers, with a particular focus on health.

Apple

At present, these are the Apple Intelligence tools Apple has announced:

Writing Tools

Writing Tools is a catch-all term for several useful features, most of which should appear in October with iOS 18.1 (and the iPad and Mac equivalents). These tools work anywhere on your device, including in Mail, Notes, Pages, and third-party apps. To use them, select a section of text and tap Writing Tools in the contextual menu.

Rewrite will take your selected text and improve it.

Proofread is like a much smarter spellchecker that checks for grammar and context.

Summarize will take any text and, well, summarize it. This also works in meeting transcripts. 

Priority notifications: Apple Intelligence understands context, which means it should be able to figure out which notifications are most important to you.

Priority messages in Mail: The system will also prioritize the emails it thinks are most important.

Smart Reply: Apple’s AI can also generate email responses. You can edit these, reject them, or write your own.

Reduce Interruptions: A new Focus mode that is smart enough to let important notifications through.

Call transcripts: It is possible to record, transcribe, and summarize audio captured in Notes or during a Phone call. When a recording is initiated during a call in the Phone app, participants are automatically notified. After the call, Apple Intelligence generates a summary to help recall key points.

Search and Memory Movies in Photos

Search is much better in Photos. It will find images and videos that fit complex descriptions and can even locate a particular moment in a video clip that fits your search description.

Search terms can be highly complex; enter a description and Apple Intelligence will identify all the most appropriate images and videos, put together a storyline with chapters based on themes it figures out from within the collection, and create a Memory Movie. The idea is that your images are gathered, collected, and presented in an appropriate narrative arc; this feature is expected to debut with iOS 18.1.

Apple

Clean Up tool in Photos

At least in my parts of social media, the Photos AI tool that most seemed to impress early beta testers was Clean Up. This super-smart implementation means Apple Intelligence can identify background objects in an image and let you remove them with a tap. I can still recall when removing items from within images required high-end software running on top-of-the-range computers equipped with vast amounts of memory.

Now you can do it in a trice on an iPhone.

Image Playground for speedy creatives

Expected to appear in iOS 18.2, Image Playground uses genAI to let you create animations, illustrations, and sketches from within any app, including Messages. Images are generated for you by Apple Intelligence in response to written commands. You can also choose between a range of themes, places, or costumes, and also create an image based on a person from your Photos library.

The feature is also available within its own app and should appear in December.

Apple

Genmoji get smarter

Genmoji uses genAI to create custom emoji. The idea is that you can type in a description of the emoji you want to use and select one of the automatically generated ones to use in a message. You will also be able to keep editing the image to get to the one you want. (The only problem is that the person on the receiving end may not necessarily understand your creative zeal.)

This feature should show in December with iOS 18.2.

Image Wand

This AI-assisted sketching tool can transform rough sketches into nicer images in Notes. Sketch an image, then select it; Image Wand will analyze the content to create a pleasing and relevant image based on what you drew. You can also select an empty space and Image Wand will look at the rest of your Note to identify a context for which it will create an image for you.

Image Wand is now expected late 2024 or early 2025.

Apple

Camera Control in iPhone 16 Pro

A new feature in iPhone 16 Pro relies on visual intelligence and AI to handle some tasks. You can point your camera, for example, at a restaurant to get reviews or menus. It will also be possible to use this feature to access third-party tools for more specific information, such as accessing ChatGPT.

Additional visual tools are coming. For example, Siri will be able to complete in-app requests and take action across apps, such as finding images in your collection and then editing them inside another app.

Coming soon: Siri gains context and ChatGPT

ChatGPT integration in Siri is expected to debut at the end of the year, with additional enhancements to follow. The idea is that when you ask Siri a question, it will try to answer using its own resources; if it is unable to do so it will ask whether you want to use ChatGPT to get the answer. You don’t have to, but you will get free access to using it if you choose. Privacy protections are built in for users who access ChatGPT — IP addresses are obscured, and OpenAI won’t store requests. 

Siri will also get significant improvements to deliver better contextual understanding and powerful predictive intelligence based on what your devices learn about you. You might use it to find a friend’s flight number and arrival time from a search through Mail or to put together travel plans — or any other query that requires contextual understanding of your situation. 

The contextual features should appear next year.

Apple

On-screen awareness, but not until 2025

A new evolution in contextual awareness is scheduled to arrive at some point in 2025. This will give Siri the ability to take and use information on your display. The idea here is that whatever is on your screen becomes usable in some way — you might use this to add addresses to your contacts book, or to track threads in an email, for example. It’s a profound connection between what you do on your device and wherever you happen to be.

Another, and perhaps even more powerful, improvement will allow Siri to control apps, and because it uses genAI, you’ll be able to pull together a variety of instructions and apps — such as editing an image and adding it to a Note without having to open or use any apps yourself. This kind of deep control builds on the accessibility tools Apple already has and leans into some of the visionOS user interface improvements.

It’s another sign of the extent to which user interfaces are becoming highly personal.

Where can I get Apple Intelligence?

Apple has always been quite clear that Apple Intelligence will first be made available in beta in US English. During beta testing, Apple adjusted this slightly so that these tools work on any compatible iPhone running US English as its language and for Siri.

The company will introduce Apple Intelligence with localized English in Australia, Canada, New Zealand, South Africa, and the UK in December. Additional language support — such as Chinese, French, Japanese, and Spanish — is coming next year.

What devices work with Apple Intelligence?

Apple Intelligence requires an iPhone 15 Pro, iPhone 15 Pro Max, or iPhone 16 series device. It also runs on Macs and iPads equipped with an M1 or later chip.

What AI is already inside Apple’s systems?

All these features are supplemented by numerous forms of AI tools Apple already has in place across its platforms, principally around image vision intelligence and machine learning. You use these built-in applications each time you use FaceID, run facial recognition in Photos, or make use of the powerful Portrait Mode or Deep Fusion features when taking a photograph.

There are many more AI tools, from recognition of addresses and dates in emails for import into Calendar to VoiceOver all the way to Door Detection, even the Measure app on iPhones. What’s changed is that while Apple’s deliberate focus had been on machine-learning applications, the emergence of genAI unleashed a new era in which the contextual understanding available to LLM models uncovered a variety of new possibilities.

The omnipresence of various kinds of AI across the company’s systems shows the extent to which the dreams of Stanford researchers in the 1960s are becoming real today.

denvit

An alternative history of Apple Intelligence

Apple Intelligence might appear to have been on a slow train coming, but the company has, in fact, been working with AI for decades.

What exactly is AI?

AI is a set of technologies that enable computers and machines to simulate human intelligence and problem-solving capabilities. The idea is that the hardware becomes smart enough to learn new tricks based on what it learns, and carries the tools needed to engage in such learning.

To trace the trail of modern AI, think back to 1963, when computer scientist and LISP inventor John McCarthy launched the Stanford Artificial Intelligence Laboratory (SAIL). His teams engaged in important research in robotics, machine-vision intelligence, and more.

SAIL was one of three important entities that helped define modern computing. Apple enthusiasts will likely have heard of the other two: Xerox’s Palo Alto Research Center (PARC), which developed the Alto that inspired Steve Jobs and the Macintosh, and Douglas Engelbart’s Augmentation Research Center. The latter is where the mouse concept was defined and subsequently licensed to Apple. 

Important early Apple luminaries who came from SAIL included Alan Kay and Macintosh user interface developer Larry Tesler — and some SAIL alumni still work at the company.

“Apple has been a leader in AI research and development for decades,” pioneering computer scientist and author Jerry Kaplan told me. “Siri and face recognition are just two of many examples of how they have put this investment to work.”

Back to the Newton…

Existing Apple Intelligence solutions include things we probably take for granted, going back to the handwriting recognition and natural language support in 1990’s Newton. That device leaned into research emanating from SAIL — Tesler led the team, after all. Apple’s early digital personal assistant first appeared in a 1987 concept video and was called Knowledge Navigator. (You can view that video here, but be warned, it’s a little blurry.)

Sadly, the technology couldn’t support the kind of human-like interaction we expect from ChatGPT, and (eventually) Apple Intelligence. The world needed better and faster hardware, reliable internet infrastructure, and a vast mountain of research-exploring AI algorithms, none of which existed at that time.  

But by 2010, the company’s iPhone was ascendant, Macs had abandoned the PowerPC architecture to embrace Intel, and the iPad (which cannibalized the netbook market) had been released. Apple had become a mobile devices company. The time was right to deliver that Knowledge Navigator. 

Apple

When Apple bought Siri

In April 2010, Apple acquired Siri for $200 million. Siri itself is a spinoff from SAIL, and, just like the internet, the research behind it emanated from a US Defense Advanced Research Projects Agency (DARPA) project. The speech technology came from Nuance, which Apple acquired just before Siri would have been made available on Android and BlackBerry devices. Apple shelved those plans and put the intelligent assistant inside the iPhone 4S (dubbed by many as the “iPhone for Steve,” given Steve Jobs’ death around the time it was released).

Highly regarded at first, Siri didn’t stand the test of time. AI research diverged, with neural networks, machine intelligence, and other forms of AI all following increasingly different paths. (Apple’s reluctance to embrace cloud-based services — due to concerns about user privacy and security — arguably held innovation back.)

Apple shifted Siri to a neural network-based AI system in 2014; it used on-device machine learning models such as deep neural networks (DNN), n-grams and other techniques, giving Apple’s automated assistant a bit more contextual intelligence. Apple Vice President Eddy Cue called the resulting improvement in accuracy “so significant that you do the test again to make sure that somebody didn’t drop a decimal place.”

But times changed fast.

Did Apple miss a trick?

In 2017, Google researchers published a landmark research paper, “Attention is All you Need.” This proposed a new deep-learning architecture that became the foundation for the development of genAI. (One of the paper’s eight authors, Łukasz Kaiser, now works at OpenAI.)

One oversimplified way to understand the architecture is this: it helps make machines good at identifying and using complex connections between data, which makes their output far better and more contextually relevant. This is what makes genAI responses accurate and “human-like” and it’s what makes the new breed of smart machines smart.

The concept has accelerated AI research. “I’ve never seen AI move so fast as it has in the last couple of years,” Tom Gruber, one of Siri’s co-founders, said at the Project Voice conference in 2023.

Yet when ChatGPT arrived — kicking off the current genAI gold rush — Apple seemingly had no response. 

The (put it to) work ethic

Apple’s Cook likes to stress that AI is already in wide use across the company’s products. “It’s literally everywhere on our products and of course we’re also researching generative AI as well, so we have a lot going on,” he said. 

He’s not wrong. You don’t need to scratch deeply to identify multiple interactions in which Apple products simulate human intelligence. Think about crash detection, predictive text, caller ID based on a number not in your contact book but in an email, or even shortcuts to frequently opened apps on your iPhone. All of these machine learning tools are also a form of AI. 

Apple’s CoreML frameworks provide powerful machine learning frameworks developers can themselves use to power up their products. Those frameworks build on the insights Adobe co-founder John Warnock had when he figured out how to automate the animation of scenes, and we will see those technologies widely used in the future of visionOS.

All of this is AI, albeit focused (“narrow”) uses of it. It’s more machine intelligence than sentient machines. But in each AI application it delivers, Apple creates useful tools that don’t undermine user privacy or security.

The secrecy thing

Part of the problem for Apple is that so little is known about its work. That’s deliberate. “In contrast to many other companies, most notably Google, Apple tends not to encourage their researchers to publish potentially valuable proprietary work publicly,” Kaplan said.

But AI researchers like to work with others, and Apple’s need for secrecy acts as a disincentive for those in AI research. “I think the main impact is that it reduces their attractiveness as an employer for AI researchers,” Kaplan said. “What top performer wants to work at a job where they can’t publicize their work and enhance their professional reputation?” 

It also means the AI experts Apple does recruit subsequently leave for more collaborative freedom. For example, Apple acquired search technology firm Laserlike in 2018, and within four years, all three of that company’s founders had quit. And Apple’s director of machine learning, Ian Goodfellow (another a SAIL alumni), left the company in 2022. I imagine the staff churn makes life tough for former Google Chief of Search and AI John Giannandrea, who is now Apple’s senior vice president of machine learning and AI strategy. 

That cultural difference between Apple’s traditional approach and the preference for open collaboration and research in the AI dev community might have caused other problems. The Wall Street Journal reported that at some point both Giannandrea and Federighi were competing for resources to the detriment of the AI team. 

Despite setbacks, the company has now assembled a large group of highly regarded AI pros, including Samy Bengio, who leads company research in deep learning. Apple has also loosened up a great deal, publishing research papers and open source AI software and machine learning models to foster collaboration across the industry.

What next?

History is always in the rear view mirror, but if you squint just a little bit, it can also show you tomorrow. Speaking at the Project Voice conference in 2023, Siri co-founder Adam Cheyer said: “ChatGPT style AI…conversational systems…will become part of the fabric of our lives and over the next 10 years we will optimize it and become accustomed to it. Then a new invention will emerge and that will become AI.”

At least one report indicates Apple sees this evolution of intelligent machinery as foundational to innovation. While that means more tools, and more advances in user interfaces, each those steps leads inevitably toward AI-savvy products such as AR glasses, robotics, health tech — even brain implants. 

For Apple users, the next step — Apple Intelligence — arrives this fall.

Please follow me on Mastodon, or join me in the AppleHolic’s bar & grill and Apple Discussions groups on MeWe.