It seems that with almost every major iOS update, Apple’s Photos app gets something new and significant, and in fact this one component has possibly seen the most frequent number of updates over the past few years, with Apple continually building on the photo management experience — iOS 5 introduced iCloud Photo Stream and basic photo editing, with iOS 6 came Shared Photo Streams, then iOS 7 gave us an entirely new “Collections” layout along with filters, and iOS 8 came along with iCloud Photo Library and advanced editing.
Although Photos skipped a major enhancement with last year’s release iOS 9, it appears that Apple will be making up for this lost time in the iOS 10 Photos app, adding new intelligent photo analysis features akin to what we’ve seen in Google Photos, but with the huge added privacy benefit of performing all of this magic directly on the user’s iPhone, iPad, or iPod touch, rather than in a cloud-based data center.
Face, Object, and Scene Recognition
iOS 10 Photos gets a new piece of magic that the company has dubbed Advanced Computer Vision — ultimately a fancy name for deep learning that is going on under the hood to provide the user with intelligent recognition of faces, objects, and scenes right on the device.
While the macOS Photos app (and iPhoto and Aperture before it) have been able to do facial recognition for some time, for seemingly mysterious reasons Apple never chose to sync this information to its iOS counterpart — even after iCloud Photo Library came along to keep everything else magically in sync. With iOS 10, however, instead of merely syncing faces recognized in the macOS Photos app, Apple has actually built in code to perform the facial recognition directly on the user’s iOS device. Apple promises the results will still sync across multiple devices via iCloud Photo Library, but this is likely more for consistency and to save duplicated computational effort.
This feature also goes beyond just recognizing faces — Apple has built in recognition algorithms for objects and scenes as well, and Apple’s SVP of Software Engineering, Craig Federeghi, explained that iOS 10 now does eleven billion computations per photo to detect and analyze this information. The real magic here is not that Apple does this — Google and others clearly got there long ago — but that Apple is relying on the silicon in the iPhone and iPad to pull this off, which will be a great bonus for anybody who has been reluctant to turn over their entire photo collection to analysis by Google’s cloud-based algorithms.
The first and most obvious result of all of this will be improvements to the organization of photos and the search experience. Faces will be grouped into People albums, and users will be able to search for not only faces and places but also objects and scenes. The new search capabilities are working well already in the first Developer Preview beta, with common object types returning as “Categories” and the content analysis, while not perfect, is surprisingly good already at this stage.
Memories
Not content with just advanced searching capabilities, however, Apple has taken Advanced Computer Vision one big step beyond with Memories, a new feature that takes this deep learning and algorithmically groups collections of photos and videos into relevant collections based on people, places, things, and times when photos were taken.
For instance, Memories can include groupings such as “On This Day” providing a retrospective of pictures taken on the same day in previous years, “Best of” collections for photos taken over a series of several months or a year, location-based groupings for vacations, event-based groupings for holidays, and more.
A new Memories tab at the bottom of the Photos app provides direct access to recent memories, beginning with memories selected for the current day at the top and working back chronologically as you scroll down.
Selecting a specific memory will go into a detailed view that provides an automatically generated movie for that memory, as well as a summary view of all of the key photos. A Show All button will let you switch to a view of all photos for the memory or back to just the Summary view. Scrolling to the bottom of the memory shows a People section for the faces that are found in the photos in that memory, a Places map showing where those photos were taken, and a Related section providing quick access to other related memories — generally those that involve the same people, were taken in the same places, or at the same time of the year. Buttons at the very bottom allow you to flag a memory as a Favorite or delete it entirely should you not want it in your library.
A new Show Holiday Events switch is also included in the Photos section of the Settings app, allowing you to decide whether Memories are automatically created and labeled based on holidays in your home country.
Memory Movies
Tapping on the video thumbnail at the top will begin playback of an automatically generated movie for the selected memory, containing a slideshow of key photos and videos set to music.
The automatically generated videos work pretty well even at this early stage. However, iOS 10’s algorithms can only guess at things like length and mood music, so once the video stops playing, sliders will appear allowing you to adjust both the mood of the video along with whether you want to generate a short, medium, or long version.
An edit button at the bottom — now represented throughout the iOS 10 Photos app as three sliders — allows you to perform more advanced customization, including changing the title wording and font or choosing the soundtrack. The latter includes the ability to select music from your own library, setting the specific length of the video clip, and choosing which specific photos get included or excluded.
Think of it as “iMovie Lite” with most of the initial composition work already done for you in advance.
Expanded Details + Related Photos
Individual photos now get a Details view that can be accessed either from a button in the top-right corner or by just swiping the photo upward to scroll down.
This view is similar to the one shown for Memories, displaying people identified in the photo, the location where the photo was taken, and Memories related to the current photo. All of these are tappable so you can go directly to a person view, find other photos taken nearby, or photos directly to a related memory.
Tapping on a heading in the main Moments timeline will also now show a similar Memory-style detail view, rather than just taking you straight to a map view.
New Smart Albums
iOS 10 Photos brings in a few new special albums for the various features. While Memories gets its own tab in the bottom navigation bar, features for Places and People become new smart albums that appear right after favorites. If you’ve marked at least one memory as a favorite, you’ll also get a Favorite Memories album.
Places
While prior versions of the Photos app allowed you to select a map view for individual photo groupings in the main Photos view, with iOS 10 you can now view your entire photo library on a map via the Places album.
This map view begins with the widest view possible to show all of your photos, and as you zoom into a given area, the larger groupings of photos expand to show smaller groupings of photos in more detailed locations on the map. A Grid view will also be available if you’d prefer to just see Moments-style groupings of photos for the selected area of the map.
People
People provides a collected view of all of the faces that Photos has detected within your library.
