Camera and Photo Technologies

Does iOS 27 tag AI-edited photos with metadata or content credentials?

• Yes, the file's metadata is modified with IPTC and EXIF metadata.
• IPTC metadata is updated based on the specific AI modification performed (spatial, reframe, or cleanup).
• In the Photos app, the info panel displays which edit was used after modification.

Will there be an API for third-party apps to use or edit keywords in Photos?

• Keywords are a feature available in the Photos app on iOS and Mac, accessible via the info panel.
• Keywords will be reflected in IPTC metadata when exporting images.
• Currently, there is no PhotoKit-level API for fetching or querying based on these keywords.

What is the optimal way to load thumbnails in a lazy grid with matched geometry?

• On the Core Graphics side, use openImage(withThumbnails:) which internally utilizes existing thumbnails or decodes and scales down the main image quickly.
• On the Core Image side, request a scale factor when opening images to instruct Core Image to scale down the image as early as possible.

Can Deferred Start cause issues where the user may attempt to capture a photo before the capture photo output has attached to the session?

• Yes, if only deferred start is used, developers may miss a shot.
• Deferred Start was introduced in iOS 26.
• To prevent this, set the isResponsiveCaptureEnabled property on the AVCapturePhotoOutput to true.
• This enables system buffering, allowing quick launches and captures even if the output hasn't fully initialized, ensuring the moment is not missed.

What is the best way to get the depth map and the image for a live viewfinder and for the taken image to create a nice 3D effect with both?

• For the preview side, enable depthDataOutputEnabled on the preview stream and use AVCaptureSynchronizer to synchronize with the RGB stream.
• For the still capture side, set depthDataDeliveryEnabled on AVCapturePhotoSettings to get depth with the still capture.
• Alternatively, for a live depth effect in preview, enable cinematicVideoCaptureEnabled on the device input for a simplified solution.
• To access actual depth samples for custom processing, use AVCaptureVideoDataOutput and AVCaptureDepthDataOutput with AVCaptureDataOutputSynchronizer.

Can final exposure settings be read before capture when quality prioritization overrides manual settings?

• When quality or balanced photo quality prioritization is used, manual controls are not guaranteed and are often overridden.
• This is because these modes involve processing algorithms, including fusion, which require capturing different exposures.
• When developers require precise control over shutter speed and ISO, developers must use speed for photo prioritization.
• There is no API to read the final exposure before capture when using quality or balanced prioritization.
• quality and balanced are treated as hints, and for photo formats, manual settings will be ignored in favor of the capture stack's best algorithms.

What is the PHAsset original resource choice used for?

• This API relates to the existing feature of RAW plus JPEG capture.
• It allows developers to choose whether the compressed image or the RAW image is considered the "original" for editing purposes.
• It also determines which resource is sourced for creating smaller image derivatives and thumbnails.
• Related APIs include PHContentEditingInput.source for choosing the resource and change requests on assets for toggling the original resource.

Will iOS 27 support linear scene-referred Bayer RAW preview streams?

• Currently, the only way to get scene-referred linear data in camera capture is through the log format (log and log2).
• Direct RAW frame output from camera capture (e.g., for ProRes RAW) includes metadata that is not compatible with what CI or filter would need for rendering.
• The idea of having RAW data from camera capture usable with CI filters is worth exploring for the future.

What are must-have recommendations for a capture-centric app?

• Performance is critical: Ensure a fast launch, responsive capture, and utilize deferred processing.
• User Experience: Focus on the specific user base and their needs; a capture experience for social media differs from a professional photography app.
• Reliability: Ensure the app is stable and dependable.
• Leverage Sample Code: Do not start from scratch; use sample code to understand best practices, such as using a dedicated serial queue for AVCaptureSession to avoid blocking the UI thread.
• Differentiation: Identify what makes the app unique and provides value beyond the basic capture functionality.
• Photos Framework Integration: Properly handle user permissions for accessing the photo library and consider performance implications for capture-to-review workflows.

What is the optimal and recommended way for streaming video and audio simultaneously, ensuring synchronization?

• Use AVCaptureSession for both audio and video by attaching device inputs for the camera and microphone.
• AVCaptureSession synchronizes audio and video sources, providing synchronized output.
• If using a different audio API (e.g., AURemoteIO), developers will need to use CMClock and CMClockConvertTime to synchronize the audio and video clocks.
• Generally, synchronize video to the audio clock to avoid audio rate conversion.
• For network streaming, ensure synchronization before transmission; playback synchronization is handled by APIs like AVPlayer or AVSampleBufferDisplayLayer.

Why does ProRAW support 48-megapixel output from a Quad Bayer sensor, whereas native Bayer RAW output is limited to binned resolution?

• Bayer RAW contains raw Bayer data, while ProRAW undergoes debayering and Photonic Engine merging, resulting in RGB linearized data.
• ProRAW's output format is independent of the sensor's native format (Bayer or Quad Bayer) because it's already processed into RGB.
• The ability to output native Quad Bayer RAW is not yet available and would require support in tools like CIImage filters.
• Debayering Quad Bayer data is significantly more complex than debayering standard Bayer data.

How should ISO-conformant gain maps and metadata be generated on iOS?

• Refer to WWDC talks from two years ago that detail the APIs for generating gain maps.
• Core Graphics and Core Image frameworks can handle gain map and ISO gain map data.
• The specifications for gain maps have been added to the HEIF and JPEG specifications.
• Understanding the metadata involves examining how the image is divided into SDR/RGB content and HDR additions.

How do Deferred Start and setPreparedPhotoSettingsArray compose for photo output warmup?

• Deferred Start primarily focuses on getting the app launch and preview running quickly by moving initialization from before preview to after preview.
• setPreparedPhotoSettingsArray allows developers to inform the photo output upfront about the worst-case scenario (e.g., quality prioritization plus other features), enabling pre-allocation for the still image pipeline.
• These two APIs are orthogonal and can complement each other.
• Deferred Start does not inherently conflict with using setPreparedPhotoSettingsArray after the preview is running.
• setPreparedPhotoSettingsArray can be called at any time, but calling it before starting the session is beneficial.

Will adjusting the PHAsset new rating property require a PHAssetChangeRequest?

• Yes, adjusting the rating property of a PHAsset requires a PHAssetChangeRequest.
• The rating property accepts values from unset to 1 through 5.

Is there a native way to obtain metadata of the original file type (PNG, JPEG, etc.) from a photo selected via the Photos picker?

• When using Transferable to set up data, ensure developers are specifying the UTType instead of leaving it as a default image type.
• Sample code for the Photos picker on developer.apple.com may provide further guidance.
• Some conversion might occur in the Photos picker if users disable captions or location data, but if developers are consistently getting PNG output, double-check the UTType for Transferable.

Is there a way to learn about the adjustment Plist format that the Apple Photos app writes, and when exporting that file, is there a way to import those adjustments back into Photos?

• There is currently no API available to decode the adjustment Plist format used by the Apple Photos app.
• If this functionality is desirable for the app, file a feedback request.

What are the biggest mistakes a developer can make when building camera-heavy apps on iPhone?

• Running AVCaptureSession on the main thread: This blocks the UI and leads to a poor user experience. Always use a dedicated serial queue for AVCaptureSession operations.
• Not using beginConfiguration and commitConfiguration for session reconfiguration: When changing multiple properties of the AVCaptureSession, use these methods to ensure the underlying graph is reconfigured only once, improving efficiency.
• Using AVCaptureVideoDataOutput for preview when AVCaptureVideoPreviewLayer suffices: When developers only need to render the preview, AVCaptureVideoPreviewLayer is highly optimized and more efficient than processing buffers from AVCaptureVideoDataOutput. Use AVCaptureVideoDataOutput only if developers need to interact with the buffers (e.g., for metadata or drawing).

What are the suggestions for a close-to-native performance way to achieve a zoom, scroll, and pan feature for photo images, up to their original max resolution without pixelating?

• Use Core Image, which is designed for UI movement, zoom, and pan by caching decoding pipeline operations.
• Core Image allows developers to decode only the visible rectangle and caches the results, enabling smooth interaction even with very large images.
• For RAW images, CIImage has a counterpart, CIImage.rawFilter, with similar capabilities for smooth zooming and panning.
• Core Image's Region of Interest (ROI) management optimizes performance by recomputing only the zoomed-in area, even for large images.

How often can Siri camera intelligence be used, and does it affect picture quality?

• Developers can use the Siri camera intelligence feature as much as the goal is to on the device.
• Captures from this feature are saved to the Siri app, not the user's normal photo library.
• The quality is screen-resolution aspect ratio, meaning developers do not get the full image quality of photo mode.

Is the spatial reframe 3D reconstruction pipeline available to developers?

• There is no developer API available for using the spatial reframe capability directly.
• ARKit framework offers some capabilities for 3D scene capture, reconstruction, and manipulation, but it's not a direct equivalent to the Photos app's reframe editing solution.
• For depth sensing, consider using the LiDAR camera on Pro phones via AVCaptureDevice or the TrueDepth camera on the front-facing camera.

How should camera capture orientation be handled across iOS, iPadOS, and macOS?

• Handling device orientation is necessary because camera mounting and device orientation can vary.
• Use AVCaptureDevice.RotationCoordinator to determine the correct rotation degree for preview or to keep an image horizon-level upright.
• Developers can also set the videoRotationAngle on the AVCaptureConnection to have the framework automatically rotate the output.
• For video data output, rotation physically rotates buffers; for photo output and movies, EXIF data is used for rotation on playback.
• Refer to the WWDC session "Support the Center Stage Front Camera in The iOS App" for detailed guidance on handling rotation correctly and defending against future changes.

Is it possible to capture 24-megapixel images with depth data on deferred photo capture?

• Yes, it is possible to capture 24-megapixel images with depth data.
• Ensure developers opt into deferred processing, as 24-megapixel processing is time-consuming.
• Verify that the selected device supports 24-megapixel maximum photo dimensions.
• Select quality prioritization and enable depthDataDeliveryEnabled to receive captures with depth data.

What APIs support multi-camera streaming apps?

• Locked Frame Duration: Use the lockedFrameDuration API to ensure a consistent frame rate (e.g., 29.97 fps) and synchronized audio.
• GenLock: Synchronize multiple iPhones to an external sync source using GenLock, often with third-party hardware like Blackmagic devices. This ensures perfect synchronization for multi-cam shoots.
• Timecode Generation: Output and associate timecode with video frames for easier post-production synchronization.
• AVPro Video Storage API: For ProRes 4K 30 video captures, this API provides deterministic file write speeds to prevent frame drops, especially on older or fragmented disks.

Is there a niche new API that may not be talked about so much?

• AVPro Video Storage API: This API is a performance optimization for ProRes video captures. It provides a pre-allocated file on disk for writing, ensuring deterministic write speeds and preventing frame drops, even on older devices.
• RAW 9 Engine: A new ML-based engine for enhancing RAW image processing (debayering, denoising) for third-party cameras, offering outstanding output quality.