How to generate an UIImage from AVCapturePhoto with correct orientation?

Question

I am calling AVFoundation's delegate method to handle a photo capture, but I am having difficulty converting the AVCapturePhoto it generates into an UIImage with the correct orientation. Although the routine below is successful, I always get a right-oriented UIImage (UIImage.imageOrientation = 3). I have no way of providing an orientation when using the UIImage(data: image) and attempting to first use photo.cgImageRepresentation()?.takeRetainedValue() also doesn't help. Please assist.

Image orientation is critical here as the resulting image is being fed to a Vision Framework workflow.

func photoOutput(_ output: AVCapturePhotoOutput, didFinishProcessingPhoto photo: AVCapturePhoto, error: Error?) {
    // capture image finished
    print("Image captured.")
    if let imageData = photo.fileDataRepresentation() {
        if let uiImage = UIImage(data: imageData){
            // do stuff to UIImage
        }
    }
}

UPDATE 1: Reading Apple's Photo Capture Programming Guide (out of date for iOS11), I did manage to find one thing I was doing wrong:

1. On every capture call (self.capturePhotoOutput.capturePhoto) one must setup a connection with the PhotoOutput object and update its orientation to match the device's orientation at the moment the picture is taken. For doing that, I created an extension of UIDeviceOrientation and used it on the snapPhoto() function I created to call the capture routine and wait for the didFinishProcessingPhoto delegate method to be executed. I've added a snapshot of the code because the code sample delimiters here don't seem to be displaying them correctly.

Update 2 Link to full project on GitHub: https://github.com/agu3rra/Out-Loud

Solution 1

Final update: I ran some experiments with the app and came to the following conclusions:

1. kCGImagePropertyOrientation doesn’t seem to influence the orientation of the captured image inside your application and it only varies with the device orientation if you update your photoOutput connection each time you are about to call the capturePhoto method. So:

   func snapPhoto() {
       // prepare and initiate image capture routine
   
       // if I leave the next 4 lines commented, the intented orientation of the image on display will be 6 (right top) - kCGImagePropertyOrientation
       let deviceOrientation = UIDevice.current.orientation // retrieve current orientation from the device
       guard let photoOutputConnection = capturePhotoOutput.connection(with: AVMediaType.video) else {fatalError("Unable to establish input>output connection")}// setup a connection that manages input > output
       guard let videoOrientation = deviceOrientation.getAVCaptureVideoOrientationFromDevice() else {return}
       photoOutputConnection.videoOrientation = videoOrientation // update photo's output connection to match device's orientation
   
       let photoSettings = AVCapturePhotoSettings()
       photoSettings.isAutoStillImageStabilizationEnabled = true
       photoSettings.isHighResolutionPhotoEnabled = true
       photoSettings.flashMode = .auto
       self.capturePhotoOutput.capturePhoto(with: photoSettings, delegate: self) // trigger image capture. It appears to work only if the capture session is running.
   }
   

2. Viewing the generated images on the debugger has shown me how they get generated, so I could infer the required rotation (UIImageOrientation) to get it displayed upright. In other words: updating UIImageOrientation tells how the image should be rotated in order for you to see it in the correct orientation. So I came to the following table:

3. I had to update my UIDeviceOrientation extension to a rather unintuitive form:

   extension UIDeviceOrientation {
       func getUIImageOrientationFromDevice() -> UIImageOrientation {
           // return CGImagePropertyOrientation based on Device Orientation
           // This extented function has been determined based on experimentation with how an UIImage gets displayed.
           switch self {
           case UIDeviceOrientation.portrait, .faceUp: return UIImageOrientation.right
           case UIDeviceOrientation.portraitUpsideDown, .faceDown: return UIImageOrientation.left
           case UIDeviceOrientation.landscapeLeft: return UIImageOrientation.up // this is the base orientation
           case UIDeviceOrientation.landscapeRight: return UIImageOrientation.down
           case UIDeviceOrientation.unknown: return UIImageOrientation.up
           }
       }
   }
   

4. This is how my final delegate method looks now. It displays the image in the expected orientation.

   func photoOutput(_ output: AVCapturePhotoOutput,
                                    didFinishProcessingPhoto photo: AVCapturePhoto,
                                    error: Error?)
   {
       // capture image finished
       print("Image captured.")
   
       let photoMetadata = photo.metadata
       // Returns corresponting NSCFNumber. It seems to specify the origin of the image
       //                print("Metadata orientation: ",photoMetadata["Orientation"])
   
       // Returns corresponting NSCFNumber. It seems to specify the origin of the image
       print("Metadata orientation with key: ",photoMetadata[String(kCGImagePropertyOrientation)] as Any)
   
       guard let imageData = photo.fileDataRepresentation() else {
           print("Error while generating image from photo capture data.");
           self.lastPhoto = nil; self.controller.goToProcessing();
           return
   
       }
   
       guard let uiImage = UIImage(data: imageData) else {
           print("Unable to generate UIImage from image data.");
           self.lastPhoto = nil; self.controller.goToProcessing();
           return
   
       }
   
       // generate a corresponding CGImage
       guard let cgImage = uiImage.cgImage else {
           print("Error generating CGImage");self.lastPhoto=nil;return
   
       }
   
       guard let deviceOrientationOnCapture = self.deviceOrientationOnCapture else {
           print("Error retrieving orientation on capture");self.lastPhoto=nil;
           return
   
       }
   
       self.lastPhoto = UIImage(cgImage: cgImage, scale: 1.0, orientation: deviceOrientationOnCapture.getUIImageOrientationFromDevice())
   
       print(self.lastPhoto)
       print("UIImage generated. Orientation:(self.lastPhoto.imageOrientation.rawValue)")
       self.controller.goToProcessing()
   }
   
   
   func photoOutput(_ output: AVCapturePhotoOutput, 
                      willBeginCaptureFor resolvedSettings: AVCaptureResolvedPhotoSettings) 
                      {
       print("Just about to take a photo.")
       // get device orientation on capture
       self.deviceOrientationOnCapture = UIDevice.current.orientation
       print("Device orientation: \(self.deviceOrientationOnCapture.rawValue)")
   }
   

Solution 2

I've had success doing this:

func photoOutput(_ output: AVCapturePhotoOutput, didFinishProcessingPhoto photo: AVCapturePhoto, error: Error?) {

        let cgImage = photo.cgImageRepresentation()!.takeRetainedValue()
        let orientation = photo.metadata[kCGImagePropertyOrientation as String] as! NSNumber
        let uiOrientation = UIImage.Orientation(rawValue: orientation.intValue)!
        let image = UIImage(cgImage: cgImage, scale: 1, orientation: uiOrientation)

}

It's based on what Apple mention in their docs:

Each time you access this method, AVCapturePhoto generates a new CGImageRef. When backed by a compressed container (such as HEIC), the CGImageRepresentation is decoded lazily as needed. When backed by an uncompressed format such as BGRA, it is copied into a separate backing buffer whose lifetime is not tied to that of the AVCapturePhoto. For a 12 megapixel image, a BGRA CGImage represents ~48 megabytes per call. If you only intend to use the CGImage for on-screen rendering, use the previewCGImageRepresentation instead. Note that the physical rotation of the CGImageRef matches that of the main image. Exif orientation has not been applied. If you wish to apply rotation when working with UIImage, you can do so by querying the photo's metadata[kCGImagePropertyOrientation] value, and passing it as the orientation parameter to +[UIImage imageWithCGImage:scale:orientation:]. RAW images always return a CGImageRepresentation of nil. If you wish to make a CGImageRef from a RAW image, use CIRAWFilter in the CoreImage framework.

Solution 3

To create our image with the right orientation we need to enter the correct UIImage.Orientation when we initialize the image.

Its best to use the CGImagePropertyOrientation that comes back from the photoOutput delegate to get the exact orientation the camera session was in when the picture was taken. Only problem here is that while the enum values between UIImage.Orientation and CGImagePropertyOrientation are the same, the raw values are not. Apple suggests a simple mapping to fix this.

https://developer.apple.com/documentation/imageio/cgimagepropertyorientation

Here is my implementation:

AVCapturePhotoCaptureDelegate

func photoOutput(_ output: AVCapturePhotoOutput, didFinishProcessingPhoto photo: AVCapturePhoto, error: Error?) {
        if let _ = error {
            // Handle Error
        } else if let cgImageRepresentation = photo.cgImageRepresentation(),
            let orientationInt = photo.metadata[String(kCGImagePropertyOrientation)] as? UInt32,
            let imageOrientation = UIImage.Orientation.orientation(fromCGOrientationRaw: orientationInt) {

            // Create image with proper orientation
            let cgImage = cgImageRepresentation.takeUnretainedValue()
            let image = UIImage(cgImage: cgImage,
                                scale: 1,
                                orientation: imageOrientation)
        }
    }

Extension for Mapping

extension UIImage.Orientation {

    init(_ cgOrientation: CGImagePropertyOrientation) {
        // we need to map with enum values becuase raw values do not match
        switch cgOrientation {
        case .up: self = .up
        case .upMirrored: self = .upMirrored
        case .down: self = .down
        case .downMirrored: self = .downMirrored
        case .left: self = .left
        case .leftMirrored: self = .leftMirrored
        case .right: self = .right
        case .rightMirrored: self = .rightMirrored
        }
    }


    /// Returns a UIImage.Orientation based on the matching cgOrientation raw value
    static func orientation(fromCGOrientationRaw cgOrientationRaw: UInt32) -> UIImage.Orientation? {
        var orientation: UIImage.Orientation?
        if let cgOrientation = CGImagePropertyOrientation(rawValue: cgOrientationRaw) {
            orientation = UIImage.Orientation(cgOrientation)
        } else {
            orientation = nil // only hit if improper cgOrientation is passed
        }
        return orientation
    }
}

Solution 4

Inside the AVCapturePhoto I’m pretty sure you will find a metadata object of the also called CGImageProperties.
Inside it you will find the EXIF dictionary for orientation the next step is just to take the orientation and create an image according to that.
I do not have experiences in using AVCapturePhotoOutput but I have some using the old way.
Pay attention that the EXIF dictionary is mapped differently in UIImageOrientation.
Here is an article I wrote a lot of time ago, but the main principle are still valid.
This question will point you to some implementations, it's pretty old too, I'm pretty sure that in the latest version they released easier API, but it will still guide you into take the issue.

Solution 5

Updated extension provided by Andre which works with Swift 4.2:

import Foundation
import UIKit

extension UIDeviceOrientation {
    var imageOrientation: UIImage.Orientation {
        switch self {
        case .portrait, .faceUp:                return .right
        case .portraitUpsideDown, .faceDown:    return .left
        case .landscapeLeft:                    return .up
        case .landscapeRight:                   return .down
        case .unknown:                          return .up
        }
    }
}

View more solutions

Comments

[Leo Dabus]

About your orientation extension. Apple has released AVCam updated. There is a similar extension. extension UIDeviceOrientation { var videoOrientation: AVCaptureVideoOrientation? { switch self { case .portrait: return .portrait case .portraitUpsideDown: return .portraitUpsideDown case .landscapeLeft: return .landscapeRight case .landscapeRight: return .landscapeLeft default: return nil } } }

[Anton Belousov]

what is getAVCaptureVideoOrientationFromDevice?

[bmjohns]

Its not a great idea to rely on the deviceOrientationOnCapture property that you set, since the delegate can come back before or after that value has already changed (like taking multiple pictures in a row). Try my answer that relies on the metadata that comes back from the actual delegate call instead stackoverflow.com/a/54225440/753964

[Stunner]

@AntonBelousov he omitted it from his answer but it can be found within his repo here: github.com/agu3rra/Out-Loud

[RawMean]

If iPhone auto rotate is tuned off, we cannot rely on the UIDevice.current.orientation because it always returns .portrait even when the user takes the picture in landscape more.

[David H]

Pretty sure that this doesn't work when rotation lock is on. But none of the other techniques work either, you have to enable the gyroscope and keep track of the device itself. What a hack. I recently burned a DTS incident to find out if there was some other way, Apple laughed at me.

[SoftDesigner]

Actually photo.metadata[kCGImagePropertyOrientation] is CGImagePropertyOrientation not UIImage.Orientation (c) developer.apple.com/documentation/imageio/…

[khjfquantumjj]

doesn't work: orientationInt always returns the same value

[mojuba]

I think it should be (rawValue: orientation.intValue - 1)