How to grab YUV formatted video from the camera, display it and process it
Answering my own question. this solved the problem I had (which was to grab yuv output, display it and process it), although its not exactly the answer to the question:
To grab YUV output from the camera:
AVCaptureVideoDataOutput *videoOut = [[AVCaptureVideoDataOutput alloc] init];
[videoOut setAlwaysDiscardsLateVideoFrames:YES];
[videoOut setVideoSettings:[NSDictionary dictionaryWithObject:[NSNumber numberWithInt:kCVPixelFormatType_420YpCbCr8BiPlanarVideoRange] forKey:(id)kCVPixelBufferPixelFormatTypeKey]];
To display it as is, use AVCaptureVideoPreviewLayer, it does not require any much code. (You can see the FindMyiCon sample in the WWDC samples pack for example).
To process the YUV y channel (bi-planer in this case so it's all in a single chunk, you can also use memcpy instead of looping) :
- (void)processPixelBuffer: (CVImageBufferRef)pixelBuffer {
CVPixelBufferLockBaseAddress( pixelBuffer, 0 );
int bufferHeight = CVPixelBufferGetHeight(pixelBuffer);
int bufferWidth = CVPixelBufferGetWidth(pixelBuffer);
// allocate space for ychannel, reallocating as needed.
if (bufferWidth != y_channel.width || bufferHeight != y_channel.height)
{
if (y_channel.data) free(y_channel.data);
y_channel.width = bufferWidth;
y_channel.height = bufferHeight;
y_channel.data = malloc(y_channel.width * y_channel.height);
}
uint8_t *yc = CVPixelBufferGetBaseAddressOfPlane(pixelBuffer, 0);
int total = bufferWidth * bufferHeight;
for(int k=0;k<total;k++)
{
y_channel.data[k] = yc[k++]; // copy y channel
}
CVPixelBufferUnlockBaseAddress( pixelBuffer, 0 );
}
Comments
-
Omry Yadan almost 2 years
I am writing an iphone (IOS 4) program that capture live video from the camera and process it in real time.
I prefer to capture in kCVPixelFormatType_420YpCbCr8BiPlanarFullRange format for easier processing (I need to process the Y Channel). how do I display data in this format? I suppose I need to somehow convert it to a UIImage and then put it in some ImageView?
Currently I have code that displays kCVPixelFormatType_32BGRA data, but naturally it does not work with kCVPixelFormatType_420YpCbCr8BiPlanarFullRange.
This is the code I use now for the transformation, any help/sample on how to do the same for kCVPixelFormatType_420YpCbCr8BiPlanarFullRange will be appreciated. (Also criticism of my current method).
// Create a UIImage from sample buffer data - (UIImage *) imageFromSampleBuffer:(CMSampleBufferRef) sampleBuffer { // Get a CMSampleBuffer's Core Video image buffer for the media data CVImageBufferRef imageBuffer = CMSampleBufferGetImageBuffer(sampleBuffer); // Lock the base address of the pixel buffer CVPixelBufferLockBaseAddress(imageBuffer, 0); // Get the number of bytes per row for the pixel buffer void *baseAddress = CVPixelBufferGetBaseAddress(imageBuffer); // Get the number of bytes per row for the pixel buffer size_t bytesPerRow = CVPixelBufferGetBytesPerRow(imageBuffer); // Get the pixel buffer width and height size_t width = CVPixelBufferGetWidth(imageBuffer); size_t height = CVPixelBufferGetHeight(imageBuffer); // Create a device-dependent RGB color space CGColorSpaceRef colorSpace = CGColorSpaceCreateDeviceRGB(); // Create a bitmap graphics context with the sample buffer data CGContextRef context = CGBitmapContextCreate(baseAddress, width, height, 8, bytesPerRow, colorSpace, kCGBitmapByteOrder32Little | kCGImageAlphaPremultipliedFirst); // Create a Quartz image from the pixel data in the bitmap graphics context CGImageRef quartzImage = CGBitmapContextCreateImage(context); // Unlock the pixel buffer CVPixelBufferUnlockBaseAddress(imageBuffer,0); // Free up the context and color space CGContextRelease(context); CGColorSpaceRelease(colorSpace); // Create an image object from the Quartz image UIImage *image = [UIImage imageWithCGImage:quartzImage]; // Release the Quartz image CGImageRelease(quartzImage); return (image); }
-
Volodymyr Kulyk almost 4 yearscould you please provide link to sample project FindMyiCon or at least WWDC video with that topic?
-
Omry Yadan almost 4 yearsSorry, it's been nearly 10 years, I don't really work on this now.