Systems and methods for decoding light field image files
IPC분류정보
국가/구분
United States(US) Patent
등록
국제특허분류(IPC7판)
G06K-009/36
H04N-007/32
G06T-009/00
G06T-009/20
G06K-009/46
출원번호
US-0955411
(2013-07-31)
등록번호
US-8831367
(2014-09-09)
발명자
/ 주소
Venkataraman, Kartik
Nisenzon, Semyon
출원인 / 주소
Pelican Imaging Corporation
대리인 / 주소
KPPB LLP
인용정보
피인용 횟수 :
120인용 특허 :
67
초록▼
Systems and methods in accordance with embodiments of the invention are configured to render images using light field image files containing an image synthesized from light field image data and metadata describing the image that includes a depth map. One embodiment of the invention includes a proces
Systems and methods in accordance with embodiments of the invention are configured to render images using light field image files containing an image synthesized from light field image data and metadata describing the image that includes a depth map. One embodiment of the invention includes a processor and memory containing a rendering application and a light field image file including an encoded image and metadata describing the encoded image, where the metadata comprises a depth map that specifies depths from the reference viewpoint for pixels in the encoded image. In addition, the rendering application configures the processor to: locate the encoded image within the light field image file; decode the encoded image; locate the metadata within the light field image file; and post process the decoded image by modifying the pixels based on the depths indicated within the depth map to create a rendered image.
대표청구항▼
1. A system for rendering an image using a light field image file including an encoded image and metadata describing the encoded image, where the metadata comprises a depth map that specifies depths from the reference viewpoint for pixels in the encoded image, the system comprising: a processor; and
1. A system for rendering an image using a light field image file including an encoded image and metadata describing the encoded image, where the metadata comprises a depth map that specifies depths from the reference viewpoint for pixels in the encoded image, the system comprising: a processor; andmemory containing a rendering application and a light field image file including an encoded image and metadata describing the encoded image, where the metadata comprises a depth map that specifies depths from the reference viewpoint for pixels in the encoded image;wherein the rendering application configures the processor to: locate the encoded image within the light field image file;decode the encoded image;locate the metadata within the light field image file; andpost process the decoded image by modifying the pixels based on the depths indicated within the depth map to create a rendered image. 2. The system of claim 1, wherein the rendering application configuring the processor to post process the decoded image by modifying the pixels based on the depths indicated within the depth map to create the rendered image comprises applying a depth based effect to the pixels of the decoded image. 3. The system of claim 2, wherein the depth based effect comprises at least one effect selected from the group consisting of: modifying the focal plane of the decoded image;modifying the depth of field of the decoded image;modifying the blur in out-of-focus regions of the decoded image;locally varying the depth of field of the decoded image;creating multiple focus areas at different depths within the decoded image; andapplying a depth related blur. 4. The system of claim 2, wherein the metadata is located within an Application marker segment within the light field image file. 5. The system of claim 4, wherein the Application marker segment is identified using the APPS marker. 6. The system of claim 4, wherein: the depth map is encoded in accordance with the JPEG standard using lossless compression; andthe rendering application configures the processor to: locate at least one Application marker segment containing the metadata comprising the depth map; anddecode the depth map using the JPEG decoder. 7. The system of claim 4, wherein: the encoded image is an image of a scene synthesized from a reference viewpoint using a plurality of lower resolution images that capture the scene from different viewpoints;the metadata in the light field image file further comprises pixels from the lower resolution images that are occluded in the reference viewpoint;the rendering application configures the processor to locate at least one Application marker segment containing the metadata comprising the pixels from the lower resolution images that are occluded in the reference viewpoint; andthe rendering application configuring the processor to post process the decoded image by modifying the pixels based on the depth of the pixel indicated within the depth map to create the rendered image comprises rendering an image from a different viewpoint using the depth map and the pixels from the lower resolution images that are occluded in the reference viewpoint. 8. The system of claim 7, wherein: the metadata in the light field image file includes descriptions of the pixels from the lower resolution images that are occluded in the reference viewpoint including the color, location, and depth of the occluded pixels; andrendering an image from a different viewpoint using the depth map and the pixels from the lower resolution images that are occluded in the reference viewpoint further comprises: shifting pixels from the decoded image and the occluded pixels in the metadata to the different viewpoint based upon the depths of the pixels;determining pixel occlusions; andgenerating an image from the different viewpoint using the shifted pixels that are not occluded and by interpolating to fill in missing pixels using adjacent pixels that are not occluded. 9. The system of claim 7, wherein the image rendered from the different viewpoint is part of a stereo pair of images. 10. The system of claim 7, wherein: the metadata in the light field image file further comprises a confidence map for the depth map, where the confidence map indicates the reliability of the depth values provided for pixels by the depth map; andrendering an image from a different viewpoint using the depth map and the pixels from the lower resolution images that are occluded in the reference viewpoint further comprises applying at least one filter based upon the confidence map. 11. The system of claim 7, wherein: the metadata in the light field image file further comprises an edge map that indicates pixels in the decoded image that lie on a discontinuity; andrendering an image from a different viewpoint using the depth map and the pixels from the lower resolution images that are occluded in the reference viewpoint further comprises applying at least one filter based upon the edge map. 12. The system of claim 1, wherein: the encoded image is an image of a scene synthesized from a reference viewpoint using a plurality of lower resolution images that capture the scene from different viewpoints;the metadata in the light field image file further comprises pixels from the lower resolution images that are occluded in the reference viewpoint; andthe rendering application configuring the processor to post process the decoded image by modifying the pixels based on the depths indicated within the depth map to create the rendered image comprises rendering an image from a different viewpoint using the depth map and the pixels from the lower resolution images that are occluded in the reference viewpoint. 13. The system of claim 12, wherein: the metadata in the light field image file includes descriptions of the pixels from the lower resolution images that are occluded in the reference viewpoint including the color, location, and depth of the occluded pixels; andrendering an image from a different viewpoint using the depth map and the pixels from the lower resolution images that are occluded in the reference viewpoint further comprises: shifting pixels from the decoded image and the occluded pixels in the metadata to the different viewpoint based upon the depths of the pixels;determining pixel occlusions; andgenerating an image from the different viewpoint using the shifted pixels that are not occluded and by interpolating to fill in missing pixels using adjacent pixels that are not occluded. 14. The system of claim 12, wherein the image rendered from the different viewpoint is part of a stereo pair of images. 15. The system of claim 12, wherein: the metadata in the light field image file further comprises a confidence map for the depth map, where the confidence map indicates the reliability of the depth values provided for pixels by the depth map; andrendering an image from a different viewpoint using the depth map and the pixels from the lower resolution images that are occluded in the reference viewpoint further comprises applying at least one filter based upon the confidence map. 16. The system of claim 12, wherein: the metadata in the light field image file further comprises an edge map that indicates pixels in the decoded image that lie on a discontinuity; andrendering an image from a different viewpoint using the depth map and the pixels from the lower resolution images that are occluded in the reference viewpoint further comprises applying at least one filter based upon the edge map. 17. The system of claim 16, wherein the edge map identifies whether a pixel lies on an intensity discontinuity. 18. The system of claim 16, wherein the edge map identifies whether a pixel lies on an intensity and depth discontinuity. 19. The system of claim 12, wherein: the metadata in the light field image file further comprises a missing pixel map that indicates pixels in the decoded image that do not correspond to a pixel from the plurality of low resolution images of the scene and that are generated by interpolating pixel values from adjacent pixels in the synthesized image; andrendering an image from a different viewpoint using the depth map and the pixels from the lower resolution images that are occluded in the reference viewpoint further comprises ignoring pixels based upon the missing pixel map. 20. The system of claim 1, wherein: the light field image file conforms to the JFIF standard and the encoded image is encoded in accordance with the JPEG standard;the memory comprises a JPEG decoder application; andthe rendering application configures the processor to: locate the encoded image by locating a Start of Image marker within the light field image file; anddecode the encoded image using the JPEG decoder.
연구과제 타임라인
LOADING...
LOADING...
LOADING...
LOADING...
LOADING...
이 특허에 인용된 특허 (67)
Wilburn, Bennett; Joshi, Neel; Levoy, Marc C.; Horowitz, Mark, Apparatus and method for capturing a scene using staggered triggering of dense camera arrays.
Boisvert, David Michael; McMahon, Andrew Kenneth John, CCD output processing stage that amplifies signals from colored pixels based on the conversion efficiency of the colored pixels.
Abell Gurdon R. (West Woodstock CT) Cook Francis J. (Topsfield MA) Howes Peter D. (Sudbury MA), Method and apparatus for arraying image sensor modules.
Sawhney,Harpreet Singh; Tao,Hai; Kumar,Rakesh; Hanna,Keith, Method and apparatus for synthesizing new video and/or still imagery from a collection of real video and/or still imagery.
Alexander David H. (Santa Monica CA) Hershman George H. (Carlsbad CA) Jack Michael D. (Carlsbad CA) Koda N. John (Vista CA) Lloyd Randahl B. (San Marcos CA), Monolithic imager for near-IR.
Hornbaker ; III Cecil V. (New Carrolton MD) Driggers Thomas C. (Falls Church VA) Bindon Edward W. (Fairfax VA), Scanning apparatus using multiple CCD arrays and related method.
Ciurea, Florian; Venkataraman, Kartik; Molina, Gabriel; Lelescu, Dan, Systems and methods for performing depth estimation using image data from multiple spectral channels.
Rieger Albert,DEX ; Barclay David ; Chapman Steven ; Kellner Heinz-Andreas,DEX ; Reibl Michael,DEX ; Rydelek James G. ; Schweizer Andreas,DEX, Watertight body for accommodating a photographic camera.
Venkataraman, Kartik; Gallagher, Paul; Jain, Ankit; Nisenzon, Semyon; Lelescu, Dan; Ciurea, Florian; Molina, Gabriel, Array cameras including an array camera module augmented with a separate camera.
Duparre, Jacques; Lelescu, Dan; Venkataraman, Kartik, Array cameras incorporating monolithic array camera modules with high MTF lens stacks for capture of images used in super-resolution processing.
Duparre, Jacques; Lelescu, Dan; Venkataraman, Kartik, Array cameras incorporating optics with modulation transfer functions greater than sensor Nyquist frequency for capture of images used in super-resolution processing.
Venkataraman, Kartik; Gallagher, Paul; Jain, Ankit K.; Nisenzon, Semyon; Lelescu, Dan; Ciurea, Florian; Molina, Gabriel, Autofocus system for a conventional camera that uses depth information from an array camera.
Venkataraman, Kartik; Jabbi, Amandeep S.; Mullis, Robert H., Capturing and processing of images captured by camera arrays including cameras dedicated to sampling luma and cameras dedicated to sampling chroma.
Venkataraman, Kartik; Jabbi, Amandeep S.; Mullis, Robert H., Capturing and processing of images captured by camera arrays including heterogeneous optics.
Venkataraman, Kartik; Jabbi, Amandeep S.; Mullis, Robert H.; Duparre, Jacques; Hu, Shane Ching-Feng, Capturing and processing of images including occlusions captured by camera arrays.
Venkataraman, Kartik; Jabbi, Amandeep S.; Mullis, Robert H.; Duparre, Jacques; Hu, Shane Ching-Feng, Capturing and processing of images including occlusions focused on an image sensor by a lens stack array.
Venkataraman, Kartik; Jabbi, Amandeep S.; Mullis, Robert H.; Duparre, Jacques; Hu, Shane Ching-Feng, Capturing and processing of images including occlusions focused on an image sensor by a lens stack array.
Venkataraman, Kartik; Jabbi, Amandeep S.; Mullis, Robert H.; Duparre, Jacques; Hu, Shane Ching-Feng, Capturing and processing of images using camera array incorperating Bayer cameras having different fields of view.
Venkataraman, Kartik; Jabbi, Amandeep S.; Mullis, Robert H., Capturing and processing of images using monolithic camera array with heterogeneous imagers.
Venkataraman, Kartik; Jabbi, Amandeep S.; Mullis, Robert H., Capturing and processing of near-IR images including occlusions using camera arrays incorporating near-IR light sources.
Srikanth, Manohar; Ramamoorthi, Ravi; Venkataraman, Kartik; Chatterjee, Priyam, System and methods for depth regularization and semiautomatic interactive matting using RGB-D images.
Ciurea, Florian; Venkataraman, Kartik; Molina, Gabriel; Lelescu, Dan, System and methods for measuring depth using an array camera employing a bayer filter.
Nayar, Shree; Venkataraman, Kartik; Pain, Bedabrata; Lelescu, Dan, Systems and methods for controlling aliasing in images captured by an array camera for use in super resolution processing using pixel apertures.
Lelescu, Dan; Venkataraman, Kartik, Systems and methods for controlling aliasing in images captured by an array camera for use in super-resolution processing.
Lelescu, Dan; Venkataraman, Kartik, Systems and methods for controlling aliasing in images captured by an array camera for use in super-resolution processing.
Duparré, Jacques, Systems and methods for correcting for warpage of a sensor array in an array camera module by introducing warpage into a focal plane of a lens stack array.
Duparre, Jacques; McMahon, Andrew Kenneth John; Lelescu, Dan; Venkataraman, Kartik; Molina, Gabriel, Systems and methods for detecting defective camera arrays and optic arrays.
Duparre, Jacques; McMahon, Andrew Kenneth John; Lelescu, Dan; Venkataraman, Kartik; Molina, Gabriel, Systems and methods for detecting defective camera arrays and optic arrays.
Lelescu, Dan; Molina, Gabriel; Venkataraman, Kartik, Systems and methods for dynamic refocusing of high resolution images generated using images captured by a plurality of imagers.
Ciurea, Florian; Venkataraman, Kartik; Molina, Gabriel; Lelescu, Dan, Systems and methods for estimating depth and visibility from a reference viewpoint for pixels in a set of images captured from different viewpoints.
Ciurea, Florian; Venkataraman, Kartik; Molina, Gabriel; Lelescu, Dan, Systems and methods for estimating depth and visibility from a reference viewpoint for pixels in a set of images captured from different viewpoints.
Venkataraman, Kartik; Lelescu, Dan; Molina, Gabriel, Systems and methods for generating compressed light field representation data using captured light fields, array geometry, and parallax information.
Venkataraman, Kartik; Lelescu, Dan; Molina, Gabriel, Systems and methods for generating compressed light field representation data using captured light fields, array geometry, and parallax information.
Ciurea, Florian; Venkataraman, Kartik; Molina, Gabriel; Lelescu, Dan, Systems and methods for generating depth maps and corresponding confidence maps indicating depth estimation reliability.
Venkataraman, Kartik; Jabbi, Amandeep S.; Mullis, Robert H., Systems and methods for generating depth maps using a camera arrays incorporating monochrome and color cameras.
Venkataraman, Kartik; Jabbi, Amandeep S.; Mullis, Robert H., Systems and methods for generating depth maps using a camera arrays incorporating monochrome and color cameras.
Venkataraman, Kartik; Jabbi, Amandeep S.; Mullis, Robert H.; Duparre, Jacques; Hu, Shane Ching-Feng, Systems and methods for generating depth maps using a set of images containing a baseline image.
Venkataraman, Kartik; Jabbi, Amandeep S.; Mullis, Robert H., Systems and methods for generating depth maps using images captured by camera arrays incorporating cameras having different fields of view.
Duparre, Jacques; McMahon, Andrew Kenneth John; Lelescu, Dan, Systems and methods for manufacturing camera modules using active alignment of lens stack arrays and sensors.
Duparre, Jacques; McMahon, Andrew Kenneth John; Lelescu, Dan, Systems and methods for manufacturing camera modules using active alignment of lens stack arrays and sensors.
Ciurea, Florian; Venkataraman, Kartik; Molina, Gabriel; Lelescu, Dan, Systems and methods for measuring depth in the presence of occlusions using a subset of images.
Ciurea, Florian; Venkataraman, Kartik; Molina, Gabriel; Lelescu, Dan, Systems and methods for measuring depth using an array of independently controllable cameras.
Venkataraman, Kartik; Jabbi, Amandeep S.; Mullis, Robert H., Systems and methods for measuring depth using images captured by a camera array including cameras surrounding a central camera.
Venkataraman, Kartik; Jabbi, Amandeep S.; Mullis, Robert H., Systems and methods for measuring depth using images captured by monolithic camera arrays including at least one bayer camera.
Venkataraman, Kartik; Jabbi, Amandeep S.; Mullis, Robert H.; Duparre, Jacques; Hu, Shane Ching-Feng, Systems and methods for normalizing image data captured by camera arrays.
Venkataraman, Kartik; Jabbi, Amandeep S.; Mullis, Robert H., Systems and methods for parallax measurement using camera arrays incorporating 3 x 3 camera configurations.
Venkataraman, Kartik; Huang, Yusong; Jain, Ankit K.; Chatterjee, Priyam, Systems and methods for performing high speed video capture and depth estimation using array cameras.
Venkataraman, Kartik; Jabbi, Amandeep S.; Mullis, Robert H., Systems and methods for performing post capture refocus using images captured by camera arrays.
Venkataraman, Kartik; Jabbi, Amandeep S.; Mullis, Robert H., Systems and methods for performing post capture refocus using images captured by camera arrays.
Lelescu, Dan; Molina, Gabriel; Venkataraman, Kartik, Systems and methods for synthesizing high resolution images using a set of geometrically registered images.
Lelescu, Dan; Duong, Thang, Systems and methods for synthesizing high resolution images using image deconvolution based on motion and depth information.
Lelescu, Dan; Molina, Gabriel; Venkataraman, Kartik, Systems and methods for synthesizing high resolution images using images captured by an array of independently controllable imagers.
Venkataraman, Kartik; Jabbi, Amandeep S.; Mullis, Robert H.; Duparre, Jacques; Hu, Shane Ching-Feng, Systems and methods for synthesizing higher resolution images using a set of images containing a baseline image.
Venkataraman, Kartik; Jabbi, Amandeep S.; Mullis, Robert H.; Duparre, Jacques; Hu, Shane Ching-Feng, Systems and methods for synthesizing higher resolution images using a set of images containing a baseline image.
Venkataraman, Kartik; Jabbi, Amandeep S.; Mullis, Robert H., Systems and methods for synthesizing higher resolution images using images captured by camera arrays.
Venkataraman, Kartik; Jabbi, Amandeep S.; Mullis, Robert H., Systems and methods for synthesizing higher resolution images using images captured by camera arrays.
Venkataraman, Kartik; Nisenzon, Semyon; Chatterjee, Priyam; Molina, Gabriel, Systems and methods for synthesizing images from image data captured by an array camera using restricted depth of field depth maps in which depth estimation precision varies.
Venkataraman, Kartik; Nisenzon, Semyon; Chatterjee, Priyam; Molina, Gabriel, Systems and methods for synthesizing images from image data captured by an array camera using restricted depth of field depth maps in which depth estimation precision varies.
※ AI-Helper는 부적절한 답변을 할 수 있습니다.