Systems and methods for synthesizing high resolution images using images captured by an array of independently controllable imagers
원문보기
IPC분류정보
국가/구분
United States(US) Patent
등록
국제특허분류(IPC7판)
G06T-003/40
H04N-013/00
G06T-005/00
H04N-005/232
출원번호
US-0519659
(2014-10-21)
등록번호
US-9361662
(2016-06-07)
발명자
/ 주소
Lelescu, Dan
Molina, Gabriel
Venkataraman, Kartik
출원인 / 주소
Pelican Imaging Corporation
대리인 / 주소
KPPB LLP
인용정보
피인용 횟수 :
44인용 특허 :
152
초록▼
Systems and methods in accordance with embodiments of the invention are disclosed that use super-resolution (SR) processes to use information from a plurality of low resolution (LR) images captured by an array camera to produce a synthesized higher resolution image. One embodiment includes obtaining
Systems and methods in accordance with embodiments of the invention are disclosed that use super-resolution (SR) processes to use information from a plurality of low resolution (LR) images captured by an array camera to produce a synthesized higher resolution image. One embodiment includes obtaining input images, determining an initial estimate of at least a portion of a high resolution image using a plurality of pixels from the input images, and determining a high resolution image that when mapped through the forward imaging transformation matches the input images to within at least one predetermined criterion using the initial estimate of at least a portion of the high resolution image. In addition, each forward imaging transformation corresponds to the manner in which each imager generates the input images, and the high resolution image has a resolution that is greater than any of the input images.
대표청구항▼
1. An array camera, comprising: an imager array including a plurality of imagers and control circuitry for independently triggering each imager, where: each of the imagers receives light through a separate lens system and includes a sensor;the control circuitry controls the imaging parameters of the
1. An array camera, comprising: an imager array including a plurality of imagers and control circuitry for independently triggering each imager, where: each of the imagers receives light through a separate lens system and includes a sensor;the control circuitry controls the imaging parameters of the sensor of each imager;a processor; andmemory containing an image processing pipeline software application and parameters defining a forward imaging transformation for the imager array;wherein the image processing pipeline software directs the processor to: trigger the imagers in the imager array to obtain a plurality of input images and store the input images in memory, where the input images capture a scene in which depths of points in the imaged scene vary and each of the input images differs from the other input images due to: scene independent geometric distortions inherent to the optics and manufacturing processes used to fabricate each of the plurality of imagers; andscene dependent geometric displacements due to parallax based upon the depths of the points in the imaged scene; anddetermine scene dependent parallax information with respect to the input images based upon disparity relative to a reference point of view resulting from the depths of points in the imaged scene, where the scene dependent parallax information comprises scene dependent geometric transformations;determine an initial estimate of at least a portion of a high resolution image from a plurality of pixels from the input images based upon a total shift for each of the plurality of pixels relative to the reference point of view, where the total shift of a given pixel location is the combination of a scene independent geometric correction determined for the given pixel using geometric calibration data and the scene dependent geometric transformation determined for the given pixel location; anddetermine a high resolution image that when mapped through the forward imaging transformation matches the input images to within at least one predetermined criterion using the initial estimate of at least a portion of the high resolution image;wherein each forward imaging transformation corresponds to the manner in which each imager in the plurality of images captures the input images and comprises applying geometric transformations related to parallax observed due to the depths of points in the imaged scene; andwherein the high resolution image has a resolution that is greater than any of the input images. 2. The array camera of claim 1, wherein the control circuitry controls imaging parameters for the sensor of each imager selected from the group consisting of: exposure times; gain; and black level offset. 3. The array camera of claim 1, wherein the plurality of imagers form a N×M array of imagers. 4. The array camera of claim 1, wherein the imager array captures a single channel of information. 5. The array camera of claim 1, wherein the imager array captures three channels of information. 6. The array camera of claim 5, wherein the imager array includes imagers that sense red light (R), imagers that sense green light (G), and imagers that sense blue light (B). 7. The array camera of claim 1, wherein the imager array captures four channels of information. 8. The array camera of claim 7, wherein the four channels of information are the three RGB channels and a fourth near-IR channel. 9. The array camera of claim 8, wherein the imager array includes imagers that sense red light (R), imagers that sense green light (G), imagers that sense blue light (B), and imagers that sense near-IR wavelengths. 10. The array camera of claim 1, the image processing pipeline software directs the processor to encode the high resolution image in accordance with the JPEG standard. 11. The array camera of claim 10, wherein the image processing pipeline software directs the processor to store the encoded high resolution image in a file format selected from the group consisting of the JPEG Interchange Format (JIF), the JPEG File Interchange Format (JFIF), or the Exchangeable image file format (Exif). 12. The array camera of claim 1, wherein imagers in the plurality of imagers have different fields of view resulting in a change in magnification of the image captured by the underlying sensor of the imager. 13. The array camera of claim 1, wherein the image processing pipeline software directs the processor to generate a depth map for the high resolution image. 14. The array camera of claim 13, wherein generating the depth map further comprises: determining depth information for pixels in the high resolution image based upon the input images, parallax information, and the characteristics of the imager array; andinterpolating the depth information to obtain depth information for every pixel in the high resolution image. 15. The array camera of claim 1, wherein the image processing pipeline software directs the processor to: generate a focus map for the high resolution image; andperform dynamic refocus of the high-resolution image by rendering the high resolution image using the focus map. 16. The array camera of claim 15, wherein the focus map identifies pixels having depths in the depth map that are within a specified depth of a defined focal plane. 17. The array camera of claim 1, wherein the forward imaging transformation further comprises applying scene independent geometric transformations related to the different geometries of each of the imagers in the plurality of imagers. 18. The array camera of claim 1, wherein the forward imaging transformation further comprises applying photometric transformations related to the different photometric characteristics of each of the imagers in the plurality of imagers. 19. The array camera of claim 1, wherein the image processing pipeline software directs the processor to: determine the initial estimate of at least a portion of the high resolution image comprises using an imaging prior including the geometric calibration data and applying scene independent geometric corrections to the input images using the geometric calibration data to obtain geometrically registered input images; anddetermine the high resolution image that when mapped through the forward imaging transformation matches the input images to at least one predetermined criterion comprises determining the high resolution image that when mapped through the forward imaging transformation matches the geometrically registered input images to within at least one predetermined criterion. 20. The array camera of claim 1, wherein: the depths of points in the imaged scene vary due to the presence of foreground and background objects;each of the input images also differ from the other input images due to occlusion zones surrounding foreground objects; andthe scene dependent parallax information also includes occlusion maps. 21. The array camera of claim 1, wherein the image processing pipeline software directs the processor to determine an initial estimate of at least a portion of a high resolution image from a plurality of pixels from the input images by fusing at least portions of the input images to form the initial estimate of at least one portion of the high resolution image. 22. The array camera of claim 21, wherein the image processing pipeline software directs the processor to fuse at least portions of the input images to form the initial estimate of at least one portion of the high resolution image by: populating a high resolution grid corresponding to the pixel locations of the at least a portion of the initial estimate of the high resolution image with pixels from the input images using the total shift for the pixels; andinterpolating the high resolution grid to obtain filtered pixel values for each pixel in the initial estimate of the high resolution image. 23. The array camera of claim 1, wherein the image processing pipeline software directs the processor to determine a high resolution image that when mapped through the forward imaging transformation matches the input images to within at least one predetermined criterion using the initial estimate of at least a portion of the high resolution image by: transforming the initial estimate of at least a portion of the high resolution image using at least one forward imaging transformation;comparing the transformed initial estimate of at least a portion of the high resolution image to at least a portion of at least one input image; andrefining the estimate of the high resolution image based upon the comparison. 24. The array camera of claim 23, wherein the image processing pipeline software directs the processor to determine a high resolution image that when mapped through the forward imaging transformation matches the input images to within at least one predetermined criterion using the initial estimate of at least a portion of the high resolution image by transforming, comparing and refining estimates until the at least one predetermined criterion is satisfied. 25. The array camera of claim 1, wherein the image processing pipeline software directs the processor to determine a high resolution image that when mapped through the forward imaging transformation matches the input images to within at least one predetermined criterion using the initial estimate of at least a portion of the high resolution image by: identifying pixels in the initial estimate of at least a portion of the high resolution image corresponding to pixels in at least one input image using at least one forward imaging transformation;comparing the corresponding pixels; andrefining the estimate of the high resolution image based upon the comparison. 26. The array camera of claim 1, wherein the image processing pipeline software directs the processor to determine a high resolution image that when mapped through the forward imaging transformation matches the input images to within at least one predetermined criterion using the initial estimate of at least a portion of the high resolution image by: generating an estimate of at least a portion of the high resolution image; andapplying an intra-channel prior filter to the estimate of at least a portion of the high resolution image, where the intra-channel prior filter is configured to preserve edges while removing noise. 27. The array camera of claim 1, wherein: the imager array captures images in multiple color channels; andthe initial estimate of at least a portion of a high resolution image is an initial estimate of at least a portion of a high resolution image in a first color channel.
연구과제 타임라인
LOADING...
LOADING...
LOADING...
LOADING...
LOADING...
이 특허에 인용된 특허 (152)
Wilburn, Bennett; Joshi, Neel; Levoy, Marc C.; Horowitz, Mark, Apparatus and method for capturing a scene using staggered triggering of dense camera arrays.
Iwase Toshihiro (Nara JPX) Kanekura Hiroshi (Yamatokouriyama JPX), Apparatus for and method of converting a sampling frequency according to a data driven type processing.
Boisvert, David Michael; McMahon, Andrew Kenneth John, CCD output processing stage that amplifies signals from colored pixels based on the conversion efficiency of the colored pixels.
Venkataraman, Kartik; Jabbi, Amandeep S.; Mullis, Robert H.; Duparre, Jacques; Hu, Shane Ching-Feng, Capturing and processing of images including occlusions captured by arrays of luma and chroma cameras.
Venkataraman, Kartik; Jabbi, Amandeep S.; Mullis, Robert H., Capturing and processing of images using monolithic camera array with heterogeneous imagers.
Venkataraman, Kartik; Jabbi, Amandeep S.; Mullis, Robert H.; Duparre, Jacques; Hu, Shane Ching-Feng, Capturing and processing of images using monolithic camera array with heterogeneous imagers.
Yamashita,Syugo; Murata,Haruhiko; Iinuma,Toshiya; Nakashima,Mitsuo; Mori,Takayuki, Device and method for converting two-dimensional video to three-dimensional video.
Pertsel, Shimon; Meitav, Ohad; Pozniansky, Eli; Galil, Erez, Digital camera with selectively increased dynamic range by control of parameters during image acquisition.
Ward, Gregory John; Seetzen, Helge; Heidrich, Wolfgang, Electronic camera having multiple sensors for capturing high dynamic range images and related methods.
Abell Gurdon R. (West Woodstock CT) Cook Francis J. (Topsfield MA) Howes Peter D. (Sudbury MA), Method and apparatus for arraying image sensor modules.
Sawhney,Harpreet Singh; Tao,Hai; Kumar,Rakesh; Hanna,Keith, Method and apparatus for synthesizing new video and/or still imagery from a collection of real video and/or still imagery.
Han, Hee-chul; Choi, Yang-lim; Cho, Seung-ki, Method of generating image data by an image device including a plurality of lenses and apparatus for generating image data.
Alexander David H. (Santa Monica CA) Hershman George H. (Carlsbad CA) Jack Michael D. (Carlsbad CA) Koda N. John (Vista CA) Lloyd Randahl B. (San Marcos CA), Monolithic imager for near-IR.
Hornbaker ; III Cecil V. (New Carrolton MD) Driggers Thomas C. (Falls Church VA) Bindon Edward W. (Fairfax VA), Scanning apparatus using multiple CCD arrays and related method.
Ciurea, Florian; Venkataraman, Kartik; Molina, Gabriel; Lelescu, Dan, System and methods for measuring depth using an array camera employing a bayer filter.
Lelescu, Dan; Molina, Gabriel; Venkataraman, Kartik, Systems and methods for dynamic refocusing of high resolution images generated using images captured by a plurality of imagers.
Ciurea, Florian; Venkataraman, Kartik; Molina, Gabriel; Lelescu, Dan, Systems and methods for generating depth maps and corresponding confidence maps indicating depth estimation reliability.
Venkataraman, Kartik; Jabbi, Amandeep S.; Mullis, Robert H.; Duparre, Jacques; Hu, Shane Ching-Feng, Systems and methods for generating depth maps using a set of images containing a baseline image.
Ciurea, Florian; Venkataraman, Kartik; Molina, Gabriel; Lelescu, Dan, Systems and methods for measuring depth in the presence of occlusions using a subset of images.
Ciurea, Florian; Venkataraman, Kartik; Molina, Gabriel; Lelescu, Dan, Systems and methods for parallax detection and correction in images captured using array cameras that contain occlusions using subsets of images to perform depth estimation.
Venkataraman, Kartik; Jabbi, Amandeep S.; Mullis, Robert H., Systems and methods for parallax measurement using camera arrays incorporating 3 x 3 camera configurations.
Ciurea, Florian; Venkataraman, Kartik; Molina, Gabriel; Lelescu, Dan, Systems and methods for performing depth estimation using image data from multiple spectral channels.
Venkataraman, Kartik; Jabbi, Amandeep S.; Mullis, Robert H., Systems and methods for performing post capture refocus using images captured by camera arrays.
Venkataraman, Kartik; Jabbi, Amandeep S.; Mullis, Robert H.; Duparre, Jacques; Hu, Shane Ching-Feng, Systems and methods for synthesizing higher resolution images using a set of images containing a baseline image.
Ludwig, Lester F., Vignetted optoelectronic array for use in synthetic image formation via signal processing, lensless cameras, and integrated camera-displays.
Rieger Albert,DEX ; Barclay David ; Chapman Steven ; Kellner Heinz-Andreas,DEX ; Reibl Michael,DEX ; Rydelek James G. ; Schweizer Andreas,DEX, Watertight body for accommodating a photographic camera.
Venkataraman, Kartik; Gallagher, Paul; Jain, Ankit K.; Nisenzon, Semyon; Lelescu, Dan; Ciurea, Florian; Molina, Gabriel, Autofocus system for a conventional camera that uses depth information from an array camera.
Venkataraman, Kartik; Jabbi, Amandeep S.; Mullis, Robert H.; Duparre, Jacques; Hu, Shane Ching-Feng, Capturing and processing of images including occlusions focused on an image sensor by a lens stack array.
Venkataraman, Kartik; Jabbi, Amandeep S.; Mullis, Robert H.; Duparre, Jacques; Hu, Shane Ching-Feng, Capturing and processing of images using camera array incorperating Bayer cameras having different fields of view.
Srikanth, Manohar; Ramamoorthi, Ravi; Venkataraman, Kartik; Chatterjee, Priyam, System and methods for depth regularization and semiautomatic interactive matting using RGB-D images.
Nayar, Shree; Venkataraman, Kartik; Pain, Bedabrata; Lelescu, Dan, Systems and methods for controlling aliasing in images captured by an array camera for use in super resolution processing using pixel apertures.
Lelescu, Dan; Venkataraman, Kartik, Systems and methods for controlling aliasing in images captured by an array camera for use in super-resolution processing.
Duparre, Jacques; McMahon, Andrew Kenneth John; Lelescu, Dan; Venkataraman, Kartik; Molina, Gabriel, Systems and methods for detecting defective camera arrays and optic arrays.
Ciurea, Florian; Venkataraman, Kartik; Molina, Gabriel; Lelescu, Dan, Systems and methods for estimating depth and visibility from a reference viewpoint for pixels in a set of images captured from different viewpoints.
Venkataraman, Kartik; Lelescu, Dan; Molina, Gabriel, Systems and methods for generating compressed light field representation data using captured light fields, array geometry, and parallax information.
Venkataraman, Kartik; Jabbi, Amandeep S.; Mullis, Robert H., Systems and methods for generating depth maps using a camera arrays incorporating monochrome and color cameras.
Venkataraman, Kartik; Jabbi, Amandeep S.; Mullis, Robert H., Systems and methods for generating depth maps using a camera arrays incorporating monochrome and color cameras.
Venkataraman, Kartik; Jabbi, Amandeep S.; Mullis, Robert H., Systems and methods for generating depth maps using images captured by camera arrays incorporating cameras having different fields of view.
Duparre, Jacques; McMahon, Andrew Kenneth John; Lelescu, Dan, Systems and methods for manufacturing camera modules using active alignment of lens stack arrays and sensors.
Venkataraman, Kartik; Jabbi, Amandeep S.; Mullis, Robert H., Systems and methods for measuring depth using images captured by a camera array including cameras surrounding a central camera.
Venkataraman, Kartik; Huang, Yusong; Jain, Ankit K.; Chatterjee, Priyam, Systems and methods for performing high speed video capture and depth estimation using array cameras.
Lelescu, Dan; Duong, Thang, Systems and methods for synthesizing high resolution images using image deconvolution based on motion and depth information.
Venkataraman, Kartik; Nisenzon, Semyon; Chatterjee, Priyam; Molina, Gabriel, Systems and methods for synthesizing images from image data captured by an array camera using restricted depth of field depth maps in which depth estimation precision varies.
Venkataraman, Kartik; Nisenzon, Semyon; Chatterjee, Priyam; Molina, Gabriel, Systems and methods for synthesizing images from image data captured by an array camera using restricted depth of field depth maps in which depth estimation precision varies.
※ AI-Helper는 부적절한 답변을 할 수 있습니다.