Systems and methods for estimating depth using ad hoc stereo array cameras
원문보기
IPC분류정보
국가/구분
United States(US) Patent
등록
국제특허분류(IPC7판)
H04N-013/02
G06T-007/00
G06T-007/20
출원번호
US-0705903
(2015-05-06)
등록번호
US-9602805
(2017-03-21)
발명자
/ 주소
Venkataraman, Kartik
Gallagher, Paul
Jain, Ankit
Nisenzon, Semyon
출원인 / 주소
FotoNation Cayman Limited
대리인 / 주소
KPPB LLP
인용정보
피인용 횟수 :
21인용 특허 :
103
초록▼
Systems and methods for stereo imaging with camera arrays in accordance with embodiments of the invention are disclosed. In one embodiment, a method of generating depth information for an object using two or more array cameras that each include a plurality of imagers includes obtaining a first set o
Systems and methods for stereo imaging with camera arrays in accordance with embodiments of the invention are disclosed. In one embodiment, a method of generating depth information for an object using two or more array cameras that each include a plurality of imagers includes obtaining a first set of image data captured from a first set of viewpoints, identifying an object in the first set of image data, determining a first depth measurement, determining whether the first depth measurement is above a threshold, and when the depth is above the threshold: obtaining a second set of image data of the same scene from a second set of viewpoints located known distances from one viewpoint in the first set of viewpoints, identifying the object in the second set of image data, and determining a second depth measurement using the first set of image data and the second set of image data.
대표청구항▼
1. An array camera, comprising: a camera array comprising a plurality of cameras that capture images of a scene from different viewpoints;a gyroscope and accelerometers;a processor; andmemory in communication with the processor;wherein software directs the processor to: obtain a first set of images
1. An array camera, comprising: a camera array comprising a plurality of cameras that capture images of a scene from different viewpoints;a gyroscope and accelerometers;a processor; andmemory in communication with the processor;wherein software directs the processor to: obtain a first set of images captured from different viewpoints at a first time using the camera array at a first location;obtain a second set of images captured from different viewpoints at a second time using the camera array at a second location distant from the first location;estimate extrinsic parameters for the camera array at the first time and at the second time based upon a combination of matching features in the first and second set of images and information from the gyroscope and accelerometers;select a reference viewpoint;determine depth estimates for pixel locations in an image from the reference viewpoint using the first set of images, wherein generating a depth estimate for a given pixel location in the image from the reference viewpoint comprises: identifying pixels in the first set of images that correspond to the given pixel location in the image from the reference viewpoint based upon expected disparity at a plurality of depths;comparing the similarity of the corresponding pixels identified at each of the plurality of depths; andselecting the depth from the plurality of depths at which the identified corresponding pixels have the highest degree of similarity as a depth estimate for the given pixel location in the image from the reference viewpoint;determine whether a depth estimate for pixel locations in an image from the reference viewpoint determined using the first set of images corresponds to an observed disparity below a threshold; andwhen the depth estimate corresponds to an observed disparity below the threshold, refining the depth estimate using images in the second set of images by: identifying pixels in the first and second sets of images that correspond to the given pixel location in the image from the reference viewpoint based upon expected disparity at a plurality of depths;comparing the similarity of the corresponding pixels form the first and second sets of images identified at each of the plurality of depths; andselecting the depth from the plurality of depths at which the identified corresponding pixels have the highest degree of similarity as a depth estimate for the given pixel location in the image from the reference viewpoint. 2. The array camera of claim 1, wherein software further directs the processor to select the plurality of depths at which pixels in images captured by the first array camera and by the second camera that correspond to the given pixel location in the image from the reference viewpoint are identified during refinement of the depth estimate based upon the depth estimate initially determined using the first set of images. 3. The array camera of claim 1, wherein software further directs the processor to: identify an object of interest in the first set of images;wherein determine depth estimates for pixel locations in an image from the reference viewpoint using the first set of images further comprises estimating a depth for the object of interest using the first set of images. 4. The array camera of claim 1, wherein software further directs the processor to calculate a confidence measure for the depth of an object of interest. 5. The array camera of claim 1, wherein software further directs the processor to generate a depth map using the depth estimates for pixel locations in an image from the reference viewpoint, where the depth map indicates distances of surfaces of scene objects from the reference viewpoint. 6. The array camera of claim 1, wherein the plurality of cameras in the camera array forms an M×N array of cameras. 7. The array camera of claim 1, wherein the camera array comprises cameras that image portions of the spectral band selected from the group consisting of red, blue, green, infrared, and extended color. 8. The array camera of claim 1, wherein the cameras in the camera array form a π filter group. 9. The array camera of claim 1, wherein: software further configures the processor to synthesize an image from the first set of images using the depth map;each image in the first set of images has a resolution; andthe image synthesized from the first set of images has a resolution that is greater than the resolution of each of the images in the first set of images. 10. The array camera of claim 1, wherein: software further configures the processor to synthesize an image from the first and second sets of images using the depth map and the extrinsic parameters of the camera array at the first and second times;each image in the first and second sets of images has a resolution; andthe image synthesized from the first set of images has a resolution that is greater than the resolution of each of the images in the first and second sets of images. 11. A method of estimating depth using an ad hoc stereo array camera, comprising: obtaining a first set of images captured from different viewpoints at a first time using a camera array;moving the camera array and obtaining a second set of images captured from different viewpoints at a second time using the camera array;obtaining gyroscope and accelerometer measurements between the first time and the second time;estimating extrinsic parameters for the camera array at the first time and at the second time based upon a combination of matching features in the first and second set of images and the gyroscope and accelerometers measurements;using software to direct a processor to: select a reference viewpoint;determine depth estimates for pixel locations in an image from the reference viewpoint using the first set of images, wherein generating a depth estimate for a given pixel location in the image from the reference viewpoint comprises: identifying pixels in the first set of images that correspond to the given pixel location in the image from the reference viewpoint based upon expected disparity at a plurality of depths;comparing the similarity of the corresponding pixels identified at each of the plurality of depths; andselecting the depth from the plurality of depths at which the identified corresponding pixels have the highest degree of similarity as a depth estimate for the given pixel location in the image from the reference viewpoint;determine whether a depth estimate for pixel locations in an image from the reference viewpoint determined using the first set of images corresponds to an observed disparity below a threshold; andwhen the depth estimate corresponds to an observed disparity below the threshold, refining the depth estimate using images in the second set of images by: identifying pixels in the first and second sets of images that correspond to the given pixel location in the image from the reference viewpoint based upon expected disparity at a plurality of depths;comparing the similarity of the corresponding pixels from the first and second sets of images identified at each of the plurality of depths; andselecting the depth from the plurality of depths at which the identified corresponding pixels have the highest degree of similarity as a depth estimate for the given pixel location in the image from the reference viewpoint. 12. The method of claim 11, further comprising using software to direct the processor to select the plurality of depths at which pixels in images captured by the first array camera and by the second camera that correspond to the given pixel location in the image from the reference viewpoint are identified during refinement of the depth estimatebased upon the depth estimate initially determined using the first set of images. 13. The method of claim 11, further comprising using software to direct the processor to: identify an object of interest in the first set of images;wherein determine depth estimates for pixel locations in an image from the reference viewpoint using the first set of images further comprises estimating a depth for the object of interest using the first set of images. 14. The method of claim 11, further comprising using software to direct the processor to calculate a confidence measure for the depth of an object of interest. 15. The method of claim 11, further comprising using software to direct the processor to generate a depth map using the depth estimates for pixel locations in an image from the reference viewpoint, where the depth map indicates distances of surfaces of scene objects from the reference viewpoint. 16. The method of claim 11, wherein the plurality of cameras in the camera array forms an M×N array of cameras. 17. The method of claim 11, wherein the camera array comprises cameras that image portions of the spectral band selected from the group consisting of red, blue, green, infrared, and extended color. 18. The method of claim 11, wherein the cameras in the camera array form a π filter group. 19. The method of claim 11, further comprising using software to direct the processor to synthesize an image from the first set of images using the depth map; wherein each image in the first set of images has a resolution; andwherein the image synthesized from the first set of images has a resolution that is greater than the resolution of each of the images in the first set of images. 20. The method of claim 11, further comprising using software to direct the processor to synthesize an image from the first and second sets of images using the depth map and the extrinsic parameters of the camera array at the first and second times; wherein each image in the first and second sets of images has a resolution; andwherein the image synthesized from the first and second sets of images has a resolution that is greater than the resolution of each of the images in the first and second sets of images.
연구과제 타임라인
LOADING...
LOADING...
LOADING...
LOADING...
LOADING...
이 특허에 인용된 특허 (103)
Wilburn, Bennett; Joshi, Neel; Levoy, Marc C.; Horowitz, Mark, Apparatus and method for capturing a scene using staggered triggering of dense camera arrays.
Iwase Toshihiro (Nara JPX) Kanekura Hiroshi (Yamatokouriyama JPX), Apparatus for and method of converting a sampling frequency according to a data driven type processing.
Boisvert, David Michael; McMahon, Andrew Kenneth John, CCD output processing stage that amplifies signals from colored pixels based on the conversion efficiency of the colored pixels.
Venkataraman, Kartik; Jabbi, Amandeep S.; Mullis, Robert H., Capturing and processing of images using monolithic camera array with heterogeneous imagers.
Ward, Gregory John; Seetzen, Helge; Heidrich, Wolfgang, Electronic camera having multiple sensors for capturing high dynamic range images and related methods.
Abell Gurdon R. (West Woodstock CT) Cook Francis J. (Topsfield MA) Howes Peter D. (Sudbury MA), Method and apparatus for arraying image sensor modules.
Sawhney,Harpreet Singh; Tao,Hai; Kumar,Rakesh; Hanna,Keith, Method and apparatus for synthesizing new video and/or still imagery from a collection of real video and/or still imagery.
Alexander David H. (Santa Monica CA) Hershman George H. (Carlsbad CA) Jack Michael D. (Carlsbad CA) Koda N. John (Vista CA) Lloyd Randahl B. (San Marcos CA), Monolithic imager for near-IR.
Hornbaker ; III Cecil V. (New Carrolton MD) Driggers Thomas C. (Falls Church VA) Bindon Edward W. (Fairfax VA), Scanning apparatus using multiple CCD arrays and related method.
Ciurea, Florian; Venkataraman, Kartik; Molina, Gabriel; Lelescu, Dan, Systems and methods for parallax detection and correction in images captured using array cameras that contain occlusions using subsets of images to perform depth estimation.
Ciurea, Florian; Venkataraman, Kartik; Molina, Gabriel; Lelescu, Dan, Systems and methods for performing depth estimation using image data from multiple spectral channels.
Rieger Albert,DEX ; Barclay David ; Chapman Steven ; Kellner Heinz-Andreas,DEX ; Reibl Michael,DEX ; Rydelek James G. ; Schweizer Andreas,DEX, Watertight body for accommodating a photographic camera.
Venkataraman, Kartik; Jabbi, Amandeep S.; Mullis, Robert H.; Duparre, Jacques; Hu, Shane Ching-Feng, Capturing and processing of images including occlusions focused on an image sensor by a lens stack array.
Srikanth, Manohar; Ramamoorthi, Ravi; Venkataraman, Kartik; Chatterjee, Priyam, System and methods for depth regularization and semiautomatic interactive matting using RGB-D images.
Duparre, Jacques; McMahon, Andrew Kenneth John; Lelescu, Dan; Venkataraman, Kartik; Molina, Gabriel, Systems and methods for detecting defective camera arrays and optic arrays.
Venkataraman, Kartik; Lelescu, Dan; Molina, Gabriel, Systems and methods for generating compressed light field representation data using captured light fields, array geometry, and parallax information.
Venkataraman, Kartik; Jabbi, Amandeep S.; Mullis, Robert H., Systems and methods for generating depth maps using a camera arrays incorporating monochrome and color cameras.
Duparre, Jacques; McMahon, Andrew Kenneth John; Lelescu, Dan, Systems and methods for manufacturing camera modules using active alignment of lens stack arrays and sensors.
※ AI-Helper는 부적절한 답변을 할 수 있습니다.