Radiance processing by demultiplexing in the frequency domain
원문보기
IPC분류정보
국가/구분
United States(US) Patent
등록
국제특허분류(IPC7판)
G06T-003/00
G06T-011/00
G06K-009/74
G03B-019/12
출원번호
US-0186392
(2008-08-05)
등록번호
US-8559756
(2013-10-15)
발명자
/ 주소
Georgiev, Todor G.
Intwala, Chintan
Babacan, Sevket Derin
출원인 / 주소
Adobe Systems Incorporated
대리인 / 주소
Wolfe-SBMC
인용정보
피인용 횟수 :
122인용 특허 :
67
초록▼
Method and apparatus for radiance processing by demultiplexing in the frequency domain. A frequency domain demultiplexing module obtains a radiance image captured with a lens-based radiance camera. The image includes optically mixed spatial and angular frequency components of light from a scene. The
Method and apparatus for radiance processing by demultiplexing in the frequency domain. A frequency domain demultiplexing module obtains a radiance image captured with a lens-based radiance camera. The image includes optically mixed spatial and angular frequency components of light from a scene. The module performs frequency domain demultiplexing on the radiance image to generate multiple parallax views of the scene. The method may extract multiple slices at different angular frequencies from a Fourier transform of the radiance image, apply a Fourier transform to each of the multiple slices to generate intermediate images, stack the intermediate images to form a 3- or 4-dimensional image, apply an inverse Fourier transform along angular dimension(s) of the 3- or 4-dimensional image, and unstack the transformed 3- or 4-dimensional image to obtain the multiple parallax views. During the method, phase correction may be performed to determine the centers of the intermediate images.
대표청구항▼
1. A system, comprising: at least one processor; anda memory comprising program instructions that are executable by the at least one processor to:obtain a radiance image of a scene captured with a lens-based radiance camera comprising an array of refracting microlenses, the radiance image including
1. A system, comprising: at least one processor; anda memory comprising program instructions that are executable by the at least one processor to:obtain a radiance image of a scene captured with a lens-based radiance camera comprising an array of refracting microlenses, the radiance image including optically mixed different spatial and angular frequency components;demultiplex the radiance image in the frequency domain to generate multiple parallax views of the scene by the program instructions further executable to:apply a Fourier transform to the radiance image to generate a transformed radiance image;extract multiple slices of the transformed radiance image each at a different angular frequency;apply the Fourier transform to each of the multiple slices of the transformed radiance image to generate an intermediate image from each of the multiple slices;stack the intermediate images to form a 3-dimensional image;apply the Fourier transform along an angular dimension of the 3-dimensional image to generate a transformed 3-dimensional image; andunstack the transformed 3-dimensional image to obtain the multiple parallax views of the scene. 2. The system as recited in claim 1, wherein the program instructions are executable by the at least one processor to: stack the intermediate images to form a 4-dimensional image;apply the Fourier transform along two angular dimensions of the 4-dimensional image to generate a transformed 4-dimensional image; andunstack the transformed 4-dimensional image to obtain the multiple parallax views of the scene. 3. The system as recited in claim 2, wherein, to apply the Fourier transform along the angular dimension of the 3-dimensional image, the program instructions are executable by the at least one processor to apply an inverse Fourier transform (IFFT) along the angular dimension of the 3-dimensional image or along the two angular dimensions of the 4-dimensional image. 4. The system as recited in claim 1, wherein the parallax views include vertical parallax views. 5. The system as recited in claim 1, wherein the parallax views include horizontal parallax views. 6. The system as recited in claim 1, wherein, to apply the Fourier transform to each of the multiple slices of the transformed radiance image, the program instructions are executable by the at least one processor to apply a 2-dimensional inverse Fourier transform (IFFT) to each of the multiple slices. 7. The system as recited in claim 1, wherein the program instructions are executable by the at least one processor to perform phase correction on the intermediate images to determine centers of the intermediate images prior to the stack of the intermediate images. 8. A computer-implemented method, comprising: obtaining a radiance image of a scene captured with a lens-based radiance camera comprising an array of refracting microlenses, the radiance image including optically mixed different spatial and angular frequency components; anddemultiplexing the radiance image in the frequency domain to generate parallax views of the scene, said demultiplexing the radiance image comprises:applying a Fourier transform to the radiance image to generate a transformed radiance image;extracting multiple slices of the transformed radiance image each at a different angular frequency;applying the Fourier transform to each of the multiple slices of the transformed radiance image to generate an intermediate image from each of the multiple slices;stacking the intermediate images to form a 4-dimensional image;applying the Fourier transform along two angular dimensions of the 4-dimensional image to generate a transformed 4-dimensional image; andunstacking the transformed 4-dimensional image to obtain the multiple parallax views of the scene. 9. The computer-implemented method as recited in claim 8, wherein said demultiplexing the radiance image in the frequency domain comprises: stacking the intermediate images to form a 3-dimensional image;applying the Fourier transform along an angular dimension of the 3-dimensional image to generate a transformed 3-dimensional image; andunstacking the transformed 3-dimensional image to obtain the multiple parallax views of the scene. 10. The computer-implemented method as recited in claim 9, wherein said applying a Fourier transform along the angular dimension of the 3-dimensional image comprises applying an inverse Fourier transform (IFFT) along the angular dimension of the 3-dimensional image or the along the two angular dimensions of the 4-dimensional image. 11. The computer-implemented method as recited in claim 8, wherein the parallax views include vertical parallax views. 12. The computer-implemented method as recited in claim 8, wherein the parallax views include horizontal parallax views. 13. The computer-implemented method as recited in claim 8, wherein said applying the Fourier transform to each of the multiple slices of the transformed radiance image comprises applying a 2-dimensional inverse Fourier transform (IFFT) to each of the multiple slices. 14. The computer-implemented method as recited in claim 8, further comprising performing phase correction on the intermediate images to determine centers of the intermediate images prior to said stacking the intermediate images to form the 3-dimensional image. 15. A non-transitory computer-readable storage medium storing program instructions, wherein the program instructions are computer-executable to implement: obtaining a radiance image of a scene captured with a lens-based radiance camera comprising an array of refracting microlenses, the radiance image including optically mixed different spatial and angular frequency components; anddemultiplexing the radiance image in the frequency domain to generate parallax views of the scene, said demultiplexing the radiance image comprises:applying a Fourier transform to the radiance image to generate a transformed radiance image;extracting multiple slices of the transformed radiance image each at a different angular frequency;applying the Fourier transform to each of the multiple slices of the transformed radiance image to generate an intermediate image from each of the multiple slices;stacking the intermediate images to form a 3-dimensional or a 4-dimensional image;applying the Fourier transform along an angular dimension of the 3-dimensional image to generate a transformed 3-dimensional image, or along two angular dimensions of the 4-dimensional image to generate a transformed 4-dimensional image; andunstacking the transformed 3-dimensional image or the transformed 4-dimensional image to obtain the multiple parallax views of the scene. 16. The non-transitory computer-readable storage medium as recited in claim 15, wherein the parallax views include vertical parallax views. 17. The non-transitory computer-readable storage medium as recited in claim 15, wherein the parallax views include horizontal parallax views. 18. The non-transitory computer-readable storage medium as recited in claim 15, wherein, in said applying the Fourier transform to each of the multiple slices of the transformed radiance image, the program instructions are computer-executable to implement applying a 2-dimensional inverse Fourier transform (IFFT) to each of the multiple slices. 19. The non-transitory computer-readable storage medium as recited in claim 15, wherein, in said applying the Fourier transform along the angular dimension of the 3-dimensional image comprises the program instructions are computer-executable to implement applying an inverse Fourier transform (IFFT) along the angular dimension of the 3-dimensional image or along the two angular dimensions of the 4-dimensional image. 20. The non-transitory computer-readable storage medium as recited in claim 15, wherein the program instructions are computer-executable to implement performing phase correction on the intermediate images to determine centers of the intermediate images prior to said stacking the intermediate images to form the 3-dimensional image. 21. A system, comprising: a mask-based radiance camera configured to capture a scene as a radiance image that includes optically different spatial and angular frequency components, the mask-based radiance camera comprising a non-sinusoidal periodic mask positioned between an objective lens and a photosensor of the mask-based radiance camera; andan image processing application configured to demultiplex the radiance image in the frequency domain to generate parallax views of the scene by the image processing application configured to:apply a Fourier transform to the radiance image to generate a transformed radiance image;extract multiple slices of the transformed radiance image each at a different angular frequency;apply the Fourier transform to each of the multiple slices of the transformed radiance image to generate an intermediate image from each of the multiple slices;stack the intermediate images to form a 3-dimensional or a 4-dimensional image;apply the Fourier transform along an angular dimension of the 3-dimensional image to generate a transformed 3-dimensional image, or along two angular dimensions of the 4-dimensional image to generate a transformed 4-dimensional image; andunstack the transformed 3-dimensional image or the transformed 4-dimensional image to obtain the parallax views of the scene. 22. The system as recited in claim 21, wherein the non-sinusoidal periodic mask comprises a grid of multiple horizontally and vertically arranged opaque linear elements that define horizontal rows and vertical columns of periodically spaced transparent openings between the opaque linear elements. 23. A system, comprising: an external mask-based radiance camera configured to capture a scene as a radiance image that includes optically different spatial and angular frequency components, the external mask-based radiance camera comprising a mask positioned in front of an objective lens of the camera; andan image processing application configured to demultiplex the radiance image in the frequency domain to generate parallax views of the scene by the image processing application configured to:apply a Fourier transform to the radiance image to generate a transformed radiance image;extract multiple slices of the transformed radiance image each at a different angular frequency;apply the Fourier transform to each of the multiple slices of the transformed radiance image to generate an intermediate image from each of the multiple slices;stack the intermediate images to form a 3-dimensional or a 4-dimensional image;apply the Fourier transform along an angular dimension of the 3-dimensional image to generate a transformed 3-dimensional image, or along two angular dimensions of the 4-dimensional image to generate a transformed 4-dimensional image; andunstack the transformed 3-dimensional image or the transformed 4-dimensional image to obtain the parallax views of the scene. 24. The system as recited in claim 23, wherein the mask comprises a grid of multiple horizontally and vertically arranged opaque linear elements that define horizontal rows and vertical columns of periodically spaced transparent openings between the opaque linear elements.
연구과제 타임라인
LOADING...
LOADING...
LOADING...
LOADING...
LOADING...
이 특허에 인용된 특허 (67)
Georgiev Todor, 3D graphics based on images and morphing.
Yamagata, Michihiro; Okayama, Hiroaki; Boku, Kazutake; Tanaka, Yasuhiro; Hayashi, Kenichi; Fushimi, Yoshimasa; Murata, Shigeki; Hayashi, Takayuki, Imaging device including a plurality of lens elements and a imaging sensor.
de Montebello Roger L. (New York NY) Globus Ronald P. (New York NY) Buck Howard S. (New York NY), Integral photography apparatus and method of forming same.
Corle Timothy R. (Santa Clara County CA) Kino Gordon S. (Santa Clara County CA) Mansfield Scott M. (San Mateo County CA), Optical recording system employing a solid immersion lens.
Patton, David L.; Spoonhower, John P.; Bohan, Anne E.; Paz-Pujalt, Gustavo R., Solid immersion lens array and methods for producing a solid immersion lens array.
Venkataraman, Kartik; Gallagher, Paul; Jain, Ankit; Nisenzon, Semyon; Lelescu, Dan; Ciurea, Florian; Molina, Gabriel, Array cameras including an array camera module augmented with a separate camera.
Duparre, Jacques; Lelescu, Dan; Venkataraman, Kartik, Array cameras incorporating monolithic array camera modules with high MTF lens stacks for capture of images used in super-resolution processing.
Duparre, Jacques; Lelescu, Dan; Venkataraman, Kartik, Array cameras incorporating optics with modulation transfer functions greater than sensor Nyquist frequency for capture of images used in super-resolution processing.
Venkataraman, Kartik; Gallagher, Paul; Jain, Ankit K.; Nisenzon, Semyon; Lelescu, Dan; Ciurea, Florian; Molina, Gabriel, Autofocus system for a conventional camera that uses depth information from an array camera.
Venkataraman, Kartik; Jabbi, Amandeep S.; Mullis, Robert H., Capturing and processing of images captured by camera arrays including cameras dedicated to sampling luma and cameras dedicated to sampling chroma.
Venkataraman, Kartik; Jabbi, Amandeep S.; Mullis, Robert H., Capturing and processing of images captured by camera arrays including heterogeneous optics.
Venkataraman, Kartik; Jabbi, Amandeep S.; Mullis, Robert H.; Duparre, Jacques; Hu, Shane Ching-Feng, Capturing and processing of images including occlusions captured by arrays of luma and chroma cameras.
Venkataraman, Kartik; Jabbi, Amandeep S.; Mullis, Robert H.; Duparre, Jacques; Hu, Shane Ching-Feng, Capturing and processing of images including occlusions captured by camera arrays.
Venkataraman, Kartik; Jabbi, Amandeep S.; Mullis, Robert H.; Duparre, Jacques; Hu, Shane Ching-Feng, Capturing and processing of images including occlusions focused on an image sensor by a lens stack array.
Venkataraman, Kartik; Jabbi, Amandeep S.; Mullis, Robert H.; Duparre, Jacques; Hu, Shane Ching-Feng, Capturing and processing of images including occlusions focused on an image sensor by a lens stack array.
Venkataraman, Kartik; Jabbi, Amandeep S.; Mullis, Robert H.; Duparre, Jacques; Hu, Shane Ching-Feng, Capturing and processing of images using camera array incorperating Bayer cameras having different fields of view.
Venkataraman, Kartik; Jabbi, Amandeep S.; Mullis, Robert H., Capturing and processing of near-IR images including occlusions using camera arrays incorporating near-IR light sources.
Georgiev, Todor G.; Chunev, Georgi N., Methods and apparatus for rendering output images with simulated artistic effects from focused plenoptic camera data.
Georgiev, Todor G.; Lumsdaine, Andrew, Methods, apparatus, and computer-readable storage media for depth-based rendering of focused plenoptic camera data.
Srikanth, Manohar; Ramamoorthi, Ravi; Venkataraman, Kartik; Chatterjee, Priyam, System and methods for depth regularization and semiautomatic interactive matting using RGB-D images.
Ciurea, Florian; Venkataraman, Kartik; Molina, Gabriel; Lelescu, Dan, System and methods for measuring depth using an array camera employing a bayer filter.
Nayar, Shree; Venkataraman, Kartik; Pain, Bedabrata; Lelescu, Dan, Systems and methods for controlling aliasing in images captured by an array camera for use in super resolution processing using pixel apertures.
Lelescu, Dan; Venkataraman, Kartik, Systems and methods for controlling aliasing in images captured by an array camera for use in super-resolution processing.
Lelescu, Dan; Venkataraman, Kartik, Systems and methods for controlling aliasing in images captured by an array camera for use in super-resolution processing.
Duparré, Jacques, Systems and methods for correcting for warpage of a sensor array in an array camera module by introducing warpage into a focal plane of a lens stack array.
Duparre, Jacques; McMahon, Andrew Kenneth John; Lelescu, Dan; Venkataraman, Kartik; Molina, Gabriel, Systems and methods for detecting defective camera arrays and optic arrays.
Duparre, Jacques; McMahon, Andrew Kenneth John; Lelescu, Dan; Venkataraman, Kartik; Molina, Gabriel, Systems and methods for detecting defective camera arrays and optic arrays.
Lelescu, Dan; Molina, Gabriel; Venkataraman, Kartik, Systems and methods for dynamic refocusing of high resolution images generated using images captured by a plurality of imagers.
Ciurea, Florian; Venkataraman, Kartik; Molina, Gabriel; Lelescu, Dan, Systems and methods for estimating depth and visibility from a reference viewpoint for pixels in a set of images captured from different viewpoints.
Ciurea, Florian; Venkataraman, Kartik; Molina, Gabriel; Lelescu, Dan, Systems and methods for estimating depth and visibility from a reference viewpoint for pixels in a set of images captured from different viewpoints.
Venkataraman, Kartik; Lelescu, Dan; Molina, Gabriel, Systems and methods for generating compressed light field representation data using captured light fields, array geometry, and parallax information.
Venkataraman, Kartik; Lelescu, Dan; Molina, Gabriel, Systems and methods for generating compressed light field representation data using captured light fields, array geometry, and parallax information.
Ciurea, Florian; Venkataraman, Kartik; Molina, Gabriel; Lelescu, Dan, Systems and methods for generating depth maps and corresponding confidence maps indicating depth estimation reliability.
Venkataraman, Kartik; Jabbi, Amandeep S.; Mullis, Robert H., Systems and methods for generating depth maps using a camera arrays incorporating monochrome and color cameras.
Venkataraman, Kartik; Jabbi, Amandeep S.; Mullis, Robert H., Systems and methods for generating depth maps using a camera arrays incorporating monochrome and color cameras.
Venkataraman, Kartik; Jabbi, Amandeep S.; Mullis, Robert H.; Duparre, Jacques; Hu, Shane Ching-Feng, Systems and methods for generating depth maps using a set of images containing a baseline image.
Venkataraman, Kartik; Jabbi, Amandeep S.; Mullis, Robert H., Systems and methods for generating depth maps using images captured by camera arrays incorporating cameras having different fields of view.
Venkataraman, Kartik; Jabbi, Amandeep S.; Mullis, Robert H., Systems and methods for generating depth maps using light focused on an image sensor by a lens element array.
Duparre, Jacques; McMahon, Andrew Kenneth John; Lelescu, Dan, Systems and methods for manufacturing camera modules using active alignment of lens stack arrays and sensors.
Duparre, Jacques; McMahon, Andrew Kenneth John; Lelescu, Dan, Systems and methods for manufacturing camera modules using active alignment of lens stack arrays and sensors.
Ciurea, Florian; Venkataraman, Kartik; Molina, Gabriel; Lelescu, Dan, Systems and methods for measuring depth in the presence of occlusions using a subset of images.
Ciurea, Florian; Venkataraman, Kartik; Molina, Gabriel; Lelescu, Dan, Systems and methods for measuring depth using an array of independently controllable cameras.
Venkataraman, Kartik; Jabbi, Amandeep S.; Mullis, Robert H., Systems and methods for measuring depth using images captured by a camera array including cameras surrounding a central camera.
Venkataraman, Kartik; Jabbi, Amandeep S.; Mullis, Robert H., Systems and methods for measuring depth using images captured by monolithic camera arrays including at least one bayer camera.
Venkataraman, Kartik; Jabbi, Amandeep S.; Mullis, Robert H.; Duparre, Jacques; Hu, Shane Ching-Feng, Systems and methods for normalizing image data captured by camera arrays.
Venkataraman, Kartik; Huang, Yusong; Jain, Ankit K.; Chatterjee, Priyam, Systems and methods for performing high speed video capture and depth estimation using array cameras.
Venkataraman, Kartik; Jabbi, Amandeep S.; Mullis, Robert H., Systems and methods for performing post capture refocus using images captured by camera arrays.
Lelescu, Dan; Molina, Gabriel; Venkataraman, Kartik, Systems and methods for synthesizing high resolution images using a set of geometrically registered images.
Lelescu, Dan; Duong, Thang, Systems and methods for synthesizing high resolution images using image deconvolution based on motion and depth information.
Lelescu, Dan; Molina, Gabriel; Venkataraman, Kartik, Systems and methods for synthesizing high resolution images using images captured by an array of independently controllable imagers.
Venkataraman, Kartik; Jabbi, Amandeep S.; Mullis, Robert H.; Duparre, Jacques; Hu, Shane Ching-Feng, Systems and methods for synthesizing higher resolution images using a set of images containing a baseline image.
Venkataraman, Kartik; Jabbi, Amandeep S.; Mullis, Robert H., Systems and methods for synthesizing higher resolution images using images captured by camera arrays.
Venkataraman, Kartik; Nisenzon, Semyon; Chatterjee, Priyam; Molina, Gabriel, Systems and methods for synthesizing images from image data captured by an array camera using restricted depth of field depth maps in which depth estimation precision varies.
Venkataraman, Kartik; Nisenzon, Semyon; Chatterjee, Priyam; Molina, Gabriel, Systems and methods for synthesizing images from image data captured by an array camera using restricted depth of field depth maps in which depth estimation precision varies.
※ AI-Helper는 부적절한 답변을 할 수 있습니다.