Automatic video system using multiple cameras
원문보기
IPC분류정보
국가/구분
United States(US) Patent
등록
국제특허분류(IPC7판)
H04N-005/225
H04N-007/18
H04N-005/232
출원번호
US-0370406
(1999-08-09)
발명자
/ 주소
Foote,Jonathan
Ahmad,Subutai
Boreczky,John
출원인 / 주소
Fuji Xerox Co., Ltd.
대리인 / 주소
Fliesler Meyer LLP
인용정보
피인용 횟수 :
252인용 특허 :
113
초록▼
A camera array captures plural component images which are combined into a single scene from which "panning" and "zooming" within the scene are performed. In one embodiment, each camera of the array is a fixed digital camera. The images from each camera are warped and blended such that the combined i
A camera array captures plural component images which are combined into a single scene from which "panning" and "zooming" within the scene are performed. In one embodiment, each camera of the array is a fixed digital camera. The images from each camera are warped and blended such that the combined image is seamless with respect to each of the component images. Warping of the digital images is performed via pre-calculated non-dynamic equations that are calculated based on a registration of the camera array. The process of registering each camera in the arrays is performed either manually, by selecting corresponding points or sets of points in two or more images, or automatically, by presenting a source object (laser light source, for example) into a scene being captured by the camera array and registering positions of the source object as it appears in each of the images. The warping equations are calculated based on the registration data and each scene captured by the camera array is warped and combined using the same equations determined therefrom. A scene captured by the camera array is zoomed, or selectively steered to an area of interest. This zooming-or steering, being done in the digital domain is performed nearly instantaneously when compared to cameras with mechanical zoom and steering functions.
대표청구항▼
What is claimed is: 1. A method, comprising: providing a camera array, the camera array including a plurality of cameras; providing a set of camera offset values for the camera array; synchronously capturing a set of images from the camera array; selecting pixels from at least one image of said s
What is claimed is: 1. A method, comprising: providing a camera array, the camera array including a plurality of cameras; providing a set of camera offset values for the camera array; synchronously capturing a set of images from the camera array; selecting pixels from at least one image of said set of images; applying a bilinear transformation to the selected pixels to transform the selected pixels from a coordinate system of the at least one image to a common coordinate system of a composite image; wherein applying a bilinear transformation includes using the set of camera offset values. 2. The method according to claim 1, wherein selecting pixels includes identifying contiguous patches in said set of images. 3. The method according to claim 2, wherein identifying contiguous patches, includes identifying pixels from the contiguous patches having registration points in common with said composite image. 4. The method according to claim 2, further comprising: combining pixels from the contiguous patches having substantially similar registration points into a common location of said composite image. 5. The method according to claim 4, wherein combining pixels from the contiguous patches includes cross-fading pixels having substantially similar registration points having a substantially similar registration points into the common location of said composite image. 6. The method according to claim 4, wherein combining pixels from the contiguous patches includes at least one of cross-fading, blurring, averaging, and other image effects to seamlessly combine said pixels from different patches having substantially similar registration points into the common location of said composite image. 7. The method according to claim 2, wherein performing a transformation includes applying a predetermined bilinear transformation matrix to each patch, said predetermined bilinear transformation matrix representing an amount of warping required for each patch to transform each patch into said common coordinate system. 8. The method according to claim 1, further comprising repeating said applying a transformative equation for each set of images synchronously captured by said camera array, each set of images representing one frame in a video stream of said scene. 9. The method according to claim 1, further comprising selecting an area of interest from the composite image; and outputting the selected area of interest to a user. 10. The method according to claim 9, wherein said outputting comprises displaying said area of interest from the combined warped images. 11. The method according to claim 9, wherein said selecting comprises: directing said area of interest to a predetermined area surrounding at least one of motion detected in said scene, audio sources detected within said scene, and proximity of objects detected in said scene. 12. A method, comprising: synchronously capturing a set of images from a camera array; selecting pixels from at least one image from the set of images; identifying contiguous patches from the selected pixels; transforming the selected pixels from a coordinate system of the at least one image to a common coordinate system; determining overlap between the contiguous patches, wherein overlapping contiguous patches include pixels having substantially similar registration points; combining overlapping contiguous patches into a common location of said a composite image; wherein combining overlapping contiguous patches includes cross-fading pixels having substantially similar registration points into the common location of said composite image; wherein cross-fading includes: varying a parameter of the pixels having substantially similar registration points from a first patch from a minimum value at a first boundary of said first patch to a maximum value at an opposite boundary of said first patch; varying said parameter of the pixels having substantially similar registration points from a second patch from said maximum value at a boundary of said second patch corresponding to said first boundary to said minimum value at a boundary of said second patch corresponding to said opposite boundary; summing corresponding pixel values of said first patch and said second patch; and placing the summed values in corresponding locations of said common coordinate system. 13. The method according to claim 12, wherein said parameter is at least one of brightness, contrast, and intensity. 14. A camera array, comprising: a set of cameras mounted in an array; a set of camera offset values for the set of cameras; an image combining mechanism configured to combine at least two of images captured from said set of cameras into a composite image, the image combining mechanism including: a warping device configured to warp patches of each image into a common coordinate system of said composite image by applying a bilinear transformation; wherein the warping device applies the set of camera offset values to warp patches; a view selection device configured to select a view from the composite image; and an output mechanism configured to display the selected view. 15. The camera array according to claim 14, wherein said image combining mechanism includes, a fading device configured to fade and combine patches having a same location in said composite image. 16. The camera array according to claim 15, wherein: said array of cameras are immovably mounted on a firm fixed base; and said warping device applies a pre-determined transformation to each of said patches. 17. The camera array according to claim 15, further comprising a registration mechanism configured to register each of said camera arrays by finding registration points in common with views of each camera and said composite image. 18. The camera array according to claim 14, wherein said view selection device includes: at least one of, a video motion detector configured to detect motion in the composite image, a sound detection device configured to determine a location of sound originating within said composite image, and a stereoscopic ranging mechanism configured to utilize at least two images from separate cameras of said camera array to determine a range of objects in said composite image; and a detection mechanism configured to automatically detect any of a face, shape, color, and motion of a subject for inclusion in said selected frame based on at least one of ranging data from said stereoscopic ranging mechanism, location data from said sound detection device, and motion detected by said video motion detector. 19. A camera array comprising: a plurality of cameras; an image combining mechanism configured to combine at least two images captured from said plurality of cameras into a composite image, the image combining mechanism including: a registration mechanism configured to register said plurality of cameras by finding registration points in common with views of the plurality of cameras and said composite image, the registration mechanism including: a registration point source directable to locations within said views of the plurality of cameras, and a detection mechanism configured to detect said registration point source and register the detected registration point source in said views of the plurality of cameras with respect to a coordinate system of said composite image. 20. The camera array according to claim 19, wherein said registration point source is at least one of a light source directed into view of said camera array and a grid placed in view of said camera array. 21. A method of registering a camera array, comprising: placing at least one registration point in a field of view of at least two cameras of said camera array; identifying a location of each registration point in a field of view of each camera of said array; calculating a warped coordinate system for placing pixels of said cameras; and maintaining a table identifying pixels of said cameras, an amount of fade, and a corresponding location in the warped coordinate system such that images can be combined in relation to said registration points. 22. The method according to claim 21, wherein said step placing comprises the steps of: sweeping an beam light source across a field of view of at least two cameras of said camera array. 23. The method according to claim 21, wherein said step of placing comprises the step of: placing a grid of known location in a field of view in at least two cameras of said array. 24. A method, comprising: providing a camera array, the camera array including a plurality of cameras; providing a set of camera offset values for the camera array; synchronously capturing a set of images from the camera array; selecting pixels from at least one image of said set of images; and applying a transformative equation to the selected pixels to transform the selected pixels from a coordinate system of the at least one image to a common coordinate system of a composite image; wherein said transformative equation is at least one of a bilinear transformation, a radial transformation, and an affine transformation; wherein applying a transformative equation includes using the set of camera offset.
연구과제 타임라인
LOADING...
LOADING...
LOADING...
LOADING...
LOADING...
이 특허에 인용된 특허 (113)
Wah Lo Allen K. (Dunwoody GA) Wu Augustine K. (Norcross GA), 2-D and 3-D multi-lens camera with one lens having a wider plane than the other lenses.
Pitteloud Dominique (Palo Alto CA) Phillips Peter (Palo Alto CA) Martin Jean-Paul H. (Oakton VA), Apparatus and method for automerging images by matching features and aligning images.
Saito Tsutomu (Shizuoka JPX), Apparatus and method for generating a pseudo camera position image from a plurality of video images from different camer.
Ohia Masashi (Tokyo JPX) Kobayashi Hiroshi (Chiba JPX) Sekiya Tsuneo (Tokyo JPX) Hamada Toshimichi (Tokyo JPX) Fukuda Kyoko (Kanagawa JPX) Iijima Koji (Kanagawa JPX), Apparatus and method for producing a panorama image using a motion vector of an image in an image signal.
Mitsutake Hideaki (Tokyo JPX) Tokumitsu Jun (Sagamihara JPX) Suda Shigeyuki (Yokohama JPX) Katayama Tatsushi (Tokyo JPX), Apparatus for producing a panoramic image using a plurality of optical systems.
Haseltine Eric C. (Manhattan Beach CA) Redmann William G. (Simi Valley CA), Electronic and computational correction of chromatic aberration associated with an optical system used to view a color v.
Lewins Lloyd J. (Marina Del Rey CA), Heads-up display (HUD) incorporating cathode-ray tube image generator with digital look-up table for distortion correcti.
Schieltz Steve W. (Palm Beach County FL) Glatt Terry Laurence (Pompano Beach FL) Kupersmit Carl (Palm Beach County FL), Image forming and processing device and method for use with no moving parts camera.
Szeliski Richard ; Shum Heung-Yeung, Image mosaic construction system and apparatus with patch-based alignment, global block adjustment and pair-wise motion-based local warping.
Lelong Pierre (Nogent/Sur/Marne FRX) Dalm Govert (Veldhoven NLX) Klijn Jan (Breda NLX), Image processing method and device for constructing an image from adjacent images.
Florent Raoul (Valenton FRX) Lelong Pierre (Nogent-Sur-Marne FRX), Image processing system comprising fixed cameras and a system simulating a mobile camera.
Glatt Terry Laurence ; Schieltz Steven W. ; Kupersmit Carl, Image splitting forming and processing device and method for use with no moving parts camera.
Fernandez Antonio (Brooklyn NY) Pineiro Gaggioni Hugo (Randolph NJ) Jaquez Martin J. (San Diego CA) Robbins John D. (Mountain Lakes NJ) Soper E. Scott (Morristown NJ), Memory system for high definition television display.
Driscoll ; Jr. Edward ; Morrow Howard ; Steinhauer Alan J. ; Lomax Willard Curtis, Method and apparatus for electronically distributing images from a panoptic camera system.
Reed Alastair (British Columbia CAX) Stansfield Peter W. (Hertfordshire GB2) Rosen Martin (Hertfordshire GB2), Method and apparatus for generating composite images.
Baker Robert G. (Delray Beach FL) Kettler Kevin (Pittsburgh PA) Suarez Gustavo A. (Boca Raton FL) Uplinger Kenneth A. (Austin TX) Freedenberg Candace J. F. (Poughkeepsie NY), Method and apparatus for hemispheric imaging which emphasizes peripheral content.
Hsu Stephen Charles ; Kumar Rakesh ; Sawhney Harpreet Singh ; Bergen James R. ; Dixon Doug ; Paragano Vince ; Gendel Gary, Method and apparatus for performing local to global multiframe alignment to construct mosaic images.
Jackson Laban Phelps ; Pecoraro Alexis S. ; Hansen Peter ; Bauer Martin L. ; Martin H. Lee, Method and apparatus for the interactive display of any portion of a spherical image.
Bender Walter R. (Auburndale MA) Teodosio Laura A. (Derby CT), Method of creating a high resolution still image using a plurality of images and apparatus for practice of the method.
Bender Walter R. ; Teodosio Laura A., Method of creating a high resolution still image using a plurality of images and apparatus for practice of the method.
Shimizu Yasushi (Hamamatsu JPX) Ando Yoshikazu (Hamamatsu JPX), Method of recording and reproducing video and sound information using plural recording devices and plural reproducing de.
Wirth Allan (Bedford MA) Jankevics Andrew J. (Acton MA) Landers Franklin M. (Wakefield MA) Bruno Theresa L. (Bedford MA) D\Amato Dante P. (Swampscott MA) Schmutz Lawrence E. (Watertown MA) Gilligan L, Micro-lens panoramic imager.
Katayama Tatsushi (Kawasaki JPX) Niwa Yukichi (Narashino JPX) Suda Shigeyuki (Yokohama JPX), Multi-lens imaging apparatus having a mechanism for combining a plurality of images without displacement of registration.
Davis John E. ; Castle Kenneth R. ; Todd Marion N. ; Stuhlinger Tilman W. ; Ruda Mitchell, Panoramic optics assembly having an initial flat reflective element.
Burt Peter J. (Princeton NJ) Irani Michal (Princeton Junction NJ) Hsu Stephen Charles (East Windsor NJ) Anandan Padmanabhan (Lawrenceville NJ) Hansen Michael W. (New Hope PA), System for automatically aligning images to form a mosaic image.
Diner Daniel B. (Altadena CA), Television monitor field shifter and an opto-electronic method for obtaining a stereo image of optimal depth resolution.
Braun David A. (Denville NJ) Nilson ; III William A. E. (Bridgewater NJ) Nelson Terence J. (New Providence NJ) Smoot Lanny S. (Morris Township ; Morris County NJ), Television system for displaying multiple views of a remote location.
Duparre, Jacques; Lelescu, Dan; Venkataraman, Kartik, Array cameras incorporating monolithic array camera modules with high MTF lens stacks for capture of images used in super-resolution processing.
Duparre, Jacques; Lelescu, Dan; Venkataraman, Kartik, Array cameras incorporating optics with modulation transfer functions greater than sensor Nyquist frequency for capture of images used in super-resolution processing.
Hamilton, Rick A.; Moskowitz, Paul A.; O'Connell, Brian M.; Pickover, Clifford A.; Walker, Keith R., Attaching external virtual universes to an existing virtual universe.
Venkataraman, Kartik; Gallagher, Paul; Jain, Ankit K.; Nisenzon, Semyon; Lelescu, Dan; Ciurea, Florian; Molina, Gabriel, Autofocus system for a conventional camera that uses depth information from an array camera.
Venkataraman, Kartik; Jabbi, Amandeep S.; Mullis, Robert H.; Duparre, Jacques; Hu, Shane Ching-Feng, Capturing and processing of images including occlusions captured by arrays of luma and chroma cameras.
Venkataraman, Kartik; Jabbi, Amandeep S.; Mullis, Robert H.; Duparre, Jacques; Hu, Shane Ching-Feng, Capturing and processing of images including occlusions focused on an image sensor by a lens stack array.
Venkataraman, Kartik; Jabbi, Amandeep S.; Mullis, Robert H.; Duparre, Jacques; Hu, Shane Ching-Feng, Capturing and processing of images including occlusions focused on an image sensor by a lens stack array.
Venkataraman, Kartik; Jabbi, Amandeep S.; Mullis, Robert H.; Duparre, Jacques; Hu, Shane Ching-Feng, Capturing and processing of images using camera array incorperating Bayer cameras having different fields of view.
Fish, Nathan J.; Williams, Robert; Case, Charlie W.; Haverty, Chris M.; Oehler, Peter R.; Kruse, Ross N., Configuration of a touch screen display with conferencing.
Jung, Edward K. Y.; Levien, Royce A.; Lord, Robert W.; Malamud, Mark A.; Rinaldo, Jr., John D., Estimating shared image device operational capabilities or resources.
Jung, Edward K. Y.; Levien, Royce A.; Lord, Robert W.; Malamud, Mark A.; Rinaldo, Jr., John D., Estimating shared image device operational capabilities or resources.
Schultz, Stephen L.; Giuffrida, Frank D.; Gray, Robert L.; Mondello, Charles, Method and apparatus for capturing, geolocating and measuring oblique images.
Schultz, Stephen L.; Giuffrida, Frank D.; Gray, Robert L.; Mondello, Charles, Method and apparatus for capturing, geolocating and measuring oblique images.
Schultz, Stephen L.; Giuffrida, Frank D.; Gray, Robert L.; Mondello, Charles, Method and apparatus for capturing, geolocating and measuring oblique images.
Hamilton, II, Rick Allen; Harrison, Steven Mark; O'Connell, Brian Marshall; Pickover, Clifford Alan; Walker, Keith Raymond, Method and system for merging disparate virtual universes entities.
Hamilton, II, Rick Allen; Harrison, Steven Mark; O'Connell, Brian Marshall; Pickover, Clifford Alan; Walker, Keith Raymond, Method and system for splitting virtual universes into distinct entities.
Hahn, Wolfgang; Weidner, Thomas, Method and system for visualizing the environment of a vehicle with a distance-dependent merging of an infrared and a visual image.
Kim, Tae-Wan; Choi, Byeong-Ho; Kim, Je-Woo; Song, Hyok, Method for synthesizing intermediate image using mesh based on multi-view square camera structure and device using the same and computer-readable medium having thereon program performing function embodying the same.
Hamilton, II, Rick Allen; Harrison, Steven Mark; O'Connell, Brian Marshall; Pickover, Clifford Alan; Walker, Keith Raymond, Method for transferring inventory between virtual universes.
Hasegawa, Hiroyuki; Hama, Hideki; Nedu, Hiroshi; Kuroya, Takeyoshi; Kurebayashi, Masaaki, Monitoring system and method, and program and recording medium used therewith.
Clodfelter, Robert M.; Bayer, Jeff; McHale, Paul; Smith, Brad, Non-linear image mapping using a plurality of non-linear image mappers of lesser resolution.
Jung, Edward K. Y.; Levien, Royce A.; Lord, Robert W.; Malamud, Mark A.; Rinaldo, Jr., John D., Preservation and/or degradation of a video/audio data stream.
Jung, Edward K. Y.; Levien, Royce A.; Lord, Robert W.; Malamud, Mark A.; Rinaldo, Jr., John D., Preservation/degradation of video/audio aspects of a data stream.
Jung, Edward K. Y.; Levien, Royce A.; Lord, Robert W.; Malamud, Mark A.; Rinaldo, Jr., John D., Preservation/degradation of video/audio aspects of a data stream.
Jung, Edward K.Y.; Levien, Royce A.; Lord, Robert W.; Malamud, Mark A.; Rinaldo, Jr., John D., Preservation/degradation of video/audio aspects of a data stream.
Routhier, Nicholas; Thibeault, Claude; Belzile, Jean; Malouin, Daniel; Carpentier, Pierre-Paul; Dallaire, Martin, Process and system for encoding and playback of stereoscopic video sequences.
Routhier, Nicholas; Thibeault, Claude; Belzile, Jean; Malouin, Daniel; Carpentier, Pierre-Paul; Dallaire, Martin, Process and system for encoding and playback of stereoscopic video sequences.
Routhier, Nicholas; Thibeault, Claude; Belzile, Jean; Malouin, Daniel; Carpentier, Pierre-Paul; Dallaire, Martin, Process and system for encoding and playback of stereoscopic video sequences.
Routhier, Nicholas; Thibeault, Claude; Belzile, Jean; Malouin, Daniel; Carpentier, Pierre-Paul; Dallaire, Martin, Process and system for encoding and playback of stereoscopic video sequences.
Dillavou, Marcus W.; Shum, Phillip Corey; Guthrie, Baron L.; Shenai, Mahesh B.; Deaton, Drew Steven; May, Matthew Benton, System and method for image registration of multiple video streams.
Dillavou, Marcus W.; Shum, Phillip Corey; Guthrie, Baron L.; Shenai, Mahesh B.; Deaton, Drew Steven; May, Matthew Benton, System and method for image registration of multiple video streams.
Dillavou, Marcus W.; Shum, Phillip Corey; Guthrie, Barton L.; Shenai, Mahesh B.; Deaton, Drew Steven; May, Matthew Benton, System and method for image registration of multiple video streams.
Dillavou, Marcus W.; Shum, Phillip Corey; Guthrie, Barton L.; Shenai, Mahesh B.; Deaton, Drew Steven; May, Matthew Benton, System and method for managing spatiotemporal uncertainty.
Srikanth, Manohar; Ramamoorthi, Ravi; Venkataraman, Kartik; Chatterjee, Priyam, System and methods for depth regularization and semiautomatic interactive matting using RGB-D images.
Nayar, Shree; Venkataraman, Kartik; Pain, Bedabrata; Lelescu, Dan, Systems and methods for controlling aliasing in images captured by an array camera for use in super resolution processing using pixel apertures.
Lelescu, Dan; Venkataraman, Kartik, Systems and methods for controlling aliasing in images captured by an array camera for use in super-resolution processing.
Duparre, Jacques; McMahon, Andrew Kenneth John; Lelescu, Dan; Venkataraman, Kartik; Molina, Gabriel, Systems and methods for detecting defective camera arrays and optic arrays.
Ciurea, Florian; Venkataraman, Kartik; Molina, Gabriel; Lelescu, Dan, Systems and methods for estimating depth and visibility from a reference viewpoint for pixels in a set of images captured from different viewpoints.
Ciurea, Florian; Venkataraman, Kartik; Molina, Gabriel; Lelescu, Dan, Systems and methods for estimating depth and visibility from a reference viewpoint for pixels in a set of images captured from different viewpoints.
Venkataraman, Kartik; Lelescu, Dan; Molina, Gabriel, Systems and methods for generating compressed light field representation data using captured light fields, array geometry, and parallax information.
Venkataraman, Kartik; Jabbi, Amandeep S.; Mullis, Robert H., Systems and methods for generating depth maps using a camera arrays incorporating monochrome and color cameras.
Venkataraman, Kartik; Jabbi, Amandeep S.; Mullis, Robert H., Systems and methods for generating depth maps using a camera arrays incorporating monochrome and color cameras.
Venkataraman, Kartik; Jabbi, Amandeep S.; Mullis, Robert H., Systems and methods for generating depth maps using images captured by camera arrays incorporating cameras having different fields of view.
Venkataraman, Kartik; Jabbi, Amandeep S.; Mullis, Robert H., Systems and methods for generating depth maps using light focused on an image sensor by a lens element array.
Duparre, Jacques; McMahon, Andrew Kenneth John; Lelescu, Dan, Systems and methods for manufacturing camera modules using active alignment of lens stack arrays and sensors.
Duparre, Jacques; McMahon, Andrew Kenneth John; Lelescu, Dan, Systems and methods for manufacturing camera modules using active alignment of lens stack arrays and sensors.
Venkataraman, Kartik; Jabbi, Amandeep S.; Mullis, Robert H., Systems and methods for measuring depth using images captured by a camera array including cameras surrounding a central camera.
Venkataraman, Kartik; Huang, Yusong; Jain, Ankit K.; Chatterjee, Priyam, Systems and methods for performing high speed video capture and depth estimation using array cameras.
Rudolph, Eric; Rui, Yong; Malvar, Henrique S; He, Li Wei; Cohen, Michael F; Tashev, Ivan, Systems and methods for real-time audio-visual communication and data collaboration in a network conference environment.
Lelescu, Dan; Duong, Thang, Systems and methods for synthesizing high resolution images using image deconvolution based on motion and depth information.
Venkataraman, Kartik; Nisenzon, Semyon; Chatterjee, Priyam; Molina, Gabriel, Systems and methods for synthesizing images from image data captured by an array camera using restricted depth of field depth maps in which depth estimation precision varies.
Shabtay, Gal; Cohen, Noy; Geva, Nadav; Gigushinski, Oded; Goldenberg, Ephraim, Thin multi-aperture imaging system with auto-focus and methods for using same.
Fish, Nathan J.; Williams, Robert; Case, Charlie W.; Haverty, Chris M.; Oehler, Peter R.; Hilsdale, Michael C., User experience for conferencing with a touch screen display.
※ AI-Helper는 부적절한 답변을 할 수 있습니다.