Methods and systems for converting 2D motion pictures for stereoscopic 3D exhibition
IPC분류정보
국가/구분
United States(US) Patent
등록
국제특허분류(IPC7판)
G06K-009/00
H04N-007/12
출원번호
US-0299909
(2007-06-25)
등록번호
US-8411931
(2013-04-02)
국제출원번호
PCT/IB2007/001726
(2007-06-25)
§371/§102 date
20090113
(20090113)
국제공개번호
WO2007/148219
(2007-12-27)
발명자
/ 주소
Zhou, Samuel
Judkins, Paul
Ye, Ping
출원인 / 주소
IMAX Corporation
대리인 / 주소
Kilpatrick Townsend & Stockton LLP
인용정보
피인용 횟수 :
54인용 특허 :
99
초록▼
The present invention discloses methods of digitally converting 2D motion pictures or any other 2D image sequences to stereoscopic 3D image data for 3D exhibition. In one embodiment, various types of image data cues can be collected from 2D source images by various methods and then used for producin
The present invention discloses methods of digitally converting 2D motion pictures or any other 2D image sequences to stereoscopic 3D image data for 3D exhibition. In one embodiment, various types of image data cues can be collected from 2D source images by various methods and then used for producing two distinct stereoscopic 3D views. Embodiments of the disclosed methods can be implemented within a highly efficient system comprising both software and computing hardware. The architectural model of some embodiments of the system is equally applicable to a wide range of conversion, re-mastering and visual enhancement applications for motion pictures and other image sequences, including converting a 2D motion picture or a 2D image sequence to 3D, re-mastering a motion picture or a video sequence to a different frame rate, enhancing the quality of a motion picture or other image sequences, or other conversions that facilitate further improvement in visual image quality within a projector to produce the enhanced images.
대표청구항▼
1. A method for converting a 2D image data sequence into a 3D image data sequence, the method comprising: receiving the 2D image data sequence;performing a temporal analysis on the 2D image data sequence to determine image motion cues;dividing the 2D image data sequence into scenes, each scene compr
1. A method for converting a 2D image data sequence into a 3D image data sequence, the method comprising: receiving the 2D image data sequence;performing a temporal analysis on the 2D image data sequence to determine image motion cues;dividing the 2D image data sequence into scenes, each scene comprising at least one object, wherein the scenes are grouped;identifying objects in each scene and determining characteristics associated with each object, wherein the characteristics comprise a geometrical position and shape;determining whether each scene is suitable for automated processing based on scene classification;assigning each object to an object layer based on the characteristics associated with each object;generating a depth map for each object based on the object layer;collecting processing information associated with each object;generating converted object images using the processing information;storing the processing information into a render data record;combining the converted object images to produce a 3D scene;combining depth maps into full frame depth maps;filling occlusion regions in the converted 3D scene;generating a 3D image sequence using the full frame depth maps and 3D scene;outputting the converted 3D image and render data record. 2. The method of claim 1, wherein generating the depth map for each object based on the object layer comprises: data;implementing a virtual camera by receiving camera parameters;identifying an object in a selected image frame;receiving an estimated dimension for the object;creating a model of the object based on the estimated dimension;positioning the model of the object at a first selected distance from a virtual camera position;computing an image of the model of the object from a virtual camera view point;comparing a size of the model of the object in the image to a size of the object in the selected image frame;positioning the model of the object at a second selected distance from the virtual camera position and computing a second image of the model of the object until the size of the model of the object in the second image matches the size of the object in the selected image frame; andestimating a real-world position of the object based on the first and second selected distances of the model of the object for computing the depth of the object. 3. The method of claim 2, wherein estimating real-world positions of objects is performed for multiple image frames. 4. The method of claim 1, further comprising: enhancing the 2D image data sequence before dividing the 2D image data sequence. 5. The method of claim 1, further comprising: generating a left eye image data sequence and a right eye image data sequence using the 2D original image data sequence, wherein the left and right eye image data sequences form a stereoscopic 3D view. 6. The method of claim 5, where the original 2D image data sequence is used as the left or right image data sequence. 7. The method of claim 1, wherein the data render records comprise data associated with processing the 2D original image sequence and generating the 3D image sequence, the associated data comprising: processing information;editorial information;computing instructions and parameters. 8. The method of claim 4, wherein performing a temporal analysis on the 2D image data sequence to determine image motion cues provides motion information to facilitate processing; and wherein enhancing the 2D image data sequence before dividing the 2D image data sequence provides enhancement information to facilitate processing in downstream stages. 9. The method of claim 1, wherein dividing the 2D image data sequence into scenes comprises creating a shot list with shot group information. 10. The method of claim 1, wherein identifying objects in each scene and determining characteristics associated with each object, further comprising: characterizing the geometrical position using match by scaling. 11. The method of claim 1, wherein determining whether each scene is suitable for automated processing is based on at least one of motion field and depth of field. 12. The method of claim 1, wherein assigning each object to the object layer based on characteristics associated with each object further comprises: assigning each object to different layers. 13. The method of claim 12, further comprising: associating a label to each object, wherein the label identifies the assigned layer for the associated object. 14. The method of claim 13, wherein the label is a color selected from a predefined color palette. 15. The method of claim 12, wherein assigning each object to different layers is based on a criteria, the criteria comprising at least one of: object transparent;reconstruction of objects with occlusion;automated object separation;description of particle objects;object size compensation; andmotion classification. 16. The method of claim 15, wherein object transparency comprises: using transparency for at least one pixel in an object;corresponding multiple pixel and depth values to different objects; andwherein the different objects are assigned to different layers. 17. The method of claim 1, wherein collecting processing information associated with each object and generating converted object images using the processing information comprises: object mask generation;object depth modeling;object re-construction;object depth map generation;object scaling; andcomputing at least one stereoscopic image. 18. The method of claim 17, wherein object re-construction comprises: detecting missing object information; andreconstructing the missing object information in a layer independent of other layers. 19. The method of claim 17, wherein object scaling comprises: compensating for miniaturization or reverse miniaturization. 20. The method of claim 17, wherein object mask generation comprises: analyzing the 2D image data sequence, layer information, and motion cues associated with the 2D image data sequence;selecting an operating mode, wherein the operating mode is selected from the group consisting of: automated;semi-automated; andmanual;wherein each operating mode comprises at least one sub-process; andgenerating an object mask using one or more sub-processes. 21. The method of claim 20, wherein the automated operating mode comprises a shape recovery sub-process. 22. The method of claim 20, wherein the semi-automated operating mode comprises: color-based separation sub-process; andgraph-cut based separation sub-process. 23. The method of claim 20, wherein generating the object mask further comprises: generating a mask at selected frames; andinterpolating the mask for all frames in a sequence based on motion tracking. 24. The method of claim 17, wherein object depth modeling comprises: analyzing the 2D image data sequence, layer information, and motion cues associated with the 2D image data sequence;selecting an operating mode, wherein the operating mode is selected from the group consisting of: automated;semi-automated; andmanual; andgenerating a rough object depth map and rough object mask using at least one sub-process. 25. The method of claim 24, wherein the automated operating mode comprises: an automated depth recovery sub-process; andwherein the automated depth recovery sub-process comprises calculating a depth value by: detecting camera motion; andobtaining camera motion using the pixel motion and temporal filtering. 26. The method of claim 24, wherein the semi-automated operation mode comprises: a bump-map depth modeling sub-process comprising: defining a best fit approximate geometrical shape of the object; andautomatically adding object details based on the best fit, color and shading;a feature-guided depth recovery sub-process comprising: calculating a depth value by detecting camera motion and obtaining camera motion from pixel motion; andwherein detecting camera motion is adapted to be at least partially controlled by a user; anda mask-guided depth recovery sub-process comprising: tracking an object mask to obtain object size and motion, wherein the tracking is adapted to be at least partially guided by a user; andcalculating depth values based on object size and motion. 27. The method of claim 24, wherein the manual operation mode comprises a solid depth modeling sub-process, wherein the model follows object motion and does not require full object detail or fit in the object boundary. 28. The method of claim 24, further comprising: using a plurality of sub-processes to generate the rough object depth map and rough object mask;scaling and combining the results of each sub-process to produce the object depth model. 29. The method of claim 24, further comprising: refining the rough object depth mask to produce a depth map. 30. The method of claim 29, further comprising: generating depth maps for at least two scenes; andmerging the depth maps for the at least two scenes by depth dissolving. 31. The method of claim 17, wherein object re-construction comprises: detecting missing information in an object; and filling missing information by: tracking object motion to determine corresponding features in other frames;registering the corresponding features; andcopying corresponding pixels from the other frames. 32. The method of claim 1, wherein combining depth maps into full frame depth maps further comprises: combining all object depth maps from all layers to create a full frame depth map; andcombining 3D scenes into full frame 3D images with occlusion regions. 33. The method of claim 1, wherein filling occlusion regions in the converted 3D scene comprises: analyzing 3D images with occlusion regions, the 2D image data sequence, and motion cues;selecting an operating mode, wherein the operating mode is selected from the group consisting of: automated;semi-automated; andmanual;wherein each operating mode comprises at least one sub-process;filling occlusion regions using at least one sub-process; andtemporally filtering the filled occlusion regions to produce the converted 3D image sequence. 34. The method of claim 33, wherein the automated operating mode comprises a pixel recovery sub-process, the pixel recovery sub-process comprising: tracking warped pixels in other frames; andcopying the tracked pixels. 35. The method of claim 33, wherein the semi-automated operating mode comprises: a feature-guided pixel recovery sub-process comprising: filling occlusion regions by copying tracked and warped pixels; andwherein filling occlusion regions is adapted to receive user input;a texture synthesis sub-process comprising: synthesizing a texture of regions surrounding each region of occlusion; andan in-painting sub-process;propagating pixel information collected from outside occlusion regions over multiple frames. 36. The method of claim 33, further comprising: filling occlusion regions using a plurality of sub-processes to produce a plurality of scenes having filled occlusion regions; andselecting the scene using a voting process, wherein the voting process is based on temporal consistency and spatial smoothness. 37. A system, comprising: a front-end system comprising an open platform adapted to: accommodate commercial software providing render scripting capability;provide at least one of parallel and distributed computing; andallow for human interaction; anda backend system adapted to: manage converting a 2D image data sequence into a 3D image data sequence by: receiving the 2D image data sequence;performing a temporal analysis on the 2D image data sequence to determine image motion cues;dividing the 2D image data sequence into scenes, each scene comprising at least one object, wherein the scenes are grouped;identifying objects in each scene and determining characteristics associated with each object, wherein the characteristics comprise a geometrical position and shape;determining whether each scene is suitable for automated processing based on scene classification;assigning each object to an object layer based on the characteristics associated with each object;generating a depth map for each object based on the object layer;collecting processing information associated with each object;generating converted object images using the processing information;storing the processing information into a render data record;combining the converted object images to produce a 3D scene;combining depth maps into full frame depth maps;filling occlusion regions in the converted 3D scene;generating a 3D image sequence using the full frame depth maps and 3D scene;outputting the converted 3D image and render data record;perform automated functions; andperform central database functions. 38. The system of claim 37, wherein the front-end system further comprises: work station terminals having a graphical user interface, wherein the number of work station terminals is scalable;a front-end system network switch adapted to interface with a backend system network switch;a pre-render adapted to provide the parallel and distributed computing; anda verification server adapted to output the verified converted image data sequence. 39. The system of claim 38, wherein the front-end further comprises a display area for allowing a converted 3D image sequence to be examined. 40. The system of claim 37, wherein the backend system further comprises: an intelligent controller server adapted to manage image sequence conversion processing;a central data storage;a data input/output device adapted to receive the 2D image data sequence;a backend system network switch adapted to interface with a front-end system network switch;one or more render cluster processors adapted to perform automated processes, wherein the number of render cluster processors is scalable; anda user controls interface adapted to allow users to monitor the image sequence conversion process.
연구과제 타임라인
LOADING...
LOADING...
LOADING...
LOADING...
LOADING...
이 특허에 인용된 특허 (99)
Sohn,Young wook, Apparatus and method for converting frame rate.
Geshwind, David Michael, Interactive computer system for creating three-dimensional image information and for converting two-dimensional image information for three-dimensional display systems.
Lubin Jeffrey ; Brill Michael Henry ; De Vries Aalbert ; Finard Olga, Method and apparatus for assessing the visibility of differences between two image sequences.
Lubin Jeffrey ; Brill Michael H. ; Pica Albert P. ; Crane Roger L. ; Paul Walter B. ; Gendel Gary A., Method and apparatus for assessing the visibility of differences between two signal sequences.
Hanna Keith James ; Kumar Rakesh ; Bergen James Russell ; Sawhney Harpreet Singh ; Lubin Jeffrey, Method and apparatus for enhancing regions of aligned images using flow estimation.
Black, Michael J.; Ju, Xuan; Minneman, Scott; Kimber, Donald G., Method and apparatus for generating a condensed version of a video sequence including desired affordances.
Lubin Jeffrey ; Peterson Heidi A. ; Spence Clay D. ; de Vries Aalbert, Method and apparatus for training a neural network to learn and use fidelity metric as a control mechanism.
Kaye, Michael C.; Best, Charles J. L.; Haynes, Robby R., Method and system for creating realistic smooth three-dimensional depth contours from two-dimensional images.
Kumar Rakesh ; Hanna Keith James ; Bergen James R. ; Anandan Padmanabhan ; Irani Michal, Method and system for image combination using a parallax-based technique.
Kumar, Rakesh; Hanna, Keith James; Bergen, James R.; Anandan, Padmanabhan; Williams, Kevin; Tinker, Mike, Method and system for rendering and combining images to form a synthesized view of a scene containing image information from a second image.
Burt Peter J. (Mercer County NJ) van der Wal Gooitzen S. (Mercer NJ) Kolczynski Raymond J. (Mercer NJ) Hingorani Rajesh (Mercer NJ), Method for fusing images and apparatus therefor.
Burt Peter J. (Princeton NJ) van der Wal Gooitzen S. (Hopewell Borough ; Mercer County NJ) Kolczynski Raymond J. (Hamilton Township ; Mercer County NJ) Hingorani Rajesh (West Windsor Township ; Merce, Method for fusing images and apparatus therefor.
Weisgerber Robert C. (245 E. 93d St. ; Suite 32A New York NY 10128), Method for imparting both high-impact cinematic images and conventional cinematic images on the same strip of motion pic.
Sezan Muhammed I. (Rochester NY) Ozkan Mehmet K. (Rochester NY) Fogel Sergei V. (Rochester NY), Method for temporally adaptive filtering of frames of a noisy image sequence using motion estimation.
Weisgerber Robert C. (246 E. 93rd St. New York NY 10128), Method for transitioning between two different frame rates during a showing of a single motion picture film.
Geshwind David M. (184-14 Midland Pkwy. Jamaica NY 11432) Handal Anthony H. (Blue Chip La. Westport CT 06880), Method to convert two dimensional motion pictures for three-dimensional systems.
Brill Michael Henry (Morrisville PA) Lubin Jeffrey (Middlesex NJ), Methods and apparatus for assessing the visibility of differences between two image sequences.
Boice Charles E. ; Hall Barbara A. ; Kaczmarczyk John M. ; Ngai Agnes Yee ; Pokrinchak Stephen P., Real-time encoding of video sequence employing two encoders and statistical analysis.
Burt Peter J. (Princeton NJ) Irani Michal (Princeton Junction NJ) Hsu Stephen Charles (East Windsor NJ) Anandan Padmanabhan (Lawrenceville NJ) Hansen Michael W. (New Hope PA), System for automatically aligning images to form a mosaic image.
Zhou, Samuel; Ye, Ping; Judkins, Paul, Systems and methods for digitally re-mastering or otherwise modifying motion pictures or other image sequences data.
Quinones Armando J. (Cincinnati OH) Rieck ; Jr. Harold P. (West Chester OH) Albrecht Richard W. (Fairfield OH) Sullivan Michael A. (Ballston Spa NY) Weisgerber Robert H. (Loveland OH) Plemmons Larry , Turbine disk cooling system.
Meade Robert J. (West Chester OH) Albrecht Richard W. (Fairfield OH) Weisgerber Robert H. (Loveland OH), Turbine disk interstage seal axial retaining ring.
Weisgerber Robert H. (Loveland OH) Albrecht Richard W. (Fairfield OH) Glynn Christopher C. (Hamilton OH) Kutney ; Jr. John T. (Cincinnati OH) Meade Robert J. (West Chester OH), Turbine disk interstage seal system.
Turner, Tara Handy; Goldberg, Evan M.; Schnittker, Matthew F.; Longson, Joseph W.; Neuman, Robert M., Apparatus and method for indicating depth of one or more pixels of a stereoscopic 3-D image comprised from a plurality of 2-D layers.
Ito, Yuichiro; Noge, Tetsuya, Computer-readable storage medium, information processing apparatus, information processing system, and information processing method.
Raymond, Mark A.; Soto, Hector Andres Porras; Karsch, Kevin, Conversion of a digital stereo image into multiple views with parallax for 3D viewing without glasses.
Mukherjee, Debargha; Wu, Chen; Wang, Meng, Depth-map generation for an input image using an example approximate depth-map associated with an example similar image.
Mukherjee, Debargha; Wu, Chen; Wang, Meng, Depth-map generation for an input image using an example approximate depth-map associated with an example similar image.
Turner, Tara Handy; Goldberg, Evan M.; Schnittker, Matthew F.; Longson, Joseph W.; Neuman, Robert M., Method and system for utilizing pre-existing image layers of a two-dimensional image to create a stereoscopic image.
Wang, Jue; Simons, David P.; Wilk, Daniel M.; Bai, Xue, Methods and apparatus for chatter reduction in video object segmentation using a variable bandwidth search region.
Wang, Jue; Simons, David P.; Wilk, Daniel M.; Bai, Xue, Methods and apparatus for chatter reduction in video object segmentation using optical flow assisted gaussholding.
Zhou, Samuel; Judkins, Paul; Ye, Ping, Methods and systems for digitally re-mastering of 2D and 3D motion pictures for exhibition with enhanced visual quality.
Ernst, Thomas Michael; Prieto, Thomas Edmund; Armstrong, Brian Stewart Randall, Motion tracking system for real time adaptive imaging and spectroscopy.
Ernst, Thomas Michael; Prieto, Thomas Edmund; Armstrong, Brian Stewart Randall, Motion tracking system for real time adaptive imaging and spectroscopy.
Ernst, Thomas Michael; Prieto, Thomas Edmund; Armstrong, Brian Stewart Randall, Motion tracking system for real time adaptive imaging and spectroscopy.
Bublitz, Daniel; Geissler, Enrico; Breuninger, Tobias; Kerwien, Norbert; Nieten, Christoph, Multifocal representation device and multifocal representation method for the three-dimensional representation of an object.
Blohmé, Patrick Ted Gert; Roush, Christopher Michael; Porseland, Karl Henrik Olav; Tavendale, Michael William; Paumgarten, Mathias; Margusity, Eric Henry; Falik, Wes; Fischer, Jens; Cave, Simon Frederick; Vicente, Israel Pastrana; Plattel, Cornelis Antonius; Brennan, Jonathan Patrick, Systems and methods for interactive video generation and rendering.
Ernst, Thomas Michael; Keating, Brian; Singh, Aditya; Zaitsev, Maxim; Herbst, Michael, Systems, devices, and methods for detecting false movements for motion correction during a medical imaging scan.
Yu, Jeffrey N.; Alameida, Jr., William Herman; Lovberg, John Arthur; Pan, Xiaojiang Jason; Engelmann, Michael, Systems, devices, and methods for tracking and compensating for patient motion during a medical imaging scan.
Yu, Jeffrey N.; Lovberg, John Arthur; Pan, Xiaojiang Jason, Systems, devices, and methods for tracking and compensating for patient motion during a medical imaging scan.
Ernst, Thomas Michael; Zahneisen, Benjamin Anton; Yu, Jeffrey N., Systems, methods, and devices for removing prospective motion correction from medical imaging scans.
※ AI-Helper는 부적절한 답변을 할 수 있습니다.