Image capturing device and digital zooming method thereof
IPC분류정보
국가/구분
United States(US) Patent
등록
국제특허분류(IPC7판)
H04N-005/345
H04N-005/265
H04N-005/232
G06T-003/00
G06T-003/40
G06T-007/00
출원번호
US-0571021
(2014-12-15)
등록번호
US-9325899
(2016-04-26)
우선권정보
TW-103139384 A (2014-11-13)
발명자
/ 주소
Chou, Hong-Long
Tseng, Yi-Hong
Chang, Wen-Yan
Wang, Yu-Chih
Wang, Tsan-Wei
Yu, Yi-Yi
출원인 / 주소
Altek Semiconductor Corp.
대리인 / 주소
J.C. Patents
인용정보
피인용 횟수 :
2인용 특허 :
3
초록▼
An image capturing device and a digital zooming method thereof are proposed in the invention. The method includes the following steps. A scene is captured by a primary lens and a secondary lens to generate a primary image and a secondary image. Image rectification is then performed on the primary an
An image capturing device and a digital zooming method thereof are proposed in the invention. The method includes the following steps. A scene is captured by a primary lens and a secondary lens to generate a primary image and a secondary image. Image rectification is then performed on the primary and the secondary images to obtain two corresponding rectified images. Feature points detected and matched from the two rectified images are used to determine a corresponding overlapping region, respectively. Pixel displacement and depth map in the two overlapping regions could be calculated and estimated. Image zooming and warping techniques are performed on the two rectified images to obtain corresponding warped images using a recalculated homography matrix according to each specific zooming factor. Finally, the two overlapping regions in the warped image are fused by a weighted blending approach to generate a digital zoomed image.
대표청구항▼
1. A digital zooming method, adapted to an image capturing device having a primary lens and a secondary lens, comprising: capturing a scene by using the primary lens and the secondary lens to generate a primary image and a secondary image;performing image rectification on the primary image and the s
1. A digital zooming method, adapted to an image capturing device having a primary lens and a secondary lens, comprising: capturing a scene by using the primary lens and the secondary lens to generate a primary image and a secondary image;performing image rectification on the primary image and the secondary image to generate a primary rectified image and a secondary rectified image;performing feature point detection on the primary rectified image and the secondary rectified image so as to detect overlapping regions respectively in the primary rectified image and the secondary rectified image, and further obtaining a plurality of pixel displacements and a depth map of the overlapping regions respectively in the primary rectified image and the secondary rectified image;when a zooming factor is between 1 and a primary-secondary image factor, performing image zooming and image warping on the primary rectified image and the secondary rectified image to generate a primary warped image and a secondary warped image according to the zooming factor, the pixel displacements, and the depth map, wherein the primary-secondary image factor is a ratio of the secondary rectified image to the primary rectified image; andperforming image fusion on overlapping regions respectively in the primary warped image and the secondary warped image to generate a digital zoomed image. 2. The digital zooming method of claim 1, wherein the step of performing image rectification on the primary image and the secondary image to generate the primary rectified image and the secondary rectified image comprises: obtaining a plurality of rectification parameters associated with the primary lens and the secondary lens; andrectifying the primary image and the secondary image to generate the primary rectified image and the secondary rectified image according to the rectification parameters. 3. The digital zooming method of claim 1, wherein the step of performing feature point detection on the primary rectified image and the secondary rectified image so as to detect the overlapping regions respectively in the primary rectified image and the secondary rectified image, and further obtaining the pixel displacements and the depth map of the overlapping regions respectively in the primary rectified image and the secondary rectified image comprises: detecting a plurality of feature points from the primary rectified image and the secondary rectified image;identifying a plurality of feature point correspondences to calculate a homography matrix according to color information of a plurality of neighboring points of each of the feature points in the primary rectified image and the secondary rectified image;obtaining the overlapping regions respectively in the primary rectified image and the secondary rectified image according to the homography matrix and accordingly obtaining each of the pixel displacements; andperforming stereo matching on each of the feature point correspondences in the primary rectified image and the secondary rectified image to obtain the depth map. 4. The digital zooming method of claim 1, wherein the primary lens and the secondary lens have different fields of view and same distortion levels, wherein the field of view of the primary lens is greater than the field of view of the secondary lens, and wherein when the zooming factor is between 1 and the primary-secondary image factor, the step of performing image zooming and image warping on the primary rectified image and the secondary rectified image to generate the primary warped image and the secondary warped image according to the zooming factor, the pixel displacements, and the depth map comprises: enlarging the primary rectified image to generate an enlarged primary rectified image according to the zooming factor;shrinking the secondary rectified image to generate a shrunken secondary rectified image according to the zooming factor; andperforming image warping on the enlarged primary rectified image and the shrunken secondary rectified image to generate the primary warped image and the secondary warped image according to the pixel displacements and the depth map, wherein a warping level is associated with the depth map. 5. The digital zooming method of claim 1, wherein the primary lens and the secondary lens have same fields of view and same distortion levels, and wherein after the step of performing image rectification on the primary image and the secondary image to generate the primary rectified image and the secondary rectified image, the digital zooming method further comprises: performing image binning on the primary rectified image to generate a binned primary rectified image;performing image cropping on the secondary rectified image to generate a cropped secondary rectified image, wherein a size of the binned primary rectified image and a size of the cropped secondary rectified image are the same; andsetting the binned primary rectified image and the cropped secondary rectified image respectively as the primary rectified image and the secondary rectified image. 6. The digital zooming method of claim 1, wherein when the zooming factor is between 1 and the primary-secondary image factor, the step of performing image zooming and image warping on the primary rectified image and the secondary rectified image to generate the primary warped image and the secondary warped image according to the zooming factor, the pixel displacements, and the depth map comprises: enlarging the primary rectified image to generate an enlarged primary rectified image according to the zooming factor;shrinking the secondary rectified image to generate a shrunken secondary rectified image according to the zooming factor; andperforming image warping on the enlarged primary rectified image and the shrunken secondary rectified image to generate the primary warped image and the secondary warped image according to the pixel displacements and the pixel depth map, wherein a warping level is associated with the depth map. 7. The digital zooming method of claim 1, wherein the primary lens and the secondary lens have same fields of view and different distortion levels, wherein the distortion level of the primary lens is much less than the distortion level of the secondary lens, and wherein after the step of performing image rectification on the primary image and the secondary image to generate the primary rectified image and the secondary rectified image, the digital zooming method further comprises: performing image cropping on a center region of the secondary rectified image to generate a cropped secondary rectified image; andsetting the cropped secondary rectified image as the secondary rectified image. 8. The digital zooming method of claim 7, wherein when the zooming factor is between 1 and the primary-secondary image factor, the step of performing image zooming and image warping on the primary rectified image and the secondary rectified image to generate the primary warped image and the secondary warped image according to the zooming factor, the pixel displacements, and the depth map comprises: enlarging the primary rectified image to generate an enlarged primary rectified image according to the zooming factor;shrinking the secondary rectified image to generate a shrunken secondary rectified image; andperforming image warping on the enlarged primary rectified image and the shrunken secondary rectified image to generate the primary warped image and the secondary warped image according to the pixel displacements and the depth map, wherein a warping level is associated with the depth map. 9. The digital zooming method of claim 1, wherein the step of performing image fusion on the overlapping regions respectively in the primary warped image and the secondary warped image to generate the digital zoomed image comprises: setting a first weight and a second weight respectively corresponding to the primary warped image and the secondary warped image according to the zooming factor;performing image fusion on the overlapping regions respectively in the primary warped image and the secondary warped image to generate a fused overlapping image based on the first weight and the second weight; andsubstituting the overlapping regions respectively in the primary warped image and the secondary warped image by the fused overlapping image to generate the digital zoomed image. 10. The digital zooming method of claim 1, wherein when the zooming factor is greater than the primary-secondary image factor, the digital zooming method further comprises: enlarging the secondary rectified image to generate an enlarged secondary rectified image according to the zooming factor;performing image warping on the shrunken secondary rectified image to generate the secondary warped image according to the pixel displacements and the depth map, wherein a warping level is associated with the depth map; andsetting the secondary warped image as the digital zoomed image. 11. The digital zooming method of claim 1, wherein when the zooming factor is less than 1, the digital zooming method further comprises: shrinking the primary rectified image to generate a shrunken primary rectified image;performing image warping on the shrunken primary rectified image to generate the primary warped image according to the pixel displacements and the pixel depth map, wherein a warping level is associated with the depth map; andsetting the primary warped image as the digital zoomed image. 12. An image capturing device comprising: a primary lens;a secondary lens;a storage unit, recording a plurality of modules; andone or more processing unit, coupled to the primary lens, the secondary lens and the storage unit, and accessing and executing the modules recorded in the storage unit, wherein the modules comprise: an image capturing module, capturing a scene by using the primary lens and the secondary lens to generate a primary image and a secondary image;an image preprocessing module, performing image rectification on the primary image and the secondary image to generate a primary rectified image and a secondary rectified image;a feature analyzing module, performing feature point detection on the primary rectified image and the secondary rectified image so as to detect overlapping regions respectively in the primary rectified image and the secondary rectified image, and further obtaining a plurality of pixel displacements and a depth map of the overlapping regions respectively in the primary rectified image and the secondary rectified image;an image zooming-warping module, when a zooming factor is between 1 and a primary-secondary image factor, performing image zooming and image warping on the primary rectified image and the secondary rectified image to generate a primary warped image and a secondary warped image according to the zooming factor, the pixel displacements, and the depth map, wherein the primary-secondary image factor is a ratio of the secondary rectified image to the primary rectified image; andan image fusion module, performing image fusion on overlapping regions respectively in the primary warped image and the secondary warped image to generate a digital zoomed image. 13. The image capturing device of claim 12, wherein the image preprocessing module obtains a plurality of rectification parameters associated with the primary lens and the secondary lens, and rectifies the primary image and the secondary image to generate the primary rectified image and the secondary rectified image according to the rectification parameters. 14. The image capturing device of claim 12, wherein the feature analyzing module detects a plurality of feature points from the primary rectified image and the secondary rectified image, identifies a plurality of feature point correspondences to calculate a homography matrix according to color information of a plurality of neighboring points of each of the feature points in the primary rectified image and the secondary rectified image, obtains the overlapping regions respectively in the primary rectified image and the secondary rectified image according to the homography matrix and accordingly obtains each of the pixel displacements, and performs stereo matching on each of the feature point correspondences in the primary rectified image and the secondary rectified image to obtain the depth map. 15. The image capturing device of claim 12, wherein the primary lens and the secondary lens have different fields of view and same distortion levels, wherein the field of view of the primary lens is greater than the field of view of the secondary lens, and wherein when the zooming factor is between 1 and the primary-secondary image factor, the image zooming-warping module enlarges the primary rectified image to generate an enlarged primary rectified image according to the zooming factor, shrinks the secondary rectified image to generate a shrunken secondary rectified image according to the zooming factor, and performs image warping on the enlarged primary rectified image and the shrunken secondary rectified image to respectively generate the primary warped image and the secondary warped image according to the pixel displacements and the depth map, wherein a warping level is associated with the depth map. 16. The image capturing device of claim 12, wherein the primary lens and the secondary lens have same fields of view and same distortion levels, and the image preprocessing module further performs image binning on the primary rectified image to generate a binned primary rectified image, performs image cropping on the secondary rectified image to generate a cropped secondary rectified image, and sets the binned primary rectified image and the cropped secondary rectified image respectively as the primary rectified image and the secondary rectified image, wherein a size of the binned primary rectified image and a size of the cropped secondary rectified image are the same. 17. The image capturing device according to claim 16, wherein when the zooming factor is between 1 and the primary-secondary image factor, the image zooming-warping module enlarges the primary rectified image to generate an enlarged primary rectified image according to the zooming factor, shrinks the secondary rectified image to generate a shrunken secondary rectified image according to the zooming factor, and performs image warping on the enlarged primary rectified image and the shrunken secondary rectified image to generate the primary warped image and the secondary warped image according to the pixel displacements and the pixel depth map, wherein a warping level is associated with the depth map. 18. The image capturing device according to claim 12, wherein the primary lens and the secondary lens have the same fields of view and different distortion levels, wherein the distortion level of the primary lens is much less than the distortion level of the secondary lens, and wherein the image preprocessing module further performs image cropping on a center region of the secondary rectified image to generate a cropped secondary rectified image, and sets the cropped secondary rectified image as the second rectified image. 19. The image capturing device according to claim 18, wherein when the zooming factor is between 1 and the primary-secondary image factor, the image zooming-warping module enlarges the primary rectified image to generate an enlarged primary rectified image according to the zooming factor, shrinks the secondary rectified image to generate a shrunken secondary rectified image, and performs image warping on the enlarged primary rectified image and the shrunken secondary rectified image to generate the primary warped image and the secondary warped image according to the pixel displacements and the depth map, wherein a warping level is associated with the depth map. 20. The image capturing device according to claim 12, wherein the image fusion module sets a first weight and a second weight respectively corresponding to the primary warped image and the secondary warped image according to the zooming factor, performs image fusion on the overlapping regions respectively in the primary warped image and the secondary warped image to generate a fused overlapping image based on the first weight and the second weight, and substitutes the overlapping regions respectively in the primary warped image and the secondary warped image by the fused overlapping image to generate the digital zoomed image. 21. The image capturing device according to claim 12, wherein when the zooming factor is greater than the primary-secondary image factor, the image zooming-warping module further enlarges the secondary rectified image to generate an enlarged secondary rectified image according to the zooming factor, performs image warping on the shrunken secondary rectified image to generate the secondary warped image according to the pixel displacements and the depth map, and sets the secondary warped image as the digital zoomed image, wherein a warping level is associated with the depth map. 22. The image capturing device according to claim 12, wherein when the zooming factor is less than 1, the image zooming-warping module shrinks the primary rectified image to generate a shrunken primary rectified image, performs image warping on the shrunken primary rectified image to generate the primary warped image according to the pixel displacements and the depth map, and sets the primary warped image as the digital zoomed image, wherein a warping level is associated with the depth map.
연구과제 타임라인
LOADING...
LOADING...
LOADING...
LOADING...
LOADING...
이 특허에 인용된 특허 (3)
Border, John N.; Enge, Amy D.; Morales, Efrain O.; Pillman, Bruce H.; Jacoby, Keith A.; Adams, Jr., James E.; Palum, Russell J.; Gallagher, Andrew C., Camera using multiple lenses and image sensors in a rangefinder configuration to provide a range map.
Parulski, Kenneth A.; Janson, Jr., Wilbert F.; Border, John N.; Fredlund, John R.; Manico, Joseph A.; Parada, Jr., Robert J., Method and apparatus for operating a dual lens camera to augment an image.
Holzer, Stefan Johannes Josef; Miller, Stephen David; Saelzle, Martin; Rusu, Radu Bogdan, Artificially rendering images using viewpoint interpolation and extrapolation.
Zhang, Jianming; Wei, Zijun; Lin, Zhe; Shen, Xiaohui; Mech, Radomir, Method for using deep learning for facilitating real-time view switching and video editing on computing devices.
※ AI-Helper는 부적절한 답변을 할 수 있습니다.