Low power framework for controlling image sensor mode in a mobile image capture device
원문보기
IPC분류정보
국가/구분
United States(US) Patent
등록
국제특허분류(IPC7판)
H04N-005/225
H04N-005/77
H04N-005/232
H04N-019/426
G06K-009/62
G06K-009/66
출원번호
US-0984869
(2015-12-30)
등록번호
US-10225511
(2019-03-05)
발명자
/ 주소
Lim, Suk Hwan
Aguera-Arcas, Blaise
출원인 / 주소
Google LLC
대리인 / 주소
Dority & Manning, P.A.
인용정보
피인용 횟수 :
0인용 특허 :
102
초록▼
The present disclosure provides an image capture, curation, and editing system that includes a resource-efficient mobile image capture device that continuously captures images. In particular, the present disclosure provides low power frameworks for controlling image sensor mode in a mobile image cap
The present disclosure provides an image capture, curation, and editing system that includes a resource-efficient mobile image capture device that continuously captures images. In particular, the present disclosure provides low power frameworks for controlling image sensor mode in a mobile image capture device. On example low power frame work includes a scene analyzer that analyzes a scene depicted by a first image and, based at least in part on such analysis, causes an image sensor control signal to be provided to an image sensor to adjust at least one of the frame rate and the resolution of the image sensor.
대표청구항▼
1. A resource-efficient image capture device, the image capture device comprising: an image sensor that provides raw image data, the image sensor operable to receive an image sensor control signal and to adjust, based at least in part on the image sensor control signal, at least one of a frame rate
1. A resource-efficient image capture device, the image capture device comprising: an image sensor that provides raw image data, the image sensor operable to receive an image sensor control signal and to adjust, based at least in part on the image sensor control signal, at least one of a frame rate and a resolution of the image sensor;an imaging pipeline that streams the raw image data and processes the raw image data to generate one or more first images;a temporary image buffer that temporarily stores the one or more first images; anda scene analyzer that analyzes a scene depicted by at least one of the one or more first images and, based at least in part on such analysis, causes the image sensor control signal to be provided to the image sensor to adjust at least one of the frame rate and the resolution of the image sensor;wherein the scene analyzer is operable to determine a desirability of the scene depicted by the at least one of the one or more first images;wherein when the scene analyzer determines that the scene depicted by the at least one of the one or more first images is desirable, the scene analyzer causes the image sensor control signal to be provided to the image sensor to increase at least one of the frame rate or the resolution at which the image sensor provides the raw image data; andwherein when the scene analyzer determines that the scene depicted by the at least one of the one or more first images is not desirable, the scene analyzer causes the image sensor control signal to be provided to the image sensor to decrease at least one of the frame rate or the resolution at which the image sensor provides the raw image data. 2. The image capture device of claim 1, wherein the image sensor is further operable to adjust, based at least in part on the image sensor control signal, at least one of an exposure time, a gain, a region of interest, a crop, and a binning mode of the image sensor, and wherein the scene analyzer causes, based at least in part on the analysis of the scene, the image sensor control signal to be provided to the image sensor to adjust at least one of the exposure time, the gain, the region of interest, the crop, and the binning mode of the image sensor. 3. The image capture device of claim 1, wherein: the image capture device is operable in at least a low resolution capture mode, a high resolution capture mode, and a burst capture mode;during a regular mode of operation, the image capture device periodically transitions between the low resolution capture mode and the high resolution capture mode; andwhen the scene analyzer determines that a particular scene depicted by a particular image is desirable, the scene analyzer causes the image capture device to transition to the burst capture mode. 4. A method for operating an image capture device, the method comprising: outputting, by an image sensor of the image capture device, raw image data, the image sensor capable of adjusting at least one of a frame rate and a resolution in response to an image sensor control signal;streaming, by an imaging pipeline of the image capture device, the raw image data from the image sensor;processing, by the imaging pipeline, the raw image data to generate a first image;analyzing, by the image capture device, a scene depicted by the first image; andproviding, by the image capture device based at least in part on the analysis of the scene depicted by the first image, the image sensor control signal to cause the image sensor to adjust at least one of the frame rate and the resolution of the image sensor;wherein analyzing, by the image capture device, the scene depicted by the first image comprises determining, by the image capture device, that the scene depicted by the first image has one or more desirable characteristics; andwherein providing, by the image capture device based at least in part on the analysis of the scene depicted by the first image, the image sensor control signal comprises providing, by the image capture device based at least in part on the analysis of the scene depicted by the first image, the image sensor control signal to cause the image sensor to increase at least one of the frame rate or the resolution. 5. The method of claim 4, further comprising: providing, by the image capture device based at least in part on the analysis of the scene depicted by the first image, the image sensor control signal to adjust at least one of an exposure time, a gain, a region of interest, a crop, and a binning mode of the image sensor. 6. The method of claim 4, wherein: analyzing, by the image capture device, the scene depicted by the first image comprises determining, by the image capture device, that the scene depicted by the first image has one or more desirable characteristics; andproviding, by the image capture device based at least in part on the analysis of the scene depicted by the first image, the image sensor control signal to cause the image sensor to adjust at least one of the frame rate and the resolution comp rises selecting, by the image capture device, one of a plurality of different capture modes in which to operate, the plurality of different capture modes respectively having a plurality of different combinations of frame rates and resolutions. 7. A resource-efficient image capture device, the image capture device comprising: an image sensor that provides raw image data, the image sensor operable to receive an image sensor control signal and to adjust, based at least in part on the image sensor control signal, a frame rate of the image sensor;an imaging pipeline that streams the raw image data and processes the raw image data to generate one or more first images;a temporary image buffer that temporarily stores the one or more first images; anda scene analyzer that analyzes a scene depicted by at least one of the one or more first images and, based at least in part on such analysis, causes the image sensor control signal to be provided to the image sensor to adjust the frame rate of the image sensor;wherein the scene analyzer is operable to determine a desirability of the scene depicted by the at least one of the one or more first images;wherein when the scene analyzer determines that the scene depicted by the at least one of the one or more first images is desirable, the scene analyzer causes the image sensor control signal to be provided to the image sensor to increase the frame rate at which the image sensor provides the raw image data; andwherein when the scene analyzer determines that the scene depicted by the at least one of the one or more first images is not desirable, the scene analyzer causes the image sensor control signal to be provided to the image sensor to decrease the frame rate at which the image sensor provides the raw image data. 8. The image capture device of claim 7, wherein the image sensor is further operable to adjust, based at least in part on the image sensor control signal, at least one of an exposure time, a gain, a region of interest, a crop, and a binning mode of the image sensor, and wherein the scene analyzer causes, based at least in part on the analysis of the scene, the image sensor control signal to be provided to the image sensor to adjust at least one of the exposure time, the gain, the region of interest, the crop, and the binning mode of the image sensor. 9. A resource-efficient image capture device, the image capture device comprising: an image sensor that provides raw image data, the image sensor operable to receive an image sensor control signal and to adjust, based at least in part on the image sensor control signal, at least one of a frame rate and a resolution of the image sensor;an imaging pipeline that streams the raw image data and processes the raw image data to generate one or more first images;a temporary image buffer that temporarily stores the one or more first images;a scene analyzer that analyzes a scene depicted by at least one of the one or more first images and, based at least in part on such analysis, causes the image sensor control signal to be provided to the image sensor to adjust at least one of the frame rate and the resolution of the image sensor, wherein, in at least one of a plurality of capture modes, the scene analyzer determines, based at least in part on the analysis of the scene depicted by the at least one of the one or more first images, whether to store the one or more first images in a non-volatile memory or to discard the one or more first images from the temporary image buffer without storing the one or more first images in the non-volatile memory;an image compression component that, in the at least one of the plurality of capture modes, compresses the one or more first images only when the scene analyzer determines that the one or more first images should be stored in the non-volatile memory, such that the image compression component expends energy on compressing only the first images selected for storage at the non-volatile memory; andthe non-volatile memory to which the one or more first images are written after compression. 10. The image capture device of claim 9, wherein, in at least a high resolution capture mode: the image sensor provides high resolution raw image data;the image capture device further comprises a temporary raw data buffer that temporarily stores the high resolution raw image data;the image capture device further comprises a down sampler that streams the high resolution raw image data from the image sensor and down samples the high resolution raw image data to output low resolution raw image data;the imaging pipeline streams the low resolution raw image data from the down sampler and processes the low resolution raw image data to generate low resolution versions of the one or more first images;the temporary image buffer temporarily stores the low resolution versions of the one or more first images; andthe scene analyzer analyzes at least one of the low resolution versions of the one or more first images to determine whether to store the one or more first images in the non-volatile memory or to discard the one or more first images. 11. The image capture device of claim 10, wherein, in at least the high resolution capture mode, when the scene analyzer determines that the one or more first images should be stored in the non-volatile memory, the imaging pipeline receives the high resolution raw image data from the temporary raw data buffer and processes the high resolution raw image data to generate high resolution versions of the one or more first images, and the image compression component compresses the high resolution versions of the one or more first images for storage at the non-volatile memory. 12. The image capture device of claim 10, wherein, in at least a low resolution capture mode: the image sensor performs binning to provide low resolution raw image data;the imaging pipeline streams the low resolution raw image data from the image sensor and processes the low resolution raw image data to generate low resolution versions of the one or more first images;the temporary image buffer temporarily stores the low resolution versions of the one or more first images; andthe scene analyzer analyzes at least one of the low resolution versions of the one or more first images to determine whether to non-temporarily store the low resolution versions of the one or more first images or to discard the low resolution versions of the one or more first image. 13. The image capture device of claim 9, wherein the temporary image buffer comprises dynamic random-access memory. 14. A resource-efficient image capture device, the image capture device comprising: an image sensor that provides raw image data, the image sensor operable to receive an image sensor control signal and to adjust, based at least in part on the image sensor control signal, at least one of a frame rate and a resolution of the image sensor;an imaging pipeline that streams the raw image data and processes the raw image data to generate one or more first images;a temporary image buffer that temporarily stores the one or more first images;a scene analyzer that analyzes a scene depicted by at least one of the one or more first images and, based at least in part on such analysis, causes the image sensor control signal to be provided to the image sensor to adjust at least one of the frame rate and the resolution of the image sensor;an image compression component that compresses images; anda non-volatile memory to which images are written after compression;wherein, in at least a burst capture mode of operation: the imaging pipeline streams the raw image data from the image sensor and processes the raw image data to generate a second image; andthe second image is provided to the image compression component for compression and then stored at the non-volatile memory without being analyzed by the scene analyzer. 15. The image capture device of claim 14, wherein, in at least the burst capture mode, the image capture device captures high resolution images at a high frame rate and performs imaging pipeline operations, compression, and storage at the non-volatile memory for all of the captured high resolution images. 16. A resource-efficient image capture device, the image capture device comprising: an image sensor that provides raw image data, the image sensor operable to receive an image sensor control signal and to adjust, based at least in part on the image sensor control signal, at least one of a frame rate and a resolution of the image sensor;an imaging pipeline that streams the raw image data and processes the raw image data to generate one or more first images;a temporary image buffer that temporarily stores the one or more first images; anda scene analyzer that analyzes a scene depicted by at least one of the one or more first images and, based at least in part on such analysis, causes the image sensor control signal to be provided to the image sensor to adjust at least one of the frame rate and the resolution of the image sensor;wherein the scene analyzer comprises: at least one neural network that receives the first image as an input and outputs at least one descriptor of a desirability of the scene depicted by the first image; anda mode controller that selects, based at least in part on the at least one descriptor, one of a plurality of capture states, wherein the image capture device thereafter operates in the capture state selected by the mode controller.
연구과제 타임라인
LOADING...
LOADING...
LOADING...
LOADING...
LOADING...
이 특허에 인용된 특허 (102)
Sharifi, Matthew; Petrou, David, Annotating relevant content in a screen capture image.
Kuberka, Cheryl J.; Barnum, David C.; Williams, Frances C.; Border, John N.; Johnson, Kenneth A., Camera configurable for autonomous self-learning operation.
Bernardi Bryan D. (Rochester NY) McIntyre Dale F. (Honeoye Falls NY) Dunsmore Clay A. (Fairport NY) Wolcott Dana W. (Honeoye Falls NY), Camera on-board voice recognition.
Kuchta Daniel W. (Brockport NY) Sucy Peter J. (Hamlin NY), Electronic still camera providing multi-format storage of full and reduced resolution images.
Strub, Henry B.; Burgess, David A.; Johnson, Kimberly H.; Cohen, Jonathan R.; Reed, David P., Hybrid recording unit including portable video recorder and auxillary device.
Ejima, Satoshi; Nozaki, Hirotake; Hiraide, Fumio, Image processing apparatus having image selection function, and recording medium having image selection function program.
Strub, Henry B.; Burgess, David A.; Johnson, Kimberly H.; Cohen, Jonathan R.; Reed, David P.; Aiello, G. Roberto, Low attention recording unit for use by vigorously active recorder.
Ostojic, Bojana; Glein, Christopher A; Gibson, Mark R.; Vong, William H; Flora, William T; Alton, Benjamin N; Newell, Mark S, Media user interface gallery control.
Schaffer, James David; Ali, Walid; Eshelman, Larry J.; Cohen-Bacrie, Claude; Lagrange, Jean-Michel; Levrier, Claire; Villain, Nicholas; Entrekin, Robert R., Method and apparatus for automatically developing a high performance classifier for producing medically meaningful descriptors in medical diagnosis imaging.
Lovell Douglas C. (40 Liss Rd. Wappinger\s Falls NY 12590-1609) Lorincz-Lovell Rose M. (40 Liss Rd. Wappinger\s Falls NY 12590-1609), Method and apparatus for producing animated drawings and in-between drawings.
Lee, Il Yong; Kim, Sung Hyun; Kim, Lag Young; Hong, Yun Pyo; Byun, Seong Chan, Method for processing image data in portable electronic device, and portable electronic device having camera thereof.
Steinberg, Eran; Prilutsky, Yury; Corcoran, Peter; Bigioi, Petronel, Perfecting of digital image capture parameters within acquisition devices using face detection.
Tedesco, Daniel E.; Jorasch, James A.; Gelman, Geoffrey M.; Walker, Jay S.; Tulley, Stephen C.; O'Neil, Vincent M.; Alderucci, Dean P., System for image analysis in a network that is structured with multiple layers and differentially weighted neurons.
Grosz, Aryk Erwin; Medina, Yedidya, Systems and methods for automatically generating a photo-based project having a flush photo montage on the front cover.
Bialynicka-Birula, Iwona; Aguera-Arcas, Blaise; Ramage, Daniel; McMahan, Hugh Brendan; Lange, Oliver Fritz; Fortuna, Emily Anne; Tyamagundlu, Divya; Holbrook, Jess; Kohlhepp, Kristine; Payne, Juston; Duleba, Krzysztof; Vanik, Benjamin; Lentz, Alison; Clapper, Jon Gabriel; Lovejoy, Joshua Denali; Donsbach, Aaron Michael, Systems and methods that leverage deep learning to selectively store images at a mobile image capture device.
Isogai, Kuniaki; Kawamura, Takashi; Kawabata, Akihiro, Video analysis apparatus and method for calculating interpersonal relationship evaluation value using video analysis.
Lin,Yun Ting; Gutta,Srinivas; Brodsky,Tomas; Philomin,Vasanth, Video monitoring system employing hierarchical hidden markov model (HMM) event learning and classification.
※ AI-Helper는 부적절한 답변을 할 수 있습니다.