Low power framework for processing, compressing, and transmitting images at a mobile image capture device
IPC분류정보
국가/구분
United States(US) Patent
등록
국제특허분류(IPC7판)
H04N-005/228
H04N-005/77
H04N-005/232
H04N-019/426
G06K-009/62
G06K-009/66
H04N-005/14
H04N-005/76
출원번호
US-0984825
(2015-12-30)
등록번호
US-9838641
(2017-12-05)
발명자
/ 주소
Lim, Suk Hwan
Aguera-Arcas, Blaise
출원인 / 주소
Google LLC
대리인 / 주소
Dority & Manning, P.A.
인용정보
피인용 횟수 :
3인용 특허 :
84
초록▼
The present disclosure provides an image capture, curation, and editing system that includes a resource-efficient mobile image capture device that continuously captures images. In particular, the present disclosure provides low power frameworks for processing, compressing, and transmitting images at
The present disclosure provides an image capture, curation, and editing system that includes a resource-efficient mobile image capture device that continuously captures images. In particular, the present disclosure provides low power frameworks for processing, compressing, and transmitting images at a mobile image capture device. One example low power framework includes a scene analyzer that analyzes a scene depicted by a first image and determines whether to store the first image in a non-volatile memory or to discard the first image from a temporary image buffer without storing the first image in the non-volatile memory.
대표청구항▼
1. An image capture device that has a low-power framework for processing imagery, the image capture device comprising: an image sensor that provides high resolution raw image data;a temporary raw data buffer that temporarily stores the high resolution raw image data;a down sampler that down samples
1. An image capture device that has a low-power framework for processing imagery, the image capture device comprising: an image sensor that provides high resolution raw image data;a temporary raw data buffer that temporarily stores the high resolution raw image data;a down sampler that down samples the high resolution raw image data to output low resolution raw image data;an imaging pipeline that streams the low resolution raw image data from the down sampler and processes the low resolution raw image data to generate low resolution versions of one or more first images;a temporary image buffer that temporarily stores the low resolution versions of the one or more first images;a scene analyzer that analyzes at least a portion of a scene depicted by at least one of the low resolution versions one or more first images and determines whether to store the one or more first images in a non-volatile memory or to discard the low resolution versions of the one or more first images from the temporary image buffer without storing the one or more first images in the non-volatile memory;an image compression component that compresses the one or more first images when the scene analyzer determines that the one or more first images should be stored in the non-volatile memory; andthe non-volatile memory to which the one or more first images are written after compression. 2. The image capture device of claim 1, wherein, when the scene analyzer determines that the one or more first images should be stored in the non-volatile memory: the image compression component compresses the low resolution versions of the one or more first images; andthe image capture device writes the low resolution versions of the one or more first images to the non-volatile memory after compression. 3. The image capture device of claim 1, wherein, in a base capture mode of operation, the image sensor periodically transitions between providing high resolution raw image data and low resolution raw image data. 4. The image capture device of claim 1, wherein, when the scene analyzer determines that the one or more first images should be stored in the non-volatile memory: the imaging pipeline receives the high resolution raw image data from the temporary raw data buffer and processes the high resolution raw image data to generate high resolution versions of the one or more first images;the image compression component compresses the high resolution versions of the one or more first images; andthe image capture device writes the high resolution versions of the one or more first images to the non-volatile memory after compression. 5. The image capture device of claim 4, wherein, when the scene analyzer determines that the one or more first images should be stored in the non-volatile memory, the low resolution versions of the one or more first images are also compressed and written to the non-volatile memory. 6. The image capture device of claim 1, wherein, in at least a second mode of operation: the imaging pipeline streams the raw image data from the image sensor and processes the raw image data to generate a second image;the second image is provided to the image compression component for compression and then storage at the non-volatile memory without being analyzed by the scene analyzer. 7. The image capture device of claim 6, wherein the second mode of operation comprises a burst mode of operation in which the image capture device captures high resolution images at a high frame rate and performs imaging pipeline operations, compression, and storage at the non-volatile memory for all of the captured high resolution images. 8. The image capture device of claim 6, wherein the second mode of operation comprises a hyper-lapse mode of operation in which the image capture device captures mid resolution images at a high frame rate and performs imaging pipeline operations, compression, and storage at the non-volatile memory for all of the captured mid resolution images. 9. The image capture device of claim 1, wherein the scene analyzer comprises: at least one neural network that receives the at least one of the one or more first images as an input and outputs at least one descriptor of a desirability of the scene depicted by at least one of the one or more first images; anda save controller that determines, based at least in part on the at least one descriptor, whether to store the one or more first images in the non-volatile memory or to discard the one or more first images from the temporary image buffer without storing the one or more first images in the non-volatile memory. 10. The image capture device of claim 1, wherein the image capture device comprises a mobile image capture device that, at least in operation, continuously captures and processes imagery. 11. A method for processing imagery, the method comprising: outputting, by an image sensor of an image capture device, high resolution raw image data;temporarily storing, by a temporary raw data buffer of the image capture device, the high resolution raw image data; anddown sampling the high resolution raw image data to output low resolution raw image data;processing, by the imaging pipeline, the low resolution raw image data to generate a low resolution version of a first image;temporarily storing, by a temporary image buffer of the image capture device, the low resolution version of the first image;analyzing a scene depicted by the low resolution version of the first image;determining based at least in part on the analysis of the scene depicted by the low resolution version of the first image whether to store the first image in the non-volatile memory or to discard the low resolution version of the first image from the temporary image buffer without storing the first image in the non-volatile memory; andwhen it is determined that the first image should be stored in the non-volatile memory: compressing the first image; andwriting the first image to the non-volatile memory after compression. 12. The method of claim 11, wherein, when it is determined that the first image should be stored in the non-volatile memory: compressing the first image comprises compressing the low resolution version of the first image; andwriting the first image to the non-volatile memory comprises writing the low resolution version of the first image to the non-volatile memory after compression. 13. The method of claim 11, further comprising, when it is determined that the first image should be stored in the non-volatile memory: receiving, by the imaging pipeline; the high resolution raw image data from the temporary raw data buffer;processing, by the imaging pipeline, the high resolution raw image data to generate a high resolution version of the first image;wherein compressing the first image comprises compressing the high resolution version of the first image; andwherein writing the first image to the non-volatile memory comprises writing the high resolution version of the first image to the non-volatile memory after compression. 14. The method of claim 11, wherein; in at least one mode of operation, the method comprises: outputting, by the image sensor of an image capture device, low resolution raw image data rather than high resolution raw image data. 15. The method of claim 11, further comprising: streaming, by the imaging pipeline, a second set of raw image data from the image sensor;processing, by the imaging pipeline, the second set of raw image data to generate a second image; andwithout providing the second image to the scene analyzer: compressing the second image; andwriting the second image to the non-volatile memory after compression. 16. A mobile image capture device that has a low power image capture framework that is operable in a plurality of different capture modes, the image capture framework comprising: an image sensor operable to output raw image data of different resolutions depending on the capture mode;one or more image processing components operable to process the raw image data into one or more images;one or more temporary buffers operable to temporarily store the one or more images;a scene analyzer operable to analyze a scene depicted by at least one of the one or more images and determine, based at least in part on such analysis, whether to store such image at a non-volatile memory or to discard such image without storing such image at the non-volatile memory;an image compression component operable to compress one or more images; andthe non-volatile memory operable to store non-temporary copies of one or more images,wherein in a first capture mode of the plurality of different capture modes: the scene analyzer analyzes each image to determine whether to store such image at the non-volatile memory or to discard such image without storing such image at the non-volatile memory; andonly images that the scene analyzer has determined should be stored in the non-volatile memory are provided to the image compression component for compression and then written to the non-volatile memory; andwherein in a second capture mode of the plurality of different capture modes, all of the images are provided to the image compression component for compression and then written to the non-volatile memory without being analyzed by the scene analyzer. 17. The mobile image capture device of claim 16, wherein, in at least one of the plurality of capture modes: the image sensor provides high resolution raw image data for each of the images;the one or more temporary buffers comprise a temporary raw data buffer that temporarily stores the high resolution raw image data for each of the images;the image capture framework further comprises a down sampler that streams the high resolution raw image data from the image sensor and down samples the high resolution raw image data to output low resolution raw image data;the one or more image processing components stream the low resolution raw image data from the down sampler and process the low resolution raw image data to generate a low resolution version of each of the images;the one or more temporary buffers temporarily store the low resolution version of each of the images;the scene analyzer analyzes the low resolution version of each of the images to determine whether to store such image at the non-volatile memory or to discard such image; andwhen the scene analyzer determines that a particular image should be stored in the non-volatile memory: the one or more image processing components receive the high resolution raw image data the temporary raw data buffer and process the high resolution raw image data to generate a high resolution version of the particular image;the image compression component compresses the high ion version of the particular image; andthe mobile image capture device writes the high resolution version of the particular image to the non-volatile memory after compression. 18. The mobile image capture device of claim 16, wherein the second capture mode comprises a burst capture mode. 19. The mobile image capture device of claim 16, wherein the first capture mode comprises a base capture mode in which the image sensor periodically transitions between providing high resolution raw image data and low resolution raw image data. 20. The mobile image capture device of claim 16, wherein the second mode of operation comprises a hyper-lapse mode of operation in which the image capture device captures mid resolution images at a high frame rate and performs imaging pipeline operations, compression, and storage at the non-volatile memory for all of the captured mid resolution images. 21. An image capture device that has a low-power framework for processing imagery, the image capture device comprising: an image sensor that performs binning to provide low resolution raw image data;an imaging pipeline that streams the low resolution raw image data and processes the low resolution raw image data to generate low resolution versions of one or more first images;a temporary image buffer that temporarily stores the low resolution versions the one or more first images;a scene analyzer that analyzes at least a portion of a scene depicted by at least one of the low resolution versions of the one or more first images and determines whether to store the lower resolution versions of the one or more first images in a non-volatile memory or to discard the low resolution versions of the one or more first images from the temporary image buffer without storing the one or more first images in the non-volatile memory;an image compression component that compresses the low resolution versions of the one or more first images when the scene analyzer determines that the low resolution versions of the one or more first images should be stored in the non-volatile memory; andthe non-volatile memory to which the low resolution versions of the one or more first images are written after compression.
연구과제 타임라인
LOADING...
LOADING...
LOADING...
LOADING...
LOADING...
이 특허에 인용된 특허 (84)
Wexler, Yonatan; Shashua, Amnon, Apparatus for adjusting image capture settings.
Kuberka, Cheryl J.; Barnum, David C.; Williams, Frances C.; Border, John N.; Johnson, Kenneth A., Camera configurable for autonomous self-learning operation.
Bernardi Bryan D. (Rochester NY) McIntyre Dale F. (Honeoye Falls NY) Dunsmore Clay A. (Fairport NY) Wolcott Dana W. (Honeoye Falls NY), Camera on-board voice recognition.
Kuchta Daniel W. (Brockport NY) Sucy Peter J. (Hamlin NY), Electronic still camera providing multi-format storage of full and reduced resolution images.
Ejima, Satoshi; Nozaki, Hirotake; Hiraide, Fumio, Image processing apparatus having image selection function, and recording medium having image selection function program.
Strub, Henry B.; Burgess, David A.; Johnson, Kimberly H.; Cohen, Jonathan R.; Reed, David P.; Aiello, G. Roberto, Low attention recording unit for use by vigorously active recorder.
Ostojic, Bojana; Glein, Christopher A; Gibson, Mark R.; Vong, William H; Flora, William T; Alton, Benjamin N; Newell, Mark S, Media user interface gallery control.
Schaffer, James David; Ali, Walid; Eshelman, Larry J.; Cohen-Bacrie, Claude; Lagrange, Jean-Michel; Levrier, Claire; Villain, Nicholas; Entrekin, Robert R., Method and apparatus for automatically developing a high performance classifier for producing medically meaningful descriptors in medical diagnosis imaging.
Lee, Il Yong; Kim, Sung Hyun; Kim, Lag Young; Hong, Yun Pyo; Byun, Seong Chan, Method for processing image data in portable electronic device, and portable electronic device having camera thereof.
Steinberg, Eran; Prilutsky, Yury; Corcoran, Peter; Bigioi, Petronel, Perfecting of digital image capture parameters within acquisition devices using face detection.
Elazar, Avishai; Back, Ayelet; Dvir, Igal; Koren-Blumstein, Guy, System and method for video content analysis-based detection, surveillance and alarm management.
Tedesco, Daniel E.; Jorasch, James A.; Gelman, Geoffrey M.; Walker, Jay S.; Tulley, Stephen C.; O'Neil, Vincent M.; Alderucci, Dean P., System for image analysis in a network that is structured with multiple layers and differentially weighted neurons.
Isogai, Kuniaki; Kawamura, Takashi; Kawabata, Akihiro, Video analysis apparatus and method for calculating interpersonal relationship evaluation value using video analysis.
Lin,Yun Ting; Gutta,Srinivas; Brodsky,Tomas; Philomin,Vasanth, Video monitoring system employing hierarchical hidden markov model (HMM) event learning and classification.
Karki, Krishna Singh; Jain, Vaibhav; Gupta, Subham; Bhalla, Poonam; Bedi, Ajay, Automatic capture and refinement of a digital image of a group of people without user intervention.
※ AI-Helper는 부적절한 답변을 할 수 있습니다.