Systems and methods that leverage deep learning to selectively store images at a mobile image capture device
원문보기
IPC분류정보
국가/구분
United States(US) Patent
등록
국제특허분류(IPC7판)
H04N-005/232
G06F-017/30
H04N-001/21
G06K-009/00
G06K-009/62
출원번호
US-0984683
(2015-12-30)
등록번호
US-9836484
(2017-12-05)
발명자
/ 주소
Bialynicka-Birula, Iwona
Aguera-Arcas, Blaise
Ramage, Daniel
McMahan, Hugh Brendan
Lange, Oliver Fritz
Fortuna, Emily Anne
Tyamagundlu, Divya
Holbrook, Jess
Kohlhepp, Kristine
Payne, Juston
Duleba, Krzysztof
Vanik, Benjamin
Lentz, Alison
Clapper, Jon Gabriel
Lovejoy, Joshua Denali
Donsbach, Aaron Michael
출원인 / 주소
Google LLC
대리인 / 주소
Dority & Manning, P.A.
인용정보
피인용 횟수 :
1인용 특허 :
80
초록▼
The present disclosure provides an image capture, curation, and editing system that includes a resource-efficient mobile image capture device that continuously captures images. The mobile image capture device is operable to input an image into at least one neural network and to receive at least one
The present disclosure provides an image capture, curation, and editing system that includes a resource-efficient mobile image capture device that continuously captures images. The mobile image capture device is operable to input an image into at least one neural network and to receive at least one descriptor of the desirability of a scene depicted by the image as an output of the at least one neural network. The mobile image capture device is operable to determine, based at least in part on the at least one descriptor of the desirability of the scene of the image, whether to store a second copy of such image in a non-volatile memory of the mobile image capture device or to discard a first copy of such image from a temporary image buffer without storing the second copy of such image in the non-volatile memory.
대표청구항▼
1. A continuous capture mobile image capture device designed for resource efficiency, the mobile image capture device comprising: a network interface;a power source;an image sensor;at least one processor; anda memory that stores a plurality of deep neural networks usable to determine a desirability
1. A continuous capture mobile image capture device designed for resource efficiency, the mobile image capture device comprising: a network interface;a power source;an image sensor;at least one processor; anda memory that stores a plurality of deep neural networks usable to determine a desirability of a scene depicted by an image, the memory comprising at least a temporary image buffer and a non-volatile memory;wherein the plurality of deep neural networks comprise one or more of: a face detection deep neural network that detects a presence of one or more faces in the scene of each input image;a face recognition deep neural network that matches one or more faces in the scene of each input image to one or more other faces;a face attributes deep neural network that detects various facial characteristics of one or more faces in the scene of each input image;an image content deep neural network that outputs one or more semantic labels that describe the scene of each input image; anda photo quality deep neural network that outputs a photo score that describes various photographic quality characteristics of each input image; andwherein the mobile image capture device is operable to: capture a first image that depicts a scene;maintain a first copy of the first image in the temporary image buffer;input the first image into at least one of the plurality of deep neural networks;receive at least one descriptor of the desirability of the scene depicted by the first image as an output from the at least one of the plurality of deep neural networks into which the first image is input; anddetermine, based at least in part on the at least one descriptor of the desirability of the scene of the first image, whether to store a second copy of the first image in the non-volatile memory of the mobile image capture device or to discard the first copy of the first image without storing a second copy of the first image. 2. The mobile image capture device of claim 1, wherein the mobile image capture device is further operable to: operate the image sensor in a plurality of different capture modes that respectively correspond to a plurality of different resolutions and frame rates,select, based at least in part on the at least one descriptor of the desirability of the scene of the first image, one of the plurality of different capture modes; andswitch operation of the image sensor to the selected capture mode. 3. The mobile image capture device of claim 1, wherein the mobile image capture device is further operable to select the at least one of the plurality of deep neural networks into which the first image is input. 4. The mobile image capture device of claim 1, wherein the plurality of deep neural networks comprise a plurality of feed-forward deep neural networks. 5. The mobile image capture device of claim 1, wherein the plurality of deep neural networks comprise at least one convolutional neural network. 6. The mobile image capture device of claim 1, wherein each of the plurality of deep neural networks outputs at least one annotation for each input image, the at least one annotation for each input image indicative of the desirability of the scene depicted by such image. 7. The mobile image capture device of claim 1, wherein the plurality of deep neural networks comprise a multi-headed deep neural network that receives a single set of inputs and provides a plurality of outputs, wherein the plurality of outputs respectively include a plurality of descriptors of the desirability of the scene of each input image. 8. The mobile image capture device of claim 1, wherein the mobile image capture device is further operable to, prior to inputting the first image into the at least one of the plurality of deep neural networks: input the first image into a recurrent deep neural network that analyzes only a portion of the first image;receive as output from the recurrent deep neural network an indication of whether the first image should be input into the at least one of the plurality of deep neural networks, for further analysis, such that die recurrent deep neural network operates as a prefilter to the plurality of deep neural networks; andreceive as output from the recurrent deep neural network a description of which portion of a next image the recurrent deep neural network should analyze. 9. The mobile image capture device of claim 1, wherein the mobile image capture device is further operable to receive data that describes a set of entities having an elevated importance to a user of the mobile image capture device, wherein the at least descriptor output by the at least one of the plurality of deep neural networks indicates whether a member of the set of entities is depicted in the scene of the first image. 10. The mobile image capture device of claim 1, wherein the mobile image capture device is further operable to: receive, from another mobile image capture device that is proximately located, an excitement signal that indicates that the other mobile image capture device has recently captured an image that depicts a desirable scene; anddetermine, based at least in part on the excitement signal, Whether to store a second Copy of a recently captured image in the non-volatile memory of the mobile image capture device or to discard a first copy of such image from the temporary image buffer without storing a second copy of such image in the non-volatile memory. 11. A resource-efficient mobile image capture device that, at least in operation, continuously captures imagery, the mobile image capture device comprising: a network interface;a power source;an image sensor;at least one processor;a memory; anda scene analyzer that includes: at least one neural network that receives a first image captured by the image sensor and outputs at least one descriptor of a desirability of a scene depicted by the first image;a save controller that determines, based at least in part on the at least one descriptor of the desirability of the scene of the first image, whether to store a second copy of the first image in the memory of the mobile image capture device or to discard the first image without storing a second copy of the first image; anda mode controller that: selects, based at least in part on the at least one descriptor of the desirability of the scene of the first image, one of a plurality of different capture modes of the mobile image capture device, wherein the plurality of different capture modes respectively correspond to a plurality of different resolutions and frame rates; andswitches operation of the image sensor to the selected capture mode. 12. The mobile image capture device of claim 11, wherein the scene analyzer further comprises an attention model that: analyzes only a portion of the first image;outputs an indication, based at least in part on the analysis of the portion of the first image, whether the first image should be further analyzed in its entirety; andoutputs a description of which portion of a second image the attention model should analyze, the second image temporally subsequent to the first image. 13. The mobile image capture device of claim 11, wherein: the scene analyzer comprises a plurality of multi-layer non-linear models; andthe scene analyzer further comprises a model selector that selects at least one of the multi-layer non-linear models into which the first image is input. 14. The mobile image capture device of claim 11, wherein the scene analyzer comprises a multi-headed deep neural network that receives a single set of inputs and provides a plurality of sets of outputs, wherein the plurality of sets of outputs respectively include a plurality of descriptors of the desirability of the scene of each input image. 15. A method to selectively retain images, the method comprising: capturing, by a mobile image capture device, an image that depicts a scene;maintaining, by the mobile image capture device, a first copy of the image in a temporary image buffer of the mobile image capture device;inputting, by the mobile image capture device, the image into at least one neural network to determine a desirability of the scene depicted by the image, the at least one neural network stored in a memory of the mobile image capture device, wherein inputting, by the mobile image capture device, the image into at least one neural network comprises: inputting, by the mobile image capture device, the image into a recurrent neural network that analyzes only a portion of the image;receiving, by the mobile image capture device, as output from the recurrent neural network a probability that the image warrants further analysis; andwhen the probability exceeds a threshold, inputting the image into at least one feed-forward deep neural network;receiving, by the mobile image capture device, at least one descriptor of the desirability of the scene depicted by the image as an output of the at least one neural network, wherein the descriptor of the desirability of the scene is received from the at least one feed-forward deep neural network; anddetermining, by the mobile image capture device based at least in part on the at least one descriptor of the desirability of the scene of the image, whether to store a second copy of the image in a nonvolatile memory of the mobile image capture device or to discard the first copy of the image from the temporary image buffer without storing a second copy of the image in the non-volatile memory. 16. The method of claim 15, wherein: inputting, by the mobile image capture device, the image into at least one neural network comprises: selecting, by the mobile image capture device, one or more of a plurality of deep neural networks; andinputting, by the mobile image capture device, the image into the selected one or more deep neural networks;receiving, by the mobile image capture device, at least one descriptor of the desirability of the scene comprises receiving, by the mobile image capture device, at least one annotation from each of the selected one or more deep neural networks, the at least one annotation for each of the selected one or more deep neural networks indicative of the desirability of the scene depicted by such image; anddetermining, by the mobile image capture device based at least in part on the at least one descriptor of the desirability of the scene of each image, whether to store a second copy of such image in the non-volatile memory of the mobile image capture device comprises determining, by the mobile image capture device based at least in part on the at least one annotation received from each of the selected one or more deep neural networks, whether to store a second copy of such image in the non-volatile memory of the mobile image capture device or to discard the first copy of such image from the temporary image buffer without storing a second copy of such image in the non-volatile memory. 17. A continuous capture mobile image capture device designed for resource efficiency, the mobile image capture device comprising: a network interface;a power source;an image sensor;at least one processor; anda memory that stores a plurality of deep neural networks usable to determine a desirability of a scene depicted by an image, the memory comprising at least a temporary image buffer and a non-volatile memory;wherein the mobile image capture device is operable to: capture a first image that depicts a scene;maintain a first copy of the first image in the temporary image buffer;input the first image into at least one of the plurality of deep neural networks;receive at least one descriptor of the desirability of the scene depicted by the first image as an output from the at least one of the plurality of deep neural networks into which the first image is input; anddetermine, based at least in part on the at least one descriptor of the desirability of the scene of the first image, whether to store a second copy of the first image in the non-volatile memory of the mobile image capture device or to discard the first copy of the first image without storing a second copy of the first image; andwherein the mobile image capture device is further operable to: operate the image sensor in a plurality of different capture modes that respectively correspond to a plurality of different resolutions and frame rates,select, based at least in part on the at least one descriptor of the desirability of the scene of the first image, one of the plurality of different capture modes; andswitch operation of the image sensor to the selected capture mode. 18. A continuous capture mobile image capture device designed for resource efficiency, the mobile image capture device comprising: a network interface;a power source;an image sensor;at least one processor; anda memory that stores a plurality of deep neural networks usable to determine a desirability of a scene depicted by an image, the memory comprising at least a temporary image buffer and a non-volatile memory;wherein the plurality of deep neural networks comprise a multi-headed deep neural network that receives a single set of inputs and provides a plurality of outputs, wherein the plurality of outputs respectively include a plurality of descriptors of the desirability of the scene of each input image; andwherein the mobile image capture device is operable to: capture a first image that depicts a scene;maintain a first copy of the first image in the temporary image buffer;input the first image into the multi-headed deep neural network;receive the plurality of descriptors of the desirability of the scene depicted by the first image as an output from the multi-headed deep neural network into which the first image is input; anddetermine, based at least in part on the plurality of descriptors of the desirability of the scene of the first image, whether to store a second copy of the first image in the non-volatile memory of the mobile image capture device or to discard the first copy of the first image without storing a second copy of the first image. 19. A continuous capture mobile image capture device designed for resource efficiency, the mobile image capture device comprising: a network interface;a power source;an image sensor;at least one processor; anda memory that stores a plurality of deep neural networks usable to determine a desirability of a scene depicted by an image, the memory comprising at least a temporary image buffer and a non-volatile memory;wherein the mobile image capture device is operable to: capture a first image that depicts a scene;maintain a first copy of the first image in the temporary image buffer, wherein to maintain the first copy the mobile image capture device maintains temporary data sufficient to generate high resolution copy of the first image;input the first image into at least one of the plurality of deep neural networks, wherein to input the first image the mobile image capture device inputs a low resolution copy of the first image into the at least one neural network;receive at least one descriptor of the desirability of the scene depicted by the first image as an output from the at least one of the plurality of deep neural networks into which the first image is input; anddetermine, based at least in part on the at least one descriptor of the desirability of the scene of the first image, whether to store a second copy of the first image in the non-volatile memory of the mobile image capture device or to discard the first copy of the first image without storing a second copy of the first image, wherein to determine whether to store the second copy, the mobile image capture device determines, based at least in part on the at least one descriptor of the desirability of the scene of the first image, whether to store the high resolution copy of the first image in the non-volatile memory of the mobile image capture device or to discard the temporary data without storing the high resolution copy of the first image. 20. A resource-efficient mobile image capture device that, at least in operation, continuously captures imagery, the mobile image capture device comprising: a network interface;a power source;an image sensor;at least one processor;a memory; anda scene analyzer that includes: at least one neural network that receives a first image captured by the image sensor and outputs at least one descriptor of a desirability of a scene depicted by the first image; anda save controller that determines, based at least in part on the at least one descriptor of the desirability of the scene of the first image, whether to store a second copy of the first image in the memory of the mobile image capture device or to discard the first image without storing a second copy of the first image;wherein the scene analyzer comprises a plurality of multi-layer non-linear models; andwherein the scene analyzer further comprises a model selector that selects at least one of the multi-layer non-linear models into which the first image is input.
연구과제 타임라인
LOADING...
LOADING...
LOADING...
LOADING...
LOADING...
이 특허에 인용된 특허 (80)
Wexler, Yonatan; Shashua, Amnon, Apparatus for adjusting image capture settings.
Kuberka, Cheryl J.; Barnum, David C.; Williams, Frances C.; Border, John N.; Johnson, Kenneth A., Camera configurable for autonomous self-learning operation.
Bernardi Bryan D. (Rochester NY) McIntyre Dale F. (Honeoye Falls NY) Dunsmore Clay A. (Fairport NY) Wolcott Dana W. (Honeoye Falls NY), Camera on-board voice recognition.
Kuchta Daniel W. (Brockport NY) Sucy Peter J. (Hamlin NY), Electronic still camera providing multi-format storage of full and reduced resolution images.
Ejima, Satoshi; Nozaki, Hirotake; Hiraide, Fumio, Image processing apparatus having image selection function, and recording medium having image selection function program.
Strub, Henry B.; Burgess, David A.; Johnson, Kimberly H.; Cohen, Jonathan R.; Reed, David P.; Aiello, G. Roberto, Low attention recording unit for use by vigorously active recorder.
Ostojic, Bojana; Glein, Christopher A; Gibson, Mark R.; Vong, William H; Flora, William T; Alton, Benjamin N; Newell, Mark S, Media user interface gallery control.
Schaffer, James David; Ali, Walid; Eshelman, Larry J.; Cohen-Bacrie, Claude; Lagrange, Jean-Michel; Levrier, Claire; Villain, Nicholas; Entrekin, Robert R., Method and apparatus for automatically developing a high performance classifier for producing medically meaningful descriptors in medical diagnosis imaging.
Lee, Il Yong; Kim, Sung Hyun; Kim, Lag Young; Hong, Yun Pyo; Byun, Seong Chan, Method for processing image data in portable electronic device, and portable electronic device having camera thereof.
Steinberg, Eran; Prilutsky, Yury; Corcoran, Peter; Bigioi, Petronel, Perfecting of digital image capture parameters within acquisition devices using face detection.
Tedesco, Daniel E.; Jorasch, James A.; Gelman, Geoffrey M.; Walker, Jay S.; Tulley, Stephen C.; O'Neil, Vincent M.; Alderucci, Dean P., System for image analysis in a network that is structured with multiple layers and differentially weighted neurons.
Isogai, Kuniaki; Kawamura, Takashi; Kawabata, Akihiro, Video analysis apparatus and method for calculating interpersonal relationship evaluation value using video analysis.
Lin,Yun Ting; Gutta,Srinivas; Brodsky,Tomas; Philomin,Vasanth, Video monitoring system employing hierarchical hidden markov model (HMM) event learning and classification.
※ AI-Helper는 부적절한 답변을 할 수 있습니다.