System for background subtraction with 3D camera
원문보기
IPC분류정보
국가/구분
United States(US) Patent
등록
국제특허분류(IPC7판)
G06K-009/00
G06T-005/50
G06K-009/38
G06T-007/12
G06T-007/194
G06T-007/174
G06T-007/90
H04N-005/222
H04N-005/272
출원번호
US-0389952
(2016-12-23)
등록번호
US-9792676
(2017-10-17)
발명자
/ 주소
Nguyen, Quang H.
Do, Minh N.
Patel, Sanjay J.
Dabbelt, Daniel P.
Lin, Dennis J.
출원인 / 주소
The Board of Trustees of the University of Illinois
대리인 / 주소
Invention Mine LLC
인용정보
피인용 횟수 :
1인용 특허 :
112
초록▼
A system for background image subtraction includes a computing device coupled with a 3D video camera, a processor o£ the device programmed to receive a video feed from the camera containing images of one or more subject that include depth information. The processor, for an image: segments pixels and
A system for background image subtraction includes a computing device coupled with a 3D video camera, a processor o£ the device programmed to receive a video feed from the camera containing images of one or more subject that include depth information. The processor, for an image: segments pixels and corresponding depth information into three different regions including foreground (FG), background (BG), and unclear (UC); categorizes UC pixels as FG or BG using a function that considers the color and background history (BGH) information associated with the UC pixels and the color and BGH information associated with pixels near the UC pixels; examines the pixels marked as FG and applies temporal and spatial filters to smooth boundaries of the FG regions; constructs a new image by overlaying the FG regions on top of a new background; displays a video feed of the new image in a display device; and continually maintains the BGH.
대표청구항▼
1. A method comprising: receiving a video feed at a computing device having a processor and memory, the video feed comprising at least one color image including depth information of one or more subjects;categorizing pixels of the at least one color image having a depth thereof less than a predetermi
1. A method comprising: receiving a video feed at a computing device having a processor and memory, the video feed comprising at least one color image including depth information of one or more subjects;categorizing pixels of the at least one color image having a depth thereof less than a predetermined threshold distance as either foreground (FG) or unclear (UC);recategorizing UC pixels as FG or to remove categorization using a function that considers at least one of (i) color and background history (BGH) information associated with the UC pixels and (ii) color and BGH information associated with pixels near the UC pixels; andconstructing at least one new color image by extracting the FG pixels of the at least one color image. 2. The method of claim 1, wherein the function for recategorizing the UC pixels is based on at least one of (i) color dissimilarity between the UC pixels and pixels near the UC pixels of the at least one colored image and (ii) color dissimilarity between the UC pixels and pixels near the UC pixels of the BGH information. 3. The method of claim 1, further comprising applying at least one of (i) spatial smoothing to a FG boundary by utilizing a median filter and (ii) temporal filtering by inspecting color images and FG assignment of a current and at least one previous frame. 4. The method of claim 1, wherein constructing the at least one new color image further comprises overlaying the FG pixels on at least one image from a second video. 5. The method of claim 1, wherein the categorizing step comprises, for each of a plurality of pixels at varying depths along a Z axis of the image: categorizing the pixel as FG if a depth thereof is less than a predetermined threshold distance from the camera and an intensity thereof is greater than a predetermined threshold intensity; andcategorizing the pixel as UC if a depth thereof is less than the predetermined threshold distance and an intensity thereof is less than the predetermined threshold intensity. 6. The method of claim 1, further comprising: detecting and labeling as connected components pixels that are adjacent to each other, are in the same categorization, and have depth values smaller than a predetermined threshold, wherein recategorizing the UC pixels comprises: removing the categorization of any FG connected component having a cross-sectional area less than a predetermined threshold area, γ;removing the categorization of any UC connected component having a cross-sectional area greater than a second predetermined threshold area, γ′; andremoving the categorization of any UC connected component having a cross-sectional area less than γ and for which no adjacent component thereof includes a FG connected component having a cross-sectional area greater than γ. 7. The method of claim 6, further comprising: detecting a FG fringe of FG pixels along at least one boundary using a morphological opening operator applied thereto;merging the FG fringe into a UC region of remaining UC pixels;comparing each of a plurality of pixels in the UC region with corresponding neighbor pixels that are not assigned as UC pixels;recategorizing each compared pixel in the UC region to FG or to remove categorization based on the neighbor pixels that best match the compared pixel; andmerging any remaining UC region with the FG regions. 8. The method of claim 7, further comprising: removing categorization of UC pixels that lie under a center of mass of each FG component; andremoving categorization of smallest UC connected components based on a median filter applied to the remaining UC pixels in the UC region. 9. The method of claim 7, further comprising, before detecting and merging the FG fringe into the UC region: warping the FG and UC regions from a depth sensor viewpoint to a color sensor viewpoint;comparing RGB values of the UC pixels with corresponding BGH information, and where a match is found, recategorizing the UC pixel to remove categorization; anddilating any remaining UC pixels to surrounding pixels thereof that are not categorized as FG pixels. 10. The method of claim 9, further comprising: propagating the depth information for the UC region and for the FG connected components from a depth sensor into a color sensor, to synchronize the depth information with corresponding pixels in the at least one color image when the color and depth sensors are respectively positioned at different locations in 3D space; andfor each of a plurality of the pixels having unknown RGB values: determining if the pixel is surrounded by other FG or UC pixels within a predetermined support window;recategorizing the unknown pixels that are surrounded by FG pixels as FG; andrecategorizing the unknown pixels that are surrounded by UC pixels as UC. 11. The method of claim 9, further comprising: computing an average depth value for each FG connected component; andselecting between a near mode and a far mode based on the average depth of the largest FG connected component, where the near mode is selected if the average depth is less than a predetermined threshold depth. 12. The method of claim 11, further comprising, when in near mode, for each FG connected component, categorizing to remove categorization a plurality of UC pixels that lie under a corresponding center of mass of corresponding one or more FG connected components before comparison thereof with the BGH information. 13. The method of claim 9, further comprising resetting the BGH information in response to detecting at least one of: an indication that the camera is shaken; one or more subjects being too close to the camera; a subject from the background moving into the foreground; and a significant lighting change. 14. The method of claim 6, further comprising: determining if a red/blue/green (RGB) value of each of a plurality of UC pixels is unchanged from a previous frame;copying only those UC pixels with RGB values that are unchanged into an updated UC region; andapplying a median filter on the FG pixels to smooth boundaries of the FG connected components. 15. A system comprising a computing device having a processor and memory, the processor programmed to receive video data, via a 3D application programming interface (API), from a camera, the video data containing (i) at least one colored image of one or more subjects and (ii) corresponding depth information; the processor further programmed to: receive video data comprising at least one color image including depth information of one or more subjects;categorize pixels of the at least one color image having a depth thereof less than a predetermined threshold distance as either foreground (FG) or unclear (UC);recategorize UC pixels to FG or to remove categorization using a function that considers at least one of (i) color and background history (BGH) information associated with the UC pixels and (ii) color and BGH information associated with pixels near the UC pixels; andconstruct at least one new color image by extracting the FG pixels of the at least one color image. 16. The system of claim 15, wherein the processor is further programmed such that the function for recategorizing the UC pixels is based on at least one of (i) color dissimilarity between UC pixels and pixels near the UC pixels of the at least one colored image and (ii) color dissimilarity between the UC pixels and pixels near the UC pixels of the BGH information. 17. The system of claim 15, wherein the processor is further programmed to apply at least one of (i) spatial smoothing to a FG boundary by utilizing a median filter and (ii) temporal filtering by inspecting color images and FG assignment of a current and at least one previous frame. 18. The system of claim 15, wherein the processor is further programmed such that constructing the at least one new color image further comprises overlaying the FG pixels on at least one image from a second video. 19. The system of claim 15, wherein the processor is further programmed such that categorizing pixels of the video data further comprises, for each of a plurality of pixels at varying depths along a Z axis of the image: categorizing the pixel as FG if a depth thereof is less than a predetermined threshold distance from the camera and an intensity thereof is greater than a predetermined threshold intensity; andcategorizing the pixel as UC if a depth thereof is less than the predetermined threshold distance and an intensity thereof is less than the predetermined threshold intensity. 20. The system of claim 15, wherein the processor is further programmed to: detect and label as connected components pixels that are adjacent to each other, are in the same categorization, and have depth values smaller than a predetermined threshold, wherein recategorizing the UC pixels further comprises: removing the categorization of any FG connected component having a cross-sectional area less than a predetermined threshold area, γ;removing the categorization of any UC connected component having a cross-sectional area greater than a second predetermined threshold area, γ′; andremoving the categorization of any UC connected component having a cross-sectional area less than γ and for which no adjacent component thereof includes a FG connected component having a cross-sectional area greater than γ.
연구과제 타임라인
LOADING...
LOADING...
LOADING...
LOADING...
LOADING...
이 특허에 인용된 특허 (112)
Cipolla Roberto (Cambridge GBX) Okamoto Yasukazu (Chiba-ken JPX) Kuno Yoshinori (Osaka-fu JPX), 3D human interface apparatus using motion recognition based on dynamic image processing.
Panahpour Tehrani, Mehrdad; Ishikawa, Akio; Sakazawa, Shigeyuki, Apparatus, method and computer program for classifying pixels in a motion picture as foreground or background.
Clanton,Charles H.; Ventrella,Jeffrey J.; Paiz,Fernando J., Cinematic techniques in avatar-centric communication during a multi-user online simulation.
DeMenthon Daniel F. (Columbia MD), Computer vision system for position monitoring in three dimensions using non-coplanar light sources attached to a monito.
Tian, Dihong; Mauchly, J. William; Friel, Joseph T., Generating and rendering synthesized views with multiple video streams in telepresence video conference sessions.
Iwamoto, Masayuki; Fujimura, Koichi, Image processing apparatus, method for processing and image and computer-readable recording medium for causing a computer to process images.
Carter, James; Yaacob, Arik; Darrah, James F., Managing the layout of multiple video streams displayed on a destination display screen during a videoconference.
Bang, Gun; Um, Gi-Mun; Chang, Eun-Young; Kim, Taeone; Hur, Nam-Ho; Kim, Jin-Woong; Lee, Soo-In, Method and apparatus for improving quality of depth image.
Berman Arie ; Vlahos Paul ; Dadourian Arpag, Method for removing from an image the background surrounding a selected subject by generating candidate mattes.
Haskell, Barin Geoffry; Puri, Atul; Schmidt, Robert Lewis, Scene description nodes to support improved chroma-key shape representation of coded arbitrary images and video objects.
Mackie, David J.; Tian, Dihong; Weir, Andrew P.; Buttimer, Maurice; Friel, Joseph T.; Mauchly, J. William; Chen, Wen-Hsiung, System and method for providing enhanced video processing in a network environment.
Prahlad, Anand; Schwartz, Jeremy A.; Ngo, David; Brockway, Brian; Muller, Marcus S., Systems and methods for classifying and transferring information in a storage network.
Do, Minh N.; Nguyen, Quang Il.; Lin, Dennis; Patel, Sanjay I., Systems and methods for embedding a foreground video into a background feed based on a control input.
Weiser, Reginald; McGravie, Richard; Diouskine, Roman; Teboul, Jeremy, Systems and methods for providing video conferencing services via an ethernet adapter.
Rudolph, Eric; Rui, Yong; Malvar, Henrique S; He, Li Wei; Cohen, Michael F; Tashev, Ivan, Systems and methods for real-time audio-visual communication and data collaboration in a network conference environment.
※ AI-Helper는 부적절한 답변을 할 수 있습니다.