초록 ▼

A system and process for selecting objects in an ubiquitous computing environment where various electronic devices are controlled by a computer via a network connection and the objects are selected by a user pointing to them with a wireless RF pointer. By a combination of electronic sensors onboard

A system and process for selecting objects in an ubiquitous computing environment where various electronic devices are controlled by a computer via a network connection and the objects are selected by a user pointing to them with a wireless RF pointer. By a combination of electronic sensors onboard the pointer and external calibrated cameras, a host computer equipped with an RF transceiver decodes the orientation sensor values transmitted to it by the pointer and computes the orientation and 3D position of the pointer. This information, along with a model defining the locations of each object in the environment that is associated with a controllable electronic component, is used to determine what object a user is pointing at so as to select that object for further control actions.

대표청구항 ▼

Wherefore, what is claimed is: 1. A system for selecting an object within an environment by a user pointing to the object with a pointing device, comprising: a pointing device comprising a radio frequency (RF) transceiver and orientation sensors, wherein the outputs of the sensors are periodically

Wherefore, what is claimed is: 1. A system for selecting an object within an environment by a user pointing to the object with a pointing device, comprising: a pointing device comprising a radio frequency (RF) transceiver and orientation sensors, wherein the outputs of the sensors are periodically packaged as orientation messages and transmitted using the RF transceiver; a base station comprising a RF transceiver which receives orientation messages transmitted by the pointing device; a pair of imaging devices each of which is located so as to capture images of the environment from different viewpoints; a computing device which is in communication with the base station and the imaging devices so as to receive orientation messages forwarded to it by the base station and images captured by the imaging devices, and which computes the orientation and location of the pointer from the received orientation message and captured images, and which also selects an object in the environment whenever the pointing device is pointed at it using the orientation and location of the pointing device. 2. The system of claim 1, wherein the pointing device is a hand-held unit resembling a wand. 3. The system of claim 1, wherein the pointing device comprises: a case having a pointing end a microcontroller; said orientation sensors which comprise, an accelerometer which is connected to the microcontroller and provides separate x-axis and y-axis orientation signals, and a magnetometer which is connected to the microcontroller and provides separate x-axis, y-axis and z-axis orientation signals; said RF transceiver which is connected to the microcontroller and which transmits the orientation messages whenever supplied to it by the microcontroller; and a power supply for powering the electronic components of the pointing device. 4. The system of claim 3, wherein the microcontroller package and transmits an orientation message at a prescribed rate. 5. The system of claim 3, wherein a command-response protocol is employed in regard to the transmission of orientation messages wherein the computing device periodically instructs the microcontroller to package and transmit an orientation message by causing the base station to transmit a request for the message to the pointing device which is provided via the pointing device's transceiver to the microcontroller. 6. The system of claim 5, wherein the computing device causes a request for an orientation message to be transmitted by the base station at a rate of approximately 50 times per second. 7. The system of claim 5, wherein the microcontroller periodically reads and stores the outputs of the orientation sensors, and wherein the orientation message generated by the microcontroller in response to a request for the same from the computing device comprises the last outputs read by the microcontroller from the accelerometer and magnetometer. 8. The system of claim 7, wherein the microcontroller reads and stores the outputs of the orientation sensors at a rate exceeding the polling rate at which the computing device requests orientation messages to be generated. 9. The system of claim 5, wherein the request for an orientation message comprises an identifier which has been assigned to the pointing device and is recognized by the microcontroller of the pointing device as indicating that the request is being directed at the pointing device, and wherein the microcontroller ignores any signal received that does not include said identifier. 10. The system of claim 9, wherein the orientation messages transmitted by the pointing device comprise said pointing device identifier so as to indicate to the computing device that the orientation message is from the pointing device. 11. The system of claim 3, wherein the pointing device further comprises a manually-operated switch which is connected to the microcontroller and which is activated and deactivated by the user for the purpose of instructing the computing device to implement a function, and wherein the state of the switch in regard to whether it is activated or deactivated at the time an orientation message is packaged for transmission is included in that orientation message. 12. The system of claim 3, wherein the pointing device further comprises a pair of visible spectrum light emitting diodes (LEDs) which are connected to the microcontroller and which are disposed so as to be visible when lit from the outside of the case, said visible spectrum LEDs being employ to provide status or feedback information to the user. 13. The system of claim 12, wherein the computing device instructs the pointing device's microcontroller to light one or both of the visible spectrum LEDs by causing the base station to transmit an command to the pointing device instructing the microcontroller to light one or both of the visible spectrum LEDs, said command being provided via the pointing device's transceiver to the microcontroller. 14. The system of claim 12, wherein the visible spectrum LEDs each emit a different color of visible light. 15. The system of claim 3, wherein the pointing device further comprises an infrared (IR) light emitting diode (LED) which is connected to the microcontroller and which is disposed so as to emit IR light when lit outside of the case, and wherein the microcontroller causes the IR LEDs to periodically flash. 16. The system of claim 15, wherein the IR LED is caused to flash for a duration of approximately 3 milliseconds at a rate of 15 flashes per second. 17. The system of claim 16, wherein each of said pair of imaging devices is a digital video camera having a IR pass filter which outputs video image frames that capture only IR light emitted or reflected in the environment toward the camera, including the emissions from the pointing device's IR LED. 18. The system of claim 17, wherein each of the cameras is fitted with a wide angle lens so as to capture a substantial portion of the environment in each video image frame. 19. The system of claim 3, wherein the orientation message further comprises error detection data computed by the microcontroller for use by the computing device to determine if a received orientation message is complete. 20. The system of claim 3, wherein the microcontroller periodically reads and stores the outputs of the orientation sensors, and wherein the microcontroller monitors the accelerometer outputs to determine if the pointing device is moving, and suspends operations whenever it determines the pointing device is not in motion for a prescribed period of time, so as to conserve power. 21. The system of claim 20, wherein the microcontroller determines whether the device has not been in motion for a prescribed period of time by comparing each new set of accelerometer outputs read to the last-stored output values and decrementing a countdown timer if the compared outputs are not substantially different, and then by checking whether the timer is equal to zero and if so powering down the pointing device. 22. The system of claim 21, wherein the microcontroller resets the countdown timer each time it is determined that the compared accelerometer outputs are substantially different. 23. The system of claim 22, wherein the countdown timer, if not reset, is decremented to zero in a period of time equal to approximately 10 minutes. 24. The system of claim 21, wherein whenever the pointing device has be powered down, the microcontroller wakes the device periodically, reads the outputs of the accelerometer, compares the new outputs to the last-stored) accelerometer output values, powers the pointing device back up and resets the countdown timer if the compared output readings differ significantly indicating that the pointing device is in motion, and powers the device back down if the compared output reading do not differ significantly indicating that the pointing device is still motionless. 25. The system of claim 3, wherein the power supply comprises a battery. 26. A computer-implemented process for selecting an object within an environment by a user pointing to the object with a pointing device, comprising using a computer to perform the following process actions: inputting orientation sensor readings generated by orientation sensors of the pointing device and provided by the device in an orientation message; deriving the orientation of the pointing device in relation to a predefined coordinate system of the environment from the inputted orientation sensor readings; inputting digital video from a pair of video cameras each of which is located so as to capture images of the environment from different viewpoints; ascertaining the location of the pointing device at a time substantially contemporaneous with the generation of the orientation message and in terms of the predefined coordinate system using the inputted digital video from the pair of video cameras; using the orientation and location of the pointing device to determine whether the pointing device is being pointed at an object in the environment that is controllable by the computer; and whenever the pointing device is being pointed at a controllable object, selecting that object for future control actions. 27. The process of claim 26, wherein the orientation sensor readings comprise the outputs of a three-axis magnetometer, and wherein the process further comprises a process action of establishing correction factors to relate the magnetometer outputs to the predefined coordinate system of the environment prior to using them to derive the orientation of the pointing device. 28. The process of claim 27, wherein the process action of establishing correction factors to relate the magnetometer outputs to the predefined coordinate system of the environment, comprises the actions of: initiating a magnetometer correction factor calibration mode; pointing the pointing device in a prescribed direction within the environment with a known orientation so as to align the axes of the magnetometer with the axes of the predefined environmental coordinate system; causing the pointing device to provide an orientation message and inputting the outputs of the magnetometer contained therein; designating the output associated with each axis of the magnetometer as the correction factor for that axis, wherein the correction factor for each axis when applied to the output of the magnetometer associated with that axis converts the output from one indicating the deviation of the magnetometer axis under consideration from magnetic north to one indicating the deviation from the corresponding axis of the predefined environmental coordinate system. 29. The process of claim 28, wherein the orientation sensor readings comprise the outputs of a two-axis accelerometer and a three-axis magnetometer, and wherein the process further comprises process actions of establishing normalization factors to normalize the outputs of the accelerometer and magnetometer prior to using them to derive the orientation of the pointing device. 30. The process of claim 29, wherein the process action of establishing normalization factors to normalize the outputs of the magnetometer, comprises the actions of: initiating a magnetometer normalization factor calibration mode; moving the pointing device for prescribed period of time in a manner that causes excursions of the magnetometer about all three of its axes; causing the pointing device to provide an orientation messages at prescribed intervals during the time the pointer is being moved; inputting and recording the outputs of the magnetometer contained in each message provided; at the end of the prescribed period of time, determining for each axis of the magnetometer which recorded output value associated with that axis is the largest of the recorded output values, designating the recorded output value determined to be the largest for each axis of the magnetometer to be the maximum possible output value for that axis, determining for each axis of the magnetometer which recorded output value associated with that axis is the smallest of the recorded output values, designating the recorded output value determined to be the smallest for each axis of the magnetometer to be the minimum possible output value for that axis, and computing normalization factors of each axis of the magnetometer that based on the designated minimum and maximum possible outputs for that axis will convert an output from the magnetometer associated with the axis under consideration to a proportional value in a range from 1.0 to-1.0. 31. The process of claim 29, wherein the accelerometer and magnetometer of the pointing device are oriented therein such that their respective first axis corresponds to an x-axis which is directed laterally to a pointing axis of the pointing device and their respective second axis corresponds to a y-axis which is directed along the pointing axis of the pointing device and the third axis of the magnetometer correspond to a z-axis which is directed vertically upward whenever the pointing device is positioned right-side up with the x and y axes lying in a horizontal plane, and wherein the process action of deriving the orientation of the pointing device, comprises the actions of: normalizing the accelerometer and magnetometer output values contained in the orientation message using the previously established normalization factors; computing the angles defining the pitch of the pointing device about the x-axis and the roll of the device about the y-axis from the normalized outputs of the accelerometer; refining the normalized magnetometer output values using the pitch and roll angles computed from the normalized accelerometer output values; applying the correction factors previously established for each axis of the magnetometer to the refined magnetometer output values; computing the yaw angle of the pointing device about the z axis using the corrected and refined magnetometer output values; tentatively designate the computed pitch, roll and yaw angles as defining the orientation of the pointing device at the time the orientation message was generated; assessing whether the pointing device was in a right-side up or up-side down position at the time the orientation message was generated; whenever it is determined that the pointing device was in the right-side up position at the time the orientation message was generated, designating the previously computed pitch, roll and yaw angles as defining the finalized orientation of the pointing device; and whenever it is determined that the pointing device was in the up-side down position at the time the orientation message was generated, correcting the tentatively designated roll angle and designating the pitch, yaw and modified roll angle as defining the finalized orientation of the pointing device. 32. The process of claim 31, wherein the process action of assessing whether the pointing device was in a right-side up or up-side down position comprises the actions of: computing the orientation of the pointer assuming the pointer is right-side up position with respect to roll; computing the orientation of the pointer assuming the pointer is up-side down position with respect to roll; estimating from the orientation of the pointer derived assuming the pointer is right-side up what the output values from the magnetometer should have been at the time the orientation message was generated; estimating from the orientation of the pointer derived assuming the pointer is up-side down what the output values from the magnetometer should have been at the time the orientation message was generated; determining whether the magnetometer output values estimated assuming the pointer was in the right-side up position or assuming the pointer was in the up-side down position exhibit the least amount of overall difference when compared to the actual output values contained in the orientation message; whenever the output values computed for the right-side up position assumption are determined to exhibit the least amount of overall difference, deeming the pointing device to have been in the right-side up position at the time the orientation message was generated; and whenever the output values computed for the up-side down position assumption are determined to exhibit the least amount of overall difference, deeming the pointing device to have been in the up-side down position at the time the orientation message was generated. 33. The process of claim 31, wherein the pointing device further comprises a roll orientation sensor whose output is included in the orientation message and which directly indicates whether the pointing device is right-side up or up-side, and wherein the process action of assessing whether the pointing device was in a right-side up or up-side down position comprises an action of determining the position from the roll orientation sensor output. 34. The process of claim 31, further comprising, prior to performing the process action of computing the angles defining the pitch and roll of the pointing device from the normalized outputs of the accelerometer, performing the process actions of: determining whether the pointing device has moved in the time between the generation of the orientation message currently under consideration and the generation of the immediately preceding orientation message; and whenever it is determined that the pointing device has moved in the time between the generation of said sequential orientation messages, ignoring the orientation message currently under consideration and waiting for the next orientation to be received. 35. The process of claim 34, wherein the process action of determining whether the pointing device has moved, comprises the process actions of: determining whether the orientation of the pointing device computed from the orientation message currently under consideration is significantly different from that computed from the orientation message received just before the current message; deeming that the pointing device has moved whenever the orientation of the pointing device computed from the orientation message currently under consideration is significantly different from that computed from the orientation message received just before the current message; and deeming that the pointing device has not moved whenever the orientation of the pointing device computed from the orientation message currently under consideration is not significantly different from that computed from the orientation message received just before the current message. 36. The process of claim 26, wherein the pointing device comprises an infrared (IR) light emitting diode (LED) which is disposed so as to emit IR light when lit outside of the case and which flashes for a prescribed duration at a prescribed rate, and wherein the pair of video cameras each capture video image frames at a rate approximately twice the prescribed flashing rate of the pointing device's IR LED and each have an IR pass filter which causes the cameras to output video image frames that capture only IR light emitted or reflected in the environment toward the camera including the emissions from the pointing device's IR LED, and wherein the process action of ascertaining the location of the pointing device using the inputted digital video from the pair of video cameras, comprises the actions of: for each video camera and each pair of video image frames produced by thereby, subtracting the pair of frames to produce a difference image, said difference image depicting substantially only the IR emissions and reflections directed toward the camera which appear in one or the other of the pair of frames but not both including the flash from the IR LED of the pointing device as it flashes at a rate approximately one-half the rate at which the image frames are captured and so will appear in only one of each pair of frames produced by the camera, and wherein said flash is presumed to be the predominate feature in the difference image, and identifying the image coordinates of the pixel in the difference image that exhibits the highest intensity using a peak detection procedure; and for each pair of approximately contemporaneous pairs of image frames generated by the pair of cameras, determining the 3D coordinates of the flash within the predefined environmental coordinate system using the image coordinates of the flash identified in the difference images associated with said approximately contemporaneous pair of image frames generated by the pair of cameras and predetermined intrinsic and extrinsic camera parameters employing a stereo image technique. 37. The process of claim 26 wherein the process action of using the orientation and location of the pointing device to determine whether the pointing device is being pointed at an object in the environment that is controllable by the computer, comprises action of: modeling the location and extent of objects within the environment that are controllable by the computer using 3D Gaussian blobs defined by a location of the mean of the blob in terms of its environmental coordinates and a covariance; for each Gaussian blob, projecting the Gaussian blob onto a plane which is normal to either a line extending from the location of the pointing device to the mean of the blob or a ray originating at the location of the pointing device and extending in a direction defined by the orientation of the device, ascertaining the value of the resulting projected Gaussian blob at a point where the ray intersects the plane, said value representing the probability that the pointing device is pointing at the object associated with the projected Gaussian blob; identifying which of the probability values associated with the projected Gaussian blobs is the largest, if any; determining if a probability value identified as the largest exceeds a prescribed minimum probability threshold; and whenever it is determined that the probability value identified as the largest exceeds the prescribed minimum probability threshold, designating the object associated with the projected Gaussian blob from which the probability value was derived as being the object that the pointing device is pointing at. 38. The process of claim 37 wherein the process action of modeling the location and extent of objects within the environment that are controllable by the computer using 3D Gaussian blobs, for each object to be modeled in the environment, comprises the actions of: the user, inputting information identifying the object that is to be modeled, and activating a switch on the pointing device, tracing the outline of the object, then deactivating the switch; and performing a target training procedure comprising, causing requests for orientation messages to be sent to the pointing device a prescribed request rate, inputting each orientation message received from the pointing device; for each orientation message input, determining whether a switch state indictor included in the orientation message indicates that the switch is activated; whenever it is initially determined that the switch is not activated, continuing the switch state determination action for each subsequent orientation message received until an orientation message is received which indicates that the switch is activated; whenever it is determined that the switch is activated, ascertaining and storing the location of the pointing device using the inputted digital video from the pair of video cameras and continuing the switch state determination action for each subsequent orientation message received; whenever it is determined that the switch has been deactivated after having been determined to be activated in the immediately preceding orientation message, deeming the tracing action to be complete and defining a 3D Gaussian blob representing the object from the previously ascertained pointing device locations stored during the tracing action. 39. The process of claim 37, wherein the process action of modeling the location and extent of objects within the environment that are controllable by the computer using 3D Gaussian blobs, comprises for each object to be modeled in the environment, the actions of: the user, inputting information identifying the object that is to be modeled, repeatedly pointing the pointing device at the object being modeled and momentarily activating a switch on the pointing device, each time pointing the device from a different location within the environment, and indicating that the target training inputs are complete; and performing a target training procedure comprising, causing requests for orientation messages to be sent to the pointing device a prescribed request rate, inputting each orientation message received from the pointing device until the user indicates the target training inputs are complete for each orientation message input, determining whether a switch state indictor included in the orientation message indicates that the switch is activated; whenever it is determined that the switch is activated, ascertaining the orientation of the pointing device using orientation sensor readings included in the orientation message and ascertaining the location of the pointing device using the inputted digital video from the pair of video cameras, and storing the orientation and location values; computing the location of the mean of a 3D Gaussian blob representing the object being modeled from the pointing device's stored orientation and location values computed for the multiple pointing locations, establishing the covariance of the Gaussian blob based on one of (i) a prescribed covariance, (ii) a user-input covariance, or (iii) a covariance computed by adding a minimum covariance to the spread of the intersection points of rays defined by the pointing device's stored orientation and location values computed for the multiple pointing locations. 40. The process of claim 39, wherein the process action of computing the location of the mean of the 3D Gaussian blob, comprises the actions of: for each pointing location where the orientation and location of the pointing device have been ascertained, establishing a ray that projects through the environment from the pointing device's location along the direction of the device's orientation; forming a linear system of equations, wherein a separate equation is created for each pointing location that defines the location of the mean of a 3D Gaussian blob as a function of the location of the pointing device, the ray and an unknown distance between the location of the pointing device and the object being modeled along the direction of the ray; solving the system of equation using a least squares approach to find the location of the mean that best fits the equations. 41. The process of claim 40, wherein the accuracy of the ray direction is a function of the distance between the location of the pointing device and the object being modeled, and wherein each equation in the system of equations is weighted so as to make equations associated with a pointing location that is closer to the object to have a greater weight in determining the location of the mean, than equations associated with a pointing location that is further away from the object. 42. The process of claim 39, wherein the process action of the user repeatedly pointing the pointing device at the object being modeled from a different location within the environment and momentarily activating the switch on the pointing device, comprises an action of pointing the pointing device at a different point along the body of the object each time. 43. The process of claim 31, wherein the pointing device further comprises a laser pointer disposes so as to project a laser beam along the pointing direction of the pointing device, and wherein the process action of establishing correction factors to relate the magnetometer outputs to the predefined coordinate system of the environment, comprises the actions of: modeling the location and extent of an object within the environment using a 3D Gaussian blob defined by a location of the mean of the blob in terms of its environmental coordinates and a covariance; initiating a pointing device angular error estimation mode; having the user repeatedly pointing the pointing device at the same point on the modeled object and momentarily activating a switch on the pointing device, each time pointing the device from a different location within the environment; causing requests for orientation messages to be sent to the pointing device a prescribed request rate; inputting each orientation message received from the pointing device; for each orientation message input, determining whether a switch state indictor included in the orientation message indicates that the switch is activated; whenever it is determined that the switch is activated, ascertaining the orientation of the pointing device using orientation sensor readings included in the orientation message and ascertaining the location of the pointing device using the inputted digital video from the pair of video cameras, and storing the orientation and location values; computing the average angular error of the pointing device as, wherein i refers to the pointing location in the environment, n refers to the total number of pointing locations, w is a ray originating at the location of the pointing device and extending in a direction defined by the orientation of the device, x is the location of the pointing device, and μ is the location of the mean of the Gaussian blob representing the modeled object.