[미국특허]
Display-independent recognition of graphical user interface control
원문보기
IPC분류정보
국가/구분
United States(US) Patent
등록
국제특허분류(IPC7판)
G06F-003/00
G06F-003/0484
G06F-003/0485
G06F-009/44
출원번호
US-0556158
(2014-11-30)
등록번호
US-9703462
(2017-07-11)
발명자
/ 주소
Butin, Emma
Keinan, Oren
Keinan, Erez
Rabinovich, Alon
출원인 / 주소
KRYON SYSTEMS LTD.
대리인 / 주소
Sherman, Vladimir
인용정보
피인용 횟수 :
0인용 특허 :
62
초록▼
A method for display-independent recognition of a graphical user interface (GUI) control, the method comprising capturing a screenshot of a GUI of a computerized application, determining a cursor position in the screenshot, analyzing an area surrounding the cursor position, to identify a GUI control
A method for display-independent recognition of a graphical user interface (GUI) control, the method comprising capturing a screenshot of a GUI of a computerized application, determining a cursor position in the screenshot, analyzing an area surrounding the cursor position, to identify a GUI control, and storing an image of the GUI control.
대표청구항▼
1. A method for automatic extraction of an image of a control in a graphical user interface (GUI), the method comprising: capturing a first screenshot of a GUI of a computerized application, and automatically analyzing said first screenshot to identify a scroll bar by extracting at least one paramet
1. A method for automatic extraction of an image of a control in a graphical user interface (GUI), the method comprising: capturing a first screenshot of a GUI of a computerized application, and automatically analyzing said first screenshot to identify a scroll bar by extracting at least one parameter of the scroll bar from within the first screenshot;if a scrollbar is identified: (1) receiving from a user coordinates of at least one position in said first screenshot;analyzing an area surrounding said at least one position, to identify at least one GUI control; and (2) storing an image of said at least one GUI control substantially in conjunction with at least one parameter of the scroll bar; andif a scrollbar is not identified: (1) receiving coordinates of at least one position from a user in said first screenshot; analyzing an area surrounding said at least one position, to identify at least one GUI control; and (2) storing an image of said at least one GUI control separate from the first screenshot of the GUI from which the image of the GUI control is taken. 2. The method according to claim 1, further comprising: creating an operation sequence of the computerized application; andassociating the at least one GUI control, with a step of an operation sequence, to produce a computerized guidance script adapted to provide guidance to a user including moving a cursor associated with an input device, to imitate user-controlled movement of said cursor via said input device. 3. The method according to claim 2, wherein the at least one GUI control is selected from a group consisting of: an icon, a text, a button, an input box, a menu, a drop-down list, a slider, another scroll bar and a bar. 4. The method according to claim 2, wherein the receiving of the coordinates comprises detecting where a user cursor click is performed. 5. The method according to claim 2, further comprising converting at least one of said first screenshot and said second screenshot to a 1-bit version. 6. The method according to claim 2, further comprising converting at least one of said first screenshot and said second screenshot to an inverted 1-bit version. 7. The method according to claim 2, wherein the storing further comprises storing an image of surroundings of the at least one GUI control. 8. The method according to claim 2, wherein the analyzing of the area surrounding each of said at least one position comprises employing optical character recognition (OCR). 9. A non-transitory computer-readable medium having stored therein instructions for automatic extraction of an image of at least one GUI control in a graphical user interface (GUI), wherein the instructions, when executed by the computer, cause the computer to: capture a first image of a GUI of a computerized application, and automatically analyze said first image, to identify a scroll bar;if a scroll bar is identified: (1) if stored scrollbar parameters indicate that scrolling is required, automatically operate said scroll bar to reveal a previously-invisible region of said GUI, and capture a second image of said previously-invisible region; (2) analyze said first or second image to identify at least one GUI control; andif a scroll bar is not identified, analyze in said first image to identify at least one GUI control. 10. The non-transitory computer readable medium according to claim 9, wherein the instructions further cause the computer to: create an operation sequence of the computerized application; andassociate the at least one GUI control, with a step of an operation sequence, to produce a computerized guidance script adapted to provide guidance to a user including moving a cursor associated with an input device, to imitate user-controlled movement of said cursor via said input device. 11. The non-transitory computer-readable medium according to claim 10, wherein said at least one GUI control is selected from a group consisting of: an icon, a text, a button, an input box, a menu, a drop-down list, a slider, another scroll bar and a bar. 12. The non-transitory computer-readable medium according to claim 10, wherein the instructions, when executed by the computer, further cause the computer to convert at least one of said first image and said second image to a 1-bit version. 13. The non-transitory computer-readable medium according to claim 10, wherein the instructions, when executed by the computer, further cause the computer to convert at least one of said first image and said second image to an inverted 1-bit version. 14. The non-transitory computer-readable medium according to claim 10, wherein the instructions, when executed by the computer, further cause the computer to identify a cursor click on said at least one GUI control. 15. The non-transitory computer-readable medium according to claim 10, wherein the instructions, when executed by the computer, further cause the computer to identify keyboard input into said at least one GUI control. 16. The non-transitory computer-readable medium according to claim 10, wherein analyzing said area surrounding said position comprises employing an OCR engine. 17. A non-transitory computer-readable medium having stored therein instructions for recording a display-independent computerized guidance script, wherein the instructions, when executed by a computer, cause the computer to: capture a screenshot of a GUI of a computerized application running on the computer;analyze said screenshot to receive from a user coordinates of at least one position in said screenshot and analyzing an area surrounding said at least one position, to identify at least one GUI control;if a scrollbar is identified, store an image of said at least one GUI control substantially in conjunction with at least one parameter of the scroll bar;andif a scrollbar is not identified, store an image of said at least one GUI control separate from the screenshot of the GUI from which the image of the GUI control is taken. 18. The non-transitory computer-readable medium according to claim 17, wherein said instructions, when executed by the computer, further cause the computer to automatically analyze said screenshot to identify a scroll bar by extracting at least one parameter of the scroll bar from within the screenshot, wherein the extracted parameter comprises at least one of an island, a triangle, or an arrow.
Mann ; II S. Edward (11020 Huebner Oak #1136 San Antonio TX 78230) Milliot Frederic (5 Rue Boulle 75011 Paris FRX), Apparatus and method for providing a generic computerized multimedia tutorial interface for training a user on multiple.
Jessen Jay Alan (Santa Clara CA) Nagarajan Palanivelu (Campbell CA) Flynn Sean Ludlow (Cupertino CA) Schneider James Alan (San Jose CA), Apparatus for causing a computer system to respond to emulated user interaction in the absence of actual user interactio.
Hejlsberg, Anders; Pepin, Brian K.; Royal, Chad W.; Anderson, Christopher L.; Della Libera, Giovanni M.; Boulter, Mark A.; Kramer, Nicholas M.; Burke, Shawn P.; Oak, Subhag P.; Nivarthi, Sreeram, Application program interface for network software platform.
Dudzik Dominic E. (Paris FRX) Bucheron Marc L. (Voisins FRX) Verhoeff Jasper (Le Mesnil St. Denis NLX), Application-external help system for a windowing user interface.
Kerr Randal H. (Richford NY) Mesnard Robert M. (Endicott NY), Automatic generation of executable computer code which commands another program to perform a task and operator modificat.
Rawat, Jai; Bhatia, Ajoy Kumar; Zissimopoulos, Vasileios Bill, Client-side form filler that populates form fields based on analyzing visible field labels and visible display format hints without previous examination or mapping of the form.
Katz,Glenn; Gough,Michael; Jacobs,Josh; Powers,John; Coleman,Patricia; Miyake,James; Ames,Greg; Kalb,Jon, Computer system for automatically instantiating tasks designated by a user.
Gough Michael Lane (Ben Lomond CA) Katz Glenn (Campbell CA) Coleman Patricia (Montara CA) Ames Greg (Redwood City CA), Computer-human interface which provides for user customization of object behavior.
Gough, Michael Lane; Koester, Stephanie Eva; McGhie, Brian Burnell; Miyake, James Ieyasu; Nygord, Richard Victor; Saxena, Samir Ranjan Tiongson, Graphical editor for program files.
Tannenbaum Alan R. (Washington Grove MD) Zetts John M. (Falls Church VA) An Yu L. (Vienna VA) Arbeitman Gordon W. (Gaithersburg MD) Greanias Evon C. (Boca Raton FL) Verrier Guy F. (Reston VA), Graphical user interface with gesture recognition in a multiapplication environment.
Habib Dennis M. ; Engelberg Mindy J. ; McNutt Kathryn A. ; Kinser Colby E. ; Lynch ; II Charles E. ; Williams Chauncey L., Help system with semitransparent window for disabling controls.
Nicol Anne (Palo Alto CA) Kenyon Lawrence A. (Sunnyvale CA) Wagner Annette (Sunnyvale CA) Sulzen James T. (Montara CA), Method and apparatus for providing help information to users of computers.
Palmer James E. (Redwood City CA) Powers ; III John R. (Monte Sereno CA) Coleman Patricia J. (Menlo Park CA) Brewer Gregory S. (Pleasanton CA) Herman Jeffrey A. (Palo Alto CA) Cochran Eli (Menlo Park, Method and apparatus for providing visual cues in a graphic user interface.
Withgott M. Margaret ; Newman William,GB2 ; Bagley Steven C. ; Huttenlocher Daniel P. ; Kaplan Ronald M. ; Cass Todd A. ; Halvorsen Per-Kristian ; Brown John Seely ; Kay Martin, Method and apparatus for supplementing significant portions of a document selected without document image decoding wit.
Xia Chenhong ; Bailey John Hudson, Method and system for aiding a user in scrolling through a document using animation, voice cues and a dockable scroll bar.
Bieniek Cynthia L. (Bellevue WA) Hunter Ross A. (Woodinville WA) McKee Timothy P. (Seattle WA) Schroeder David H. (Redmond WA), Method and system for instructing a user of a computer system how to perform application program tasks.
Amro Hatim Yousef ; Dao Dan Lee ; Dodson John Paul, Method and system for searching and retrieving specific types of objects contained within a compound document.
Eisen Ivan R. (Flower Mound TX) Morris Ruth A. (Dallas TX), Method for automatically adjusting help information displayed in an online interactive system.
Johnson Steven E. (Austin TX) Karat John (Austin TX) Ruiz Thomas M. (Austin TX), Method for providing an on line help facility for interactive information handling systems.
Stearns Glenn (Scotts Valley CA) Packard Barbara B. (Los Altos Hills CA) Watson Ralph T. (San Jose CA), Method for recording and replaying mouse commands by recording the commands and the identities of elements affected by t.
McKiel Frank A. (Trophy Club TX) Trowbridge G. Michael (Trophy Club TX) Cavendish Cathy J. (Dallas TX) Catino Robert J. (Southlake TX), Method of and system for demonstrating a computer program.
Hopkins, Rosalie Marie; Chodrow, Sarah Elizabeth; Danner, Kevin Rory, System and method for providing help/training content for a web-based application.
Adkins, David L.; Adkins, Mark R.; Alton, Jeffrey W., Systems and methods for teaching a person to interact with a computer program having a graphical user interface.
※ AI-Helper는 부적절한 답변을 할 수 있습니다.