초록 ▼

A network repository service supplements the functions of a web server to enable an increase in the efficiency of web crawling. The repository service: (a) automatically maintains a file modification list that contains the names of files on the server that have been modified (i.e., added, deleted, o

A network repository service supplements the functions of a web server to enable an increase in the efficiency of web crawling. The repository service: (a) automatically maintains a file modification list that contains the names of files on the server that have been modified (i.e., added, deleted, or otherwise modified), together with the date and time of the file modification; and (b) provides a requesting crawler with the file modification list (or a portion of the list corresponding to a time period specified by the crawler). The repository service may also (c) limit or restrict access privileges of crawlers that do not request the file modification list prior to crawling, thereby protecting the server from overcrawling. The repository service enables a crawler to request the file modification list, and avoid unnecessarily recrawling files that have not been modified since its last visit, thereby preventing considerable waste of time, network bandwidth, server processing resources, and crawler processing resources. Using the file modification list, the crawler can remove all prior references to deleted files, and efficiently recrawl only those files that have been added or changed since the crawler last visited the web server.

대표청구항 ▼

1. In a computer network comprising a web server machine and a web crawler machine, a method comprising:a) automatically maintaining a file modification list on the web server, wherein the file modification list comprises a plurality of modification entries describing modifications made to web conte

1. In a computer network comprising a web server machine and a web crawler machine, a method comprising:a) automatically maintaining a file modification list on the web server, wherein the file modification list comprises a plurality of modification entries describing modifications made to web content files stored on the web server machine, wherein each modification entry comprises a file name, a file modification time, and a file modification type; b) transmitting from the web crawler machine to the web server machine a request for file modification information, wherein the request comprises a last visited time indicating when the web crawler machine last visited the web server machine; c) generating the requested file modification information and transmitting the requested file modification information to the web crawler machine, wherein the requested file modification information comprises file names corresponding to file modification times later than the last visited time of the web crawler machine; d) generating from the requested file modification information transmitted to the web crawler machine a list of new and changed web content files stored on the web server machine; and e) transmitting from the web crawler machine to the web server machine requests for the new and changed web content files.