초록 ▼

A web crawler downloads documents from among a plurality of host computers. The web crawler enqueues document addresses in a data structure called the Frontier. The Frontier generally includes a set of queues, with all document addresses sharing a respective common host component being stored in a r

A web crawler downloads documents from among a plurality of host computers. The web crawler enqueues document addresses in a data structure called the Frontier. The Frontier generally includes a set of queues, with all document addresses sharing a respective common host component being stored in a respective common one of the queues. Multiple threads substantially concurrently process the document addresses in the queues. The Frontier includes a set of parallel "priority queues," each associated with a distinct priority level. Queue elements for documents to be downloaded are assigned a priority level, and then stored in the corresponding priority queue. Queue elements are then distributed from the priority queues to a set of underlying queues in accordance with their relative priorities. The threads then process the queue elements in the underlying queues. When performing a continuous crawl, the web crawler reinserts the queue element for a downloaded document into the Frontier in accordance with a download priority level associated with the downloaded document. For example, the download priority level may be determined as a function of an expiration date and time associated with document whose document address is denoted by the queue element.

대표청구항 ▼

[ What is claimed is:] [1.]1. A method of performing a continuous crawl for locating and downloading documents from among a plurality of host computers, comprising:(a) obtaining at least one referring document set that includes addresses of one or more referred documents; each referred document addr

[ What is claimed is:] [1.]1. A method of performing a continuous crawl for locating and downloading documents from among a plurality of host computers, comprising:(a) obtaining at least one referring document set that includes addresses of one or more referred documents; each referred document address including a host component;(b) enqueuing queue elements in a plurality of queues, each queue element denoting one of the referred document addresses;(c) substantially concurrently operating a plurality of threads;(d) while operating each thread, repeatedly performing steps of:(d1) identifying a queue element in a selected one of the queues, downloading a referred document corresponding to a referred document address in the identified queue element, and dequeuing the identified queue element; and(d2) executing at least one application program for processing the downloaded document;the plurality of queues including a plurality of parallel priority level queues, each having a distinct associated download priority level, the download priority level corresponding to a probability of the queue elements enqueued in the associated priority level queue therein being processed by the threads; andstep (d) including determining a download priority level for maintaining freshness of the downloaded document and re-enqueuing the queue element for the downloaded document in one of the parallel priority level queues in accordance with the determined download priority level.