题目内容
As long as a crawler downloads pages, it is sure to be able to handle them.
查看答案
搜索结果不匹配？点我反馈
更多问题
The robots exclusion protocol, or the robots.txt protocol, is able to solve the problem of the costs of using Web crawlers.
“Crawl-delay:” parameter in the robots.txt file asks crawlers to delay for a number of seconds between requests.
A parallel crawler can run multiple processes in parallel, as a result of which the download rate is maximized, the overhead is minimized and repeated downloads of the same page are avoided.
The new URLs discovered during the crawling process are assigned so that the same page is not downloaded more than once.
微信支付 
支付宝支付