题目内容
To use Web crawlers will cost a lot of network resources.
查看答案
搜索结果不匹配？点我反馈
更多问题
If the frequency of accesses to a given server is too high, it will suffer from severoverload.
As long as a crawler downloads pages, it is sure to be able to handle them.
The robots exclusion protocol, or the robots.txt protocol, is able to solve the problem of the costs of using Web crawlers.
“Crawl-delay:” parameter in the robots.txt file asks crawlers to delay for a number of seconds between requests.
微信支付 
支付宝支付