题目内容
It's difficult for a server to satisfy multiple requests per second and/or downloading large files from a single crawler or the requests from multiple crawlers.
查看答案
搜索结果不匹配？点我反馈
更多问题
To use Web crawlers will cost a lot of network resources.
If the frequency of accesses to a given server is too high, it will suffer from severoverload.
As long as a crawler downloads pages, it is sure to be able to handle them.
The robots exclusion protocol, or the robots.txt protocol, is able to solve the problem of the costs of using Web crawlers.
微信支付 
支付宝支付