爬虫协议-反爬虫

# 例如

https://130.14.250.12/robots.txt

User-agent: *
Disallow: /

http://www.robotstxt.org/robotstxt.html

The "User-agent: *" means this section applies to all robots. The "Disallow: /" tells the robot that it should not visit any pages on the site.