现在来看看两个最终的例子。

例1


# Don't come to this site

User-agent: *

Disallow: / # Disallows anything


例2


# robots.txt for XYZcorp

# webmaster: John Doe contact JohnD@xyzcorp.com

User-agent: * # Applies to all robots except next record

Disallow: /store/order/ # No robot should visit any URL starting with

/store/order/

Disallow: /admin/ # Disallow any pages in the admin folder

Disallow: /world_domination.htm # Disallow world_domination.htm


好，这就是有关robot.txt文件的全部内容。