2818204

IETF RFC 9309 [ Active ]

Robots Exclusion Protocol

standard by Internet Engineering Task Force , 09/01/2022

View all product details

Most Recent

Track It

Language:
    • Available Formats
    •  
    • Availability
    • Priced From ( in USD )
    • Printed Edition
    • Ships in 1-2 business days
    • $33.00
    • Add to Cart

Customers Who Bought This Also Bought

 

About This Item

 

Full Description

This document specifies and extends the "Robots Exclusion Protocol" method originally defined by Martijn Koster in 1994 for service owners to control how content served by their services may be accessed, if at all, by automatic clients known as crawlers. Specifically, it adds definition language for the protocol, instructions for handling errors, and instructions for caching.

 

Product Details

Edition:
22#
Published:
09/01/2022
Number of Pages:
12
File Size:
1 file , 170 KB