Skip to content

[Bots] Clarify Web Bot Auth draft support #31306

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
thibmeu wants to merge 1 commit into
base: production
Choose a base branch
from
Open

[Bots] Clarify Web Bot Auth draft support #31306

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 7 additions & 3 deletions src/content/docs/bots/reference/bot-verification/web-bot-auth.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@ import { GlossaryTooltip, Steps } from "~/components";

Web Bot Auth is an authentication method that leverages cryptographic signatures in HTTP messages to verify that a request comes from an automated bot. Web Bot Auth is used as a verification method for [verified bots](/bots/concepts/bot/verified-bots/) and [signed agents](/bots/concepts/bot/signed-agents/).

It relies on two active IETF drafts: a [directory draft](https://datatracker.ietf.org/doc/html/draft-meunier-http-message-signatures-directory) allowing the crawler to share their public keys, and a [protocol draft](https://datatracker.ietf.org/doc/html/draft-meunier-web-bot-auth-architecture) defining how these keys should be used to attach crawler's identity to HTTP requests.
It relies on IETF drafts: a [directory draft](https://datatracker.ietf.org/doc/html/draft-meunier-http-message-signatures-directory-03) allowing the crawler to share their public keys, and a [protocol draft](https://datatracker.ietf.org/doc/html/draft-meunier-web-bot-auth-architecture-02) defining how these keys should be used to attach the crawler's identity to HTTP requests.

This documentation goes over specific integration within Cloudflare.

Expand Down Expand Up @@ -55,7 +55,7 @@ Many existing [JWK libraries](https://jwt.io/libraries) support WebCrypto API fo
## 2. Host a key directory

You need to host a key directory which creates a way for your bot to authenticate its requests to Cloudflare.
This directory should follow the definition from the active IETF draft [draft-meunier-http-message-signatures-directory-01](https://datatracker.ietf.org/doc/html/draft-meunier-http-message-signatures-directory-01).
This directory should follow the definition from [draft-meunier-http-message-signatures-directory-03](https://datatracker.ietf.org/doc/html/draft-meunier-http-message-signatures-directory-03).

<Steps>
1. Host a key directory at `/.well-known/http-message-signatures-directory` (note that this is a requirement). This key directory should serve a JSON Web Key Set (JWKS) including the public key derived from your signing key.
Expand Down Expand Up @@ -175,10 +175,13 @@ Construct a [`Signature` header](https://www.rfc-editor.org/rfc/rfc9421#name-the

#### `Signature-Agent` header

Construct a [`Signature-Agent` header](https://www.ietf.org/archive/id/draft-meunier-http-message-signatures-directory-01.html#name-header-field-definition) that points to your key directory. Note that Cloudflare will fail to verify a message if:
Construct a [`Signature-Agent` header](https://datatracker.ietf.org/doc/html/draft-meunier-http-message-signatures-directory-03#name-header-field-definition) that points to your key directory. Cloudflare implements the `Signature-Agent` format from `draft-meunier-http-message-signatures-directory-03`, where the header value is a structured string such as `"https://signature-agent.test"`.

Cloudflare will fail to verify a message if:

- The message includes a `Signature-Agent` header that is not an `https://`.
- The message includes a valid URI but does not enclose it in double quotes. This is due to Signature-Agent being a structured field.
- The message uses the dictionary form from later drafts, such as `sig2="https://signature-agent.test"`.
- The message has a valid `Signature-Agent` header, but does not include it in the component list in `Signature-Input`.

### 4.4. Add the headers to your bot's requests
Expand Down Expand Up @@ -229,6 +232,7 @@ The following component parameters defined in IETF RFC 9421 are not supported, a
If your message is failing validation, the cause(s) may include:

- Ensure you have a [`Signature-Agent` header](/bots/reference/bot-verification/web-bot-auth/#signature-agent-header), and that its value is in double-quotes.
- Ensure your [`Signature-Agent` header](/bots/reference/bot-verification/web-bot-auth/#signature-agent-header) uses a structured string, not a dictionary.
- Ensure you include `signature-agent` in the component list in your [`Signature-Input` header](/bots/reference/bot-verification/web-bot-auth/#signature-agent-header).
- Ensure your `expires` timestamp is not too short, such that, by the time it arrives at Cloudflare servers, it has already expired. A minute is often sufficient.
- Ensure you are not signing components containing non-ASCII values, or on the unsupported list.
Expand Down
Loading