Skip to content

Commit 17ad071

Browse files
ralyodioclaude
andauthored
The pass we sell is now a rung on the ladder (#185)
* The pass we sell is now a rung on the ladder /crawl has sold a day pass since 2026-09-05. It bought a training crawler past the 402 and then met the same 600-an-hour throttle as an anonymous curl, so the thing we sell bought nothing a caller could feel. Three changes, all of them rate: - TIERS.pass, at the sponsor ceiling: 120,000 an hour and 2,000 a minute. Deliberately the same numbers as sponsor rather than higher, because a sponsored key is a relationship and a pass is a dollar, and there is no honest reason the dollar should out-rank the relationship. - The proxy places a pass holder on that rung. The gate has already read the same token and said nothing about it, because "should this be charged" and "what allowance is this" are different questions; reading it again is one HMAC and only for a request that presents a token. - The 429 names the pass, with its price and how to buy it. Every branch of the upgrade path ended at something only a person can do: read an email, fill in a form, ask us. An agent that hits the wall at three in the morning had nowhere to go but slow down or spread itself over a proxy pool. Now it has a rung it can climb alone. The sales page also lists what a pass buys, via the benefits list added in x402-gateway 0.4.0 (bumped here from 0.3.0). WHAT IS NOT IN THIS, AND WHY. Every line of that list is rate. The directory's position, argued at length in lib/tiers.js, is that the data stays open to everyone -- no key, no account, no field withheld -- and money buys speed. Withholding fields from the free tier would sell the opposite of what this site is for, and would cost us the search and AI-answer traffic that is the only way a person finds us. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TKuF2jCbRj3GZTQVtwhi5m * Let the lockfile carry x402-gateway 0.4.0 pnpm 11 refuses a lockfile entry published inside minimumReleaseAge, and 0.4.0 is hours old. The exclude list already names every earlier version of this package for the same reason. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TKuF2jCbRj3GZTQVtwhi5m --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
1 parent 555f523 commit 17ad071

7 files changed

Lines changed: 246 additions & 11 deletions

File tree

‎apps/web/package.json‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -12,7 +12,7 @@
1212
"dependencies": {
1313
"@profullstack/player": "^0.3.1",
1414
"@profullstack/rssamplifier": "workspace:*",
15-
"@profullstack/x402-gateway": "0.3.0",
15+
"@profullstack/x402-gateway": "0.4.0",
1616
"@rssamplifier/auth": "workspace:*",
1717
"@rssamplifier/db": "workspace:*",
1818
"@rssamplifier/feed": "workspace:*",

‎apps/web/src/lib/crawl-gateway.js‎

Lines changed: 62 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
import { createGateway, isTrainingAgent, RETRIEVAL_AGENTS } from '@profullstack/x402-gateway';
1+
import { createGateway, isTrainingAgent, readPass, RETRIEVAL_AGENTS } from '@profullstack/x402-gateway';
22
import { crawlSales } from '@rssamplifier/db';
33

44
import { db } from './db.js';
@@ -186,6 +186,26 @@ export const gateway = createGateway({
186186
*/
187187
chargeSpoofedBrowsers: true,
188188
exempt,
189+
/*
190+
* What a pass actually buys, printed on the sales page and in every 402.
191+
*
192+
* Every line is rate, and that is not a hedge — it is the directory's whole
193+
* position, argued in lib/tiers.js: the data stays open to everyone, no key,
194+
* no account, no field withheld, and money buys speed. A benefits list that
195+
* promised fields or endpoints the free tier cannot have would be selling the
196+
* opposite of what this site is for, and would cost us the search and AI
197+
* answer traffic that is the only way a person finds us.
198+
*
199+
* Written as the numbers rather than as adjectives, because the reader is
200+
* deciding between paying, waiting, and rotating addresses, and only the
201+
* numbers settle that.
202+
*/
203+
benefits: [
204+
'120,000 requests an hour, up from 600 anonymous or 6,000 signed in',
205+
'2,000 requests a minute of burst, up from 120',
206+
'No throttle on any page, feed, or API route, for the whole day',
207+
'Every field and every entry point stays open to everyone, paid or not: a pass buys rate, never access',
208+
],
189209
/*
190210
* Book the sale.
191211
*
@@ -228,3 +248,44 @@ export const gateway = createGateway({
228248
* @type {(request: Request) => Promise<Response | undefined>}
229249
*/
230250
export const gate = x402Proxy(gateway);
251+
252+
/** Where a pass is presented: the named header, or a bearer token. */
253+
const PASS_HEADER = 'x-crawl-pass';
254+
const BEARER_PASS = /^Bearer\s+(cp_[A-Za-z0-9_-]+\.[A-Za-z0-9_-]+)$/i;
255+
256+
/**
257+
* Whether this request carries a crawl pass that is genuinely ours and still
258+
* live.
259+
*
260+
* The gate already reads the pass, and answers `undefined` for a holder exactly
261+
* as it does for a person — which is correct for deciding whether to charge,
262+
* and loses the one fact the throttle needs: that this caller has paid. Rather
263+
* than have the gate report it, which would grow a contract five other sites
264+
* depend on, the pass is read again here. It is an HMAC over a short string
265+
* with no database behind it, so reading it twice costs a hash.
266+
*
267+
* The secret is the CoinPay key, which is what `createGateway` defaults to when
268+
* no explicit `secret` is passed, so this verifies exactly the tokens the
269+
* gateway mints and nothing else. With no key configured there are no valid
270+
* passes to find, and every caller falls through to the free ladder.
271+
*
272+
* @param {Request} request
273+
* @returns {Promise<boolean>}
274+
*/
275+
export async function hasValidPass(request) {
276+
const secret = env['COINPAY_X402_KEY'];
277+
if (!secret) return false;
278+
279+
const direct = request.headers.get(PASS_HEADER);
280+
const bearer = BEARER_PASS.exec(request.headers.get('authorization') ?? '');
281+
const token = direct ? direct.trim() : bearer ? bearer[1] : null;
282+
if (!token) return false;
283+
284+
try {
285+
return Boolean(await readPass(token, { secret }));
286+
} catch {
287+
// A malformed token is not an error worth failing a request over; it is
288+
// simply not a pass, and the caller lands on the free ladder.
289+
return false;
290+
}
291+
}

‎apps/web/src/lib/tiers.js‎

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -101,6 +101,22 @@ export const TIERS = {
101101
hourly: FREE_HOURLY * AUTH_MULTIPLIER,
102102
},
103103
sponsor: { name: 'sponsor', burst: envInt('TIER_SPONSOR_BURST', 2_000), hourly: SPONSOR_HOURLY },
104+
/**
105+
* A bought crawl pass, at the sponsor's ceiling.
106+
*
107+
* The rung the ladder was missing. Until now the top of it was "ask us and we
108+
* will sponsor you", which is not something a program can do at three in the
109+
* morning, and the pass that /crawl already sold bought a training crawler
110+
* past a 402 and then met the same 600-an-hour throttle as everyone else. So
111+
* the thing we sell bought nothing a heavy caller could feel.
112+
*
113+
* Deliberately the same numbers as `sponsor` rather than higher: a sponsored
114+
* key is a relationship and a pass is a dollar, and there is no honest reason
115+
* the dollar should out-rank the relationship. The reasoning for that ceiling
116+
* being bounded rather than infinite is in SPONSOR_HOURLY above and applies
117+
* unchanged here.
118+
*/
119+
pass: { name: 'pass', burst: envInt('TIER_SPONSOR_BURST', 2_000), hourly: SPONSOR_HOURLY },
104120
};
105121

106122
/**

‎apps/web/src/proxy.js‎

Lines changed: 39 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1,10 +1,10 @@
11
import { NextResponse } from 'next/server';
22

33
import { SIGNED_IN_HINT_COOKIE, hintToRestore } from './lib/session-hint.js';
4-
import { gate } from './lib/crawl-gateway.js';
4+
import { gate, hasValidPass } from './lib/crawl-gateway.js';
55
import { attempt, callerIdentity } from './lib/crawlThrottle.js';
66
import { countRequest } from './lib/trafficCounter.js';
7-
import { tierFor } from './lib/tiers.js';
7+
import { TIERS, tierFor } from './lib/tiers.js';
88

99
/**
1010
* The one thing that runs in front of every request.
@@ -63,7 +63,19 @@ export async function proxy(request) {
6363
* unlimited, and nothing above it can be worth paying for. Signed in is now
6464
* a large budget rather than no budget.
6565
*/
66-
const tier = tierFor(request);
66+
/*
67+
* A bought pass is the top rung, and it is checked before the rest of the
68+
* ladder because it is the only one someone paid for. Without this the pass
69+
* we sell at /crawl bought a crawler past the 402 and then met the same
70+
* 600-an-hour throttle as an anonymous curl, which is to say it bought
71+
* nothing anyone could feel.
72+
*
73+
* The gate above has already read the same token and said nothing about it,
74+
* because "should this be charged" and "what allowance is this" are different
75+
* questions. Reading it again is one HMAC, and only for a request that
76+
* actually presents a token.
77+
*/
78+
const tier = (await hasValidPass(request)) ? TIERS.pass : tierFor(request);
6779

6880
const verdict = attempt(callerIdentity(request), Date.now(), tier);
6981

@@ -112,6 +124,29 @@ function tooMany(verdict, tier) {
112124
? 'Create an API key at https://rssamplifier.com/account and send it as a bearer token; a sponsored key raises the ceiling further.'
113125
: 'This is the sponsor ceiling. If you need more than this, ask and we will raise it.';
114126

127+
/*
128+
* The rung a program can climb on its own.
129+
*
130+
* Every branch above ends at something only a person can do: read an email,
131+
* fill in a form, ask us. An agent that hits this wall at three in the
132+
* morning has nowhere to go, and the honest options left to it are to slow
133+
* down or to spread itself over a proxy pool. So the pass is named here, with
134+
* its price, as the one upgrade that needs no human on either side.
135+
*
136+
* Said to the paid tier too, where it reads as "you already have this",
137+
* because a caller at the sponsor ceiling asking what is above it should be
138+
* told there is nothing rather than sold something twice.
139+
*/
140+
const buyable =
141+
tier.name === 'pass' || tier.name === 'sponsor'
142+
? undefined
143+
: {
144+
url: 'https://rssamplifier.com/crawl',
145+
price: '1.00 USD per day, USDC, settled by CoinPay',
146+
buys: '120,000 requests an hour and 2,000 a minute, for the whole day',
147+
how: 'Fetch https://rssamplifier.com/crawl with Accept: application/json for an x402 offer, pay it, then send the pass in the x-crawl-pass header. No account and no human needed.',
148+
};
149+
115150
return NextResponse.json(
116151
{
117152
error: 'rate limit exceeded',
@@ -121,6 +156,7 @@ function tooMany(verdict, tier) {
121156
tier: tier.name,
122157
hourlyLimit: Number.isFinite(tier.hourly) ? tier.hourly : null,
123158
upgrade: nextRung,
159+
...(buyable ? { buy: buyable } : {}),
124160
retryAfter: verdict.retryAfter,
125161
},
126162
{

‎apps/web/test/paid-tier.test.js‎

Lines changed: 122 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,122 @@
1+
import test from 'node:test';
2+
import assert from 'node:assert/strict';
3+
4+
import { mintPass } from '@profullstack/x402-gateway';
5+
6+
import { TIERS } from '../src/lib/tiers.js';
7+
8+
/**
9+
* The pass rung, and the thing it fixes.
10+
*
11+
* Until now /crawl sold a pass that bought a training crawler past the 402 and
12+
* then met the same 600-an-hour throttle as an anonymous curl. The pass is now
13+
* a rung on the ladder, so the thing we sell buys something a caller can feel.
14+
*
15+
* `hasValidPass` is imported lazily inside each test: lib/crawl-gateway.js
16+
* reads COINPAY_X402_KEY at module scope through `env`, and these set it first.
17+
*/
18+
const SECRET = 'cp_live_test_secret_0123456789';
19+
20+
async function passModule() {
21+
process.env.COINPAY_X402_KEY = SECRET;
22+
return import('../src/lib/crawl-gateway.js');
23+
}
24+
25+
const withHeaders = (headers) =>
26+
new Request('https://rssamplifier.com/topics/anything', { headers: new Headers(headers) });
27+
28+
async function livePass() {
29+
const now = Math.floor(Date.now() / 1000);
30+
const { token } = await mintPass({ secret: SECRET, ref: 'test', expiresAt: now + 3600, now });
31+
return token;
32+
}
33+
34+
test('a pass sits at the sponsor ceiling, not above it', () => {
35+
// A sponsored key is a relationship and a pass is a dollar. There is no
36+
// honest reason the dollar should out-rank the relationship.
37+
assert.equal(TIERS.pass.hourly, TIERS.sponsor.hourly);
38+
assert.equal(TIERS.pass.burst, TIERS.sponsor.burst);
39+
});
40+
41+
test('a pass is worth far more than signing in, or nobody would buy one', () => {
42+
assert.ok(
43+
TIERS.pass.hourly > TIERS.session.hourly,
44+
'a bought pass must beat the free account tier',
45+
);
46+
assert.ok(TIERS.pass.hourly > TIERS.anon.hourly * 100);
47+
});
48+
49+
test('a live pass in the header is recognised', async () => {
50+
const { hasValidPass } = await passModule();
51+
const token = await livePass();
52+
assert.equal(await hasValidPass(withHeaders({ 'x-crawl-pass': token })), true);
53+
});
54+
55+
test('a live pass presented as a bearer token is recognised', async () => {
56+
// The gateway accepts both spellings, so the throttle must agree with it or
57+
// a caller doing exactly what the sales page said gets throttled anyway.
58+
const { hasValidPass } = await passModule();
59+
const token = await livePass();
60+
assert.equal(await hasValidPass(withHeaders({ authorization: `Bearer ${token}` })), true);
61+
});
62+
63+
test('no pass is not a pass', async () => {
64+
const { hasValidPass } = await passModule();
65+
assert.equal(await hasValidPass(withHeaders({})), false);
66+
});
67+
68+
test('a forged pass is refused, which is the whole point of signing them', async () => {
69+
const { hasValidPass } = await passModule();
70+
const token = await livePass();
71+
const tampered = `${token.slice(0, -1)}${token.endsWith('A') ? 'B' : 'A'}`;
72+
assert.equal(await hasValidPass(withHeaders({ 'x-crawl-pass': tampered })), false);
73+
});
74+
75+
test('a pass from a different secret is refused', async () => {
76+
const { hasValidPass } = await passModule();
77+
const now = Math.floor(Date.now() / 1000);
78+
const { token } = await mintPass({
79+
secret: 'someone-elses-key',
80+
ref: null,
81+
expiresAt: now + 3600,
82+
now,
83+
});
84+
assert.equal(await hasValidPass(withHeaders({ 'x-crawl-pass': token })), false);
85+
});
86+
87+
test('an expired pass is refused', async () => {
88+
const { hasValidPass } = await passModule();
89+
const now = Math.floor(Date.now() / 1000);
90+
// Minted live, then judged from a moment after it lapsed.
91+
const { token } = await mintPass({ secret: SECRET, ref: null, expiresAt: now + 1, now });
92+
await new Promise((resolve) => setTimeout(resolve, 1100));
93+
assert.equal(await hasValidPass(withHeaders({ 'x-crawl-pass': token })), false);
94+
});
95+
96+
test('garbage in the header is not an error, just not a pass', async () => {
97+
const { hasValidPass } = await passModule();
98+
for (const value of ['', ' ', 'cp_', 'cp_nodot', 'not-a-pass', 'Bearer nonsense']) {
99+
assert.equal(
100+
await hasValidPass(withHeaders({ 'x-crawl-pass': value })),
101+
false,
102+
JSON.stringify(value),
103+
);
104+
}
105+
});
106+
107+
test('the sales page lists what a pass buys, and every line of it is rate', async () => {
108+
// The directory's position is that data is open and money buys speed. A
109+
// benefits list promising fields or endpoints the free tier cannot have would
110+
// sell the opposite of what this site is for.
111+
const { gateway } = await passModule();
112+
const benefits = gateway.options.benefits;
113+
assert.ok(Array.isArray(benefits) && benefits.length > 0, 'a pass has to say what it buys');
114+
assert.ok(
115+
benefits.some((line) => /120,000 requests an hour/.test(line)),
116+
'the headline number belongs in the list',
117+
);
118+
assert.ok(
119+
benefits.some((line) => /rate, never access/.test(line)),
120+
'and so does the promise that nothing is withheld',
121+
);
122+
});

‎pnpm-lock.yaml‎

Lines changed: 5 additions & 5 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎pnpm-workspace.yaml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -13,7 +13,7 @@ allowBuilds:
1313
msgpackr-extract: false
1414
minimumReleaseAgeExclude:
1515
- '@profullstack/player@0.2.0 || 0.3.1'
16-
- '@profullstack/x402-gateway@0.1.0 || 0.2.1 || 0.3.0'
16+
- '@profullstack/x402-gateway@0.1.0 || 0.2.1 || 0.3.0 || 0.4.0'
1717
- '@profullstack/x402-client@0.2.0'
1818
- '@profullstack/leaderboard@0.3.0 || 0.3.1'
1919
- '@profullstack/partners@0.2.0'

0 commit comments

Comments
 (0)