Severity: High
Evidence:
README.md:143-144 claims an automatic illegal-content blacklist.references/safety.md:5 says the filter runs at the transport layer on every fetch() call and search result.references/safety.md:53 says torcore.fetch() blocks if the URL, title, or body text matches the blacklist.scripts/torcore.py:293 performs sess.get(...) before any URL safety check, using requests' default redirect-following behavior.scripts/torcore.py:295-302 detects an onion-to-clearnet redirect only after the redirected response has already been fetched.scripts/torcore.py:304-329 checks only the request URL plus title/final URL. The extracted body at scripts/torcore.py:333-349 is returned without a body safety check.scripts/crawl.py:103-114 passes fetched text directly into the crawler and persists it through scripts/db.py:197-203.
Deterministic reproduction without Tor:
import sys
sys.path.insert(0, 'scripts')
import torcore
class Resp:
status_code = 200
url = 'http://example.onion/page'
encoding = 'utf-8'
apparent_encoding = 'utf-8'
text = '<html><head><title>ordinary title</title></head><body>BLOCK_ME appears only in body</body></html>'
class Session:
def get(self, *args, **kwargs):
return Resp()
torcore.tor_session = lambda: Session()
torcore.is_content_safe = lambda text: 'BLOCK_ME' not in text
result = torcore.fetch('http://example.onion/page')
print({'error': result['error'], 'text': result['text'], 'title': result['title']})Actual output:
{'error': None, 'text': 'ordinary title\nBLOCK_ME appears only in body', 'title': 'ordinary title'}
Impact:
A page whose title and URL look safe but whose body contains blocked content is returned by fetch(), can be scraped through batch_scrape(), can be stored by crawl(), and can be exported later. Separately, the onion redirect guard does not prevent the redirected request; it only reports after the clearnet destination has already been contacted through Tor.
Smallest credible fix:
- Check the normalized URL before issuing the request.
- Use
allow_redirects=False; validate each Location before deciding whether to follow.
- Stream and cap response bytes before decoding/parsing so very large responses are not fully loaded.
- Run
is_content_safe() on title and extracted body before returning text or saving crawl results.
- Add mocked tests for URL-only, title-only, body-only, and onion-to-clearnet redirect cases.
Severity: High
Evidence:
README.md:143-144claims an automatic illegal-content blacklist.references/safety.md:5says the filter runs at the transport layer on everyfetch()call and search result.references/safety.md:53saystorcore.fetch()blocks if the URL, title, or body text matches the blacklist.scripts/torcore.py:293performssess.get(...)before any URL safety check, using requests' default redirect-following behavior.scripts/torcore.py:295-302detects an onion-to-clearnet redirect only after the redirected response has already been fetched.scripts/torcore.py:304-329checks only the request URL plus title/final URL. The extracted body atscripts/torcore.py:333-349is returned without a body safety check.scripts/crawl.py:103-114passes fetched text directly into the crawler and persists it throughscripts/db.py:197-203.Deterministic reproduction without Tor:
Actual output:
Impact:
A page whose title and URL look safe but whose body contains blocked content is returned by
fetch(), can be scraped throughbatch_scrape(), can be stored bycrawl(), and can be exported later. Separately, the onion redirect guard does not prevent the redirected request; it only reports after the clearnet destination has already been contacted through Tor.Smallest credible fix:
allow_redirects=False; validate eachLocationbefore deciding whether to follow.is_content_safe()on title and extracted body before returning text or saving crawl results.