Diagnostics for intermittent CI failures - #89
Closed
AndrewSav wants to merge 5 commits into
Closed
Conversation
reverse and ldap2 run with DISABLE_DNS_RESOLVER=true and query dbl.spamhaus.org through whatever resolver the host provides. Spamhaus refuses some resolvers with 127.255.255.254, which the unqualified reject_rhsbl_sender in main.cf treats as a listing, so every fixture sender is rejected and the suite fails in ways that name delivery, rspamd and zeyple rather than DNS. A passing run recorded nothing about the blocklist at all, so it could not be told apart from a run where the DBL answered nothing. fixtures_reverse, fixtures_ldap2 and fixtures_default (the unbound control) now print, before any mail is sent, the DNS status, A and TXT answers for dbltest.com (the permanent test listing), gmail.com (the fixtures' sender domain, the name postfix actually looks up) and example.com (unlisted): 127.0.1.2 healthy 127.255.255.254 refused; the TXT names the address Spamhaus refused NXDOMAIN answered nothing; the blocklist is inert The probe is diagnostic only and cannot fail a run.
A healthy Spamhaus answer does not say which resolver produced it; only a refusal names the egress address it refused. So on a passing run the probe could not tell which resolver the runner had used. It now prints, first, the nameserver the container is configured with and the resolver's egress address as seen by two whoami services (whoami.akamai.net and o-o.myaddr.l.google.com), and, for every DBL answer, the remaining TTL: a full value is a fresh answer, a lower one was served from cache and shows how long ago it was fetched.
master.yml only ran on pushes to master, so a PR could never exercise the single-runner all-suites job that failed on master. With pull_request added it runs on the PR alongside the per-suite workflows, and re-running it samples a fresh runner each time.
This reverts commit 060bd07.
make dnsbl starts a container like default's, without ClamAV and with DISABLE_DNS_RESOLVER=true so postfix queries dbl.spamhaus.org through the host's resolver, then runs test/dnsbl-loop.sh: before every attempt it logs the resolver and the blocklist's answers, sends one message with the fixtures' sender (user@gmail.com), and reports the reply. It stops at the first refusal, exits 1 so the run is red with the probe output directly above the 554, and exits 0 after DNSBL_ITERATIONS (20) accepted attempts DNSBL_INTERVAL (15 s) apart. Both outcomes are logged. Spamhaus publishes no numeric limit for its public mirrors, only that volume must stay within what non-commercial use reasonably needs, and answers 127.255.255.255 to excessive queries. An iteration costs 9 queries (8 from the probe, 1 from postfix), so a run is 180 spaced over five minutes; the loop stops immediately if it ever sees 127.255.255.255. The interval exceeds the 10 s negative TTL observed on the sender-domain answer, so each attempt reaches Spamhaus rather than the resolver's cache. The probe moves from the Makefile into test/share/tests/dnsbl-probe.sh, which is mounted into every suite's container, so the fixtures and the loop share it. The suite is not part of make all; it has its own workflow.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The issue I'm hitting at home, and tried to address with #87 also is affecting the
CI intermittently, see 34746135638. This PR is to add some logging, to be able to reason about it a bit better.
Log what the DBL answers the test resolver before sending fixtures reverse and ldap2 run with DISABLE_DNS_RESOLVER=true and query dbl.spamhaus.org through whatever resolver the host provides. Spamhaus refuses some resolvers with 127.255.255.254, which the unqualified reject_rhsbl_sender in main.cf treats as a listing, so every fixture sender is rejected and the suite fails in ways that name delivery, rspamd and zeyple rather than DNS. A passing run recorded nothing about the blocklist at all, so it could not be told apart from a run where the DBL answered nothing.
fixtures_reverse, fixtures_ldap2 and fixtures_default (the unbound control) now print, before any mail is sent, the DNS status, A and TXT answers for dbltest.com (the permanent test listing), gmail.com (the fixtures' sender domain, the name postfix actually looks up) and example.com (unlisted):
127.0.1.2 healthy
127.255.255.254 refused; the TXT names the address Spamhaus refused
NXDOMAIN answered nothing; the blocklist is inert
The probe is diagnostic only and cannot fail a run.