Skip to content

BIP93: Fix checksum selection bounds and restrict ms sizes - #2258

Open
BenWestgate wants to merge 2 commits into
bitcoin:masterfrom
BenWestgate:bip93-checksum-boundary
Open

BIP93: Fix checksum selection bounds and restrict ms sizes#2258
BenWestgate wants to merge 2 commits into
bitcoin:masterfrom
BenWestgate:bip93-checksum-boundary

Conversation

@BenWestgate

@BenWestgate BenWestgate commented Aug 17, 2026

Copy link
Copy Markdown
Contributor

Motivation

BIP93 currently admits expanded codewords beyond the checksum’s stated period, so its detection guarantee no longer applies to the complete codeword. This is a problem for a generic codex32_decode function in later PRs. Both regular and long checksums permitted codewords in excess of their period.

Meanwhile, fixing the above will invalidate very uncommon ms strings or leave an ugly disjoint checksum selection rule for len(hrp) == 2. A better solution is immediately follow up with by restricting master seeds to the most commonly used seed and entropy sizes. This reduces target length ambiguity for optional insert/delete correcting wallets and my benchmarks show improvements in both false-correction risk and performance at the same indel search depths.

Changes

  • ms32_verify_checksum and ms32_create_checksum now include the expanded "ms" HRP length when selecting regular vs. long checksums.
  • ms32_create_checksum uses if 5 + len(data) > 80: as the switch to ms32_create_long_checksum.
  • added ms32_create_regular_checksum and ms32_verify_regular_checksum.
  • The Python reference code, specification, rationale, and vectors are updated.
  • ms32_decode enforces the new ms string lengths.
  • Obsolete tests/vectors from the seed size restrictions are removed.

codex32 lengths:

  • Regular codewords will now be <= 93 expanded values
  • Long codewords will now be >= 96 and <= 1023 expanded values

invalid codex32 lengths:

  • Expanded values 94, 95 and > 1023.

codex32-encoded master seed sizes:

  • 16-, 20-, 24-, 28-, 32- and 64-bytes

unsupported ms sizes:

  • All from 16- to 64-bytes besides the six above.

Compatibility

This is a breaking change for old strings encoding deprecated byte-length seeds. However the authors do not see this is a problem due to the exceedingly rare possibility these have ever been created and used.

Testing

Thoroughly reviewed the complete diff.
Checked the new vectors.
Checked that expanded length 1023 verifies and 1024 fails.
Checked that the legacy short-checksum vectors should fail.
Checked that ms32_decode rejects newly invalid lengths, even when they have valid header, incomplete group and checksum.

Discussion

Proposal: #2040 (comment)
cACK #2040 (comment)
Reference impl issue: BlockstreamResearch/codex32#75
rust-codex32 PR: BlockstreamResearch/codex32#76
Restrict ms sizes suggestion: #2258 (comment)

@vitrixLab

This comment was marked as low quality.

@murchandamus murchandamus added Proposed BIP modification PR by non-owner to update BIP content Pending acceptance This BIP modification requires sign-off by the champion of the BIP being modified labels Aug 17, 2026
@murchandamus

Copy link
Copy Markdown
Member

cc: @roconnor-blockstream, @apoelstra

@BenWestgate

BenWestgate commented Aug 20, 2026

Copy link
Copy Markdown
Contributor Author

Remaining gate: reproducible boundary vectors.

I added "reproducible boundary vectors."

Edit: Unsure if I should have done this to satisfy a review comment that was later marked "low quality".

For vectors, we need at minimum:

  • add regular checksum 44-, 45-, and 46-byte seeds to These examples use the wrong checksum for their given data sizes
  • add Expanded values 94, 95 and > 1023 to These examples have improper lengths
  • remove any vectors that are now accepted

Willing to revert the "reproducible" vectors added in a9d8246 if reviews also find it overkill or open to another suggestion.

What I would keep:
Separated checksum verification into regular/long primitives that check expanded length is within the checksum period and residue is valid. While ms32_verify_checksum owns the switch to long primitive at 96 expanded symbol codewords and calls the above.

Similarly ms32_create_regular_checksum creates the short checksum for data, while ms32_create_checksum selects the required checksum by expanded codewode length and creates it.

It should be easier to generate test vectors and test them now, as well as to reuse our checksums for non-codex32 data.

BenWestgate added a commit to BenWestgate/codex32 that referenced this pull request Aug 20, 2026
Account for the HRP expansion when selecting and slicing regular and long
checksums. Enforce the invalid 94–95 value gap and update the affected test
vectors.

Aligns the reference implementation with bitcoin/bips#2258.
@roconnor

roconnor commented Aug 25, 2026

Copy link
Copy Markdown

Based on my comment at #2040 (comment) which received some thumbs up by all parties here, I wonder if it everyone would find it preferable to simply restrict BIP-93 to seed lengths 128, 160, 192, 224, 256 and 512. I think I slightly prefer that, and if we are going to make such a change it would be better to do it from the get go.

@BenWestgate

Copy link
Copy Markdown
Contributor Author

cACK

simply restrict BIP-93 to seed lengths 128, 160, 192, 224, 256 and 512. I think I slightly prefer that, and if we are going to make such a change it would be better to do it from the get go.

To prepare to generalize BIP-93 we still must also correct the checksum-period coverage or it leaves the unsafe checksum-selection question for every later application:

For an ms application to restrict to these lengths, codex32_decode must verify the separator, header and checksum, then return HRP and u5 data. If expanded codewords may exceed the checksum period, detection guarantees won't apply, risking undetected cross-application/profile confusion.

Since restricting ms seed lengths can simplify the test vectors, rationale and compatibility sections I will add it here as a separate commit rather than revive #2077.

  • Checksum verification primitives enforce their maximum 93/1023 periods.
  • Checksum verification selects Long from 96 expanded length.
  • Checksum creation selects Long above 80 expanded hrp plus data length.
  • Master seed format rules will update and state its valid lengths.
  • “Long codex32 only for 512 bits” is an ms-profile rule, not a general rule.
  • Compatibility mentions the removed previously valid non-multiple of 4-byte and 33-63-byte sizes and their previous checksum selections.
  • SLIP-0039 permits additional lengths, compatibility needs "interconvertible with SLIP-0039 master seeds of supported lengths".
  • 43–47-byte master-seed vectors will be removed.
  • Checksum-layer assertions for 93/94/95/96/1023/1024 will be added while ms tests will enforce the six seed lengths.

Include the expanded human-readable part when selecting the codex32
checksum. This keeps regular codewords within the 93-symbol BCH period
and rejects the 94- and 95-symbol gap before the long checksum starts
at 96 symbols.

Split regular and long checksum construction and verification so their
periods can be tested independently. Add assertions for both selection
boundaries and the long checksum upper limit.

Refs: bitcoin#2258 (comment)
BenWestgate added a commit to BenWestgate/bips that referenced this pull request Aug 26, 2026
Include the expanded human-readable part when selecting the codex32
checksum. This keeps regular codewords within the 93-symbol BCH period
and rejects the 94- and 95-symbol gap before the long checksum starts
at 96 symbols.

Split regular and long checksum construction and verification so their
periods can be tested independently. Add assertions for both selection
boundaries and the long checksum upper limit.

Refs: bitcoin#2258 (comment)
@BenWestgate
BenWestgate force-pushed the bip93-checksum-boundary branch 2 times, most recently from 53203a9 to c201536 Compare August 26, 2026 20:18
@BenWestgate

Copy link
Copy Markdown
Contributor Author

c201536 is ready for review. I drafted the next commit "BIP93: Restrict ms seed lengths", working on the vectors.

Each commit is stand alone complete.

Restrict codex32-encoded BIP32 master seeds to 16, 20, 24, 28,
32, or 64 bytes. These sizes correspond to the BIP39 entropy
sizes and the 512-bit BIP32 seed produced by BIP39 recovery and provide
minimum six-character length gaps to reduce target length ambiguity
during optional insertion/deletion correction.

Move payload decoding out of the generic codex32 secret definition
and make it application-specific. Define the byte conversion and
length requirements under the master seed format instead.

Additionally:
- Enforce the new valid encoded lengths (48, 54, 61, 67, 74, 127) in
  the Python `ms32_decode` in-line reference.
- Remove obsolete short-checksum backward compatibility constraints and
  unsupported length test vectors.
- Add test vectors for 160, 192, and 224-bit master seeds
- Update rationale, compatibility, and reference implementation links.
@BenWestgate BenWestgate changed the title BIP93: Fix checksum selection boundaries BIP93: Fix checksum selection bounds and restrict ms sizes Aug 26, 2026
@apoelstra

Copy link
Copy Markdown
Contributor

ACK 5117f58

This looks good to me! Lol I think my hideous BIP39->codex32 conversion scheme is now permissible under "application-specific rules".

@jonatack jonatack removed the Pending acceptance This BIP modification requires sign-off by the champion of the BIP being modified label Aug 28, 2026
@BenWestgate

Copy link
Copy Markdown
Contributor Author

This looks good to me! Lol I think my hideous BIP39->codex32 conversion scheme is now permissible under "application-specific rules".

It indeed is! I used your bip39_12w profile's non-byte aligned payload as a major consideration here and in my rewrite of #1958. (payload length parameter, not a byte length) And for bip39_24w our regular checksum fully covers the codewords.

@roconnor

roconnor commented Aug 29, 2026

Copy link
Copy Markdown

I had thought that SLIP-0039 supported just 128 and 256, but now I see that it supports lengths [128,144,160,176,192,208,224,240,256].

This PR for BIP-93 supports [128, 160, 192, 224, 256] (and 512).

I think we should at give some consideration in our discussion here in supporting all SLIP-39 lengths. It would cause no problems for us regarding long and short codex32 checksums. It would make insertion and deletion recovery somewhat more difficult.

With the full SLIP-93 lengths [128,144,160,176,192,208,224,240,256] we would have codex32 encodings of lengths [48,51,54,58,61,64,67,70,74] with gaps of 3 or 4 characters between valid lengths.

Thinking out loud, if you take a 160 bit, 54 length ms codex32 string and delete 3 random characters, can you actually able to recover that by trial and error inserting 3 characters? There are 54 choose 3 positions to try inserting, and 32^3 combinations to try in each position which is a total of 812 777 472 possible combinations to try. That does seem to be a feasible number to try, especially since you are looking for one with a valid checksum.

So we are considering a trade off here of enhanced SLIP-39 compatibility for secret lengths that probably no one ever uses versus allowing recovering an from up to 3 characters being omitted from a codex32 string, which feels like also an unlikely number of characters to forget to jot down.

I'm genuinely torn here. What are the thoughts of other folks here?

@roconnor

roconnor commented Aug 29, 2026

Copy link
Copy Markdown

Maybe another angle to consider is, would users want to try 144 bit secrets with codex32? The argument is that Bitcoin has 128-bit security so you really want to debias your dice / entropy source when making a 128 bit secret. An alternative would be to make a longer secret and avoid debiasing your entropy with the idea that your longer secret would still contain 128-bits of entropy. This was my main motivation for considering 160 bit secrets (which was the next step up in BIP-39 secret sizes). But shorter is better because it is less work to create checksums and fewer characters means fewer errors to correct.

The question is can users use regular biased dice to get 128 bits of entropy in 144 bits?

The AI machine suggests an air bubble trapped in a cheap opaque die could bias one face up to 25% of the time. If the distribution were [25%, 16.75%, 16.75%, 16.75%, 16.75%, 8%], that would reduce the entropy of the dice from 2.58 bits per roll to 2.518 bits per roll. If the distribution were even more unreasonable at [33%, 16.66%, 16.66%, 16.66%, 16.66%, 0%] (this is also a fair die with one number repeated on a face). We'd have 2.25 bits per roll. Treating that die as fair would leave you with only than 125 bits of entropy in a 144 bit secret, but would give you 138 bits for a 160 bit secret.

So I don't know. Maybe 144 bits is still to small to eliminate debiasing?

@roconnor

roconnor commented Aug 29, 2026

Copy link
Copy Markdown

As a point of information, as a child I had a PC game of monopoly where the RNG had an off by 1 error and 6s were never rolled. This gives a "five" sided dice with entropy of 2.32 per roll, which gives just over 128 bits of entropy for a 144 bit secret. We did eventually notice the missing 6, but it took quite a surprising amount of time to realize.

I think I'd argue that 144 bit is inadequate for generating 128 bits of entropy using dice without debiasing.

@roconnor

roconnor commented Aug 29, 2026

Copy link
Copy Markdown

As counter argument for why 144 bits could be adequate, if you are building a 2 out of n secret share, you are xoring 2 biases secrets together which, generally speaking, reduces the bias of the resulting master secret.

That is to say that while generating a 144 bit master secret form a plausibly biased die seems too close to borderline unsafe, generating a 144-bit master secret from 2 randomly generated shares feels a lot safer.

(As a counter-counter argument, leaking k-1 shares reduces the security back down to a single share, though that involves the attacker both loading the dice and accessing k-1 shares).

@roconnor

Copy link
Copy Markdown

We could consider a compromise where ms secrets are limited to between 128 and 256 bits (i.e. 16-32 bytes) (plus the 512 bit long format), but generating bit sizes outside [128, 160, 192, 224, 256] is discouraged? I don't know if that is sensible.

@BenWestgate

BenWestgate commented Aug 29, 2026

Copy link
Copy Markdown
Contributor Author

We could consider a compromise where ms secrets are limited to between 128 and 256 bits (i.e. 16-32 bytes) (plus the 512 bit long format), but generating bit sizes outside [128, 160, 192, 224, 256] is discouraged? I don't know if that is sensible.

Adding 160, 192 and 224 already required my single threaded python indel correction to search 160, 192 and 224 one indel less deeply to stay under 10 seconds on my laptop.

It can fill 4 omissions in 44 character observed in about 5 seconds. Since omissions are marked as erasures and algebraically solved, its only 149k combinations of 4 in 45 since deletions are marked as erasures and algebraically solved. 250k for 4 in 51 and 1.1 million for 4 in 71. (128, 160 and 256-bit respectively)

The limit for filling omissions is false-correction. Intuitively it feels wrong to correct more deletions than substitutions so I prefer the symmetric "up to 4 extra or omitted characters, and mixtures" I.e. I+O <= 4.

Removing inserted characters is faster and CPU limited rather than accuracy limited.

I will simulate how much adding the 4 extra lengths increases ambiguity assuming non 128 and 256 remain searched to 3 indels.

Lastly we say "we do not recommend interconversion and suggest users generate a fresh master seed" so this potentially harms an optional feature to better support a not recommended feature.

We said wallets may assume the true length is 48 or 74. But if weird lengths get corrected to 48 or 74 they're not well supported so they need to be checked at some depth before "assuming 48 or 74".

For avoiding transcription errors and having fast corrections of them, the best recommendation is avoid generating sizes besides 128-bit, the 4-indel correction is necessarily slower on 256-bit and they've more chances to make mistakes, we already say " shorter strings are better."

However if not generating 128-bit, 256-bit is the next best size to reduce search ambiguity.

SLIP 39 requires 128 and 256 and requires 16-bit multiples so if the argument is slip39 compatibility, its either 128 and 256 alone or every even byte size 16-64, not just those 16-32.

@BenWestgate

Copy link
Copy Markdown
Contributor Author

I had thought that SLIP-0039 supported just 128 and 256, but now I see that it supports lengths [128,144,160,176,192,208,224,240,256].

It supports all even byte lengths 16-64 inclusive.

It would cause no problems for us regarding long and short codex32 checksums. It would make insertion and deletion recovery somewhat more difficult.

One of these lengths switched checksums in my first commit here.

I will test indel recovery, if it's not meaningfully worse than the 6 lengths in my 2nd commit, I have no objection.

gaps of 3 or 4 characters between valid lengths.

Thinking out loud, if you take a 160 bit, 54 length ms codex32 string and delete 3 random characters, can you actually able to recover that by trial and error inserting 3 characters? There are 54 choose 3 positions to try inserting, and 32^3 combinations to try in each position which is a total of 812 777 472 possible combinations to try.

Only the positions need search, the characters are marked erasures and filled without brute force.

3 is very fast, 4 omitted is seconds on a laptop.

So we are considering a trade off here of secret lengths that probably no one ever uses versus allowing recovering an from up to 3 characters being omitted from a codex32 string, which feels like also an unlikely number of characters to forget to jot down.

Its possible to recover from 5 omitted characters too if 0.1% chance of false corrections is acceptable. Limiting to 4 is safer.

We dont recommend interconversion, indel recovery is optional.

We could use a new profile SLIP39_EMS and let that one support every even byte length and also the passphrase feature from SLIP39, while using our SSS and checksums.

I also don't oppose every even byte length if indel recovery is unharmed.

@BenWestgate

Copy link
Copy Markdown
Contributor Author

Maybe another angle to consider is, would users want to try 144 bit secrets with codex32?

I dont think a lower bound can be assumed on biased entropy. If the goal is less work, more efficient entropy extractors and 128-bit secrets are better.

But shorter is better because it is less work to create checksums and fewer characters means fewer errors to correct.

Fewer lengths means fewer length errors to correct, fewer worksheets to print.

The question is can users use regular biased dice to get 128 bits of entropy in 144 bits?

This would require modifying the "uniform random" part of share generation.

So I don't know. Maybe 144 bits is still to small to eliminate debiasing?

Biased shares means we can't say nothing is learned about the secret at K-1, if smallest strings are better, so is debasing.

Debiasing doesn't take too long. I can generate 128-bits with a quarter in 1/2 hour. But I would not trust 160 or even 256 flips to be 128-bit entropy.

@BenWestgate

Copy link
Copy Markdown
Contributor Author

xoring 2 biases secrets together which, generally speaking, reduces the bias of the resulting master secret.

My thought experiment was if my dice only give 1s and 2s treating that like log2(6) bits and xoring with another share produced with only 1s and 2s is not going to improve much. they're biased in the same way so entropy does not increase.

@roconnor

Copy link
Copy Markdown

I had thought that SLIP-0039 supported just 128 and 256, but now I see that it supports lengths [128,144,160,176,192,208,224,240,256].

It supports all even byte lengths 16-64 inclusive.

Ah, my whole premise was mistaken then. Okay, I'm less inclined to support the full range of SLIP-39 seeds. The most common sizes of 128-bit and 256-bit seeds will still be interconvertable. And, as noted in this PR, all of the seed sizes supported by this amendment, are also interconvertable.

Now I'm inclined to keep this aspect of the PR as is.

Comment thread bip-0093.mediawiki
*** We do not define how to choose the identifier, beyond noting that it SHOULD be distinct for every master seed and master seed share set the user may need to disambiguate.
** The share index "s".
** A conversion of the 16-to-64-byte BIP-0032 HD master seed to bech32:
** A conversion of a 16-, 20-, 24-, 28-, 32-, or 64-byte BIP-0032 HD master seed to bech32:

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would you be willing to rephrase this in terms of bits? I personally almost always think of seed sizes in terms of bits.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sure, the original used bytes to concisely imply multiples of 8-bits.

I think of bits when generating, but if encoding an existing, I think bytes as thats how the software moves seeds around.

Now that its noncontiguous the word savings is minimal.

Comment thread bip-0093.mediawiki
* Re-arrange those bits into groups of 8 bits. Any incomplete group at the end MUST be 4 bits or less, and is discarded.

Unlike the decoding process in BIP-0173, master-seed decoding does not require that the discarded incomplete group contain only zero bits.
The decoded master seed MUST be exactly 16, 20, 24, 28, 32, or 64 bytes.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this is fine to stay as bytes here since the decoding process is producing bytes as output.

Comment thread bip-0093.mediawiki
The 13 character checksum design only supports expanded codewords of up to 93 values.
After accounting for the expanded <code>ms</code> human-readable part, header, and checksum, this limits the payload of a regular codex32 string to 69 characters.
While this is enough to support the 32-byte advised size of BIP-0032 master seeds, BIP-0032 allows seeds to be up to 64 bytes in size.
We define a long codex32 format to support these longer seeds by defining an alternative checksum.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"While this is enough to support the 256-bit advised size of BIP-0032 master seeds, BIP-0032 allows seeds to be up to 512 bits in size. In particular, BIP-0039 derives 512 master seeds from its mnemonic.
We define a long codex32 format to support 512 bit seeds by defining an alternative checksum."

@BenWestgate BenWestgate Aug 31, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why don't we change bytes to bits in another PR?

It touches lines outside these two commits and is behavior neutral. I have a refactor for BIP93 queued based on this (with typos and reordering), a bit vs bytes commit could go there if you want to propose one over the full text.

I'm sure we made arbitrary choices and clarity could be improved.

Comment thread bip-0093.mediawiki

def ms32_encode(data):
combined = data + ms32_create_checksum(data)
return "ms" + "1" + ''.join([CHARSET[d] for d in combined])

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I recommend returning None if the constructed string it doesn't have a valid length.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I copied the style of bech32_encode https://github.com/sipa/bech32/blob/master/ref%2Fpython%2Fsegwit_addr.py#L68-L71

Its intentional to not validate data on encode and create_checksum functions.

I sort of wanted to write encode from bytes for the "ms" secret, which would call decode to verify itself (like encode/decode, in segwit_addr.py) but I kept this since master seed shares also need an encode function but must not be encoded from bytes.

Comment thread bip-0093.mediawiki
A codex32 string is similar to a bech32 string defined in [https://github.com/bitcoin/bips/blob/master/bip-0173.mediawiki BIP-0173].
It reuses the base-32 character set from BIP-0173, and consists of:

* A human-readable part, which is the string "ms" (or "MS").

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we want to rephrase this in this PR to say the HRP is application specific, but will be "ms" (or "MS") for the application defined in this document ... or something like that?

@BenWestgate BenWestgate Aug 31, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To keep this focused I left HRP generalizing out of this PR. #2040, would say something like BIP173:

The human-readable part, which is intended to convey the type of data, or anything else that is relevant to the reader. This part MUST contain 1 to 500 US-ASCII characters, with each character having a value in the range [33-126]. HRP validity may be further restricted by specific applications.

The new vague sentence "string validity may be further restricted by specific applications" can then be moved to payload. "Payload validity may be further restricted by specific applications".

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also before #2040 we should probably be conceptually clear on application vs profile vs HRP.

"cl" is one of each, "ms" is one of each. "bip39_" is an application with "12w" and "24w" profiles.

If I propose codex32 xpubs and prepend key origin info in their HRP, that's millions of HRP but one profile of a BIP32 key serialization application.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Okay. I'm happy to postpone this until a future PR.

@roconnor

roconnor commented Sep 6, 2026

Copy link
Copy Markdown

I got a bunch of emails from @BenWestgate last Wednesday on this PR but that do not appear here. I'm very confused.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Proposed BIP modification PR by non-owner to update BIP content

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants