Let DTLS max proto and options lift an earlier cap

wolfSSL_set_max_proto_version() and wolfSSL_clear_options() only moved a
DTLS SSL down. After a maximum of DTLS 1.2, a maximum of DTLS 1.3 or 0
returned success and left the SSL on DTLS 1.2, and clearing
SSL_OP_NO_DTLSv1_3 did not bring DTLS 1.3 back. When a maximum loosens
the mask, or wolfSSL_clear_options() clears a DTLS version bit, the
version is now walked down from the CTX method version, so the SSL moves
back up. A lift never moves an allowed version down, and keeps an
allowed version when the walk finds none, so a CTX maximum lowered after
wolfSSL_new() does not reach the SSL that way. A resuming client keeps
its session's version while the mask allows it. wolfSSL_set_options()
and the checks at handshake start still only step down.
wolfSSL_SetSession() now fails when the suites cannot be rebuilt for the
version the mask moved it to; a session with no version left is still
reported when the handshake starts.

The DTLS maximum shares the WOLFSSL_OP_NO_TLSv1_2/_3 bits with
SSL_OP_NO_DTLSv1_2/_3. The new options field dtlsMaxMask records the
bits the maximum added itself, and a later maximum clears only those, so
a version the application or the CTX masked off stays off.
wolfSSL_set_options() and wolfSSL_clear_options() take a bit out of the
record.

The mask no longer moves the version while the first handshake is under
way. DtlsHandshakeStarted() reports that from handShakeDone, the connect
and accept states, got_client_hello, buffered output and, with async
crypto, an async state other than TLS_ASYNC_BEGIN or handler args kept
in ssl->async, since DoTls13ClientHello() clears got_client_hello when
it pends and can pend before it leaves TLS_ASYNC_BEGIN. Before, an async
accept that re-entered could switch to DTLS 1.2 partway through a DTLS
1.3 ClientHello, and a client could have its version raised between
WANT_WRITE and the retry that sent the ClientHello. In that window a
DTLS wolfSSL_set_max_proto_version() that would change the mask fails
without changing anything, since it could no longer take effect, and one
that changes nothing succeeds. A maximum of 0 on a DTLS SSL means the
newest DTLS version, so it fails there too instead of settling on the
current cap. The maximum also refuses a change after the handshake: a
lift there would make a later DTLS 1.2 renegotiation treat the server's
downgrade sentinel as an attack. wolfSSL_clear() lets it apply again,
and now resets the async handler state in builds without TLS 1.3 too,
where a stale one would leave the SSL looking started.
wolfSSL_clear_options() reports a failed suite rebuild but not
VERSION_ERROR, which the handshake start reports, as
wolfSSL_set_options() does.

wolfSSL_CTX_set_max_proto_version() raises a fixed-version DTLS method
to DTLS 1.3, as it does fixed TLS methods. A DTLS 1.3 method lowered to
DTLS 1.2 could not be restored before. A wolfDTLSv1_2_method() CTX given
a maximum of 0 now becomes DTLS 1.3-only, as a wolfTLSv1_2_method() CTX
becomes TLS 1.3-only.

wolfSSL_CTX_set_max_proto_version() with DTLS1_2_VERSION or
DTLS1_VERSION also sets WOLFSSL_OP_NO_TLSv1_3 and WOLFSSL_OP_NO_TLSv1_2,
as the TLS versions do (WOLFSSL_OP_NO_TLSv1_3 only when DTLS 1.3 is
built, at both levels, so a maximum of 0 adds no bit that means
nothing), so an SSL keeps the CTX maximum it was created with when the
CTX maximum is raised later. Before, such an SSL stayed on DTLS 1.2
while TLSv1_3_Capable() read DTLS 1.3 from the shared method, and it
rejected the server's downgrade sentinel.
wolfSSL_CTX_get_max_proto_version() returns DTLS versions for a DTLS
CTX, so its result can be passed back to the setter. It honors the CTX
minimum, and for a fixed-version DTLS method it returns that version, or
0 when the mask or the minimum excludes it. It shares the version walk
with DtlsAllowedMinor() through the new DtlsMaskAllowedMinor(), which
takes the mask directly.

wolfSSL_CTX_set_min_proto_version() and
wolfSSL_CTX_set_max_proto_version() fail on a DTLS CTX when the mask and
the minimum leave no DTLS version, or, for a fixed-version DTLS method,
leave out the method's own version, as the TLS setters do. Before, they
returned success and every later wolfSSL_new() failed with
VERSION_ERROR. A rejected DTLS call restores the mask, the minimum, the
method version and the min/max flags, so the CTX keeps working.

wolfSSL_set_max_proto_version() refuses a TLS version on a DTLS SSL,
like the CTX setter and OpenSSL. On a fixed-version DTLS SSL it also
refuses a maximum below the SSL's version, judged by the bits the
maximum adds, so an option the application set does not make every later
maximum fail. Before, the bit was set and the SSL kept its version, so
the ClientHello went out with an empty supported_versions list.

DtlsCheckVersionMask() takes a mode (handshake start, lift, or the
server's step-down inside DoClientHello()), so every caller goes through
the same checks, and the DTLS mask helpers move above the OPENSSL_EXTRA
block so the setters can use them. DtlsMaskVersion() no longer records
VERSION_ERROR itself, so the option setters that ignore it leave nothing
on the error queue; DoClientHello() records it where it sends the alert.

When DtlsCheckVersionMask() lifts a server SSL to DTLS 1.3, it runs
InitSSL_DtlsServer() again, since the DTLS 1.3 cookie secret and
ClientHello fragmentation are only set up for the version the SSL was
created with. Without them every DTLS 1.3 handshake of a server created
under SSL_OP_NO_DTLSv1_3 and lifted later fails with COOKIE_ERROR.
InitSSL_DtlsServer() only fills in what is missing.

Tests: a server created under a CTX SSL_OP_NO_DTLSv1_3 and lifted, with
a plain and a fragmented post-quantum ClientHello, a maximum of 0 after
a handshake and a lower one after wolfSSL_clear(), a DTLS 1.2 cap lifted
by a maximum, by a maximum of 0 and by clearing the option, a maximum
lifted by clearing the option, a DTLS 1.0 cap lifted to DTLS 1.2, a lift
under a CTX maximum lowered to DTLS 1.0 that keeps DTLS 1.2, the CTX
maximum of DTLS 1.0, fixed DTLS 1.3 and DTLS 1.2 CTXs raised to DTLS
1.3, a TLS 1.2 maximum refused on a DTLS SSL, CTX and user options that
survive later maximums, a maximum and option calls while the first
ClientHello waits on WANT_WRITE, a maximum after the ClientHello was
sent, maximums after the handshake and after wolfSSL_clear(), maximums
that repeat, lift or tighten from a server SNI callback, a maximum of 0
mid-handshake, the CTX maximum read back and set again, a lower maximum
refused and an equal one or 0 accepted on a fixed DTLS 1.3 SSL with
SSL_OP_NO_DTLSv1_3 set and on a fixed DTLS 1.2 SSL, a DTLS 1.3 CTX
minimum refused under SSL_OP_NO_DTLSv1_3 and a DTLS 1.2 CTX maximum
refused under a DTLS 1.3 minimum, both leaving a working CTX, a minimum
refused on a fixed DTLS 1.3 method that masks its own version, the CTX
maximum of a fixed DTLS 1.3 method that masks its own version, a CTX
maximum raised after wolfSSL_new(), option and maximum calls after the
CTX maximum was lowered, the return value of wolfSSL_clear_options()
when no version is left, the exact error for a DTLS 1.3-only client
against a DTLS 1.2 server, and a TLS 1.3 minimum refused on a DTLS
object without DTLS 1.3.

Author: Frauschi

src/internal.c

@@ -2818,10 +2818,6 @@ void InitSSL_Method(WOLFSSL_METHOD* method, ProtocolVersion pv)
     method->downgrade  = 0;
 }
 
-#if defined(WOLFSSL_DTLS) && !defined(NO_WOLFSSL_SERVER)
-static int InitSSL_DtlsServer(WOLFSSL* ssl);
-#endif /* WOLFSSL_DTLS && !NO_WOLFSSL_SERVER */
-
 #if defined(OPENSSL_EXTRA) || defined(WOLFSSL_EITHER_SIDE) || \
     defined(WOLFSSL_EXTRA) || defined(WOLFSSL_WPAS_SMALL)
 int InitSSL_Side(WOLFSSL* ssl, word16 side)
@@ -8218,15 +8214,8 @@ byte DtlsMinDowngrade(byte minDowngrade)
 
 /* Returns the newest built DTLS minor, no newer than maxMinor, that the version
  * mask and minDowngrade allow, or 0. DTLS minors count down. */
-byte DtlsAllowedMinor(const WOLFSSL* ssl, byte maxMinor, byte minDowngrade)
+byte DtlsMaskAllowedMinor(unsigned long mask, byte maxMinor, byte minDowngrade)
 {
-#if defined(OPENSSL_EXTRA) || defined(HAVE_WEBSERVER) || \
-    defined(WOLFSSL_WPAS_SMALL)
-    unsigned long mask = ssl->options.mask;
-#else
-    unsigned long mask = 0;
-#endif
-
 #ifdef WOLFSSL_DTLS13
     if (maxMinor <= DTLSv1_3_MINOR &&
             (mask & WOLFSSL_OP_NO_TLSv1_3) == 0) {
@@ -8246,12 +8235,23 @@ byte DtlsAllowedMinor(const WOLFSSL* ssl, byte maxMinor, byte minDowngrade)
         return DTLS_MINOR;
     }
 #endif
-    (void)ssl;
     (void)mask;
     (void)maxMinor;
     (void)minDowngrade;
     return 0;
 }
+
+/* DtlsMaskAllowedMinor() with the SSL's own version mask. */
+byte DtlsAllowedMinor(const WOLFSSL* ssl, byte maxMinor, byte minDowngrade)
+{
+#if defined(OPENSSL_EXTRA) || defined(HAVE_WEBSERVER) || \
+    defined(WOLFSSL_WPAS_SMALL)
+    return DtlsMaskAllowedMinor(ssl->options.mask, maxMinor, minDowngrade);
+#else
+    (void)ssl;
+    return DtlsMaskAllowedMinor(0, maxMinor, minDowngrade);
+#endif
+}
 #endif
 
 /* This function inherits a WOLFSSL_CTX's fields into an SSL object.
@@ -9288,7 +9288,7 @@ static void InitSSL_Tls13Options(WOLFSSL* ssl, WOLFSSL_CTX* ctx)
  * @return  0 on success, and when there is nothing to do.
  * @return  A negative error code when a secret cannot be generated.
  */
-static int InitSSL_DtlsServer(WOLFSSL* ssl)
+int InitSSL_DtlsServer(WOLFSSL* ssl)
 {
     int ret;
 
@@ -44460,8 +44460,9 @@ static int AddPSKtoPreMasterSecret(WOLFSSL* ssl)
         }
 
     #ifdef WOLFSSL_DTLS
-        ret = DtlsCheckVersionMask(ssl);
+        ret = DtlsCheckVersionMask(ssl, DTLS_MASK_IN_HELLO);
         if (ret == WC_NO_ERR_TRACE(VERSION_ERROR)) {
+            WOLFSSL_ERROR_VERBOSE(VERSION_ERROR);
             /* propagate socket errors to avoid re-calling send alert */
             if (SendAlert(ssl, alert_fatal, wolfssl_alert_protocol_version)
                     == WC_NO_ERR_TRACE(SOCKET_ERROR_E)) {