obs-outputs: upgrade mbedtls dependency to v4

[dsimidzija]

Description

The official mbedtls migration guide was used for the following:

• update cmake target modules and versions (mbedcrypto => tfpsacrypto)
• remove deprecated entropy configuration (ctr_drbg + entropy)
• migrate MD5 and HMAC to the new PSA API

The official migration guide:

https://github.com/Mbed-TLS/mbedtls/blob/development/docs/4.0-migration-guide.md

I've done the best I can to cover the guide material, but there are some concerns:

• I've never used cmake, so I don't know if any further changes are needed there, for example, there is a message in FindMbedTLS.cmake which states:

  Use of the custom CMake find module for MbedTLS versions >= 3.6.0 is not supported - build errors might occur!

  Does this mean that this file should be removed, as it would now be obsolete?

• I've added calls to psa_crypto_init() for both HMAC and MD5 macros, which should be called before any crypto functions according to the new docs. Should this be moved to obs_module_load, as a counterpart to mbedtls_psa_crypto_free? My gut tells me yes, but I wanted to double check what is the convention in OBS.

• Old code doesn't seem to have any error handling, so I've replicated that, but that makes me uneasy, the new PSA API can fail on each step (you can see it in the hmac_demo.c with PSA_CHECK). However, I'm not sure how to implement that cleanly with the way these macros are used. I guess I'm just asking for confirmation that this is okay as-is.

Motivation and Context

This aims to close #13601 .

How Has This Been Tested?

I've tested RTMP output with a local test server, with and without TLS (self-signed certificate). While I was able to establish a handshake in both versions, I was unable to view the stream when using RTMPS, but I suspect this might be due to the test server malfunctioning, as RTMP ingestion seemed to work without issues. Unfortunately I don't have any other place to test this on a live/prod server.

Additionally, I only have a Linux machine, so I have no means to check if this code works on other platforms.

Types of changes

• Bug fix (non-breaking change which fixes an issue)
• Breaking change (fix or feature that would cause existing functionality to change) - breaking in the sense of backward compatibility with mbedtls v2/v3

Checklist:

• I have read the contributing document.
• My code has been run through clang-format.
• My code follows the project's style guidelines
• My code is not on the master branch.
• My code has been tested.
• All commit messages are properly formatted and commits squashed where appropriate.
• I have included updates to all appropriate documentation.

[tytan652]

Keeping compat with v3 at build time is required.

[dsimidzija]

I was afraid that might be a requirement. 😀

I don't think that will be a problem to accomplish with the code itself (I'm guessing it's enough to just rely on MbedTLS_VERSION_MAJOR). But I might need some help with the cmake stuff, does this mean that FindMbedTLS.cmake shouldn't be changed at all, provided it won't ever be used for v4?

[tytan652]

You should rebase, we recently drop the custom finder (there was already things in the work to remove it).

Compatibility with MbedTLS 3 is required, we guarded to exclude 4 to reduce the number of report about not supporting 4 and having hopefully a better CMake error than include and linking issue.

[dsimidzija]

While I test the changes locally, a question regarding cmake updates, since it looks like this is not supported:

find_package(MbedTLS 3...<5 REQUIRED)

I came up with this:

find_package(MbedTLS REQUIRED)
if(NOT DEFINED MbedTLS_VERSION_MAJOR OR MbedTLS_VERSION_MAJOR VERSION_LESS "3")
  message(
    FATAL_ERROR
    "MbedTLS >= 3 is required, but found "
    "${MbedTLS_VERSION}"
  )
else()
  add_definitions(-DMbedTLS_VERSION_MAJOR=${MbedTLS_VERSION_MAJOR})
endif()

The question is, do we want this check only on obs-outputs CMakeLists.txt, or also in the feature-whatsnew.cmake and os-windows.cmake?

Also, is there a better way to do this? If my understanding is correct, this doesn't guarantee that you'll link against v4 if both v4 and v3 are both installed.

[tytan652]

Don't over-engineer just strip the version range for now that were added to block version 4. (The codebase still support version 2 anyway.)

[dsimidzija]

My bad, I was under the impression that we're ditching v2. I've pushed the changes, and tested locally with mbedtls v3 and v4, LGTM but it might be a good idea for someone else to test it as well.

[RytoEX]

Don't over-engineer just strip the version range for now that were added to block version 4. (The codebase still support version 2 anyway.)

Perhaps out of scope for this PR, but do we actually build against MbedTLS 2.x in any of our first-party builds anymore? As far as I can tell, we only built against MbedTLS 2.x for Ubuntu 24.04, which we have dropped. Ubuntu 26.04 is now on MbedTLS 3.6.5.

[RytoEX]

While I test the changes locally, a question regarding cmake updates, since it looks like this is not supported:

find_package(MbedTLS 3...<5 REQUIRED)

Why is this not supported?

[dsimidzija]

Why is this not supported?

According to this SO answer, it looks like cmake is doing some semantic validation with the default finder. I'm really not familiar with cmake, so I wasn't able to find a way to turn this off, but maybe it's irrelevant in this case if v2 is not being removed?

That is, until mbedtls v5 gets released. 😁