build(configure): enable more sanitizers for --enable-sanitize

[tueda]

With this PR, --enable-sanitize leads to -fsanitize=address,pointer-compare,pointer-subtract,undefined,bounds-strict,float-divide-by-zero,float-cast-overflow on supported compilers.

For example, ASAN_OPTIONS=detect_invalid_pointer_pairs=2 /path/to/vorm test.frm reports the following error for the example in #866:

==563070==ERROR: AddressSanitizer: invalid-pointer-pair: 0x7aafb45f3a50 0x7bafb59e007c
    #0 0x567d4a700039 in CheckWild /home/tueda/work/form/sources/wildcard.c:1876
    #1 0x567d4a6aae05 in RunReplace /home/tueda/work/form/sources/transform.c:1624
    #2 0x567d4a6d8cc1 in RunTransform /home/tueda/work/form/sources/transform.c:782
...

[jodavies]

Maybe the CI should also run the test suite with sanitizers enabled? Adding FORMapart tests will slow the CI down a lot, removing the -w2 tests after Ana's PR is ready will speed it up again.

[coveralls]

coverage: 65.034% (-0.01%) from 65.048% — tueda:pr/build/more-sanitizers into form-dev:master

[tueda]

Maybe the CI should also run the test suite with sanitizers enabled?

This sounds good, but to replace (most of) Valgrind Memcheck's functionality, we also need to set up MemorySanitizer to detect uninitialised memory use (this is a very powerful check for FORM, as you know). This requires the dependent libraries (GMP, ..., FLINT) to be built with MemorySanitizer as well. Once this is done, we can run separate CI jobs with --enable-sanitize and --enable-sanitize=memory. Optionally, we can also add jobs with --enable-sanitize=thread.

[tueda]

...and I get many failures in the test suite if the sanitiser is enabled, particularly with invalid-pointer-pair.

I haven't checked all the failures, but what I can say for now is that I wasn't aware that pointer arithmetic outside an object's bounds (except one past the end) is undefined behaviour, even without dereferencing it. This makes some sense for non-flat address spaces, but such address arithmetic shouldn't be a problem on modern hardware.

Simple example

S x,n;
L F = 1+x;
id x^n? = n*x^n/x;
.end

==713755==ERROR: AddressSanitizer: invalid-pointer-pair: 0x6cc02b3e05bf 0x6cc02b3e05c0
    #0 0x60f77313c33e in tokenize /home/tueda/work/form/sources/token.c:623
    #1 0x60f7729eb0a4 in CompileAlgebra /home/tueda/work/form/sources/compiler.c:548
    #2 0x60f77295ccbe in CoIdExpression /home/tueda/work/form/sources/comexpr.c:1001
    #3 0x60f77295d1d9 in CoId /home/tueda/work/form/sources/comexpr.c:397
    #4 0x60f7729ccc7b in CompileStatement /home/tueda/work/form/sources/compiler.c:696
    #5 0x60f772ee0f3d in PreProcessor /home/tueda/work/form/sources/pre.c:1130
    #6 0x60f773097a3a in main /home/tueda/work/form/sources/startup.c:1820
    #7 0x70102c03cfcf in __libc_start_call_main (/home/linuxbrew/.linuxbrew/opt/glibc/lib/libc.so.6+0x29fcf)
    #8 0x70102c03d088 in __libc_start_main@@GLIBC_2.34 (/home/linuxbrew/.linuxbrew/opt/glibc/lib/libc.so.6+0x2a088)
    #9 0x60f7728dba24 in _start (/home/tueda/work/form/build/sanitize/sources/vorm+0x101ba24)

0x6cc02b3e05bf is located 1 bytes before 104-byte region [0x6cc02b3e05c0,0x6cc02b3e0628)
allocated by thread T0 here:
    #0 0x70102df2c1df in malloc (/home/linuxbrew/.linuxbrew/lib/gcc/current/libasan.so.8+0x12c1df)
    #1 0x60f773143a5d in Malloc1 /home/tueda/work/form/sources/tools.c:2263
    #2 0x60f7731514cc in DoubleBuffer /home/tueda/work/form/sources/tools.c:2918
    #3 0x60f77313645b in tokenize /home/tueda/work/form/sources/token.c:69
    #4 0x60f7729eb0a4 in CompileAlgebra /home/tueda/work/form/sources/compiler.c:548
    #5 0x60f7729544e0 in DoExpr /home/tueda/work/form/sources/comexpr.c:251
    #6 0x60f7729555d0 in CoLocal /home/tueda/work/form/sources/comexpr.c:66
    #7 0x60f7729ccc7b in CompileStatement /home/tueda/work/form/sources/compiler.c:696
    #8 0x60f772ee0f3d in PreProcessor /home/tueda/work/form/sources/pre.c:1130
    #9 0x60f773097a3a in main /home/tueda/work/form/sources/startup.c:1820
    #10 0x70102c03cfcf in __libc_start_call_main (/home/linuxbrew/.linuxbrew/opt/glibc/lib/libc.so.6+0x29fcf)

0x6cc02b3e05c0 is located 0 bytes inside of 104-byte region [0x6cc02b3e05c0,0x6cc02b3e0628)
allocated by thread T0 here:
    #0 0x70102df2c1df in malloc (/home/linuxbrew/.linuxbrew/lib/gcc/current/libasan.so.8+0x12c1df)
    #1 0x60f773143a5d in Malloc1 /home/tueda/work/form/sources/tools.c:2263
    #2 0x60f7731514cc in DoubleBuffer /home/tueda/work/form/sources/tools.c:2918
    #3 0x60f77313645b in tokenize /home/tueda/work/form/sources/token.c:69
    #4 0x60f7729eb0a4 in CompileAlgebra /home/tueda/work/form/sources/compiler.c:548
    #5 0x60f7729544e0 in DoExpr /home/tueda/work/form/sources/comexpr.c:251
    #6 0x60f7729555d0 in CoLocal /home/tueda/work/form/sources/comexpr.c:66
    #7 0x60f7729ccc7b in CompileStatement /home/tueda/work/form/sources/compiler.c:696
    #8 0x60f772ee0f3d in PreProcessor /home/tueda/work/form/sources/pre.c:1130
    #9 0x60f773097a3a in main /home/tueda/work/form/sources/startup.c:1820
    #10 0x70102c03cfcf in __libc_start_call_main (/home/linuxbrew/.linuxbrew/opt/glibc/lib/libc.so.6+0x29fcf)

SUMMARY: AddressSanitizer: invalid-pointer-pair /home/tueda/work/form/sources/token.c:623 in tokenize

Undefined behaviour here (according to C23 draft N3096 §6.5.6 ¶9):

form/sources/token.c

Line 623 in 82407dd

while ( out >= AC.tokens ) { *tt-- = *out--; }

The compiler is then free to do anything, summon nasal demons, or delete all your files.

[jodavies]

Maybe the CI should also run the test suite with sanitizers enabled?

This sounds good, but to replace (most of) Valgrind Memcheck's functionality, we also need to set up MemorySanitizer to detect uninitialised memory use (this is a very powerful check for FORM, as you know). This requires the dependent libraries (GMP, ..., FLINT) to be built with MemorySanitizer as well. Once this is done, we can run separate CI jobs with --enable-sanitize and --enable-sanitize=memory. Optionally, we can also add jobs with --enable-sanitize=thread.

At least for now, I would suggest to keep the valgrind tests in addition.

[jodavies]

So if I understand correctly, that needs to be something like while ( out > AC.tokens ) { *tt-- = *out--; } *tt-- = *out;, and presumably similar for the line after as well?