Fix VDF prover failures on Windows

[wjblanke]

The root issue is that labs() was being used instead of llabs(). labs() is undefined (on windows only apparently) if the value tested is too large. This was causing the Pullmark reducer to misbehave and reduction not to happen correctly in rare cases. That resulted in too large values and sometimes buffer overflows causing heap corruption in some cases. In the process fixed some other things the compiler was not happy about but doesn't really affect the behavior.

[arvidn]

the memory leak is in a test, so not a big deal probably.
It might deceive a future reader of the code though

[arvidn]

technically it should be delete[] ba.data;, but I was also under the impression that we had a function to free a ByteArray object. mostly just to ensure we use the correct heap in case this is exported over a DLL boundary.
A minor issue though.