Change CLI to employ multithreading by default

[daniellerozenblit]

Summary

This PR modifies ZSTDCLI_NBTHREADS_DEFAULT so that multithreading is used by default when the library is compiled with multithreading support.

We use a conservative value of max(1, min(4, nbCores/4)) worker threads for this default value.

Testing

I benchmarked on an Intel(R) Xeon(R) Gold 6138 CPU @ 2.00GHz machine with 80 logical cores. I observed that the default behavior is now similar to the behavior when explicitly doing multi-threaded compression using the -T flag.

~/zstd$ ./zstd -b3 ~/silesia/silesia/*
 3# 12 files         : 211938580 ->  66216521 (x3.201),  358.2 MB/s,  891.2 MB/s
~/zstd$ ./zstd -b3 -T4 ~/silesia/silesia/*
 3# 12 files         : 211938580 ->  66216521 (x3.201),  323.2 MB/s,  885.1 MB/s
~/zstd$ ./zstd -b3 -T2 ~/silesia/silesia/*
 3# 12 files         : 211938580 ->  66216521 (x3.201),  266.1 MB/s,  885.0 MB/s
~/zstd$ ./zstd -b3 -T1 ~/silesia/silesia/*
 3# 12 files         : 211938580 ->  66137723 (x3.205),  189.6 MB/s,  863.7 MB/s

[Cyan4973]

~/zstd$ ./zstd -b3 ~/silesia/silesia/*
 3# 12 files         : 211938580 ->  66216521 (x3.201),  358.2 MB/s,  891.2 MB/s
~/zstd$ ./zstd -b3 -T4 ~/silesia/silesia/*
 3# 12 files         : 211938580 ->  66216521 (x3.201),  323.2 MB/s,  885.1 MB/s

The difference between these 2 measurements is > 10%, which is quite substantial if they are expected to be equal.
Is it repeatable, or is it a one-off ?

Also:
I would expect -vv to specify clearly the nb of threads used.

[Cyan4973]

Also as a follow-up,
it would be good to clarify the (potential) impact on the benchmark mode (-b).

[daniellerozenblit]

~/zstd$ ./zstd -b3 ~/silesia/silesia/*
 3# 12 files         : 211938580 ->  66216521 (x3.201),  358.2 MB/s,  891.2 MB/s
~/zstd$ ./zstd -b3 -T4 ~/silesia/silesia/*
 3# 12 files         : 211938580 ->  66216521 (x3.201),  323.2 MB/s,  885.1 MB/s

The difference between these 2 measurements is > 10%, which is quite substantial if they are expected to be equal. Is it repeatable, or is it a one-off ?

Also: I would expect -vv to specify clearly the nb of threads used.

I re-did this experiment, compressing silesia with the default number threads (which on my machine evaluates to 4) and then explicitly passing -T4 and got more consistent results.

Overall it seems that there is more noise in the benchmarks with multithreading, but the behavior looks overall as expected.

# -T4
 3# 12 files         : 211938580 ->  66216521 (x3.201),  310.7 MB/s,  861.7 MB/s
 3# 12 files         : 211938580 ->  66216521 (x3.201),  306.1 MB/s,  862.6 MB/s
 3# 12 files         : 211938580 ->  66216521 (x3.201),  318.3 MB/s,  868.3 MB/s

# default
 3# 12 files         : 211938580 ->  66216521 (x3.201),  316.1 MB/s,  871.2 MB/s
 3# 12 files         : 211938580 ->  66216521 (x3.201),  315.3 MB/s,  867.9 MB/s
 3# 12 files         : 211938580 ->  66216521 (x3.201),  307.5 MB/s,  866.8 MB/s

[Cyan4973]

Yes, level 3 seems better