Rsync 3.2.3 vs rsync 3.1.2 benchmarks

Folks may know I’m an avid user and tester of zstd compression and I’ve now integrated xxHash into my backup script routines. So when I totally missed the memo that rsync 3.2 added support for faster checksum hashing via xxHash and added lz4 and zstd compression support, I had to build a custom rsync 3.2.3 RPM for CentOS 7 and benchmark compared it to native CentOS 7’s rsync 3.1.2 and the resulted gains with rsync 3.2.3 are awesome https://community.centminmod.com/threads/custom-rsync-3-2-3-rpm-builds-with-xxhash-zstd-support-for-centos-7.20768/ !

Knowing how many folks have multiple servers here and that they need to transfer data amongst those servers, I thought this would be of interest. So enjoy! :smiley:

rsync 3.2.3

rsync --version
rsync  version 3.2.3  protocol version 31
Copyright (C) 1996-2020 by Andrew Tridgell, Wayne Davison, and others.
Web site: https://rsync.samba.org/
Capabilities:
    64-bit files, 64-bit inums, 64-bit timestamps, 64-bit long ints,
    socketpairs, hardlinks, hardlink-specials, symlinks, IPv6, atimes,
    batchfiles, inplace, append, ACLs, xattrs, optional protect-args, iconv,
    symtimes, prealloc, stop-at, no crtimes
Optimizations:
    SIMD, asm, openssl-crypto
Checksum list:
    xxh128 xxh3 xxh64 (xxhash) md5 md4 none
Compress list:
    zstd lz4 zlibx zlib none

rsync comes with ABSOLUTELY NO WARRANTY.  This is free software, and you
are welcome to redistribute it under certain conditions.  See the GNU
General Public Licence for details.

rsync 3.1.2

rsync --version
rsync  version 3.1.2  protocol version 31
Copyright (C) 1996-2015 by Andrew Tridgell, Wayne Davison, and others.
Web site: http://rsync.samba.org/
Capabilities:
    64-bit files, 64-bit inums, 64-bit timestamps, 64-bit long ints,
    socketpairs, hardlinks, symlinks, IPv6, batchfiles, inplace,
    append, ACLs, xattrs, iconv, symtimes, prealloc

rsync comes with ABSOLUTELY NO WARRANTY.  This is free software, and you
are welcome to redistribute it under certain conditions.  See the GNU
General Public Licence for details.

FYI, I plan to add Rsync 3.2+ support to my Centmin Mod LEMP stack after doing more testing too :sunglasses:

9 Likes

Did more benchmarks without pre-compressed log files in the mix due to rsync 3.2.3 zlib bug. Also added zstd fast negative compression levels up to -8000 yes rsync zstd has up to -131072 to 22 compression levels https://community.centminmod.com/threads/custom-rsync-3-2-3-rpm-builds-with-xxhash-zstd-support-for-centos-7.20768/#post-87375

At rsync 3.2.3 zstd -8000, transfer speed is almost as fast as no compressed rsync 3.2.3 and is faster than no compressed rsync 3.1.2 - 277MB/s vs 284MB/s ! Compared to rsync 3.1.2 compressed at 6.9MB/s !

/usr/bin/rsync -avzi --stats --log-file=/home/rsync-benchmarks/rsyncbench-rsync-native-compressed-051220-224524.log /home/rsync-test-no-precomp/ /home/rsync-test-no-precomp-dst/
[rsync 3.1.2 native compressed] files: 2,842 reg: 2,841, dir: 1
[rsync 3.1.2 native compressed] total bytes: 510556971 sent bytes: 55648190 (6553198.59 per second)
[rsync 3.1.2 native compressed] transfer speed (MB/s): 6.8922 speedup: 9.17
[rsync 3.1.2 native compressed] real: 7.70s user: 7.66s sys: 0.50s cpu: 106% maxmem: 2332 KB cswaits: 5855

/usr/bin/rsync -avi --stats --log-file=/home/rsync-benchmarks/rsyncbench-rsync-native-no-compress-051220-224524.log /home/rsync-test-no-precomp/ /home/rsync-test-no-precomp-dst/
[rsync 3.1.2 native no-compress] files: 2,842 reg: 2,841, dir: 1
[rsync 3.1.2 native no-compress] total bytes: 510556971 sent bytes: 510977977 (204412790.00 per second)
[rsync 3.1.2 native no-compress] transfer speed (MB/s): 213.7309 speedup: 1.00
[rsync 3.1.2 native no-compress] real: 2.28s user: 1.66s sys: 0.79s cpu: 107% maxmem: 1988 KB cswaits: 16267
/usr/local/bin/rsync -avzi --stats --cc xxh128 --zc zstd --zl -8000 --debug=NSTR --log-file=/home/rsync-benchmarks/rsyncbench-xxh128-zstd-051220-233002.log /home/rsync-test-no-precomp/ /home/rsync-test-no-precomp-dst/
[rsync 3.2.3 xxh128-zstd--8000 files: 2,842 reg: 2,841, dir: 1
[rsync 3.2.3 xxh128-zstd--8000 total bytes: 510556971 sent bytes: 502664954 (335145968.00 per second)
[rsync 3.2.3 xxh128-zstd--8000 transfer speed (MB/s): 277.0974 speedup: 1.02
[rsync 3.2.3 xxh128-zstd--8000 real: 1.73s user: 0.34s sys: 0.62s cpu: 56% maxmem: 3888 KB cswaits: 13102
/usr/local/bin/rsync -avzi --stats --cc xxh128 --zc none --debug=NSTR --log-file=/home/rsync-benchmarks/rsyncbench-xxh128-none-051220-233220.log /home/rsync-test-no-precomp/ /home/rsync-test-no-precomp-dst/
[rsync 3.2.3 xxh128-none] files: 2,842 reg: 2,841, dir: 1
[rsync 3.2.3 xxh128-none] total bytes: 510556971 sent bytes: 510977988 (340687990.67 per second)
[rsync 3.2.3 xxh128-none] transfer speed (MB/s): 284.9746 speedup: 1.00
[rsync 3.2.3 xxh128-none] real: 1.71s user: 0.18s sys: 0.69s cpu: 51% maxmem: 2744 KB cswaits: 19841