Feb 25, 2018

A small sender CPU can cause slow rsync tranfers

.. when (accidentially) resuming large file transfers.

I was transfering an .iso image from a weakish Atom N450 CPU to a bigger host and noticed a ~75% decrease in transfer rate whenever I stopped the transfer and continued. First I tested for disk/network I/O upper limits with dd/netcat. I got curious and replicated the same behaviour between two Azure Standard A1 machines - the smallest instance type available on the platform. Verifying on more powerful systems, the behaviour vanished.

Seeking an answer, I found somebody else posting a similar question. Reading rsyncs architecture Documentation and judging from the behaviour, this quote explains it

Generating the rolling checksums and searching for matches in the checksum set sent by the generator require a good deal of CPU power. Of all the rsync processes it is the sender that is the most CPU intensive.

And the advice in the superuser answer:

Rsync does a delta-transfer by default if it sees a partial file on the receiver (leftover from the scp test?). [...] Explicitly telling rsync to not to use partial transfers (-W|--whole-file) will put transfer rates in the same ballpark again. If needed, --append|--append-verify is another way to resume an interrupted file transfer without causing a delta-transfer. get -a in sftp can resume a download too.

I wondered if raising the --block-size=BLOCKSIZE value would cause less computation on big files, but it got an upper limit in a recent release. Using an older version with multiple-megabyte blocksizes didn't made a difference though.

because graphs..

Logs

Send file with netcat for reference

user@remote:~$ nc -q 0 rmaschine 60000 < bionic-desktop-amd64.iso

listen on port for file, ~85 MB/s

~$ nc -l -p 60000 | time pv -s 1762754560 -r -p -b > bionic-desktop-amd64.iso

1,64GiB [84,6MiB/s] [==========================================================================>] 100%

0.53user 5.20system 0:19.86elapsed 28%CPU (0avgtext+0avgdata 2184maxresident)k
0inputs+3442888outputs (0major+93minor)pagefaults 0swaps

Test the impact of a computationally "cheap" ssh cipher, but it barely made a difference, ~22 MB/s

~$ scp -c arcfour -o Compression=no remote:/home/user/bionic-desktop-amd64.iso .
bionic-desktop-amd64.iso                                            100% 1681MB  22.4MB/s   01:15

transfer the whole file without interruption, ~21 MB/s

~$ rsync --info=progress2 -e ssh remote:/home/user/bionic-desktop-amd64.iso .
  1,762,754,560 100%   20.89MB/s    0:01:20 (xfr#1, to-chk=0/1)

partial transfer, 7.5MB/s is an aggregate value, transfer rate for the chunk is ~4.2 MB/s

~$ rsync --partial --inplace --stats --info=progress2 -e "ssh -o Compression=no -c arcfour -x" remote:/home/user/bionic-desktop-amd64.iso .
  1,762,754,560 100%    7.45MB/s    0:03:45 (xfr#1, to-chk=0/1)

Total transferred file size: 1,762,754,560 bytes
Literal data: 1,028,723,936 bytes
Matched data: 734,030,624 bytes

sent 189,730 bytes  received 1,029,083,574 bytes  4,408,022.72 bytes/sec