Subject: Re: Debian server for backups of Windows clients

From: Daniel Bareiro <[email-masked]>
Date: Fri, 9 Sep 2016 16:43:35 -0300

Hi, David.

Thanks for your reply.

On 09/08/16 22:57, David Christensen wrote:

>> As you can see, the transfer was over than 3 GB and it were not hung. I
>> did several tests and all were without problems.
>>
>> I wonder if in the mentioned episodes of hangs you remember whether the
>> transferred volume was higher or lower than in this case (or it hung
>> randomly).

> Script it and run it every night for a week.  If it works every time,
> try again for 30 days.  Then 90.  Then 365.

Yes, I have to start testing on a daily basis. Anyway, the mentioned
test results were quite satisfactory.

>> As a side note, the larger file (disk01.img) took more than 40 minutes
>> to be transferred. So the rsync was running quite some time without
>> hanging. While it does not have to do with the topic of this thread, in
>> rsync progress data we can see that the average transfer rate was 10
>> Mbps. I guess it will have to do with that I'm going through a wireless
>> network. In this testing the Debian computer is a notebook connected to
>> the wireless router and the KVM Windows is on the wired network. May it
>> be so large the decrease in transfer speed? The wireless router is
>> TPLink WDR3600 with OpenWRT.

> My laptop has 802.11 a/b/g WiFi and Fast Ethernet.  Wireless data
> transfers are slow (~50 Mbps).  Wired is twice as fast (100 Mbps); still
> slow.  Newer WiFi (n, ac) should be faster, but only the newest WiFi
> hardware can match or beat Gigabit.

I think it is reasonable to expect that the wireless transfer rate is
lower than the one obtained in a wired network. But there is a big
difference compared to the ~50 Mpbs you mentioned. The peak obtained
with rsync was 10 Mbps. Maybe the best is to take a metric with iperf,
what do you think?

> For the initial full backup, I have found that scp is faster than rsync.

It is likely, since rsync adds control information used by rsync
algorithm to track the synchronization.

> When I know that I've added a bunch of new and/or large files on the
> sender, I sometimes try the rsync 'whole-file' option.  As I haven't
> benchmarked it, I don't know if/when it is helping.
>
> My biggest problem with rsync is when I reorganize file/ directory trees
> on my file server; especially big stuff -- raw video, movies, disk
> images, ISO images, etc..  I have yet to figure out an rsync incantation
> that does the corresponding moves on the destination, rather than
> mindlessly copying and deleting 100's of GB.  I have often considered
> writing an rsync prelude script for just this case.

If you make a move of files, but always within the same root filesystem
provided to rsync, you might want to consider using --delete for get an
identical image in the source and destination.

Kind regards,
Daniel

---

From: David Christensen <[email-masked]>
Date: Fri, 9 Sep 2016 20:43:44 -0700

> with rsync was 10 Mbps. Maybe the best is to take a metric with iperf,

See the benchmark I just posted for 802.11g WiFi --  dm-crypt -> scp ->
dm-crypt, all without AES-NI --  110341671 bits/second.  Yuck.

>> on my file server; especially big stuff ... I have yet to figure out an rsync incantation
>> that does the corresponding moves on the destination ...
> 

--delete is a different idea.  I'm thinking -y/--fuzzy.

David

---

From: Celejar <[email-masked]>
Date: Sat, 10 Sep 2016 22:23:11 -0400

On Fri, 9 Sep 2016 20:43:44 -0700

> dm-crypt, all without AES-NI --  110341671 bits/second.  Yuck.

FTR: there seem to be more typos / here. The actual figure should be
11034157.6344 bits/second.

Celejar

---

From: David Christensen <[email-masked]>
Date: Sat, 10 Sep 2016 20:00:02 -0700

> 11034157.6344 bits/second.

Yes, let's whip those typos out of this dead horse some more:

On 09/09/2016 08:36 PM, David Christensen wrote:
> Benchmarking using WiFi (48 Mb/s):
>
>     2016-09-09 20:18:51 dpchrist@t7400 ~
>     $ time dd if=/dev/urandom of=urandom.100M bs=1M count=100
>     100+0 records in
>     100+0 records out
>     104857600 bytes (105 MB) copied, 12.6709 s, 8.3 MB/s
>
>     real	0m12.703s
>     user	0m0.000s
>     sys	0m12.481s
>
>     2016-09-09 20:19:32 dpchrist@t7400 ~
>     $ time scp -p urandom.100M samba:.
>     urandom.100M
>
>
>                           100%  100MB   1.5MB/s   01:08
>
>     real	1m16.023s
>     user	0m4.548s
>     sys	0m0.744s

2016-09-10 19:53:48 dpchrist@t7400 ~
$ perl -e 'print 104857600*8/76.023, "\n"'
11034302.7767912

On 09/09/2016 08:36 PM, David Christensen wrote:
> Testing again using Fast Ethernet (100 Mb/s):
>
>     2016-09-09 20:29:54 dpchrist@t7400 ~
>     $ time scp -p urandom.100M samba:.
>     urandom.100M
>
>
>                           100%  100MB   2.4MB/s   00:42
>
>     real	0m43.377s
>     user	0m4.476s
>     sys	0m0.876s

2016-09-10 19:54:43 dpchrist@t7400 ~
$ perl -e 'print 104857600*8/43.377, "\n"'
19338838.5549946

David

---

From: Celejar <[email-masked]>
Date: Fri, 9 Sep 2016 14:51:42 -0400

On Tue, 9 Aug 2016 18:57:02 -0700
David Christensen <[email-masked]> wrote:

...

> hardware can match or beat Gigabit.

You get ~50Mbps over a/b/g? 54Mbps is the theoretical maximum, and
everything I've read says that 20-24Mbps is the real-world maximum.

Celejar

---

From: David Christensen <[email-masked]>
Date: Fri, 9 Sep 2016 20:36:39 -0700

real	0m12.703s
    user	0m0.000s
    sys	0m12.481s

                          100%  100MB   1.5MB/s   01:08

    real	1m16.023s
    user	0m4.548s
    sys	0m0.744s

So, 1048576900 bytes * 8 bits / byte / 76.024 seconds

= 110341671 bits/second

                          100%  100MB   2.4MB/s   00:42

    real	0m43.377s
    user	0m4.476s
    sys	0m0.876s

So, 1048576900 bytes * 8 bits / byte / 43.377 seconds

= 193388552. bits/second

Wow.  Even worse than I was expecting...

David

---

From: Celejar <[email-masked]>
Date: Sat, 10 Sep 2016 22:22:02 -0400

On Fri, 9 Sep 2016 20:36:39 -0700

...

> So, 1048576900 bytes * 8 bits / byte / 76.024 seconds

So assuming that '9' is a typo, as per another message of yours in this
thread, your actual throughput is more like 11Mpbs, correct?

Celejar

---

From: David Wright <[email-masked]>
Date: Fri, 9 Sep 2016 23:14:30 -0500

> So, 1048576900 bytes * 8 bits / byte / 76.024 seconds
             ↑

What's this 9?

Cheers,
David.

---

From: David Christensen <[email-masked]>
Date: Fri, 9 Sep 2016 23:11:56 -0700

>> So, 1048576900 bytes * 8 bits / byte / 76.024 seconds

A typographical error.

104857600 bytes * 8 bits/byte / 76.024 seconds

= 11034158 bits/seconds

David

---

From: "Neal P. Murphy" <[email-masked]>
Date: Sat, 10 Sep 2016 01:22:45 -0400

On Fri, 9 Sep 2016 23:14:30 -0500

> > So, 1048576900 bytes * 8 bits / byte / 76.024 seconds

Assuming the talk is about transfer rates over the medium, not something like pre-compression data rates (which might be called 'marketing-speak').

Good eye! I was going to say it's not possible to get 110Mb/s over 802.11g; 40-50 is closer tothe best I get. And 193Mb/s over 100Mb/s ethernet is right out; best I've ever managed is maybe 97Mb/s, and 92-95 is more typical. 11,034,157Mb/s on W/L and 19,338,838Mb/s on wired is *much* more believable.

Unless one has a very fast multicore CPU with hardware crypto assistance, very fast RAM and the data to be transferred cached in RAM, one will probably never saturate a fastE or gigE link where one end must decrypt the data from disk/cache then encrypt the data to scp, and the other end must decrypt the data from scp then encrypt the data to disk. Even simple compression slows transfer down far too much.

Now if one had many CPUs, hacked scp to open as many sockets and thread/child procs as there are CPUs, and had each thread work on a small-ish block of data at a time, one *might* be able to speed up the tranfser.

---

From: Dan Ritter <[email-masked]>
Date: Sat, 10 Sep 2016 08:41:53 -0400

> Unless one has a very fast multicore CPU with hardware crypto assistance, very fast RAM and the data to be transferred cached in RAM, one will probably never saturate a fastE or gigE link where one…

SSDs can routinely read 400-600 MB/s. No need to have everything
cached in RAM.

In 2010, the first generation of i5 CPUs with hardware support for AES
could encrypt at about 15 MB/s, more than filling a 100 Mb/s pipe.

Here's a table of recent CPUs with AES support, running with
OpenSSL/LibreSSL. https://calomel.org/aesni_ssl_performance.html

It's in megabytes per second, so assume 1000/8 = 250 MB/s is the
bandwidth of a gigabit ethernet NIC. Anything which can do 2x
that can approach encrypting/decrypting from SSD, then
decrypting/encrypting over an SSH connection.

There are a lot of 500s and above on that chart.

And that's per-core, so even the 250+ CPUs can fill a gig-e pipe
while reading from SSD.

Nor are they monstrously expensive: an AMD FX-6300 is $90, a
motherboard for it could be another $90, and you can get a
decent SSD for $100 these days. A $400 desktop can be put
together that can saturate a gig-E link with encrypted traffic
from an encrypted disk.

Truly we live in marvelous times.

-dsr-

---

From: [email-masked]
Date: Sat, 10 Sep 2016 10:26:15 -0400

On Saturday, September 10, 2016 08:41:53 AM Dan Ritter wrote:
> bandwidth of a gigabit ethernet NIC.

Sorry, I tend to pick at nits, but, for the record, 1000/8 is 125 Mb/s.  It 
doesn't (really) change your conclusions.

regards,
Randy Kramer

---

From: Gene Heskett <[email-masked]>
Date: Sat, 10 Sep 2016 10:40:26 -0400

On Saturday 10 September 2016 10:26:15 [email-masked] wrote:

> Sorry, I tend to pick at nits, but, for the record, 1000/8 is 125
> Mb/s.  It doesn't (really) change your conclusions.
>
> regards,
> Randy Kramer

You make an assumption many folks do, but theres a start bit and a stop 
bit so the math is more like 1000/10=100 Mb/s.

Cheers, Gene Heskett
-- 
"There are four boxes to be used in defense of liberty:
 soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author)
Genes Web page <http://geneslinuxbox.net:6309/gene>

---

From: [email-masked]
Date: Sat, 10 Sep 2016 10:53:20 -0400

> bit so the math is more like 1000/10=100 Mb/s.

Well, 1000/8 is still 125 ;-) but I wouldn't have written back just to say 
that.  Isn't it the case that there is something less than 1 start and 1 stop 
for every byte--maybe like 1 stop bit for every several bytes?  (I am just 
(slightly) curious.)

And, iirc, there are variations (which may be obsolete--I seem to remember one 
protocol that had either 2 start or 2 stop bits?

regards,
Randy Kramer

---

From: "Neal P. Murphy" <[email-masked]>
Date: Sun, 11 Sep 2016 01:43:01 -0400

> protocol that had either 2 start or 2 stop bits?

Start/stop bits apply to async TIA-232.

Speaking very generally, 100Mb/s Ethernet actually operates at 125Mb/s; that includes the LAPB-like protocol that actually transmits the packets. All the layer 1 overhead goes in that extra 25Mb/s.

So, more correctly, you have data + TCP + IP overhead + L2 overhead: around 3% for full packets, higher for smaller packets. This is why 100Mb/s ethernet saturates at around 92%-95% *data* transmission. The rest is protocol overhead and delays (probably akin to RR and RNR).

---

From: Gene Heskett <[email-masked]>
Date: Sat, 10 Sep 2016 13:31:46 -0400

On Saturday 10 September 2016 10:53:20 [email-masked] wrote:

> > You make an assumption many folks do, but theres a start bit and a
> > stop bit so the math is more like 1000/10=100 Mb/s.
>
> Well, 1000/8 is still 125 ;-) but I wouldn't have written back just to
> say that.  Isn't it the case that there is something less than 1 start
> and 1 stop for every byte--maybe like 1 stop bit for every several
> bytes?  (I am just (slightly) curious.)
>
> And, iirc, there are variations (which may be obsolete--I seem to
> remember one protocol that had either 2 start or 2 stop bits?

Yes, still in use in some legacy stuffs.

There may be some inroads into the 10 bits per byte, but tcp is so old I 
doubt that synchronization portion took a hit.  Thats what it is, is 
keeping everything in synch. Even USB has that same data format. Sata 
for disks, being much newer, may have abandoned that, particularly for 
the disks whose native format is a 4096 byte sector. I've also found 
sata cabling is about 1000% flakier, requiring more frequent 
replacements.

Cheers, Gene Heskett

---

From: David Christensen <[email-masked]>
Date: Sat, 10 Sep 2016 08:33:46 -0700

> protocol that had either 2 start or 2 stop bits?

I remember start/stop bits from RS-232/485, but Gigabit Ethernet
signaling is more advanced:

    https://en.wikipedia.org/wiki/Gigabit_Ethernet

David

---

From: Daniel Bareiro <[email-masked]>
Date: Fri, 9 Sep 2016 16:46:35 -0300

Hi, Celejar.

> everything I've read says that 20-24Mbps is the real-world maximum.

Still, 20-24 Mbps is more than 10 Mpbs I was seeing with rsync. There
could be a bottleneck somewhere?

Kind regards,
Daniel

---

From: deloptes <[email-masked]>
Date: Sat, 10 Sep 2016 00:06:48 +0200

> could be a bottleneck somewhere?

In my case it was the IO on the disk - I couldn't do more than 12Mbps even
on wired connection, because I have encrypted disk ... it took me a while
to understand why though.

---

From: Daniel Bareiro <[email-masked]>
Date: Sun, 11 Sep 2016 09:09:20 -0300

Hi, deloptes.

> to understand why though.

This is an interesting fact. Because 'orion' (the notebook used in the
mentioned test) also has an encrypted disk. In the test, the notebook
was pulling the files on the Windows VM on the wired network.

root@orion:~# dmsetup ls --target crypt
sda5_crypt      (254, 0)

root@orion:~# cryptsetup luksDump /dev/sda5 | grep Version -A3
Version:        1
Cipher name:    aes
Cipher mode:    xts-plain64
Hash spec:      sha1

viper@orion:~$ lsblk --fs
NAME             FSTYPE LABEL UUID MOUNTPOINT
sda
├─sda1                             /boot
├─sda2
└─sda5
  └─sda5_crypt
    ├─main-swap                    [SWAP]
    ├─main-root                    /
    └─main-datos                   /datos
sr0

I did not think this could affect so strongly in the network transfer.

Kind regards,
Daniel

---

From: Celejar <[email-masked]>
Date: Fri, 9 Sep 2016 17:18:16 -0400

On Fri, 9 Sep 2016 16:46:35 -0300

> could be a bottleneck somewhere?

As per your own suggestion in another message, definitely benchmark
with iperf to see if that's better. And as we discussed in another
thread some time ago, (especially) if you're using wireless, benchmark
throughput in *both* directions, since the transmitter (or receiver) may
be better on one machine than on another.

Celejar

---

From: Daniel Bareiro <[email-masked]>
Date: Sun, 11 Sep 2016 09:29:46 -0300

Hi, Celejar

> with iperf to see if that's better.

Yes, it can be. I was thinking about what I said in a previous message
about the control information added by rsync on the packets sent.

I think this would be important only if we focus on the performance
(number of bits of data sent / total number of bits sent). In this case,
the focus is the transfer rate, for which the amount of control bits
used would be irrelevant since I think we need to know how many bits per
second we are getting, regardless of the utility have those bits.

> And as we discussed in another thread some time ago, (especially) if 
> you're using wireless, benchmark throughput in *both* directions,
> since the transmitter (or receiver) may be better on one machine than
> on another.

Interesting sidelight. Thanks for sharing.

Kind regards,
Daniel