Morgan Tocker ([info]mtocker) wrote,
@ 2009-05-28 09:35:00
Previous Entry  Add to memories!  Tell a Friend  Next Entry
Entry tags:mysql

Efficient way to copy large amounts of data?
Dear Lazyweb,

Yesterday I tried to Rsync a MySQL data directory from serverA to serverB on the same network. I thought that if out of a few hunded gigabytes maybe 2% changed, this should work, right? Wrong. Rsync is designed to minimize bandwidth, so in my case it was *much* quicker to wipe the data and start again (I feel this is something I should have known earlier, but it doesn't hurt to try and share your mistakes).

Which gets me thinking - it doesn't have to be this way. Bit-torrent works in a similar way to Rsync, but it's certainly not network efficient. Are there any projects similar to Rsync that are using network-hungry algorithms to try and make sure that two directories are in sync with the goal just being as fast as possible?


(15 comments) - (Post a new comment)

Compression
[info]ronaldbradford
2009-05-28 04:52 pm UTC (link)
A long time ago, I remember that disabling compression within rsync helped for a given situation.

A look at man gives

-z, --compress compress file data during the transfer
-a, --archive archive mode; same as -rlptgoD (no -H)

Doesn't appear enabled by default.
It was some time ago, so perhaps the client was using -z specifically.

(Reply to this) (Thread)
Re: Compression
[info]mtocker
2009-05-28 05:02 pm UTC (link)
I disabled compression ;) That was one of the first things I did.

If you look here:
http://www.mysqlperformanceblog.com/2008/06/05/how-would-you-compress-your-mysql-backup/

You can see that by these tests, gzip in a single threaded execution can only do 37MB/s - which is less than what gigabit ethernet can do.

(Reply to this) (Parent)(Thread)
Re: Compression
(Anonymous)
2009-06-07 08:00 am UTC (link)
Bump up the blocksize!!!

When you are on gigabit networks, there is no point of making checksums of small blocks. The maximum block size used by rsync is a mere 16K, that will be transferred in miliseconds.

So bump up the block size (-B) option to 1-10Mb and you will see huge speed boost on big files (on fast networks, that is).

(Reply to this) (Parent)

[info]demonbane
2009-05-28 05:32 pm UTC (link)
If you're trying to sync a small number of very large files, this should be very quick. I can sync a 16 GB file with ~100 MB of changes across a gigabit network in about 10-15 seconds.

Rsync suffers from two basic performance issues: a large number of small files (since it has to do a full inventory before it starts the sync) and transferring *changes* in compressed files. (A compressed file can have significant differences even though the change in the compressed data is minimal.) The large number of files problem you really can't get around. When you need to sync 500k+ files, it'll take a while. (We recently did a transfer of 24 *million* files, and the inventory alone took nearly 8 hours. Plus another 24 hours for the data transfer.)

In terms of compressed files, there is something you can do. We use rsync for some of our database backups, and have found that it is generally much faster to rsync the raw SQL database dump rather than trying to rsync a gzipped copy of it (even with the --rsyncable flag to gzip). So if you're trying to rsync a compressed version of the file, don't. If you need to gzip it, do it on the receiving end.

(Reply to this) (Thread)

[info]mtocker
2009-05-28 05:41 pm UTC (link)
There were only a few hundred files in the complete backup - and about 5-10 large files accounted for most of it.

I'm surprised you could do the 16GB file so fast, it must have at least been in the filesystem cache, while in my case (InnoDB O_DIRECT) it's not going to be.

I'm sure it works beautifully with SQL dumps... my only problem is how long that would take to import at the other end ;)

(Reply to this) (Parent)
rsync can work
[info]jeremycole
2009-05-28 05:42 pm UTC (link)
Hi Morgan,

Rsync can do it, but its waste usually comes in two areas:

1. Trying to compute differences. This is the "save the network" strategy. It wastes a ton of IOPS and CPU to compute differences between blocks, when, on gigabit ethernet it would've been much faster just to send the block. Most of the time wasting comes from having to read on both sides (== IO) rather than just read on one side and write on the other. You can disable this with the -W option, which only sends whole files, and does not attempt to compute differences. Note that it still checks file mtime and size, so it only sends files which are missing, not necessarily all files. This makes it a bit better at resuming broken transfers.

2. Tunneling through SSH. SSH is a pretty inefficient protocol for large file transfers on a local network, but it's the typical transport for rsync. You can get around this by running an rsync daemon on the remote side, thus using rsync protocol over a raw socket.

All that said, I find for most transfers, "rsync -avPW" over ssh is just fine. If I need a bit more of a boost, run rsync daemon. If you need even more speed, use tar and nc piped together, e.g.: "tar -cv /path | nc host port" on the sending side, and "nc -l 5000 | tar -xv" on the receiving side.

(Reply to this) (Thread)
Yeah, I should have used -W in this case.
[info]mtocker
2009-05-28 06:08 pm UTC (link)
If every file has just changed a couple of bytes, then don't you find it a sad loss to not be able to compute differences?

I suggested "like bittorent", because with a 16K page, it should actually be fairly trivial to create checksums and report what has changed between files (provided the checksum chunk-size is also 16K, or has a good alignment).

- Morgan

(Reply to this) (Parent)(Thread)
Re: Yeah, I should have used -W in this case.
[info]jeremycole
2009-05-28 06:22 pm UTC (link)
A sad loss of what?

Computing differences and applying changes means:

1a. Read block on sending side
1b. Read block on receiving side
2a. Compute checksum on sending side
2b. Compute checksum on receiving side
3. Send checksum, if different, continue
4. Send full block from sending side
5. Write full block on receiving side

The cost of step 1b is nearly as much as the cost of step 5, and introduces more delay from synchronization. Why not skip 1b through 3?

(Reply to this) (Parent)(Thread)
Re: Yeah, I should have used -W in this case.
[info]mtocker
2009-05-28 06:38 pm UTC (link)
Seeing it put like that makes me think what our new world of parallelism can do.

If you assume both sender and receiver are identical, both sender and receiver are dedicated for only this purpose, and that you don't know how much work you have to do until you've done it, then the only overhead should be*:

- the latency to compute the checksum.
- the latency to send the checksum across the network to one side.
- the latency to compare the checksum.

You can just sync the file as you read through blocks (okay, that's a little bit different from bittorrent).

* I'm not sure if reality is quite as straight forward as my assumptions, since the receiver may not have the same capacity for reads as the writes may effect it. But it could be a hell of a lot faster than it is currently.....

(Reply to this) (Parent)

(Anonymous)
2009-05-28 07:10 pm UTC (link)
Maybe you should have got a look at DRBD, this is "Raid-1 over network"

(Reply to this) (Thread)
Won't work :(
[info]mtocker
2009-05-29 05:49 pm UTC (link)
To start with, the old server and new server were running different filesystems ;)

I'd also need to install a kernel module, and DRBD typically works via an initial sync process (which would have the same cost or higher, since I have to do the whole partition).

Technically I think I could fool the initial sync process and done an online device verification, but that wouldn't work because the physical layout of the blocks will be completely different between serverA and serverB even though the data is similar!

- Morgan

(Reply to this) (Parent)

[info]mike503
2009-05-28 08:45 pm UTC (link)
there are some pretty big changes in rsync 3.x. for example, for large numbers of files, it is much more efficient. no longer building an entire file list on the source and destination before it does work, but starts working instantly, creating filelists in the background continuously.

might be something to make sure you're using the latest and greatest too.

(Reply to this)


(Anonymous)
2009-05-29 06:39 pm UTC (link)
netcat is your friend. Wire speed, since your disks will keep up.

On the destination machine:
nc -l -p 9999 | tar -xv

on the source machine:
tar -c | nc dest.ip 9999


(Reply to this) (Thread)

[info]mtocker
2009-05-29 09:19 pm UTC (link)
That probably would have been the most efficient method given my description (provided every file changed by a small amount).

The problem is my disks can still read/write faster large amounts of data than my network can send... so it's not the best case.

(Reply to this) (Parent)(Thread)

[info]interra
2009-05-29 10:05 pm UTC (link)
Given you have spare CPU to burn add compression to tar/gzip/pigz. And with series of compression level experiments you'll find most efficient compression ratio to your combination of disk/bus/cpu/network bandwidth and load.

(Reply to this) (Parent)

(15 comments) - (Post a new comment)

Create an Account
Forgot your login or password?
Login w/ OpenID
English • Español • Deutsch • Русский…