Morgan Tocker (mtocker) wrote,
Morgan Tocker
mtocker

Efficient way to copy large amounts of data?

Dear Lazyweb,

Yesterday I tried to Rsync a MySQL data directory from serverA to serverB on the same network. I thought that if out of a few hunded gigabytes maybe 2% changed, this should work, right? Wrong. Rsync is designed to minimize bandwidth, so in my case it was *much* quicker to wipe the data and start again (I feel this is something I should have known earlier, but it doesn't hurt to try and share your mistakes).

Which gets me thinking - it doesn't have to be this way. Bit-torrent works in a similar way to Rsync, but it's certainly not network efficient. Are there any projects similar to Rsync that are using network-hungry algorithms to try and make sure that two directories are in sync with the goal just being as fast as possible?
Tags: mysql
Subscribe

  • On Testing MySQL Knowledge

    I can't read posts like this one without at least a little chuckle. Is the number one question you should be asking people how to start and stop…

  • My MySQL Conference Submission 2010: Workarounds

    I just submitted an abstract for the MySQL Conference 2010. What do I want to talk about? Workarounds. Domas likes to do tricks with GDB. Sheeri…

  • Hidden gems in 5.1

    I think 5.1 gets some bad press for not being a compelling upgrade. It's not the big features that make the difference, but the subtle ones. I…

  • Post a new comment

    Error

    default userpic

    Your reply will be screened

    Your IP address will be recorded 

    When you submit the form an invisible reCAPTCHA check will be performed.
    You must follow the Privacy Policy and Google Terms of use.
  • 15 comments

  • On Testing MySQL Knowledge

    I can't read posts like this one without at least a little chuckle. Is the number one question you should be asking people how to start and stop…

  • My MySQL Conference Submission 2010: Workarounds

    I just submitted an abstract for the MySQL Conference 2010. What do I want to talk about? Workarounds. Domas likes to do tricks with GDB. Sheeri…

  • Hidden gems in 5.1

    I think 5.1 gets some bad press for not being a compelling upgrade. It's not the big features that make the difference, but the subtle ones. I…