Skip to content

Copy Large Amounts of Data

June 1st, 2020 - IT(1 min)

Last week I managed to configure Plex on my server, as a FreeNAS jail. Now I want to copy the data from my Synology box.

cp

I mounted my synology share containing multimedia files onto the FreeNAS, because I figured it’s easier using a cp and SMB mount than doing it via e.g. ssh. The command is like this:

# mount_smbfs -I <SMB IP ADDRESS> //[email protected]/share /path/to/local/mnt
mount_smbfs -I 10.0.0.30 //[email protected]/share /mnt/test

I have also needed to create a ~/.nsmbrc file (for root) contining the password:

[DISKSTATION:MEDIA]
password=changeme

I tried copying the files as a bulk:

cd /mnt/test
cp -R ./Movies /mnt/Main/media/tmp

My shell session timed out before it could finish (it copied about 20%). I figured I could go further if I would go letter by letter:

for i in A B C D E; do
  echo "$i"
  cp -R ./$i /mnt/Main/media/tmp
done

It worked for small groups, but I still needed a way to resume

xargs

My second attempt was to use a pipe finishing with xargs to copy the rest. I have tried:

diff -qr ./L /mnt/Main/media/tmp/L | \
  awk '{print $4}'| \
  xargs root  wheel   667 May 23 18:24 .zshrc -n1 -J% echo "%"

The command identified the missing files. However it did not give me the proper results because most directories have spaces.

rsync

I have tried to avoid rsync (because it’s a bit of black magic for me how it works), but failing the other means and not willing to go through my windows machine to do some ol’ drag-and-drop…

rsync -vrh --append-verify ./ /mnt/Main/media/tmp

This would:

  • append to the data (if it has been partially copied)
  • archive (not really since most files are video)
  • recurse through directories
  • present results in human-readable form

You could also execute a --checksum to verify if the file-level checksums match (might do it later, to double-check all went ok), but this is veery slow.

Conclusion

If you have few data, use cp. If you have lots (I found out that I have 1TB+) and the process may fail, use rsync. You’ll be able to resume the transfers.

HTH,

Share on
Reddit
Linked in
Whatsapp

A little experiment: