bbcp Usage Details

Introduction

Basically, you can execute bbcp on either our submit nodes or the storage server on your side. However, since bbcp need to be present by both side, please make sure you have a bbcp executable binary copy already and can be reached in the PATH.

The syntax of bbcp is similar to the syntax of scp, with some special options specifying how to run ssh/bbcp. Since bbcp utilizes multiple streaming to tune up transferring speed, please test first before you start use it for massive data moving.

How-To

The bbcp has been configured in our module environment. The simplest method to start a data transfer on our site is like this:

$ module add bbcp
$ bbcp -P 5 -w 1M <source file> <destination IP>:<destination path>

It means to send file with window size 1MB and show a progress report every 5 seconds. User also can tune the stream number. The default stream number is 4 and it should be alright for transfers over the wide-area network (WAN). To transfer to Guillimin please be sure to specify a log in node i.e guillimin[4, 5, 6, 7 or 8] to avoid the round robin complication on bbcp protocol whereby no streams are opened for file transfer.

Since you need a bbcp copy on your destination server. You can either download binary file or compile from the source. Please refer following web link for installation:
http://www.nics.tennessee.edu/computing-resources/data-transfer/bbcp

To transfer data from your storage server on your site:

Since bbcp is in the module and it won't be present in the remote shell environment, please add the module in your ~/.bashrc which is located in your home directory in Guillimin so the bbcp module will be loaded in remote shell automatically:

$ cat ~/.bashrc
$ module add bbcp

All the useful parameters are listed below. However, just be careful not to set stream number more than 4 since it will cost memory usage a lot:

-w : window size. the window size can be calculated by 'Round Time trip * Bandwidth'.

-P : progress report interval

-v : verbose

-f : remove file if it exist on destination

-F : forget disk space check on destination

-s : how many stream to use. Default is 4 and it can go to 16 or even 32. However, more stream doesn't mean higher transfering rate but more memory consumption. must be used carefully.

-r: copy subdirectories and their contents (actual files only)

-I : (capital i) filelist

Calculating the Window Size

When choosing the window size for the -w option of bbcp, use the "ping" command between the machines you are transferring files to and from. Multiply half of the ping time (the Round Trip Time) by the capacity of the link. For example, if the ping time is 100 milliseconds and the network is a 10Gbits/sec infrastructure:

Window Size = 0.5 * 100msec * 10Gbits/sec / 8bits/byte
            = 0.5 * 0.1sec * 10000Mbits/sec / 8bits/Byte = 62.5MBytes

This is the number you would use for window size of bbcp