LFTP
Command line based FTP client
LFTP is a fully featured FTP client with a CLI interface:
- Multiple segmented Download
- Automation via Bash script
- Multiple simultaneous downloads
- Directory Navigation
Multiple Segmented Downloads
Clients that have an issue with SFTP download speeds, can take advantage of LFTP's feature to download data in multiple segments. It basically breaks down a file into smaller pieces and downloads them separately to give a boost to your overall download speed.
Each piece is downloaded using it's own connection which means that on paper your speed will be multiplied by the number of segments that are there. However, in the real world that is not the case and you should experiment a little to figure out the number of segments that give you the best performance.
- Connect to your Ultra service via SSH.
- The above is necessary so that your local machine is able to verify the host key of your Ultra.cc service when you use LFTP in the future.
- Exit your service's SSH with the following command:
exit - Use LFTP to access your service's SFTP server.
lftp sftp://username@servername.usbx.me
Example:
lftp sftp://ultradocs@oblivion.usbx.me- Navigate to the directory on your service that contains the data that you wish to download.
By default, data is saved in
~/downloads/<torrent_client_name>. Given below are example commands to change the directory to~/downloads/rtorrentand then view the data that it contains.
cd ~/downloads/rtorrent
ls- Navigate to the directory on your local machine where you wish to save the downloaded data. Given below is an example to change the local directory to the Desktop folder and view its contents.
cd C:\\Users\\ultradocs\\Desktop
ls- Use 5 segments to download a single file.
pget -n 5 <filename>
Example:
pget -n 5 ubuntu.isoThe above will download ~/downloads/rtorrent/ubuntu.iso to the Desktop folder on your local machine.
- Use 5 segments to download a directory.
mirror --use-pget-n=5 <directory_name>
Example:
mirror --use-pget-n=5 MyLinuxISOs- Use segmented downloads along with parallel downloading for best download speeds. In the example given below, we are using 5 parallel downloads along with 5 segments.
mirror --parallel=5 --use-pget-n=5 MyLinuxISOsThe above will download ~/downloads/rtorrent/MyLinuxISOs to the Desktop folder on your local machine.
As mentioned previously, you can experiment with the number of segments and parallel downloads to get maximum download speeds. We would not recommend setting a number higher than 10.
Push files from your slot using LFTP
This is for those who would like to automate downloading or "pushing" files off your Ultra.cc Slot to another FTP server. For example, you may run a Network Attached Storage device at home which is able to run a simple SFTP/FTP server. This command below can be set up to run automatically and send anything in one folder to your Server at your residence while you sleep. Great for grabbing the latest Nightlight Linux distro's.
lftp -u destinationusername,destinationpassword -e 'mirror --reverse --parallel=3 --continue --only-missing /home/usbdocs/files/LFTP /home/destinationusername/LFTP; quit' sftp://destinationip:portThe Path /home/usbdocs/files/LFTP is used purely as an example. You could change this to any path on your Ultra.cc Slot and LFTP would push that path to the target. You may also add multiple copies of the above line when creating your bash script (details below) to sync multiple folders to multiple target servers and/or folders.
Argument | Effect |
use-pget-n=2 | Number of Segments per file, This is best matched to the number of Parallel downloads. For example, if your parallel=2, then pget-n=2 too. Don't aim for too high a number for example 4 and 4 would be 4 downloads each in 4 segments for 16 threads even this may be higher than required. Experiment to see what works best for your target connection |
continue | Create .part files in case of accidental disconnection. This will allow the next run of lftp to pick up where it left off. This may need disabling if you find the end transfer to be corrupted (just delete --continue) |
mirror | specify a source and target directory |
parallel=number | Number of files to Push to target at once |
reverse | reverse mirror i.e. put all files from an Ultra.cc slot to a Target Server |
only-missing | Push only files not already present on the target server |
sftp://destinationip:port | This will need to be changed to reflect the type of Server your connecting to SFTP/FTP the example here connects to an SFTP server. destinationip and port will need filling based on your target connection details. |
Upload files to your slot using LFTP
Uploading to your slot is just as simple as downloading from your slot "pushing". Using the same command above will work as long as it is run from the location you want your files to arrive at.
lftp -u usbslotusername,usbslotpassword -e 'mirror --reverse --parallel=3 --continue --only-missing /home/destinationusername/LFTP /home/usbdocs/files/LFTP; quit' sftp://destinationip:22Notice that the main changes are the order of the directory and the address at the end. The function of the command is exactly the same; It is just the direction that has changed.
Automate the Process
- Your first step is to find the full path of your home directory
pwdwill output your full home path. - Something like this will be outputted. Be sure to make a note of it.
/home/username- Next, create the following directories.
mkdir ~/lock && mkdir ~/scripts- Then, navigate to the
~/scriptsdirectory.
cd ~/scripts- And create the script file
nano LFTP.sh- Paste the following lines into the editor:
- Make sure to replace
usernamewith your Ultra username.
- Save the script and exit the nano editor by pressing
Ctrl+XthenYthenENTER. - Now we need to set a few things that will be passed to LFTP automatically ready for automation.
- Create a config directory for lftp.
mkdir ~/.lftp- Next, create and open the lftp rc file in the nano editor.
nano ~/.lftp/rc- Paste these lines in and save
set net:limit-total-rate 0:41943040
set ssl:verify-certificate no- Save and exit the nano editor by pressing
Ctrl+XthenYthenENTER.
Test Script
- First, navigate to the
~/scriptsfolder.
cd ~/scripts- Next, we need to make the script executable.
chmod +x LFTP.sh- The script is now ready to be executed, which can be done with the following command.
./LFTP.shIf the script is running and you were to rerun it, you may see an error message ERROR: flock() failed. This is a file lock to stop multiple script executions running simultaneously. If you are sure the script is not in fact running, you can delete the lock file from the ~/lock directory.
Check Progress
You can also check the progress of the backup script, which is running in a screen session.
- Attach to the screen session with the following command.
screen -rd lftp- To detach from the screen session and leave the script running in the background, press
ctrl+aand then pressd.
Automate Script
Once you have concluded the script to be running successfully, you can schedule it to be automatically executed via crontab.
- Open crontab with the following command:
crontab -e- Inside the crontab add the following line at the bottom of the crontab file.
- Make sure to replace
usernamewith your Ultra username.
0 */4 * * * /home/username/scripts/LFTP.sh- The script will now run every 4 hours, checking for files that have changed and sync them to the target server. Please be aware that anything removed from Target will be synced again. Remove the item from the source or move it to a non synced folder.
- Save and exit crontab by pressing
Ctrl+XthenYandENTER.
If you require further assistance, you can open a support ticket here!
Feedback and suggestions regarding this article are welcome on our Discord server