Speeding up lftp with parallelization

published on | edited on

By default, lftp doesn’t use parallelization and you need to specifically mention in your command arguments or in settings. I use a simple Makefile to upload this website to a FTP server. In that file, I have the following line.

install:
	lftp -e "set ftp:ssl-allow true; \
	     set ssl:verify-certificate yes; \
		 open ftp://john@example.com; \
		 mirror -X .* -X .*/ --reverse --parallel=8 --verbose --delete public/ httpdocs/; \
		 bye"

The secret is that, you need to use both reverse and parallel flags.

Every upload used to take 200-250 seconds and now only 60-90 seconds. I could make more parallelization; however, a minute and half is better than waiting for 4 minutes. 2.5-3 times improvements!

I recorded results of different settings. I only changed the parallel flag and results are in in seconds.

Setting Result
1 207
2 182
4 109
8 65
16 56
32 57
64 79