Fast transfers with dumb tools

This web site is built using jekyll and markdown, and is completely static (yay!). The site is hosted at my university which provides access through the sftp protocol only (so no rsync!). To minimize bandwidth and time (from outside, the connection is quite slow), the sftp client lftp has a “mirror –only-newer” command that is perfect if you are building your site manually, as you would maintain access times in sync with the server and only upload modified files. Unfortunately, when building the site with jekyll all files are regenerated and –only-newer is ineffective! I solved this (admittedly small) problem in a simple, clean and general way, and as someone asked me how did I do it, I thought it would be nice to add a note here. Note that Jekyll 3.0 has an “–incremental” option that supposedly modifies only changed files and could then be used with the “mirror –only-newer” feature of lftp, but this option is not completely reliable in my experience, in particular with style changes. The trick is to mantain a second local copy of the site that is perfectly in sync with the hosting site. Then you can use rsync with your local copy (this just updates the modified files in the second local copy), and then use “mirror –only-newer” in lftp to push the changes. Note that to allow a passwordless connection with rsa cryptography, the syntax of lftp is slightly convoluted. The two commands are:

rsync -cr _site/ _sitecopy 
lftp -u uname,placeholder -e "set sftp:connect-program 'ssh -a -x -i rsa_key';\\
  mirror -e -R --only-newer _sitecopy htdocs; quit" sftp://host.site

here rsa_key is my private key, uname is the username and host.site is the hosting site. Needless to say, I have the above commands in a Makefile so I need only a “make push” to send only changed files over the wire to the hosting site.