Making a Double Commander button that copies selected files to cloud storage via lftp.

Double Commander, ADrive and Lftp.

Ok, so Double Commander is the file manager I am using on linux, having converted from Windows where I always used Total Commander. 

ADrive is the cloud storage I use because its an easy name to remember.

Lftp is my preferred ftp program since it saved my bacon by allowing mirroring with lots of options, in particular one which automatically deletes the source files once they have been transferred. This sounds dangerous, and it normally would be, but it allowed me to read many, many magnetic tapes with tar on a very old computer which had a disk with a storage capacity that was much less than any of the tapes I had to read. As the files were read off each tape, they were periodically synchronised to a USB disk on a laptop via ethernet-crossover cable and scrubbed off the old computer's disk to create space for more files coming off the tape. This sounds dangerous but it worked pretty well since, as each file is read off tape, even if it is deleted by lftp on the laptop while it is being written to by tar on the old computer, it will stay on the old machine's disk so tar will keep writing to it and close the file perfectly normally. Lftp on the laptop will then re-mirror it later on and overwrite the partially recorded file on the USB disk. As a result only 2 - 3 files out of a couple of terabytes of data overall were actually corrupted. I tried many times to do it the proper way e.g. by piping output from tar to ftp, cpio, etc, as rightly suggested by people on a forum I sought advice from, but failed. However, lftp mirroring was a triumph for a real computer numpty, partly since the sysadmin had been promising to do this job for years and never did. I'll add more on this later - I am so proud!!

Trying to connect to ADrive from Double Commander

A separate problem was wanting to connect DC with a cloud storage site which allows sftp and rsync, but not much else. This can be done very well with FileZilla, but its a tiny bit of a chore to start it up and select the files you want to backup, etc, so if it can be done within DC, that would be great. The Network Connect button in DC allows you to create a new ftp connection using the following windows: 

But, after much trying with different options in these boxes, I couldn't get it to work and indeed the web proves that sftp isn't an option in DC, so how can we do it??

At this point, I remembered my fondness for lftp and tried to get it to work with ADrive. Eventually after lots of trying I found I could do it in a command shell as follows: 

lftp sftp://jon.doe@rabbits.com@sftp.adrive.com

What's so special about that then? Well, it took absolutely ages to work out how to connect with a username that contains an @ symbol, since ADrive gives you an account where your username is your e-mail address. The web suggested a number of solutions (e.g. replacing the first @ with %40) but none of them would work, in my hands. However, the above turned-out to be the only combination of commands that would let me login. It also turned out to be useful later on that lftp allows you to give it a string of all the commands you want it to execute once logged-in, for example:

lftp -e 'cd jbcdocu2; ls; bye;' sftp://jon.doe@rabbits.com@sftp.adrive.com

The lftp instructions to be executed have to be within single or double speech marks. As said above, lftp allows mirroring and that is something which I would like to explore with DC later on. But for now lets work out a way of getting lftp to run from DC e.g. by triggering it with a button so that it backs up the selected file or files. This required learning a bit about the way to set up custom Toolbar buttons. If you look in the DC Configuration Options, you will see the following for the Terminal:

Basically when you set up a Toolbar button to execute a shell command, that command gets slotted into the {command} part of the strings shown above and the -e sh -c tells the shell to run whatever commands come after it, providing they are separated by semi-colons. So if we want to make a button that runs a command line program, we can right-click to copy and paste an existing button and right-click again, followed by Edit to give us the following: 

 

You need to select External command, as above, and type the actual command (in this case mocp) in the Command: box. You can enter a helpful Tooltip and use DC to find the relevant icon (usually in /usr/share/icons). In the Parameters: box you can select what type of terminal window you want; %t1 makes the terminal stay up as long as the command is running whereas %t0 makes the terminal window disappear as soon as its not needed and leaving it blank means that a terminal window is not needed at all. 

Now that worked, so this looks pretty promising, but how can we get it to work with lftp when we want to automatically back up the selected file list to the cloud with minimum hassle to the user? Basically, I tried lots of things for doing that and, its probably my incompetence, but I came across an issue that when the shell command contains spaces everything goes a bit haywire because DC seems to split your command up into separate words and send them to standard input of the shell (stdin) as individual command lines. I tried speech marks, etc, and couldn't really get anything to work really, sorry!! So what we want is to have is a shell command that is one word which will start lftp and send the selected file list to ADrive. Now, this is where my real numptiness comes in because I really am sh*te at bash scripting so I try to avoid it as much as possible and use python instead. Anyhow, with my friend, google, I eventually worked out how to make a simple bash script which reads the selected file list from DC and sends it to a python script which does the lftp stuff. This is all really provisional and open to suggestions for improvement, but here is the bash script: 

You need to chmod u+x it first. Basically the read command reads one line of input from stdin and we feed a file into it which we have simply called jim within the script. To make it read more than one line from this file, we change the delimiter that it uses to work out where the end of the line is, hence the -d ''. This sets the delimiter to nothing rather than the default, which is the end-of-line (EOL) marker, so instead it ignores the EOL's and reads the whole file in as one line, including carriage returns "\n", but that doesn't matter because we can split them out in python. This line has to be fed into python as a string, hence the "$jim" thing which I don't really understand because I am sh*te at scripting, but never mind.

OK, so we have got a command file that sends the list of files selected in DC into a python script as a string. But where does this list of filenames come from? In short, the answer is here: 

I think everything is explained in the above figure. We have a one-word command: 

./testpy2.com<%L>adrive.log 

which runs the simple script we described earlier by re-directing the list of DC-selected filenames to it (<%L) as stdin and re-directing the stdout to a log-file so we can check if its worked, or not. The %L thing is a DC symbol for the name of the temporary file (incl. its path) which contains the user-selected filenames, one per line (incl. their paths). We need to set the home directory which contains the script (test2py.com) and the python file; the log file will also be written here. The way this runs is that a terminal window comes up into which you have to type your ADrive password, but currently the window is blank (need help on all this) so under Parameters: the %[Type...] gives the user a hint on this. The %? gives a preview of the command that will be sent to the shell, which is helpful, and the %t1 makes the terminal persist, although it wants to die at the end, but remember I am still very vague on all this and need help! 

As mentioned, my bash scripting is sh*te so we'll do it with python3 and here is the relevant script:

And, basically it works, i.e. use the keyboard insert button to select a bunch of files in DC, click the custom lftp button we have made, type your cloud storage password into the terminal window which comes up, and bingo, the files are copied to the cloud by lftp. 

What I am interested in is doing it better, because it is all a bit rubbish, i.e. I don't get much coming up on the screen telling me what is happening, so any advice would be very much appreciated. 

OK, so I fiddled about with this with help from the Double Commander forum and it looks like I can't easily get it dramatically better than it is. I have therefore posted the scripts which work reasonably well below, in case they are ever of any use to someone. One thing I noticed while tweaking is that the command stated above and below: 

lftp -e 'cd jbcdocu2; ls; bye;' sftp://jon.doe@rabbits.com@sftp.adrive.com

only works when typed in the command line. For some reason which I don't fully understand, it has to be given without the single quotes when run as a python subprocess. 

Anyhow, the final python version of the shell script is called: lftp_selected.com since it allows us to upload the files we have selected in DC by pressing insert on them. This command file has to reside in the home directory and has to be made executable (chmod u+x lftp_selected.com) so we can call it from within DC by using the Command: 

~/lftp_selected.com<%L>lftp_selected.log

 

This feeds in the file containing a list of the files that were are going to backup (%L) and writes out a log file (lftp_selected.log) in the present working directory of DC. 

This feeds in the file containing a list of the files that were are going to backup (%Lb) and writes out a log file (lftp_selected.log) in the present working directory of DC. Note that the b in %Lb makes it backup the red-selected files from both DC windows. Might be handy, or might be annoying (in which case delete the b), dunno??

The set-up of the the toolbar button is as shown: 

The Parameters: line of the toolbar options widget says:

 

%[ * * Type remote password <return> when blank terminal window comes up * * ] %t1

as a reminder to enter the remote password in the terminal that pops up.

The script file lftp_selected.com looks like this: 

#!/bin/bash

read -d '' selected_files

python3 ~/lftp_selected.py "$selected_files"

Note that the read -d is followed by two single speech marks, not a double speech mark. 

This command file runs the python program: lftp_selected.py which also has to reside in the home directory. The python file looks like this: 

remote_site="sftp://jon.doe@rabbits.com@sftp.adrive.com"
remote_folder="lftpdc"
local_editor="featherpad"
#
import subprocess,sys
from pathlib import Path
filelist=sys.argv[1].split("\n")
putstring,findstring="",""
for filename in filelist:
    if filename!="":
       putstring+="put \""+filename+"\"; "
       findstring+="\""+Path(filename).name+"\" "
lftp_command_string="-e cd "+remote_folder+"; "+putstring+\
    " echo Remote folder:; pwd; echo Transferred files:; find -l "\
    +findstring+"; bye;"
lftp_commands=["lftp", lftp_command_string, remote_site]
subprocess.run(lftp_commands)
subprocess.run([local_editor,"lftp_selected.log"])

The first three lines should be changed to specify the remote server, the folder to use on the remote server (which must exist) and the local file display- or editor-utility to bring-up the log file of the transfer. They must be entered as strings, i.e. with double quotes. 

It all seems to work OK. Select files in either window (not both) to backup remotely and click the lftp icon. There is a pop-up to to remind you to enter the remote password in the blank terminal window (see second picture below) or nothing will happen. 

Remember, its all a bit rubbish because nothing is echoed as you type in the terminal, etc! 

Bingo, and little file editor window pops up showing first the remote folder to which the files have been sent followed by a list of the transferred files on the remote server. 

I might do a bit more work on this to tinker with the mirror options in lftp. In fact, I found my old notes on this:

sudo ifconfig enp1s0 128.40.202.209

ping 128.40.202.208

lftp jbc@128.40.202.208

repeat 10m mirror --Move

  -  -  -  OR  -  -  -  

repeat 10m mirror --Remove-source-files

Better Lua version. I'll come back to the mirroring a bit later, hopefully, but I have been learning how to do this sort of thing with Lua which is the proper scripting language for DC. The manual explains how to set up Lua to work with DC here. I have cobbled together a script which does the job reasonably neatly, I think, which is below. Hopefully, just the top 4 lines would need adapting by anyone trying this. It reads in %L i.e. the file containing the full paths of the files we want to upload, and %F which is a file containing just the names of the files (i.e. no path) which it will use for an lftp find command to sort-of check that they have been uploaded. 

In my case, this file is called /home/jon/lftpdc.lua and we need to set it up as a DC command by filling in a new Options, Toolbar widget almost as before, but this time we set an Internal Command (rather than external) and there is no need for redirects or reminder popups.

It seems to work OK i.e. use the insert button to select a group of files that you want to backup in the cloud then click on the new lftp icon.

It prompts you for the remote password, which is masked.

DC hangs a bit while the upload is happening and then it gives you a log widget to confirm the files have arrived at the remote site. 

So, this probably is quite a lot neater than doing it the python way. If anyone wants to give it a bash, the Lua code is below and a prettier version is here. 

remote_site="sftp://sftp.adrive.com"

remote_login="jon.doe@rabbits.com"

remote_folder="lftpdc"

local_logfile="/home/jon/lftp.log"

--

params = {...}

iterator1 = io.lines(params[1])

iterator2 = io.lines(params[2])

putstring,findstring="",""

for line in iterator1 do

    if line~="" then

       putstring=putstring.."put \""..line.."\"; "

    end

end

for line in iterator2 do

    if line~="" then

       findstring=findstring.."\""..line.."\" "

    end

end

click_received, remote_pw=Dialogs.InputQuery("Password","Enter remote password:",true,"")

if click_received==true and remote_pw~="" then

    remote_login=remote_login..","..remote_pw

    lftp_command_string="lftp -e \'cd "..remote_folder.."; "..putstring..

        " echo Transferred files:; find "..findstring.."; bye;\' -u "..remote_login..

        " "..remote_site.." > "..local_logfile

    os.execute(lftp_command_string)

    io.input(local_logfile)

    Dialogs.MessageBox(io.read("*a"), "lftp results")

    io.close()

end

I'll try to do something simple with lftp mirroring soon. 

Thanks to skif_off on the DC forum, there is an even neater way of doing this which avoids feeding two files into the Lua script!! There is a DC utility to extract the filename from its full path called SysUtils.ExtractFileName() so we can use this to generate the list of files which lftp will 'find' as confirmation that they have arrived from the %L file. Change the options toolbar as below so that we just feed %L into the script. 

Here is the new Lua file: 

remote_site="sftp://sftp.adrive.com"

remote_login="jon.doe@rabbits.com"

remote_folder="lftpdc"

local_logfile="/home/jon/lftp.log"

--

params = {...}

iterator1 = io.lines(params[1])

putstring,findstring="",""

for line in iterator1 do

    if line~="" then

       putstring=putstring.."put \""..line.."\"; "

       findstring=findstring.."\""..SysUtils.ExtractFileName(line).."\" "

    end

end

click_received, remote_pw=Dialogs.InputQuery("Password","Enter remote password:",true,"")

if click_received==true and remote_pw~="" then

    remote_login=remote_login..","..remote_pw

    lftp_command_string="lftp -e \'cd "..remote_folder.."; "..putstring..

        " echo Transferred files:; find "..findstring.."; bye;\' -u "..remote_login..

        " "..remote_site.." > "..local_logfile

    os.execute(lftp_command_string)

    io.input(local_logfile)

    Dialogs.MessageBox(io.read("*a"), "lftp")

    io.close()

end 

And a prettified version is here. DC does hang while the upload is happening and there isn't a massive amount of error-checking, so its best to use it only for a small number of files.