Reducing the toil of executing one off code on a remote server with SSH and SCP

This is a little Linux tip that I've started using quite frequently on one of my projects: how to transfer a script and execute it with ssh. I use it when I need to execute a script on a remote server, to do something as a one-off like build a particular data output file, or perform a one off database procedure.

The issue is that I often need to tweak and experiment, but the code being remote means I don't have all my local tools available and transferring the code with git commits is clunky. This is not an everyday occurrence. Typically, I am transferring the main codebase to remote servers with a pipeline using git, but if this one-off script needs to be developed iteratively on (e.g.) a remote staging/debugging environment, and it doesn't belong in the repo long term, then it would clog up the commit history with runtime bugs, and 'Added semicolon'
commits etc. Generally the tweak, test, tweak method of working on a script is not what git repositories are used for!

So, to achieve this quickly, there are a number of existing workflows that I used, all of which involve a certain amount of toil.

  1. Fire up local dev environment, develop and test script on local machine, upload script to remote environment and run.
  2. If no local dev environment, or setup costly, develop directly on remote debug server with vim and console
  3. If vim development at distinct disadvantage to ide, use scp to transfer code changes to remote debug server

I would love to get better at vim, but having spent years using IDEs, it's difficult to give up on my finely tuned mouse skills and keyboard shortcuts for navigating through codebases. I learn a bit more vim each time I use it, but I'm not 'there' yet.

So typically I was using option 3 which involves toil, e.g. if developing a php script and I make a syntax error, I have to do a whole manual copy to server process just for a tiny change.

My aha moment was realising I can create a general purpose script to upload and execute a script on a remote debug server, automating all parts of my option 3 workflow.

So when to use this pattern:

Here is a general pattern

#!/bin/bash

# Copy the script to the server
scp my_script.php server:~/my_script.php

# Run the script which generates downloadable file my_report.csv
ssh server /bin/bash << EOF
cd ~
echo 'Creating report'
php my_script.php
echo 'Done'
EOF

# Copy the file back to local machine
today=$(date +"%Y-%m-%d")
scp server:~/my_report.csv "my_report_${today}.csv"

# Optionally remove all trace of existence
ssh server "rm -f ~/my_report.csv ~/my_script.php"

In my IDE, I can now tweak the PHP, and double click to execute the new file on the server, saving me countless clicks and shell commands.

The other advantage, is that I can then run the script from my local machine on the production environment when it is ready, simply by changing the ssh destinations, making it much less likely for human error!