Prevent Process Termination over SSH Connection

When running heavy long-time running processes on a remote server through ssh, it is quite annoying that your processes are terminated when you are disconnected. Here is a tip just for this.

Let's assume you ssh into your server
$ ssh USERNAME@SERVER_IP

You want to run some program, which will run for quite some time.
$ ./a.out arg1 arg2 ...

What happens if your ssh connection is lost, perhaps due to unstable network connection, etc. Well, your program a.out will terminate itself, so you will have to re-run it. In fact, you will need to keep your ssh connection until the program completes, which is quite annoying.

One solution is using nohup
$ nohup ./a.out arg1 arg2 ... &

This will redirect the output to nohup.out file. With this, even if you disconnect from ssh connection, your program will keep running in the background.

Another solution, which is probably much more sophisticated, is to use screen. It is your work session where you can run processes, and you can easily detach from it to do other things while your processes running inside the session are intact and not terminated. You can always re-attach to the screen session and resume your processes still running.

To use this, you need to first create a new screen session
$ ssh USERNAME@SERVER_IP
$ screen -S SESSION_NAME

This will create a new screen. Here, you can run your program
$ ./a.out arg1 arg2 ...

Next, you can always detach from this current session with <CTRL + a> d. That is, press <CTRL> and <a> keys together, release the keys and then press <d> key and release. This will detach from the session. At this point, your program will keep running regardless of whether you disconnect from your ssh connection to the server.

To list running screen sessions, run
$ screen -list

To re-attach to the session, you simply run
$ screen -r SESSION_NAME

To exit the screen, you run
$ exit

By the way, you may notice that within screen session, your scroll will register as up/down arrow keys. If you want your scroll to work as scroll terminal output, you can run the following (credit to pistos)
$ echo 'termcapinfo xterm* ti@:te@' >> ~/.screenrc

You can also press <CTRL> <a> and then <ESC> to enter copy mode and do scroll freely. To exit copy mode, simply press <ESC>

Happy hacking!