SSH Jump Host and Connection Multiplexing

Jump Hosts

While working with libvirt as my primary hypervisor to launch test VMs I need a way to connect to the VMs easily over SSH. As libvirt uses private network and SNAT for connecting the VMs to external world getting SSH access to the VMs requires Port Forwarding or DNAT.

I recently came to know about SSH Jump Host configuration. It which uses the SSH ProxyCommand to tunnel the SSH connection through intermediate hosts. I found it very useful to connect to my VMs hosted by Libvirt KVM on private network. Here is the command that I use to connect to the VMs

ssh -t -o ProxyCommand='ssh hypervisor_user@my-hypervisor1 nc vm1 22' vm_user@vm1

What is more amazing is SSH allows multiple intermediate Jump Hosts in the path.

Here is another trick taken from Gentoo wiki. Add the following configuration to your ssh config file at ~/.ssh/config

Host *+*
   ProxyCommand ssh $(echo %h | sed 's/+[^+]*$//;s/\([^+%%]*\)%%\([^+]*\)$/\2 -l \1/;s/:/ -p /') exec nc -w1 $(echo %h | sed 's/^.*+//;/:/!s/$/ %p/;s/:/ /')

with this config in place we can specify multiple intermediate jump hosts in the following format

ssh user1%host1:port1+user2%host2:port2+ host3:port3 -l user3

Connection Multiplexing

Connection multiplexing is a way to optimize creation of SSH connection between the client and server when frequent requests are made from the client to the server. Instead of creating a new SSH connection for each request and closing it down which incurs delays, it is easier to reuse an existing SSH connection.

ssh -M -S ~/.ssh/controlmasters/user1@server1:22 server1
ssh -S ~/.ssh/controlmasters/user1@server1:22 server1

It is easier to set this up with the ssh config file, here is an example:


Host Server1
       HostName server1
       ControlPath ~/.ssh/controlmasters/%r@%h:%p
       ControlMaster auto
       ControlPersist 10m





Port forwarding with ssh for VMs on libvirt NAT network

In my previous post I explored  port forwarding with IPTables to make VMs on a NAT Network (libvirt) accessible to the external world.

The same result can be achieved by using SSH  to forward a port to the VM on the private network(accessible only within the hypervisor). This is specially handy if you don’t have root level access to change IPTables setting on the hypervisor. This is achieved by creating a ssh tunnel to connect a port on the hypervisor to a port on the VM.

Two things to keep in mind to make this work.

  • you must have a ssh login on the VM
  • the port forwarding works as long as the ssh connection is alive

Here is the SSH command that sets up the tunnel between the hypervisor and the VM

ssh -L<hypervisor_ip>:<hypervisor_port>:<vm_ip>:<vm_port> <user_on_vm>@<vm_ip>

an example:

ssh -L0.0.0.0:10022:vm1:22 user1@vm1

I have used as the hypervisor so that the tunnel listen to all the interfaces on the hypervisor.

Execute this command on the hypervisor, this command will open a ssh connection to vm1 and prompt for password for user1. Once the connection is established, the tunnel is created from port 10022 on the hypervisor to port 22(ssh port) on the VM. In this example connecting to port 10022 on the hypervisor will actually access the port 22 on the VM. Users can now SSH to the VM from the external world by connecting to port 10022 on the hypervisor.

Certificate Based SSH User Authentication

SSH server can authenticate user based on Certificates. This post describes the process to setup user authentication using Certificates.

Setting Up Certificate Authority Infrastructure


CA Machine

  • Generate CA key (user_ca) for signing user ssh keys

root@CA:~# ssh-keygen -f user_ca
Generating public/private rsa key pair.
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in user_ca.
Your public key has been saved in
The key fingerprint is:
b3:af:e8:ef:c4:5d:90:f8:be:16:99:74:f2:39:3a:3e root@CA
The key's randomart image is:
+---[RSA 2048]----+
|                 |
|         . .     |
|        . o      |
|         .o..    |
|        S..*..   |
|       . =+.+    |
|        + oo .   |
|       o .E.     |
|     .oo+++o     |
root@CA:~# ls

Server Machine

  • Transfer and add CA public key ( as Trusted Key in the ssh server machines and restart openssh server

# vi /etc/ssh/sshd_config

TrustedUserCAKeys /etc/ssh/

Authorizing SSH Users


Client Machine

  • ssh key from user1 is generated (make sure to have a pass-phrase for the key, else it will be rejected during authentication)

user1@client1:~$ ssh-keygen -trsa
Generating public/private rsa key pair.
Enter file in which to save the key (/home/user1/.ssh/id_rsa):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /home/user1/.ssh/id_rsa.
Your public key has been saved in /home/user1/.ssh/
The key fingerprint is:
6f:d0:27:36:69:80:e4:ad:0c:5c:f9:d9:d8:af:a9:8d user1@client1
The key's randomart image is:
+---[RSA 2048]----+
|      o.         |
|   . +.o         |
|    o o.o=       |
|     o .+oo.     |
|      o S B..    |
|         = +.    |
|          oo     |
|         +o      |
|        E..      |

  • ssh key transferred to CA to be signed with user_ca Private Key

user1@client1:~$ scp .ssh/ root@CA:~/
root@ca's password:                                                                                                                               100%  395     0.4KB/s   00:00    

CA Machine

  • User ssh public key signed by CA

ssh-keygen -s user_ca -I user_user1 -n user1 -V +52w
Signed user key id "user_user1" serial 0 for user1 valid from 2014-12-30T16:18:00 to 2015-12-29T16:19:57

Client Machine

  • ssh key transferred back to client after signing with user_ca Private Key

user1@client1:~$ scp root@CA:~/ .ssh/
root@ca's password: 

Testing the result

Client Machine

  • Once the above is completed user1 will be able to login to server1 without any password.
user1@client1:~$ ssh user1@server1
Enter passphrase for key '/home/user1/.ssh/id_rsa':
Welcome to Ubuntu Vivid Vervet (development branch) (GNU/Linux 3.13.0-43-generic x86_64)

 * Documentation:

The programs included with the Ubuntu system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Ubuntu comes with ABSOLUTELY NO WARRANTY, to the extent permitted by
applicable law.



SSH Infect

SSH Infect


A script to manage ssh public key install on servers

#set -x

packup_dir="${DB_PATH}/$(basename $0)_packup"

which ssh-add >/dev/null 2>&1

function list_infected()
    cd ${DB_PATH}/
    hosts=`ls ${DB_HOST_PREFIX}* 2>/dev/null`

    for host in $hosts; do
            echo "- ${host##${DB_HOST_PREFIX}_}"
    cd - >/dev/null 2>&1

function clear_infected()
    cd ${DB_PATH}
    hosts=`ls ${DB_HOST_PREFIX}_$1 2>/dev/null`
    if [[ "$hosts" == "" ]]; then
        echo "No matching record found for host: $1"
        exit 1
    echo "#=================================================="
    echo "Following hosts will be cleared"
    echo "#=================================================="
    echo ${hosts##${DB_HOST_PREFIX}_}|tr " " "\n"
    echo "#=================================================="
    read -p "Confirm [y/n]: " ans
    if [[ "$ans" == "y" ]]; then
        key=$(awk '{print $NF}' ~/.ssh/
        $(ssh $1 "sed -i \"/$key/d\" ~/.ssh/authorized_keys") && echo "Key removed"
        rm -f $hosts

        ssh_key_test $1
        if [[ "$?" != "0" ]]; then
            echo "Host $1 cleaned"
    cd - >/dev/null 2>&1

function packup()

    rm -rf $packup_dir*
    packup_dir=${packup_dir}_$(date +%F)
    mkdir -p ${packup_dir}/{$(basename ${DB_PATH}),$(basename ${bin_dir})}
    cp -rp ${DB_PATH}/${DB_HOST_PREFIX}* ${packup_dir}/$(basename ${DB_PATH})
    cp -rp $0 ${packup_dir}/$(basename ${bin_dir})
    cp -rp ~/.ssh ${packup_dir}
    tar -czf ${packup_dir}.tgz $packup_dir 2>/dev/null
    echo "Package: ${packup_dir}.tgz"

function startup()
    if [[ ! -d $DB_PATH ]]; then
        echo "DB path is not present"
        mkdir -p $DB_PATH


function ssh_key_gen()
    if [[ ! -f ~/.ssh/id_rsa ]]; then
        [[ $SSH_ADD_PRESENT -eq 0 ]] && ssh-add -D
        ssh-keygen -t rsa -N '' -q -f ~/.ssh/id_rsa
        [[ $SSH_ADD_PRESENT -eq 0 ]] && ssh-add

function ssh_key_test()
    ssh -oPasswordAuthentication=no $1 exit 0;
    return $?


function ssh_key_copy()
    SSH_KEY_COPY_CMD="umask 0022; mkdir -p ~/.ssh; chmod 700 ~/.ssh; cat >> ~/.ssh/authorized_keys"
    ssh_key_test $DESTHOST
    if [[ "$?" != "0" ]]; then
        $(cat ~/.ssh/ |ssh $DESTHOST $SSH_KEY_COPY_CMD) \
                && touch $DB_PATH/${DB_HOST_PREFIX}_${DESTHOST}
        touch $DB_PATH/${DB_HOST_PREFIX}_${DESTHOST}


if [[ "$1" == "-l" ]]; then
    echo "#=================================================="
    echo "Infected hosts"
    echo "#=================================================="
elif [[ "$1" == "-c" ]]; then
    clear_infected $2
elif [[ "$1" == "-p" ]]; then
    echo "#=================================================="
    echo "Packing `basename $0` database"
    echo "#=================================================="