I have been working on a project which requires me to connect to my test environment deployed on GCP. We don’t have public IPs available for all the VMs in the test environment, but one of the VMs in the deployment is configured as a JumpHost i.e it has a public IP available. We need to jump through this VM to get to any other VMs in the test environment.
This is quite a standard setup for many companies who deploy software on cloud computes. It helps us to protect the test VMs from getting exposed to the external world thereby reducing the attack surface and also Public IPs are not necessary for the tests we run in our environment.
SSh client JumpHost configuration
Normally it’s just fine to configure your ssh client to connect to the VMs not having the public IPs using a JumpHost configuration. Quoting directly from Gentoo wiki
### First jumphost. Directly reachable Host betajump HostName jumphost1.example.org ### Host to jump to via jumphost1.example.org Host behindbeta HostName behindbeta.example.org ProxyJump betajump
The issue we have is these VMs often gets re-deployed with dynamic IPs. This makes the ssh client configuration invalide after each deployment.
Site-to-Site Cloud VPN
We can of course use a cloud VPN to get access to these VM with private IPs but getting access to Public IP in the office network is beyond my current role 🙂 . Although there are a lot of discussion happening about the shiny new VPN just got included in the linux Kernel. I hope to give it a try sometime soon.
Need for lightweight client VPN
With all the above constraints at hand, I was looking for a lightweight VPN solution that just connects my Dev VM on the laptop to the test environment. That’s when I discovered sshuttle. A nifty little VPN that uses SSH to setup a tunnel to the remote network. What is amazing about this tool is that it does not need me to have root access to the remote end point of the VPN(which is always an issue), neither does it need any Public IP for the client. As long as I have root/sudo access on my Dev VM and I can ssh to the JumpHost, I am able to connect to the remote network. As this is based on SSH no setup is needed on the server side. Merely a valid ssh credential is sufficient. The only requirement is to have python available on the client and server.
I used the following command to connect to the remote network.
sshuttle -e "ssh -o ConnectTimeout=1 -o ConnectionAttempts=1 -o ServerAliveInterval=5 -o ServerAliveCountMax=3" -r myuser@33.XX.XXX.XX --dns 10.XXX.X.0/24
The above command connects to the public IP of the JumpHost (33.XX.XXX.XX) as myuser. It uses the regular ssh command to setup the tunnel so you can pass various options to SSH. In the above case I setup SSH client options to detect idle and quick disconnect detection.
Once the above command succeeds I am able to connect to any of the private IPs of the VMs in my test environment. There is no need to any further configuration. What is interesting is that even the DNS resolution works and the VMs can be accessed with their names, when you use the –dns option.
Solving the name resolution issue with selective query forwarding
One issue that is faced with the setup is that all my dns queries started getting redirected to the VPN remote end point. This is a bit inconvenient as you lose the ability to resolve local machines in office. Although public DNS names gets resolved by the cloud provider’s DNS. To resolve this I applied a patch to sshuttle to forward only the queries of the remote domain to the VPN endpoint while all other queries continue to get resolved by the DNS configured in my Dev VMs resolv.conf. Unfortunately I could only implement this for Linux using NAT. My for is available at
Distributing the VPN software.
As sshuttle is written in pure python, distributing it is really easy. I packaged it into a single file executable using PEX. The final size of the executable is less than 500K (336K in my case). To generate the PEX binary I used the following command.
pex -v sshuttle -o sshuttle.pex -e sshuttle.cmdline:main
Now you can use sshuttle.pex as the sshuttle command and it can be distributed as this single file.
Overall sshuttle is an amazing tool for Devs working in a hybrid cloud environment. Its lightweight, does not introduce any dependency on the server or client side and immensely helpful in providing remote access and name resolution. Thanks to https://forwardingplane.net/2020/01/17/nix4neteng-8-sshuttle/ for introducing me to Sshuttle.