Designing a L2 virtual network

In the previous blog, we saw how we can connect two devices in the same subnet over a L3 device. With some configuration to enable ARP Proxying and host routes on the L3 device we were able to simulate a L2 network over a L3 device.

The important thing to note from the previous blog is that we did not make any change on the device nodes. All changes were made to the router.

So, what can we achieve with this capability?

In this blog, I will be taking that experience to the next level by designing a virtual L2 network using routers.

But most importantly we will be using only common Linux utilities and kernel features to achieve this.

Connecting nodes across multiple routers

Let’s start with setting up a simple topology for this experiment. We have two routers each connecting 2 device nodes. The routers themselves are connected to each other using a L3 link.

Slide2

In a way, each router with nodes are replica of the setup from my first blogs. The end nodes are devices while the middle node are the routers.

Device Interface IP Address
Node_1 node_if0 20.0.0.1/24
Node_2 node_if0 20.0.0.2/24
Vrouter1 vr_if1

vr_if2

eth1

1.0.0.1/24

Node_3 node_if0 20.0.0.3/24
Node_4 node_if0 20.0.0.4/24
Vrouter2 vr_if1

vr_if2

eth1

1.0.0.2/24

Look at the commands from my previous blog to see how to bring up this topology.

Making it work

To make it work each router need to be configured with Proxy ARP (refer to previous blog). Each router must be configured with host routes for all the device nodes. This means we have host routes for devices directly connected to the router and also the ones that are connected via the second router.

Here is an example routing table listing with routes for directly connected devices.

vrouter1 > route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
1.0.0.0         0.0.0.0         255.255.255.0   U     0      0        0 eth1
20.0.0.1        0.0.0.0         255.255.255.255 UH    0      0        0 vr_if1
20.0.0.2        0.0.0.0         255.255.255.255 UH    0      0        0 vr_if2

Use the following commands to setup remote routes

vrouter1 > ip route add 20.0.0.3/32 nexthop via 1.0.0.2
vrouter1 > ip route add 20.0.0.4/32 nexthop via 1.0.0.2

Similar route configuration will be needed for vrouter2 to reach node_1(20.0.0.1) and node_2(20.0.0.2)

Caveat: I found that L2 learning did not work properly when I connected multiple routers. To rescue the situation, I added static ARP entries.

Manually creating the ARP entries for directly connected devices on the router. Use the following commands to setup the static ARP entries.

arp -s 20.0.0.${node_no} $node_mac

With the above taken care you should be able to ping nodes across the multiple routers.

node_1 > ping 20.0.0.3
PING 20.0.0.3 (20.0.0.3) 56(84) bytes of data.
64 bytes from 20.0.0.3: icmp_seq=1 ttl=62 time=437 ms
64 bytes from 20.0.0.3: icmp_seq=2 ttl=62 time=1.02 ms
64 bytes from 20.0.0.3: icmp_seq=3 ttl=62 time=0.947 ms
^C
--- 20.0.0.3 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2002ms
rtt min/avg/max/mdev = 0.947/146.511/437.563/205.804 ms

L3 fabric as a transport

Now that we have a workable L2 network that can span across multiple routers, is there anything else required to make this solution work in a generic environment?

If you look closely, you will realize that although the routers in our experiment are connected to each other over a L3 link, the host routes in our end routers depend on the knowledge of exact path connecting them, e.g. if we introduce a third router in between them our solution will not work. As the remote host routes now needs to be adjusted to account for the extra hop that was introduced by the third router. Additionally, all the routers in the path connecting the end routers need to be aware of the host routes for the device nodes.

The VPN tunnels to rescue

The way to solve this situation is to setup tunnels between the routers connecting the device nodes. In this way, the L3 path is isolated from knowing the host routes and the Host routes are isolated from the impact of changes in the L3 path that connects the end routers.

We can setup GRE tunnels between the end routers, but we can also make the connections over SSH tunnels.

Use the following commands for setting up GRE

vrouter1 > ip tunnel add tun0 mode gre remote 1.0.0.2 local 1.0.0.1 ttl 255
vrouter1 > ifconfig tun0 2.0.0.1/24 up

Or

Use the SSH based tunnel as described at

http://man.openbsd.org/ssh#SSH-BASED_VIRTUAL_PRIVATE_NETWORKS

So now we have out L3 Virtual Private Network which isolates us from impact of the changes in the path. Make sure that you use the tunneled path for reaching the remote nodes. To do this adjust the host routs to use the tunnel paths

vrouter1 > ip route add 20.0.0.3/32 nexthop via 2.0.0.2
vrouter1 > ip route add 20.0.0.4/32 nexthop via 2.0.0.2

Putting it all together

Finally, this is how the complete topology looks.

Slide3

Here is a summary of the IP addresses and interfaces on each node.

 

Device Interface IP Address
Node_1 node_if0 20.0.0.1/24
Node_2 node_if0 20.0.0.2/24
Vrouter1 vr_if1

vr_if2

eth1

tun0

1.0.0.1/24

2.0.0.1/24

Node_3 node_if0 20.0.0.3/24
Node_4 node_if0 20.0.0.4/24
Vrouter2 vr_if1

vr_if2

eth1

tun0

1.0.0.2/24

2.0.0.2/24

The routing table with tunnel paths enabled looks like the following.

vrouter-routing-table

And the pings are still working 🙂

node_1 > ping 20.0.0.3
PING 20.0.0.3 (20.0.0.3) 56(84) bytes of data.
64 bytes from 20.0.0.3: icmp_seq=1 ttl=62 time=290 ms
64 bytes from 20.0.0.3: icmp_seq=2 ttl=62 time=1.01 ms
64 bytes from 20.0.0.3: icmp_seq=3 ttl=62 time=0.973 ms
64 bytes from 20.0.0.3: icmp_seq=4 ttl=62 time=1.17 ms
^C
--- 20.0.0.3 ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3004ms
rtt min/avg/max/mdev = 0.973/73.499/290.832/125.477 ms

 

Advertisements

Published by

Chandan Dutta Chowdhury

Software Engineer

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s