Test-driving arbitrary data publishing over BGP

BGP is a routing protocol known for its strength in scaling and resilience. It is also flexible and extensible.  With its Multi-Protocol extension BGP can support distribution of various data types. Still to extend BGP for every new route data type  requires introduction of new address family(AFI/SAFI) and making BGP aware of the new data type.

What if we could just distribute any arbitrary data over BGP without the need for introduction of changes to BGP? Of-course the BGP end point itself cannot decipher the arbitrary data but the data can be passed on to other applications which can make sense of it.

Well this feature is proposed in the following IETF draft


The draft proposes introduction of a generic opaque Address family to allow distributing arbitrary key value pair over BGP.

Testing opaque data dissemination over BGP

While investigating GoBGP for my previous experiment I realized that GoBGP has support for Key Value based opaque data dissemination. So let’s see how we can disseminate a Key Value pair of arbitrary data over BGP.

For this experiment I am using a simple linear topology as shown below.


Also I have used mininet to simulate the above topology using Linux Network Name Spaces as the BGP speakers.

Starting the topology

Install mininet using apt-get commands as follows

$ sudo apt-get install mininet

Start the default linier topology as follows. This topology has two hosts H1( and H2 ( connected through a switch

$ sudo mn

Start GoBGP instances on the hosts. To do this create 2 config files one for each GoBGP instance. The following is an example config file for H1(

  as = 64512
  router-id = ""
  neighbor-address = ""
  peer-as = 64512
  afi-safi-name = "opaque"

To start the GoBGP instance use the following command

mininet> h1 ./GO_WA/bin/gobgpd -f ./gobgpd_h1.conf &
mininet> h2 ./GO_WA/bin/gobgpd -f ./gobgpd_h2.conf &

Check the BGP neighbor status

mininet> h1 /home/ubuntu/GO_WA/bin/gobgp nei
Peer        AS  Up/Down State       |#Advertised Received Accepted 64512 00:00:24 Establ      |          0        0        0

Also make sure that the neighbor had the capability to publish and consume opaque data

mininet> h1 ./GO_WA/bin/gobgp neigh
BGP neighbor is, remote AS 64512
  BGP version 4, remote router ID
  BGP state = BGP_FSM_ESTABLISHED, up for 1d 05:45:43
  BGP OutQ = 0, Flops = 0
  Hold time is 90, keepalive interval is 30 seconds
  Configured hold time is 90, keepalive interval is 30 seconds
  Neighbor capabilities:
        opaque: advertised and received
    BGP_CAP_ROUTE_REFRESH:      advertised and received
    BGP_CAP_FOUR_OCTET_AS_NUMBER:       advertised and received
  Message statistics:
                         Sent       Rcvd
    Opens:                  1          1
    Notifications:          0          0
    Updates:                4          0
    Keepalives:          3572       3572
    Route Refesh:           0          0
    Discarded:              0          0
    Total:               3577       3573
  Route statistics:
    Advertised:             3
    Received:               0
    Accepted:               0

Publishing an opaque Key-Value pair

Use the following command to publish any key value pair over BGP connection.

mininet> h1 ./GO_WA/bin/gobgp global rib add key key1 value value1 -a opaque
mininet> h1 ./GO_WA/bin/gobgp global rib add key key2 value value2 -a opaque
mininet> h1 ./GO_WA/bin/gobgp global rib add key key3 value value3 -a opaque

Verify that the opaque data is received on the BGP peers i.e. H2

mininet> h2 ./GO_WA/bin/gobgp global rib -a opaque

  Network Next Hop  Age      Attrs
*>key1  00:01:10 [{Origin: ?} {LocalPref: 100} {Value: value1}]
*>key2  00:00:09 [{Origin: ?} {LocalPref: 100} {Value: value2}]
*>key3  00:00:03 [{Origin: ?} {LocalPref: 100} {Value: value3}]


Publishing arbitrary data over BGP can be used to distribute application specific configuration.  BGP will not try to interpret the opaque data. The data must be used by the application. The draft proposes BGP implementation may provide APIs for the application to publish and consume opaque data.



Test-driving EVPN route publishing with GoBGP

In recent times there has been a lot of interest in tunnel based L2 networks, especially for Cloud Networks implemented with VXLAN.  The tunnel based networks were initially proposed with the idea of alleviating the 4k limit imposed with VLAN based networks.

EVPN based VXLAN tunneled networks use BGP as control plane for L2 learning. In this port we will test GoBGP for publishing L2 routes.

L2 learning in tunneled networks

The tunnel based networks need special handling of L2 learning. L2 learning consists of learning how to reach certain MAC (and IP address) in the network. In a normal VLAN based network this is done using the data path, i.e. when two IP addresses communicate in a VLAN the switch in the data path use the source MAC address of the packet to learn which port of the switch is connected a MAC address. This normally happens during the ARP resolution phase that converts the IP(L3) address to MAC(L2) address. It involves flooding all ports of the switch to query for the MAC address associated with the requested IP address.

In a tunnel based networks each of the endpoint form a full mesh of tunnels with all other endpoints.  Unfortunately, data path learning is a very costly option for tunnel based network as it requires flooding of all the tunnels.

Control Plane based L2 learning

The tunnel based networks instead implement a control plane based L2 learning. The control plane based learning work by proactively publishing the location of a MAC(L2) address on the overlay network to all the tunnel end-points.

To implement a control plane based learning someone with the knowledge of location of the MAC(L2) address on the overlay network must send this information to all the tunnel endpoints. This can be done by using a central controller which has a complete knowledge of the overlay network. This is how SDN controllers work, these controllers have the complete view of the network and can publish L2 reachability information to all tunnel endpoints

The EVPN solution uses a combination of data path and control plane learning. Each tunnel endpoint implements a local data path learning and publishes the learned L2 reachability information as L2 routes to the remote endpoints. It uses BGP to publish local and learn remote L2 addresses.

Publishing a L2 route

The L2 route consist of a MAC address (and optionally the associated IP address) and a next-hop IP address of the VTEP. This essentially means that any endpoint trying to send a packet to a MAC on the overlay network should look up its local L2 route table for the destination MAC and send the encapsulated L2 packet to the next hop VTEP IP.

Publishing L2 routes with GoBGP

Now that we have the theory of L2 route publishing cleared, let’s look at some real example of L2 routes published over BGP.

GoBGP is an open source BGP implementation which supports publishing EVPN routes. The following figure represents the test topology. It consists of three BGP peers connected in full mesh.  The VTEP IPs are used to create the BGP peers.


The Router details are as Follows(These are KVM instances started using libvirt and use the management IP for router-id)

Device VTEP-IP

To install GoBGP refer to the install documentation available at


We are going to use the following configuration file to start BGP service Router1. Similar configuration files will be used to start the gobgpd instances on the other routers

  as = 64512
  router-id = ""
  neighbor-address = ""
  peer-as = 64512
  afi-safi-name = "l2vpn-evpn"
  neighbor-address = ""
  peer-as = 64512
  afi-safi-name = "l2vpn-evpn"

To start the gobgpd instance use the following command

# gobgpd -f gobgpd.conf


Check that all the neighbors have established BGP connection and EVPN route are supported


To publish and view a L2 route use the following commands.

# gobgp global rib add macadv aa:bb:cc:dd:ee:04 1 1 rd 64512:10 rt 64512:10 encap vxlan -a evpn
# gobgp global rib -a evpn

Here is an example

Router3 >gobgp global rib add macadv aa:bb:cc:dd:ee:05 1 1 rd 64512:10 rt 64512:10 encap vxlan -a evpn
Router3 >gobgp global rib -a evpn
    Network                                                                                            Next Hop             AS_PATH              Age        Attrs
*>  [type:macadv][rd:64512:10][esi:single-homed][etag:1][mac:aa:bb:cc:dd:ee:05][ip:][labels:[1]]                                   00:00:02   [{Origin: ?} {Extcomms: [64512:10], [VXLAN]}]

The following screen shout captures the output


The above route tells the router to send any L2 packet destined to “aa:bb:cc:dd:ee:05” to the VTEP IP of Router3 over a VXLAN tunnel.

Let’s check that the L2 routes show up on the other Routers. The following route is learned through BGP on Router1.

Router1 >gobgp global rib -a evpn
    Network                                                                                            Next Hop             AS_PATH              Age        Attrs
*>  [type:macadv][rd:64512:10][esi:single-homed][etag:1][mac:aa:bb:cc:dd:ee:05][ip:][labels:[1]]                           00:00:38   [{Origin: ?} {LocalPref: 100} {Extcomms: [64512:10], [VXLAN]}]

The following screen shout captures the output


The route says: a  L2 packet destined for “aa:bb:cc:dd:ee:05” should be encapsulated as in  VXLAN tunnel and sent to VTEP IP of Router3).


From the above demonstration is can be seen how BGP based control plane is used by EVPN to distribute the L2 routes. This approach removes the need for a centralized controller which can become a single point of failure and bottleneck while scaling overlay networks