Test Driving Etcd – part 3 (Service Discovery and Proxy)

This blog is part of 3 part series on etcd

Part1: Cluster Setup

Part2: API and HA Application

Part3: Service Discovery and Proxy

Service Discovery

In a clustered environment services can be dynamically created and moved around due to reasons line node failure, maintenance or load rebalancing etc. In each of these cases the client need to discover the address the services running on the cluster. Using a service-registry to which the service must register itself once started normally solves this problem.

Etcd being a distributed and highly available data-store can be used as a service-registry to maintain records of services available in the cluster. Directory APIs exposed by Etcd provide a convenient way to group related configuration records.

Lets take a brief look at these APIs and how these APIs can be used to provide a Service Discovery mechanism.

The Directory APIs

Etcd stores data in a file-system like hierarchy, the key-value pairs can be stored in a directory and a directory can be nested within another. Directories are automatically created if they are part of the path of a key.

client > ./etcdctl set /Dir1/Dir2/key value
client > ./etcdctl ls /Dir1
client > ./etcdctl ls /Dir1/Dir2

Directories can also be created explicitly

client > ./etcdctl mkdir /dir3

Directories can have TTL values associated, when the timer expires the directory and its contents are recursively removed

client > ./etcdctl mkdir /dir4 --ttl 10

We can list and delete the contents of the Directory recursively

client > ./etcdctl ls /Dir1 --recursive=True

To create a hidden directory(or key) its name should start with an “_”

client > ./etcdctl mkdir /_dir5
client > ./etcdctl ls -p /

Another interesting directory feature is the support for ordered keys. This API allows the Directory to record the order in which the key value pair is added to it.

client > curl -XPOST -d value=Job1

client > curl -XPOST -d value=Job2                  

client > curl -XPOST -d value=Job3

Lets do a listing to view the keys

client > curl

Using Directory APIs for discovery

As seen in the previous section the directory APIs can be used to group together related records and create a hierarchy of records. These features of the data-store can be used to store information about service, the hierarchy of these records helps in creating a dependency tree like structure. As etcd is able to maintain the order of entries added to the directory it help is creating a queue like structure.


Lets try to create a hypothetical service record using these APIs. For our example lets, assume that all services will be registered under the “_services” directory. When the mysql service is deployed in the cluster the deployment process will create a service record for the database service as a directory ”db” with records such as “mysql:host” with value to the mysql server host amd “mysql:port” with the value of the port used by mysql and so on.

Similarly other services deployed in the cluster should register themselves by creating directories under the “_services” directory.

├── db
│   ├── mysql:host
│   ├── mysql:password
│   ├── mysql:port
│   └── mysql:user
└── web
    ├── apache:host
    └── apache:port

For a client to discover all the services available on the cluster all it needs to do is to list the contents of “_services” directory.

Etcd Proxy

With service discovery we solved the problem of publishing the services available on various nodes on the cluster, but as etcd is a distributed services itself the client needs to know which nodes to send the Etcd commands to and what id the node that the client was communicating with fails.

When a client tries to connect to an Etcd cluster it must know about the cluster Nodes and try connecting to each of them till the connection succeeds. If the cluster is augmented with more nodes at a later time, the clients need to be updated about the new nodes


Etcd proxy tries to resolve this problem by running an instance of etcd locally on the client node. The Etcd instance on the client node acts as a proxy to the real Etcd cluster and forwards all calls to the real cluster. The proxy instance does not participate in quorum formation or replication of configuration data.

This means that the client need not know about the cluster nodes or which node in the cluster is actually active and which one has failed.

To start a etcd instance as a proxy the –proxy flag should be set to “on” or “readonly. In addition the proxy needs to be told about the initial cluster members with the –initial-cluster flag. More details can be found in the documentation.


In this post we looked at the Directory APIs provided by Etcd and how those APIs can be used to provide a service discovery mechanism. We also looked at the proxy mode in Etcd, which helps in simplifying client configuration.


Published by

Chandan Dutta Chowdhury

Software Engineer

2 thoughts on “Test Driving Etcd – part 3 (Service Discovery and Proxy)”

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s