Test Driving Etcd – part 2 (API and HA Application)

This blog is part of 3 part series on etcd

Part1: Cluster Setup

Part2: API and HA Application

Part3: Service Discovery and Proxy

In my previous blog, we looked at Etcd a highly available distributed configuration data-store, in this post we will look at the API exposed by Etcd. We will use our previous setup to try these APIs. Finally we will develop a very simple HA application based on Etcd APIs and test service failover.

Etcd data API Overview


Etcd exposes REST based APIs for accessing the data-store, cluster configuration and statistics. We will mostly focus on the data-store API for this post.

Storing data in Etcd

Etcd stores data in key-value pairs; in the previous blog we have already looked at using etcdctl to store a key-value pair in the data-store. In this post we will use curl to interect with etcd. To store a key-value pair of “my_key”, “my_value” we use the following command.


client $ curl -L http://1.1.1.1:4001/v2/keys/my_key -XPUT -d value="my_value"

{"action":"set","node":{"key":"/my_key","value":"my_value","modifiedIndex":446,"createdIndex":446}}

Retrieving data

To read value of a key stored in the data-store use HTTP GET. Here is an example


client $ curl -v -L http://1.1.1.1:4001/v2/keys/my_key

{"action":"get","node":{"key":"/my_key","value":"my_value","modifiedIndex":476,"createdIndex":476}}

Deleting the data

Deleting data is as simple as using the DELETE verb with HTTP.


client $ curl -L http://1.1.1.1:4001/v2/keys/my_key -XDELETE

{"action":"delete","node":{"key":"/my_key","modifiedIndex":457,"createdIndex":446},"prevNode":{"key":"/my_key","value":"my_value","modifiedIndex":446,"createdIndex":446}}

Apart from the normal read/write/delete operation, Etcd also support some interesting data manipulation operation, which can be used to make cluster aware application. Lets look at some of this operation

Storing data with a TTL value

Etcd allow storage of key-value pair with an associated timeout. Here is an example usage.

client $ curl -L http://1.1.1.1:4001/v2/keys/my_key -XPUT -d value="my_value" -d ttl=10

{"action":"set","node":{"key":"/my_key","value":"my_value","expiration":"2015-05-27T06:02:06.385658433Z","ttl":10,"modifiedIndex":693,"createdIndex":693}}

This means once the data is stored in the data-store it will only be available for certain period of time(till the timeout values expires) following which the key-value pair is automatically removed from the data-store

client $ curl -L http://1.1.1.1:4001/v2/keys/my_key 

{"errorCode":100,"message":"Key not found", "cause":"/my_key","index":701}

Conditional update of data

Etcd allows conditional updated of key-value pair. This means a data update is allowed only if given criteria are met.

E.g. compare-and-swap operation which takes two values the previous value (P) and the new value(N) and allows updating the value associated with a key only if the previous value (P) matches the value of the key in the data-store

Lets take an example to explain this

client $ curl -L http://1.1.1.1:4001/v2/keys/my_key -XPUT -d value="my_value"

{"action":"set","node":{"key":"/my_key","value":"my_value","modifiedIndex":787,"createdIndex":787}}

client $ curl -L http://1.1.1.1:4001/v2/keys/my_key -XPUT -d value="my_value1" -d prevValue="my_value"

{"action":"compareAndSwap","node":{"key":"/my_key","value":"my_value1","modifiedIndex":796,"createdIndex":787},"prevNode":{"key":"/my_key","value":"my_value","modifiedIndex":787,"createdIndex":787}}

client $ curl -L http://1.1.1.1:4001/v2/keys/my_key -XPUT -d value="my_value1" -d prevValue="my_value"

{"errorCode":101,"message":"Compare failed","cause":"[my_value != my_value1]","index":797}

Another case of compare-and-update is the creation of new key, to make sure that the key did not exist in the date-store before the operation the prevExist attribute of the key should be set to “false”

client $ curl -L http://1.1.1.1:4001/v2/keys/my_key -XPUT -d value="my_value1" -d prevExist="false"

{"errorCode":105,"message":"Key already exists","cause":"/my_key","index":819}

The same logic applies for compare-and-delete operation which deletes the key-value pair if the old value supplied to the operation matches the one stored in the data-store

client $ curl -L http://1.1.1.1:4001/v2/keys/my_key -XDELETE -d prevValue="my_value1"
{"action":"delete","node":{"key":"/my_key","modifiedIndex":824,"createdIndex":787},"prevNode":{"key":"/my_key","value":"my_value1","modifiedIndex":796,"createdIndex":787}}</pre>

We have looked at very few of the data-store APIs, for a complete description of Etcd APIs have a look at the documentation

A Highly Available application using Etcd

Now that we have looked at the basic data-store APIs of Etcd let use these APIs to build a simple HA application. We will use the Etcd cluster setup that we built in the previous blog and extend it to include two server node which will be running our HA application. Following is the cluster setup for this test.

etcd-ha-app-setup

The application simple makes the shared service IP address of 1.1.1.50 highly available.

The application code is available here. The code will be running on both the servers (Server_1 and Server_2) one of them will act as the master node and host the service (Shared Service IP) in case of the master nodes failure the service will be moved to the backup node.

The following diagram shows the working of the application.

etcd-ha-app-flowchart

The Client node constantly tries to use the service (ping the Shared Service IP). Here is an example of service disruption during node switchover.

failover

The key to running a shared service by the two-node cluster in our example is acquiring the lock and the heart-beat mechanism. Lets look at the application code and see how Etcd API provides these clustering mechanisms.

function acquire_lock()
{
    $ETCDCTL mk /my_service $NODE --ttl $RES_TIME
    return $?
}

function heartbeat()
{
    $ETCDCTL set /my_service $NODE --swap-with-value $NODE --ttl $RES_TIME
    return $?
}

The application uses compare-and-update operation to setup a new key (my_service) it then uses data update with TTL value to keep holding the key. This prevents the second instance of the application to start the service. Etcd thus provides a distributed locking mechanism for the clustered application.

Conclusion

In this post, we briefly looked at some of the interesting data manipulation API exposed by Etcd. We used these APIs to develop a simple HA application. In the following post, we will look Etcd and service discovery.

Advertisements

Published by

Chandan Dutta Chowdhury

Software Engineer

2 thoughts on “Test Driving Etcd – part 2 (API and HA Application)”

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s