Removing a stale CoreOS node from discovery.etcd.io

coreosetcd

I removed a node from a CoreOS cluster and brought up a fresh one in its place. Unfortunately, it picked up the same IP as the old machine.

This stopped etcd from discovering the rest of the cluster, with messages like this in the journal: "failed: fail on join request: Peer address has existed"

How can I remove the old node from discovery.etcd.io?

Best Answer

Short answer if you know the machine id

Send an HTTP DELETE request to https://discovery.etcd.io/<cluster-id>/<machine-id>

e.g.

curl https://discovery.etcd.io/abcdef0123456789abcdef0123456789/7cf9a5cf5e3b4baea82e28618bffeaf5 -XDELETE

Longer answer - how to find the machine id to delete

First, fetch the JSON object from your discovery URL, e.g. 

https://discovery.etcd.io/abcdef0123456789abcdef0123456789

It will look a bit like this:

{
  "action": "get",
  "node": {
    "key": "\/_etcd\/registry\/abcdef0123456789abcdef0123456789",
    "dir": true,
    "nodes": [
      {
        "key": "\/_etcd\/registry\/abcdef0123456789abcdef0123456789\/6148dbb812a44dbe8773bebf329634e7",
        "value": "http:\/\/10.132.47.218:7001",
        "expiration": "2015-04-15T17:58:12.753046544Z",
        "ttl": 598570,
        "modifiedIndex": 453369429,
        "createdIndex": 453369429
      },
      {
        "key": "\/_etcd\/registry\/abcdef0123456789abcdef0123456789\/646fbdaee73544e6ac289894e935f0c7",
        "value": "http:\/\/10.132.47.218:7001",
        "expiration": "2015-04-15T18:30:03.08506867Z",
        "ttl": 600480,
        "modifiedIndex": 453418705,
        "createdIndex": 453418705
      },
      {
        "key": "\/_etcd\/registry\/abcdef0123456789abcdef0123456789\/05e0decf1d9240819382db7a7f8ff2e7",
        "value": "http:\/\/10.132.58.166:7001",
        "expiration": "2015-04-14T23:47:31.402148037Z",
        "ttl": 533129,
        "modifiedIndex": 451690943,
        "createdIndex": 451690943
      },
      {
        "key": "\/_etcd\/registry\/abcdef0123456789abcdef0123456789\/af2783b2327e4f3a9b6e7ea169814a06",
        "value": "http:\/\/10.132.58.167:7001",
        "expiration": "2015-04-14T23:47:38.676204353Z",
        "ttl": 533136,
        "modifiedIndex": 451691169,
        "createdIndex": 451691169
      }
    ],
    "modifiedIndex": 426955695,
    "createdIndex": 426955695
  }
}

Let's say our reused IP is 10.132.47.218 - we want to identify the machine id which corresponds to that. We can see from the key this is 6148dbb812a44dbe8773bebf329634e7

Now we can delete this simply using a DELETE verb 

curl https://discovery.etcd.io/abcdef0123456789abcdef0123456789/7cf9a5cf5e3b4baea82e28618bffeaf5 -XDELETE
Related Topic