CoreOS: Inject a real private IP in etcd2 cloud-config

cloud-initcoreosetcdrackspace-cloud

I have three networks on Rackspace:

  1. Public network
  2. Service network 10.181.XXX.XXX (they call it a private network sometimes but it is not private per-se, it is datacenter-wide private so their tenants may share it)
  3. A real private network of 192.168.3.0/24 created via the networking UI

My plan is to launch a secure CoreOS cluster whose etcd peers and clients are limited to the real private networking so it is not available for the outside world.

So, I launched a CoreOS server and it had three interfaces:

  1. eth0 has the IP address of the public network – good
  2. eth1 has the IP address of the service network – good
  3. eth2 has the IP address of the private network – great!

All looked well but etcd is bound to the service network instead of the private one I created manually. Here's a piece of cloud-config:

#cloud-config
    coreos:
      etcd2:
        discovery: https://discovery.etcd.io/XXX
        advertise-client-urls: http://$private_ipv4:2379,http://$private_ipv4:4001
        initial-advertise-peer-urls: http://$private_ipv4:2380
        listen-client-urls: http://$private_ipv4:2379,http://$private_ipv4:4001
        listen-peer-urls: http://$private_ipv4:2380

Which means that $private_ipv4 variable gets expanded to eth1 IP address, here's the contents of /etc/environment:

COREOS_PUBLIC_IPV4=166.78.XXX.XXX
COREOS_PRIVATE_IPV4=10.181.XXX.XXX

Ok, looks like Rackspace injects its own networks, it is explainable. But does this mean that etcd is locked only to the first two networks and there's no way to configure it to use the real private one?

I've tried a bunch of hacks such as this:

listen-peer-urls: http://`/usr/bin/ifconfig eth2 | /usr/bin/grep --word-regexp inet | /usr/bin/awk '{print $2}'`:2380

and others but neither got properly executed nor substituted in the cloud-config.

My questions are:

  1. Am I doing something weird in terms on CoreOS cluster setup field, so that I can not achieve the target easily and naturally? If so, how would you lay out a secure cluster whose etcd operates within a real private network?
  2. Are there ways to "inject" dynamic values into cloud-init file so it gets interpolated at runtime? The issue is that the IP address is not known beforehand so it needs to get gotten and injected somehow I think.

Best Answer

They also provide a way to create your own private network and attach your server to it. As explained KB Article ID: 2163. If you did that and had the servicenet connected interface removed, it would no longer be an issue. However, the caveat to this would be that you would no longer have a way to utilize servicenet related services. More on intricacies about that is their KB Article ID: 2250.

Or, you could configure your firewall so that only the services that you use/need servicenet for are able to reach your server. You could even venture into the world of orchestrating a true private network through a cloud vyatta appliance and still retain access to Rackspace services as explained in KB Article ID: 3454.

All of those articles can be found at: http://www.rackspace.com/knowledge_center/

Ultimately, you'd have to weigh what option works best for you.

HTH

Related Topic