Docker – CoreOS, Docker networking, more than one bridge

bridgecoreosdocker

I have a server with 4 ethernet-cards. With the switching hardware I put all 4 cards in different VLANs. With the default networking on CoreOS I get single bridge for the containers, with a private IP address range. And besides that, I can control the hosts networking, and forward ports.

What I want is to control which container connects to which physical port, without exposing the hosts networking stack to the containers. I thought of multiple bridges: one for each physical card, making the physical port of each card to a member of the respective bridge, and connect containers to the bridge I specify. I would have control over the VLAN membership of the containers.

But i'm not sure if docker can handle this. I know how to connect Docker to a custom bridge, but I couldn't find any hints how to connect it to several bridges. Can it do something like that? Is there any other solutions on how to control which VLAN my containers belong to?

thanks,

__
s.

Best Answer

It's not an easy solution but its doable with openvswitch running on your host machines. You want to do something like

http://fbevmware.blogspot.com/2013/12/coupling-docker-and-open-vswitch.html

Related Solutions

CoreOS – Bare-Metal VLAN Networking with CoreOS

Put something like this (adjusted for your nic) in your cloud-config file:

- name: 00-vlan1.netdev
  runtime: true
  content: |
    [NetDev]
    Name=vlan1
    Kind=vlan

    [VLAN]
    Id=1
- name: 00-vlan2.netdev
  runtime: true
  content: |
    [NetDev]
    Name=vlan2
    Kind=vlan

    [VLAN]
    Id=2
- name: 30-ens192.network
  runtime: true
  content: |
    [Match]
    Name=ens192

    [Network]
    DHCP=yes
    VLAN=vlan1
    VLAN=vlan2

This information come from two sources: the manpages for systemd-networkd and the coreos blog https://coreos.com/blog/intro-to-systemd-networkd

I tried this on one of my own systems since I also will need vlans soon. Works very well. Here is the result:

core@localhost ~ $ ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN 
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: ens192: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 00:0c:29:48:9e:49 brd ff:ff:ff:ff:ff:ff
    inet 10.9.8.123/24 brd 10.9.8.255 scope global dynamic ens192
       valid_lft 3200sec preferred_lft 3200sec
    inet6 fe80::20c:29ff:fe48:9e49/64 scope link 
       valid_lft forever preferred_lft forever
3: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN 
    link/ether 56:84:7a:fe:97:99 brd ff:ff:ff:ff:ff:ff
    inet 172.17.42.1/16 scope global docker0
       valid_lft forever preferred_lft forever
4: vlan1@ens192: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP 
    link/ether 42:cf:24:c7:c6:bd brd ff:ff:ff:ff:ff:ff
    inet6 fe80::40cf:24ff:fec7:c6bd/64 scope link 
       valid_lft forever preferred_lft forever
5: vlan2@ens192: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP 
    link/ether da:0d:c5:73:91:1d brd ff:ff:ff:ff:ff:ff
    inet6 fe80::d80d:c5ff:fe73:911d/64 scope link 
       valid_lft forever preferred_lft forever
core@localhost ~ $

Nginx – How to use nginx with coreos docker containers and fleet

You have three main options:

Use a specific host for routing

This would be similar to your set up described above, but I'd make sure your unit file stays on that host with MachineID=abc123.... It's not super HA however, since you have a single point of failure.

Use a Cloud Load Balancer

If you're on a cloud provider, use a LB and set up a health check on port 80. Only the machine running on 80 will be given traffic. You'll have a bit of downtime (seconds to minutes) if the LB doesn't detect your failure quickly enough.

Two Layered Routing

Something similar to https://github.com/robszumski/varnish_etcd, but you can do it with varnish/nginx/haproxy and confd, or vulcand. Basically, you have a routing layer on each machine, that redirects to the backend's location dynamically.

Best Answer

Related Solutions

CoreOS – Bare-Metal VLAN Networking with CoreOS

Nginx – How to use nginx with coreos docker containers and fleet

Related Topic