Hostnames in HAProxy configuration file

haproxyhostnamehostsload balancing

My haproxy.cfg file has two backend servers using hostnames:

server ops-ca-revealv2e-prod-1 ops-ca-revealv2e-prod-1:443 cookie ops-ca-revealv2e-prod-1 ssl weight 1 maxconn 512 check

server ops-ca-revealv2e-prod-2 ops-ca-revealv2e-prod-2:443 cookie ops-ca-revealv2e-prod-2 ssl weight 1 maxconn 512 check

These hostnames are part of Amazon OpsWorks and are injected into /etc/hosts automatically whenever an instance comes up or down. If I attempt to restart HAProxy when one of the instances is down, I receive the error:

[ALERT] 362/225440 (27202) : parsing [/opt/haproxy-ssl/haproxy.cfg:42] : 'server ops-ca-revealv2e-prod-2' : invalid address: 'ops-ca-revealv2e-prod-2' in 'ops-ca-revealv2e-prod-2:443'
[ALERT] 362/225440 (27202) : Error(s) found in configuration file : /opt/haproxy-ssl/haproxy.cfg
[ALERT] 362/225440 (27202) : Fatal errors found in configuration.

Is there a way to tell HAProxy to check if a hostname is valid? If it is valid, use it, if not, ignore it.

Best Answer

In haproxy >= 1.7, you should be able to use the init-addr option, specifying none to prevent DNS resolution at startup.

From the docs:

init-addr {last | libc | none | <ip>},[...]*

Indicate in what order the server's address should be resolved upon startup if it uses an FQDN. Attempts are made to resolve the address by applying in turn each of the methods mentionned in the comma-delimited list. The first method which succeeds is used. If the end of the list is reached without finding a working method, an error is thrown. Method "last" suggests to pick the address which appears in the state file (see "server-state-file"). Method "libc" uses the libc's internal resolver (gethostbyname() or getaddrinfo() depending on the operating system and build options). Method "none" specifically indicates that the server should start without any valid IP address in a down state. It can be useful to ignore some DNS issues upon startup, waiting for the situation to get fixed later. Finally, an IP address (IPv4 or IPv6) may be provided.

So your config line could be:

server s1 myhostname init-addr none

1. Did you check that it works correctly?

Changes to hosts should take effect immediately, but Windows caches name resolution data so for some time the old records may be used. Open a command line (Windows+R, cmd, Enter) and type:

ipconfig /flushdns

To drop the old data. To check if it works, use (assuming you have an ipv4 entry in your hosts for www.example.com, or an ipv6 entry in your hosts for ipv6.example.com):

ping -4 www.example.com -n 1
ping -6 www.example.com -n 1

And see if it uses the correct IP. If yes, your hosts file is fine and the problem is elsewhere.

Also, you can reset the NetBios cache with (open the console as an admin or it will fail):

nbtstat -R

You can check the current data in the DNS cache with:

ipconfig /displaydns | more

NB: nslookup does not look at the hosts file. See NSLOOKUP and NBLOOKUP give one IP address; PING finds another

2. Basics

Is your hosts file named correctly? It should be hosts and not host, etc.
Is the extension correct? It should have no extension (hosts not hosts.txt) - be careful if you have configured windows to hide known extensions, check the properties to be sure: The correct hosts file's type will show up as just "File".
Did you follow the correct syntax? Did you accidentally prefix lines with a hash (#) which indicates comments?
Did you take care of all variants (www.example.com and example.com - safest to just add both)?

3. Whitespace

The format for each line is IP address, then a horizontal tab (escape code \t, ASCII HT, hex 0x09) or a single space (hex 0x20), then the host name, ie. www.example.com, then finally a carriage return followed by a line feed, (escape codes \r\n, ASCII CRLF, hex 0x0d 0x0a).

Sample entries, using Unicode control pictures to indicate control characters. (Don't copy and paste these into your hosts file!)

192.0.2.1␉www.example.com␍␊
2001:db8:8:4::2␉ipv6.example.com␍␊

The individual bytes may be viewed in Notepad++ with the hex editor plugin. Notepad++ will also show special characters (View -> Show Symbol) so you can easily inspect the number and kind of whitespace characters.

If you copied and pasted hosts entries from somewhere, you may end up with multiple spaces. In theory hosts supports multiple spaces separating the two columns, but it's another thing to try if nothing else works.

To be on the safe side, make sure all lines in your hosts file either use tabs or spaces, not both.

Lastly, terminate the file with a blank line.

4. Registry Key

There is a registry key specifying the location of the hosts file. Supposedly, Windows doesn't actually support putting the hosts file in other locations, but you might want to check. The key is:

\HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters\DataBasePath

The entry should be:

%SystemRoot%\System32\drivers\etc

Or, in a Command Prompt window, type:

reg query HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters -v DataBasePath

which should display something similar to:

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters
DataBasePath    REG_EXPAND_SZ    %SystemRoot%\System32\drivers\etc

5. Permissions

Sometimes there are issues with permissions on the file, the file attributes, and similar things. To recreate the file with default permissions:

Create a new text file on your desktop.
Copy and paste the contents of your current hosts file into this file in Notepad.
Save the new text file and rename it to hosts.
Copy (do not move) the file to your %SystemRoot%\System32\drivers\etc directory, and overwrite the old file.

Last point is important: Copying works, moving doesn't.

The local Users account must be able to read the hosts file. To make sure (in Windows 7):

Navigate to %SystemRoot%\System32\drivers\etc in Windows Explorer.
If you can't see the hosts file, ensure you can see hidden and system files.
Right-click on the hosts file and select Properties from the context menu.
In the hosts Properties window, click on the Security tab.
Examine the list of names in the Group or user names: box. If %COMPUTERNAME%\Users is present, click on it to view permissions.
If Users is not present, or is present but does not have Read permission, click Edit....
If Users is not present, click Add..., type Users, click Check Names, and click OK or press Enter.
Select Users, and ensure Read & execute is checked in the Allow column. Click OK. If a Windows Security alert box pops up, choose Yes to continue.
Click OK to close the hosts Properties window.
Go up to section 1 of this answer and follow the directions to check if it's working now.

Or, in a Command Prompt window, type:

icacls %SystemRoot%\System32\drivers\etc\hosts

which should display something like:

C:\WINDOWS\System32\drivers\etc\hosts NT AUTHORITY\SYSTEM:(F)
                                      NT AUTHORITY\SYSTEM:(I)(F)
                                      BUILTIN\Administrators:(I)(F)
                                      BUILTIN\Users:(I)(RX)
                                      APPLICATION PACKAGE AUTHORITY\ALL APPLICATION PACKAGES:(I)(RX)
                                      APPLICATION PACKAGE AUTHORITY\ALL RESTRICTED APPLICATION PACKAGES:(I)(RX)

You should see an (R) after BUILTIN\Users.

6. Encoding

The hosts file should encoded in ANSI or UTF-8 without BOM. You can do this with File -> Save As.

7. Proxies

If you have a proxy configured, it may bypass the hosts file. The solution is to not use the proxy, or configure it to not do this.

To check, go to your Internet Explorer -> Internet Options -> Connections -> LAN settings. If everything is blank and "Automatically detect settings" is checked, you aren't using a proxy.

If you rely on a proxy to access the web and therefore don't want to disable it, you can add exceptions by going to Internet Explorer -> Internet Options -> Connections -> LAN settings -> Proxy Server / Advanced. Then add your exceptions to the Exceptions text box. e.g. localhost;127.0.0.1;*.dev

8. DNS address

(This may also resolve proxy issues.)

Go to your network connections properties, then TCP/IP settings, and change the first DNS server to 127.0.0.1 (localhost). The second should probably be your actual DNS's IP.

This is not necessary for the hosts file to work, but it may help in your case if something is configured strangely.

9. .local addresses

If you are using a .local domain entry in the form of myhost.local and it gets ignored please try the following:

x.x.x.x myhost.local www.myhost.local

even if the www.myhost.local does not exist. Windows somehow does not append its workgroup or localdomain.

10. Line / count limits

(added to this answer to make it visible as it's been mentioned a few times)

Windows hosts file seems to have a line or host limit. If you have more than 150 characters on a line or more than 8 hosts entries for an IP create a new line

e.g. instead of:

1.2.3.4 host1.com host2.com host3.com host4.com host5.com host6.com host7.com host8.com host9.com

Try this:

1.2.3.4 host1.com host2.com host3.com host4.com host5.com
1.2.3.4 host6.com host7.com host8.com host9.com

Dynamic HAProxy configuration

If you authorize your stats socket, you can send commands to it.

From the 1.4 documentation

disable server <backend>/<server>

Mark the server DOWN for maintenance. In this mode, no more checks will be performed on the server until it leaves maintenance. If the server is tracked by other servers, those servers will be set to DOWN during the maintenance.

In the statistics page, a server DOWN for maintenance will appear with a "MAINT" status, its tracking servers with the "MAINT(via)" one.

Both the backend and the server may be specified either by their name or by their numeric ID, prefixed with a sharp ('#').

This command is restricted and can only be issued on sockets configured for level "admin".

You can use this to effectively add and remove servers from active service.

Barring that, you can use option httpchk and block the check requests on the appserver side when a server should leave or join the balancing group. This requires some means to centrally manage that, of course.