Freebsd – One domain on Apache goes to default page without www

apache-2.4freebsdvirtualhost

I'm using Apache/2.4.16 on FreeBSD 10.2. I have two instances of Apache running, each serving several domain names. The server has two network interfaces, one for the public Internet, one only accessible through my internal LAN. All of the virtual host configuration files are nearly identical except for the domain names and directories. All of the sites work as expected, except for one: If I try to go to adultemployment.net, I get the server's default page, but if I specify any of the machine names configured in the DNS (www, ftp, mail, etc.) I get the site's page as expected, e.g., www.adultemployment.net displays correctly.

This is the virtual host configuration file for adultemployment.net:

<VirtualHost 100.0.193.102:80 192.168.14.202:80>
    ServerName AdultEmployment.net
    ServerAlias *.AdultEmployment.net
    ServerAdmin webmaster@AdultEmployment.net
    DocumentRoot "/path.to/adultemploy/www"
    <Directory "/path.to/adultemploy/www">
        Options All MultiViews
        AllowOverride All
        Require all granted
    </Directory>
    ErrorDocument 404 /cgibin/badurl.pl
    ErrorLog "|/usr/local/sbin/rotatelogs /path.to/adultemploy/logs/error_log 86400"
    CustomLog "|/usr/local/sbin/rotatelogs /path.to/adultemploy/logs/access_log 86400" combined
    <IfModule mod_alias.c>
        ScriptAlias /cgibin/ "/path.to/adultemploy/cgibin/"
        <Directory "/path.to/adultemploy/cgibin">
            AllowOverride None
            Options None
            Require all granted
        </Directory>
    </IfModule>
</VirtualHost>

This is the virtual host configuration file for another site (which works correctly), bidsadult.com:

<VirtualHost 100.0.193.102:80 192.168.14.202:80>
    ServerName BidsAdult.com
    ServerAlias *.BidsAdult.com
    ServerAdmin webmaster@BidsAdult.com
    DocumentRoot "/path.to/bidsadult/www"
    <Directory "/path.to/bidsadult/www">
        Options All MultiViews
        AllowOverride All
        Require all granted
    </Directory>
    ErrorDocument 404 /cgibin/badurl.pl
    ErrorLog "|/usr/local/sbin/rotatelogs /path.to/bidsadult/logs/error_log 86400"
    CustomLog "|/usr/local/sbin/rotatelogs /path.to/bidsadult/logs/access_log 86400" combined
    <IfModule mod_alias.c>
        ScriptAlias /cgibin/ "/path.to/bidsadult/cgibin/"
        <Directory "/path.to/bidsadult/cgibin">
            AllowOverride None
            Options None
            Require all granted
        </Directory>
    </IfModule>
</VirtualHost>

The default and virtual host configurations are set in httpd.conf:

# 'Main' server configuration
#
# The directives in this section provide defaults for any
#  <VirtualHost> containers you may define later in the file.
#
# All of these directives may appear inside <VirtualHost> containers,
# in which case these default settings will be overridden for the
# virtual host being defined.
#
ServerAdmin webmaster@occupant591.net
ServerName generic.occupant591
#
<Directory />
    AllowOverride none
    Require all denied
</Directory>
#
<IfModule dir_module>
    DirectoryIndex index.php index.cgi index.shtml index.htm index.html
</IfModule>
#
<Files ".ht*">
    Require all denied
</Files>
#
LogLevel info

<IfModule log_config_module>
    LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combined
    LogFormat "%h %l %u %t \"%r\" %>s %b" common
    LogFormat "%{Referer}i -> %U" referer
    LogFormat "%{User-agent}i" agent
    <IfModule ssl_module>
        LogFormat "%t %h %{SSL_PROTOCOL}x %{SSL_CIPHER}x \"%r\" %b" ssl_request
    </IfModule>
    <IfModule logio_module>
        LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\" %I %O" combinedio
    </IfModule>
</IfModule>

<IfModule cgid_module>
    Scriptsock logs/cgisock
</IfModule>

<IfModule mime_module>
    TypesConfig conf/mime.types
    AddType application/x-httpd-php         .php
    AddType application/x-tar               .tgz
    AddType text/plain                      .xml
    AddType image/x-icon                    .ico
    AddEncoding x-compress .Z
    AddEncoding x-gzip .gz .tgz
    AddType application/x-compress .Z
    AddType application/x-gzip .gz .tgz
    AddHandler cgi-script .cgi
    AddHandler cgi-script .pl
    AddType text/html .shtml
    AddOutputFilter INCLUDES .shtml
</IfModule>

<FilesMatch "\.php$">
    SetHandler application/x-httpd-php
</FilesMatch>
<FilesMatch "\.phps$">
    SetHandler application/x-httpd-php-source
</FilesMatch>

<Location /cgi-bin/phf*>
    Deny from all
    ErrorDocument 403 http://phf.apache.org/phf_abuse_log.cgi
</Location>
<Location /cgibin/phf*>
    Deny from all
    ErrorDocument 403 http://phf.apache.org/phf_abuse_log.cgi
</Location>

# 'Main' server configuration
#
# The directives in this file set up the values used by the 'main'
# server, which responds to any requests that aren't handled by any
# other <VirtualHost> definition.
#
Include /path.to.apache.config/conf/generic.site

# Virtual hosts
Include /path.to.apache.config/conf/ActiveSites/*.conf

The default site configuration is set by generic.site:

<VirtualHost 100.0.193.102:80 192.168.14.202:80>
    ServerAdmin webmaster@occupant591.net
    DocumentRoot "/path.to/generic/www"
    <Directory "/path.to/generic/www">
        Options All MultiViews
        AllowOverride All
        Require all granted
    </Directory>
    ErrorDocument 404 /cgibin/badurl-occ.pl
    ErrorLog "|/usr/local/sbin/rotatelogs /path.to/generic/logs-occ/error_log 86400"
    CustomLog "|/usr/local/sbin/rotatelogs /path.to/generic/logs-occ/access_log 86400" combined
    <IfModule mod_alias.c>
        ScriptAlias /cgibin/ "/path.to/generic/cgibin/"
        <Directory "/path.to/generic/cgibin">
            AllowOverride None
            Options None
            Require all granted
        </Directory>
    </IfModule>
</VirtualHost>

I've tried to fix this by copying a functional configuration file onto the one for adultemployment.net and updating the domain name and directory entries. That didn't change anything. I've tried several different sets of ServerName and ServerAlias entries, such as

    ServerName www.AdultEmployment.net
    ServerAlias *.AdultEmployment.net AdultEmployment.net

and

    ServerName www.AdultEmployment.net
    ServerAlias *.AdultEmployment.net

I thought the problem might be because adultemployment.net was alphabetically first in the ActiveSites directory, so I added another domain to precede it – and that site works as expected, but adultemployment.net still doesn't return its page without a machine name.

At first I thought it might be a browser cache issue, but I get the same behavior accessing the site from Firefox and IE on my w2K machine, Firefox on my w7 laptop, and lynx ssh'd into an outside hosting company's shell. I also tried clearing the arp entry for the domain on the server, to no avail.

I've tried tweaking the DNS entries, and looked for symlinks as suggested here or .htaccess entries as suggested here for similar problems, but nothing seems to fix this problem where one site on the server fails, but all of the others work correctly.

I've been trying to fix this problem for over a week, intermittently when I first discovered it, and almost continuously for the past two days. Any hints or pointers would be greatly appreciated!!

Best Answer

I stumbled across a "solution" (a work-around, actually, the fundamental cause is still unknown):

For this configuration (the second Apache configuration loaded at boot time), I had to add a ServerName entry duplicating the information in httpd.conf to the generic.site configuration file:

ServerName generic.occupant591

After adding the "default" ServerName entry, all of the sites served by this Apache configuration work correctly. If a similar entry is added to the other configuration (the first one loaded at boot time), the sites matching the "default" ServerName entry fail.

TL;DR;

If a visitor attempted to access a non-existent page on an unconfigured domain (i.e., one properly being served through the generic.site configuration) such as mature.bid/videos, a [core:info] entry was written into the error log with the message File does not exist: /web/generic-occ/www/videos because the generic site doesn't have a file or directory named videos and so Apache has nothing to serve to answer the request. While the error log entries illuminate the fact non-existent pages are being requested, they provide no domain information about the requests and are therefore really just noise.

I was using the stock Apache "combined" log format to generate the access logs. Again, the access logs indicated the pages being accessed, but not the domain they were being requested from. For normal configurations where the logs are recording information for a single domain, that's not a problem, but the generic.site configuration is designed to answer for all domains that are not otherwise configured, so logging the domain information is required in this instance to be useful. Fortunately, Apache provides options for logging domain info, and %v for "the canonical ServerName of the server serving the request" seemed just what I needed. I created a new LogFormat statement:

LogFormat "%h %l %u %t %v \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" host_combined

I restarted Apache, tried to load http://adultemployment.net again, and found the access log contained an entry including AdultEmployment.net "GET / HTTP/1.1" Great, I thought, that worked! - until I noticed the same text was logged when an unconfigured domain was accessed. Perhaps I should be using %V for "the server name according to the UseCanonicalName setting to get the domain information? I changed my LogFormat command to

LogFormat "%h %l %u %t %V \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" host_combined

then restarted Apache - and found the domains with non-existent pages were being logged as I expected. Now I have a record of which of the unconfigured sites are being accessed, and what pages visitors expect them to have.

This left me wondering why the LogFormat %v string logged "AdultEmployment.net" as "the canonical ServerName of the server serving the request" - none of the configuration files are written such that AdultEmployment.net should be the default server name. (For the other Apache configuration on this machine, the %v string logged is "Dreamer" - the O/S machine name for the server.) I can't find the page now, but during my earlier research I came across a situation where a similar problem was fixed by changing the domain name to not be the same as the operating system's server name. I tried ssh'ing in to the IP address for this Apache configuration to see if AdultEmployment.net was showing up as part of the operating system configuration - but it wasn't, as I expected.

As an experiment I expected to cause problems, I tried adding a ServerName entry to the generic.site configuration file:

ServerName Dreamer.occupant591.net

I thought this might fix adultemployment.net (it did), but I expected it would also break occupant591.net, causing it to display the unconfigured site page. However, it did not break the "host" site - in fact, I haven't been able to find anything failed now.

Since apache2ctl -S (as mentioned here) isn't available with Apache 2.4 on FreeBSD, I read the httpd man page, and found -S is an option I can pass directly to the daemon. With the ServerName entry in the generic.site configuration, the httpd -S report starts with

VirtualHost configuration:
192.168.14.202:80      is a NameVirtualHost
         default server Dreamer.occupant591.net (/path.to.apache.config/conf/generic.site:6)
         port 80 namevhost Dreamer.occupant591.net (/path.to.apache.config/conf/generic.site:6)
         port 80 namevhost AAAAAAAAA-Dummy.com (/path.to.apache.config/conf/ActiveSites/AAAAAAAA-Dummy.conf:2)
                 wild alias *.AAAAAAAAA-Dummy.com
         port 80 namevhost AdultEmployment.net (/path.to.apache.config/conf/ActiveSites/AdultEmployment.net.conf:3)
                 wild alias *.AdultEmployment.net

When I remove the ServerName entry from the generic.site configuration, the httpd -S report starts with

VirtualHost configuration:
192.168.14.202:80      is a NameVirtualHost
         default server AdultEmployment.net (/path.to.apache.config/conf/generic.site:6)
         port 80 namevhost AdultEmployment.net (/path.to.apache.config/conf/generic.site:6)
         port 80 namevhost AAAAAAAAA-Dummy.com (/path.to.apache.config/conf/ActiveSites/AAAAAAAA-Dummy.conf:2)
                 wild alias *.AAAAAAAAA-Dummy.com
         port 80 namevhost AdultEmployment.net (/path.to.apache.config/conf/ActiveSites/AdultEmployment.net.conf:3)
                 wild alias *.AdultEmployment.net

Note that without the ServerName entry, the default server is still supposedly being set at generic.site:6 even though there is no ServerName entry in the file, at line 6 or any other line in the file. Clearly something is broken but I don't have time to chase this ghost any more right now.

When I was initially setting up this Apache configuration with two instances on the server, I had a ServerName entry in the other instance's configuration that included the full domain name. That caused the "host" site to break, and I fixed that problem by removing the top-level domain component from the ServerName entry in httpd.conf, and removing the ServerName entry altogether from the generic.site configuration file. I made the same changes to this configuration, which seemed to work until approximately when I reconfigured my network to get rid of my ISP's router. The next time I tried to access adultemployment.net, I found the problem described in my question.

Now I'm puzzled by why the two instances of Apache behave differently - but I'm going to have to put off that investigation for when I have nothing to do and a staff to do it with...

Related Topic