Ubuntu – apache listening on 80, but not on any other port

i wanted to change a working apache configuration of example.com (exampleIP = 1.2.3.4 ) to change from the default port 80 to port 8001, such that http://example.com:8001 should work. I wasn't able to do so, and will document what i have attempted. I think I might need help on iptables.

my /etc/hosts firstly is fine

1.2.3.4    example.com

I started with replacing the port 80 with 8001 in the following places

1. /etc/apache2/ports.conf

   Listen 1.2.3.4:8001

2. /etc/apache2/conf.d/virtual.conf

   NameVirtualHost 1.2.3.4:8001

3. /etc/apache2/sites-enabled/example.com

   <VirtualHost 1.2.3.4:8001>

         ServerName example.com:8001
         #UPDATE: ServerName example.com doesn't make a difference either
   

   </VirtualHost>

When i replace 8001 in the above 3 cases with 80, it works. with 8001 i can't establish a connection. tcpdump also doesnt show incoming requests.

Since apache daemon when restarted did not throw any errors, i tried confirmation if the webserver was listening on 8001

$ sudo lsof -i |grep 8001
apache2     731     root    3u  IPv4 46858730       TCP example.com:8001 (LISTEN)
apache2     734 www-data    3u  IPv4 46858730       TCP example.com:8001 (LISTEN)
apache2     736 www-data    3u  IPv4 46858730       TCP example.com:8001 (LISTEN)
apache2     737 www-data    3u  IPv4 46858730       TCP example.com:8001 (LISTEN)
apache2     738 www-data    3u  IPv4 46858730       TCP example.com:8001 (LISTEN)
apache2     739 www-data    3u  IPv4 46858730       TCP example.com:8001 (LISTEN)

It seemed to listen on 8001. I even attempted to apply the following iptable rules into the terminal:

$ sudo iptables -A INPUT -p tcp --dport 8001 -j ACCEPT
$ sudo iptables -A OUTPUT -p tcp --sport 8001 -j ACCEPT
$ sudo iptables -A INPUT -m limit --limit 5/min -j LOG --log-prefix "iptables denied:  " --log-level 7

i never got a denied log. Here is what iptables verbose mode shows

$ sudo iptables -L -v -n
Chain INPUT (policy ACCEPT 3052M packets, 785G bytes)
pkts bytes target     prot opt in     out     source               destination         
   25  1690 ACCEPT     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0           tcp dpt:8001 
    0     0 ACCEPT     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0           tcp dpt:8001 
   92  6484 LOG        all  --  *      *       0.0.0.0/0            0.0.0.0/0           limit: avg 5/min burst 5 LOG flags 0 level 7 prefix `iptables denied:  ' 

Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         

Chain OUTPUT (policy ACCEPT 4317M packets, 695G bytes)
 pkts bytes target     prot opt in     out     source               destination         
   20  1760 ACCEPT     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0           tcp spt:8001 

I'm not familiar with iptable rules, so hints on any rules i missed above will be great. Reading other threads i also wondered if this could be a reason.

$ cat /proc/sys/net/ipv4/conf/eth0/forwarding 
0

Given this information, any hints on what could be stopping the webserver running on 8001 but not on 80 ?

UPDATE 1:
i tried flushing all iptable rules

$ sudo iptables -X
$ sudo iptables -F

i also tried to see if tcpdump was catching anything

 sudo tcpdump -i eth0 port 8001 -v

It doesn't on 8001, but change the port to 80 ( at the ^ 3 places again ) and it does.

i tried to look for established and listening process's

$ netstat -an
tcp        0      1.2.3.4:8001            0.0.0.0:*               LISTEN   

Lastly from my local machine, i tried curl as well

curl http://1.2.3.4:8001 -v
* About to connect() to 1.2.3.4 port 8001 (#0)
*   Trying 1.2.3.4... No route to host
* couldn't connect to host
* Closing connection #0
curl: (7) couldn't connect to host

UPDATE 2

In addition my /etc/log/messages that catches iptables errors seemed to throw the following. but it turns out that these appear when you enter and exit tcpdump.

Jan 22 09:30:31 node1 kernel: device eth0 entered promiscuous mode
Jan 22 09:30:31 node1 kernel: audit(1264152631.798:54): dev=eth0 prom=256 old_prom=0 auid=4294967295
Jan 22 09:30:33 node1 kernel: device eth0 left promiscuous mode

UPDATE 3
found this on tcpdump faq's

…This might be because the interface
on which you're capturing is plugged
into a switch; on a switched network,
unicast traffic between two ports will
not necessarily appear on other ports
– only broadcast and multicast traffic will be sent to all ports…

…which would indicate that if you
sniff on a 10Mb port, you will not see
traffic coming sent to a 100Mb port,
and vice versa…

…If your machine is not plugged into
a switched network or a dual-speed
hub, or it is plugged into a switched
network but the port is set up to have
all traffic replicated to it, the
problem might be that the network
interface on which you're capturing
doesn't support "promiscuous" mode, or
because your OS can't put the
interface into promiscuous mode…

so to summarize this peculiar case :

• iptables have been flushed with -F, -X
• requests fine with apache listening at 1.2.3.4:80
• tcpdump catches nothing with apache listening at 1.2.3.4:8001 ( see UPDATE 1,3 )