Port 80 connection refused

Question

i can not connect to port 80 on my webserver. my iptables are in the default state:

Chain INPUT (policy ACCEPT)
target     prot opt source               destination         

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination         

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination 

when i start different servers (nginx, nodejs ...) i can make them listen to port 80, but trying to access, i always get "connection refused". Listening to any other port (81,8080 whatever) works perfectly fine. Only port 80 is somehow blocked. Accessing port 80 via localhost does work, so for testing purpose i even switched of the external firewall, still no luck. What can i do to find out who is blocking this port 80?

as requested the output of netstat -tlpn (when running nginx on port 80):

Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 0.0.0.0:21              0.0.0.0:*               LISTEN      710/vsftpd      
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      1179/sshd       
tcp        0      0 0.0.0.0:25              0.0.0.0:*               LISTEN      1661/master     
tcp        0      0 0.0.0.0:5984            0.0.0.0:*               LISTEN      980/beam.smp    
tcp        0      0 87.106.64.11:3306       0.0.0.0:*               LISTEN      1346/mysqld     
tcp        0      0 0.0.0.0:80              0.0.0.0:*               LISTEN      3274/nginx: master 
tcp6       0      0 :::22                   :::*                    LISTEN      1179/sshd       
tcp6       0      0 :::25                   :::*                    LISTEN      1661/master  

Solution 1

i did a tcptraceroute 87.106.64.11 80 and its definitely my server that is blocking the port. At some point in the trace i got this:

s18132051.onlinehome-server.info (87.106.64.11) [closed]

So i reset my ip tables, and that did the trick.

So either there where some hidden rule in the table, or iptable -L did not give me all rules. I will mark this as the answer as it fixes the problem.
I would still love to hear, how come i didn't get any blocking rules when doing iptables -L

Solution 2

I experienced the same problem, but on Debian 8.4 (Jessie). Like the above, the solution was the IPTables flush script as listed at http://insanelabs.net/linux/linux-reset-iptables-firewall-rules/. Though iptables reported no rules, there must have been some "hidden" rules, or otherwise a bug in iptables itself. I am reporting this bug to the Debian maintainers.

In case the linked site goes down, here is the full text of the script in question, reproduced here for convenience.

#!/bin/sh
echo "Flushing iptables rules..."
sleep 1
iptables -F
iptables -X
iptables -t nat -F
iptables -t nat -X
iptables -t mangle -F
iptables -t mangle -X
iptables -P INPUT ACCEPT
iptables -P FORWARD ACCEPT
iptables -P OUTPUT ACCEPT

Solution 3

You didn't list all of your rules. I've found a good way to get any active netfilter tables is:

for blah in sudo /bin/cat /proc/net/ip_tables_names; do sudo /sbin/iptables -t $blah -L -vn --line-numbers|sed "s/^/$blah: /" ; done |less -RXF

Another options is to run:

/sbin/iptables-save

If you have the patience, you can go through each rule one at a time and change any DENY targets to ACCEPT and see which one fixes the problem.

Comments

[mekdigital]

I have tried so many things, thanks!!! I believe I put myself in a "broken" state by playing around with Microk8s. I created an ingress for the cluster and that action must have written the hidden rule! I wasted two days :)