Unable to start nginx due to permission denied on a port

Question

I used to be able to start nginx on my AWS EC2, but now I get bind() to 0.0.0.0:3008 failed (13: Permission denied). This happens when nginx calls bind() in response to the configuration listen 3008 default_server, in /etc/nginx/nginx.conf.

Possible causes I have looked for are that AWS is blocking port 3008, that the port is in use or that the user running the service has insufficient privileges.

The answers to the following two related questions all come down to the second or third of those possibilities:

https://stackoverflow.com/questions/48478869/cannot-bind-to-some-ports-due-to-permission-denied https://stackoverflow.com/questions/39586692/nginx-error-bind-to-0-0-0-080-failed-permission-denied

As for the first of the three possible causes, I checked whether AWS is blocking port 3008 by deleting all but one security group other than the default (which allows all traffic). The inbound rules for this security group allow incoming TCP traffic on port 3008:

HTTP            TCP 80  0.0.0.0/0
HTTP            TCP 80  ::/0
SSH             TCP 22  0.0.0.0/0
SSH             TCP 22  ::/0
Custom TCP Rule TCP 3000 - 3030 0.0.0.0/0
Custom TCP Rule TCP 3000 - 3030 ::/0
HTTPS           TCP 443 0.0.0.0/0
HTTPS           TCP 443 ::/0
All ICMP - IPv4 All N/A 0.0.0.0/0
All ICMP - IPv4 All N/A ::/0
Custom TCP Rule TCP 8080 - 8084 0.0.0.0/0
Custom TCP Rule TCP 8080 - 8084 ::/0

To check whether port 3008 is in use, I ran $ sudo netstat -anp | grep 3008 and there is no output.

To ensure that privileges are sufficient, I added User=root and Group=root to the service file, ran ls -l on the service file to verify that its user and group are already root, and I ran systemctl start with sudo. Details below, in the form of commands and output, should answer any routine questions about the above, and about the overall environment in which this problem occurs. Thanks in advance for any suggestions.

Operating system:

$ uname -a
Linux ip-172-31-40-184.ec2.internal 3.10.0-957.12.1.el7.x86_64 #1 SMP Wed Mar 20 11:34:37 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

Attempting to start nginx:

$ sudo systemctl start nginx
Job for nginx.service failed because the control process exited with error code. See "systemctl status nginx.service" and "journalctl -xe" for details.

Checking status of nginx:

$ sudo systemctl status nginx
â— nginx.service - nginx - high performance web server
   Loaded: loaded (/usr/lib/systemd/system/nginx.service; disabled; vendor preset: disabled)
   Active: failed (Result: exit-code) since mié 2019-09-04 13:59:24 UTC; 32s ago
     Docs: http://nginx.org/en/docs/
  Process: 24450 ExecStart=/usr/sbin/nginx -c /etc/nginx/nginx.conf (code=exited, status=1/FAILURE)

sep 04 13:59:24 ip-172-31-40-184.ec2.internal systemd[1]: Failed to start nginx - high performance web server.
sep 04 13:59:24 ip-172-31-40-184.ec2.internal systemd[1]: nginx.service failed.

Errors logged by nginx (port 80 error was fixed by rebooting):

$ tail /var/log/nginx/error.log
2019/09/03 19:58:09 [emerg] 17319#17319: bind() to 0.0.0.0:3008 failed (13: Permission denied)
2019/09/03 19:58:59 [emerg] 17381#17381: bind() to 0.0.0.0:80 failed (98: Address already in use)
2019/09/03 19:58:59 [emerg] 17381#17381: bind() to 0.0.0.0:3008 failed (13: Permission denied)
2019/09/03 21:18:17 [alert] 7491#7491: unlink() "/var/run/nginx.pid" failed (2: No such file or directory)
2019/09/03 21:25:32 [emerg] 11207#11207: bind() to 0.0.0.0:3008 failed (13: Permission denied)
2019/09/03 22:30:21 [emerg] 16333#16333: bind() to 0.0.0.0:3008 failed (13: Permission denied)
2019/09/03 22:50:51 [emerg] 15980#15980: bind() to 0.0.0.0:3008 failed (13: Permission denied)
2019/09/04 01:31:57 [emerg] 9819#9819: bind() to 0.0.0.0:3008 failed (13: Permission denied)
2019/09/04 01:32:07 [emerg] 10095#10095: bind() to 0.0.0.0:3008 failed (13: Permission denied)
2019/09/04 01:32:12 [emerg] 10264#10264: bind() to 0.0.0.0:3008 failed (13: Permission denied)

The nginx service file (with User and Group added by hand):

$ ls -l /usr/lib/systemd/system/nginx.service 
-rw-r--r--. 1 root root 420 sep  3 22:50 /usr/lib/systemd/system/nginx.service

$ cat /usr/lib/systemd/system/nginx.service
[Unit]
Description=nginx - high performance web server
Documentation=http://nginx.org/en/docs/
After=network-online.target remote-fs.target nss-lookup.target
Wants=network-online.target

[Service]
User=root
Group=root
Type=forking
PIDFile=/var/run/nginx.pid
ExecStart=/usr/sbin/nginx -c /etc/nginx/nginx.conf
ExecReload=/bin/kill -s HUP $MAINPID
ExecStop=/bin/kill -s TERM $MAINPID

[Install]
WantedBy=multi-user.target

Configuration of nginx (including /etc/nginx/conf.d/default.conf, which specifies ):

$ cat /etc/nginx/nginx.conf

user  nginx;
worker_processes  1;

error_log  /var/log/nginx/error.log warn;
pid        /var/run/nginx.pid;


events {
    worker_connections  1024;
}


http {
    include       /etc/nginx/mime.types;
    default_type  application/octet-stream;

    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';

    access_log  /var/log/nginx/access.log  main;

    sendfile        on;
    #tcp_nopush     on;

    keepalive_timeout  65;

    #gzip  on;

    include /etc/nginx/conf.d/*.conf;

    server {
        listen 3008 default_server;
        root /home/ec2-user/webapp/debug/build;
        server_name search-demo.net;
        index index.html index.htm;
        location / {
        }

    listen 443 ssl; # managed by Certbot
    ssl_certificate /etc/letsencrypt/live/search-demo.net-0001/fullchain.pem; # managed by Certbot
    ssl_certificate_key /etc/letsencrypt/live/search-demo.net-0001/privkey.pem; # managed by Certbot
    include /etc/letsencrypt/options-ssl-nginx.conf; # managed by Certbot
    ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; # managed by Certbot

}
}

Other than the first part shown below, the included default nginx configuration is commented out:

$ cat /etc/nginx/conf.d/default.conf
server {
    listen       80;
    server_name  localhost;

    #charset koi8-r;
    #access_log  /var/log/nginx/host.access.log  main;

    location / {
        root   /usr/share/nginx/html;
        index  index.html index.htm;
    }

    #error_page  404              /404.html;

    # redirect server error pages to the static page /50x.html
    #
    error_page   500 502 503 504  /50x.html;
    location = /50x.html {
        root   /usr/share/nginx/html;
    }
...
}

Solution

To unlock port 3008:

sudo semanage port -a -t http_port_t  -p tcp 3008

The question could be marked as a duplicate. The way I searched was not bringing up solutions involving SELinux. After I started to suspect SELinux, a colleague found:

https://stackoverflow.com/questions/18480201/ubuntu-nginx-emerg-bind-to-0-0-0-080-failed-13-permission-denied#answer-39971725

Even after the port fix, SELinux was not done messing with nginx. Visitors would get a 403 (forbidden) error from nginx. To enable the nginx user to read the directory:

sudo setsebool -P httpd_can_network_connect on
chcon -Rt httpd_sys_content_t /path/to/www