PostgreSQL Connection Timed Out OperationalError on Django and new server

Question

After my server got compromised, I've had to migrate to a new droplet - both on DigitalOcean, Debian Jessie, so I shouldn't have the Heroku issues that a lot of the similar questions seem to cover.

I copied the database over using pg_dump, and it seems intact. PostgreSQL is installed, and running, but whenever I try to utilise the database from the Django ORM, I get the error -

OperationalError: could not connect to server: Connection timed out
Is the server running on host "localhost" (127.0.0.1) and accepting
TCP/IP connections on port 5432?

So, some obvious things first.

Netstat shows:

tcp        0      0 127.0.0.1:5432          0.0.0.0:*               LISTEN      613/postgres    
tcp        0      0 127.0.0.1:6379          0.0.0.0:*               LISTEN      554/redis-server 12

I can see several PostgreSQL process in top and using ps auxw | grep postgres:

 1072 postgres  20   0  234680 129672 126676 S   6.0  3.2   3:06.98 postgres: me databasename [local] COPY                                          +
  640 postgres  20   0  227000 119564 117876 S   0.3  2.9   0:03.77 postgres: checkpointer process                                                  +
  613 postgres  20   0  226712  21252  19824 S   0.0  0.5   0:00.05 /usr/lib/postgresql/9.4/bin/postgres -D /var/lib/postgresql/9.4/main -c config_f+
  641 postgres  20   0  226712   5556   4080 S   0.0  0.1   0:02.59 postgres: writer process  

systemctl shows

postgresql.service                   loaded active exited 
[email protected]          loaded active running   

psycopg2 is installed, and I'm working from inside the virtualenv.

Any queries I run from shell using the Django ORM return the above error. I can access and run queries from psql, and it returns data fine, so PostgreSQL appears to be running, but Django can't reach it for whatever reason. The password and username are correct, although I think that would throw a different error in anycase. Restarting the service it makes no difference.

My Django settings are:

DATABASES = {
"default": {
    "ENGINE": "django.db.backends.postgresql_psycopg2",
    "NAME": "databasename",
    "USER": "databaseusername",
    "PASSWORD": "databasepasswordhere",
    "HOST": "localhost",
    "PORT": "5432",
    "CONN_MAX_AGE":360,
}
}

(I've tried both localhost and 127.0.0.1 as host, in case it was an IPv4/IPv6 resolution thing, albeit unlikely) pip freeze shows:

Django==1.9.6
argparse==1.2.1
click==6.6
django-extensions==1.6.7
django-redis==4.4.3
django-redis-cache==1.6.5
django-rq==0.9.1
django-rq-dashboard==0.3.0
gunicorn==19.5.0
hiredis==0.2.0
newrelic==2.64.0.48
nltk==3.2.1
oauthlib==1.1.1
psycopg2==2.6.1
requests==2.10.0
requests-oauthlib==0.6.1
rq==0.6.0
six==1.10.0
tweepy==3.5.0
wsgiref==0.1.2

Per this question, I've checked that PostgreSQL, PostgreSQL-contrib, libpq-dev Python-dev are installed.

Testing the connection on command-line as per this question throws the same error.

The real kicker here is - this is all working on the old server, and I can run ORM queries just fine - but it's not a secure server anymore, package manager is broken, so I need to abandon it, and need the new one to work. I can't see any differences in configuration between the two - I wondered if there was a permissions issue, but I get the same issue running as root/via sudo as my regular user.

I'm sure there's a simple configuration setting I've missed, but I'm pretty stumped at the moment!

Edit:

Grabbing the PID of postgres main process and running lsof gives this, though:

postgres 613 postgres 6u IPv4 11589 0t0 TCP localhost:postgresql (LISTEN) postgres 613 postgres 7u unix 0xffff8800d8e1cc00 0t0 11591 /var/run/postgresql/.s.PGSQL.5432

Which seems weird. Is it defaulting to a Unix socket there?

Solution

A connection timeout error suggests that the database server is not accessible with your settings over the network in a way that does not return an immediate error. The best approach, in my experience, is to troubleshoot the two sides of the connection separately. This means testing the server-side of the connection from the psql command line first, and then testing the django side once that is resolved.

I assume here you can install psql.on the server.

Immediate checks:

1. Can you ping the server from the server? If not, start there in troubleshooting the network. (I see this is localhost but still try -- it takes very little time and at least verifies assumptions).
2. Can you connect to the server via psql using the same port, ip address, etc? If not correct errors you get while trying that, try to fix those first. Chances are the same problems will occur here too. You may need to check firewalling and postgresql configuration if this times out.
3. Once these are working, then start trying from django again. Then you know it is Django/Python specific and no server-side.