Cannot connect to Cloudera Manager, not listening on port 7180

Question

I'd really appreciate some help to get cloudera manager running on AWS EC2. Its my first install, and I'm aiming to use the AWS Free Tier to spin up a few nodes and do some training on Hadoop cluster and the cloudera distribution. I'm using the RedHat RHEL 7.2 image on AWS EC2.

I am following the instructions here... Cloudera Manager installation

I have installed cloudera manager OK, and get to the screen where it invites you to use a browser to log-in to the cloudera manager server. But that's where the problem starts. It seems the app is not listening on port 7180, so there's no hope of connecting from another machine across the network. I can't even connect locally, on the server, yet the service appears to be running OK. But its not listening on port 7180.

Q1 - How can I confirm the config is set to use port 7180.?

Q2 - are there obvious steps that I'm missing here ?

Thanks in advance,

[Edit..] I'm beginning to wonder if the Free EC2 host is running short on memory to run cloudera manager. I saw one comment that implied that....AWS Forum post . But the process doesn't crash or report any problems in its logfile. So it must be OK, right?

[Edit.... with more diagnostic info....]

Here's a list of the diagnostics I've checked:-

• SELinux is not running [for install and testing purposes],
• WAN firewalls,
• EC2 firewall/Security group,
• Local firewall on server,
• Cloudera manager log,
• Is the service up and running?
• Can you connect locally?

Securtity group on the EC2 instance, it contains:- SSH and Port 7180,

Firewall/iptables/firewalld on the RedHat instance, tried:- adding ports to iptables, then dissabling iptables, then adding ports to firewalld, then dissabling the firewalld service,

$ sudo iptables -L
Chain INPUT (policy ACCEPT)
target     prot opt source               destination         
ACCEPT     all  --  anywhere             anywhere             ctstate RELATED,ESTABLISHED
ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:ssh
ACCEPT     tcp  --  anywhere             anywhere             state NEW tcp dpt:7180
ACCEPT     tcp  --  anywhere             anywhere             state NEW tcp dpt:7182

But I'm getting the feeling that the installation of cloudera manager is not happy, or not running correctly.

I've checked the cloudera manager log, and it ends with the following.

$ tail /var/log/cloudera-scm-server/cloudera-scm-server.log
2016-02-25 11:02:23,581 INFO main:com.cloudera.cmon.components.MetricSchemaUpdate: persisting 19264 new metrics
2016-02-25 11:02:28,920 INFO main:com.cloudera.cmon.components.MetricSchemaUpdate: persisting 0 updated metrics
2016-02-25 11:02:28,924 INFO main:com.cloudera.cmon.components.MetricSchemaManager: Cross entity aggregates processed.

And when I use tail -f, and restart the cloudera-scm-server service, the log scrolls a lot, and comes back the same state. If I search for ERROR, there are no lines with "ERR".

$ sudo service cloudera-scm-server start
Starting cloudera-scm-server (via systemctl):              [  OK  ]

$ sudo systemctl status cloudera-scm-server
● cloudera-scm-server.service - LSB: Cloudera SCM Server
   Loaded: loaded (/etc/rc.d/init.d/cloudera-scm-server)
   Active: active (exited) since Thu 2016-02-25 12:23:03 EST; 44s ago
     Docs: man:systemd-sysv-generator(8)
  Process: 747 ExecStart=/etc/rc.d/init.d/cloudera-scm-server start (code=exited, status=0/SUCCESS)

So, if I try to test the service, by connecting from the local machine I get the sort of behavious that makes me thing its just not listening, and maybe not started correctly.

Try poke it with a curl from the same shell as the cloudera-scm-server service was started

$ curl localhost:7180
curl: (7) Failed connect to localhost:7180; Connection refused

$ wget localhost:7180
--2016-02-25 08:00:16--  http://localhost:7180/
Resolving localhost (localhost)... ::1, 127.0.0.1
Connecting to localhost (localhost)|::1|:7180... failed: Connection refused.
Connecting to localhost (localhost)|127.0.0.1|:7180... failed: Connection refused.

Try check what ports are listening on that machine, no 7180 , what's up with that???

$ netstat -nltp
(No info could be read for "-p": geteuid()=1000 but you should be root.)
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name    
tcp        0      0 0.0.0.0:7432            0.0.0.0:*               LISTEN      -                   
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      -                   
tcp        0      0 127.0.0.1:25            0.0.0.0:*               LISTEN      -                   
tcp6       0      0 :::7432                 :::*                    LISTEN      -                   
tcp6       0      0 :::22                   :::*                    LISTEN      -                   
tcp6       0      0 ::1:25                  :::*                    LISTEN      -  

Solution

Here's what to look for, and a possible solution - give it more memory...

Check the status of the cloudera-scm-server service using [depending on your flavour of linux]

$ sudo service cloudera-scm-server status

OR

$ sudo systemctl status cloudera-scm-server

Look for the status - Active: active (running) But if you find - Active: active (exited) you may have a problem during the startup of the cloudera-scm-server.

In which case, look at the log files for cloudera-scm-server

$sudo ls -l /var/log/cloudera-scm-server

$sudo cat /var/log/cloudera-scm-server/cloudera-scm-server.out 
JAVA_HOME=/usr/java/jdk1.7.0_67-cloudera
Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x000000078dc58000, 265809920, 0) failed; error='Cannot allocate memory' (errno=12)
#
# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (malloc) failed to allocate 265809920 bytes for committing reserved memory.
# An error report file with more information is saved as:
# /tmp/hs_err_pid831.log
[ec2-user@ip-172-31-31-166 ~]$ sudo tail -100 /var/log/cloudera-scm-server/cloudera-scm-server.out
JAVA_HOME=/usr/java/jdk1.7.0_67-cloudera
Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x000000078dc58000, 265809920, 0) failed; error='Cannot allocate memory' (errno=12)

Use the command top to indicate how much memory is available to your system.

Possible solution - have a look at this discussion at Cloudera forum

In this case the java heap size was too small.

As we see that heap was exhausted, assuming this is not a memory leak or something of the sort, Cloudera Manager may need more heap to operate. This can be configured in: /etc/default/cloudera-scm-server You could, for instance, change "-Xmx2G" to "-Xmx3G" or "-Xmx4G" If the problem still happens, perhaps the heap dumps will yeild some clues.

Comments

[Hadoop_noob]

Thanks for your comment @OkezieE. CM seems to get stuck at that same place. Its been around 3-4 hrs since starting, and the log file hasn't moved from {code}2016-02-25 12:19:47,762 INFO main:com.cloudera.cmon.components.MetricSchemaManager: Cross entity aggregates processed. {code}

[Hadoop_noob]

I also discovered that the database is running. I was beginning to think maybe I forgot to install a db for Cloudera to persist its data. # Auto-generated by initialize_embedded_db.sh # # 20160225-073401 # # These are database settings for CM Manager # com.cloudera.cmf.db.type=postgresql com.cloudera.cmf.db.host=localhost:7432 com.cloudera.cmf.db.name=scm com.cloudera.cmf.db.user=scm

[Okezie]

How does the resource usage look on that server. Frankly, I think the free tier ec2 instance (t2.micro, t1.micro e.t.c) are too underpowered both in terms of CPU and Memory. It has a single core and memory is ~500MB available for non-system usage. I don't think that is sufficient based on the system requirements of Cloudera Manager. Check this page before going any further: cloudera.com/documentation/enterprise/5-4-x/topics/…