How can I get Openfire 3.7.1 to 'remember' its configuration?

Question

I have (well, had up until this morning) an Openfire intranet XMPP server that had been running merrily along for 370 days without being touched, and a total of about 3 years of running. Users authenticate to AD on an adjacent server, and are divided by OU into groups. All that functionality work[ed|s] great.

My problem is that the service crashed this morning, my assistant got a 'no Java heap space' error (and no screenshot, sorry) when attempting to use the admin console to determine what the problem was. We went to the physical console and restarted the service, and it restarted fine, or at least without visible error.

But now, trying to go to the admin console brings up the initial setup page. I groaned, but dug out my notes and re-entered all the LDAP info and DB connection info. With some satisfaction I hit the 'Log into Admin Console' button, and waited...only to be returned to the setup screen I had just left, asking me what my preferred system language was. So:

The MySQL server is running, and the connection info is correct in the /opt/openfire/conf/openfire.xml file. I tested the connection by using the credentials in that file to view the ofProperty table. Checking the contents of the ofProperty table reveals that the settings were saved there, including all the LDAP info, the name and other fields, the group definitions, everything. But Openfire refuses to load those from the database.

Things I have checked:

• Examining /var/log/messages shows nothing out of the ordinary.
• MySQL is running fine and able to take connections from localhost as well as the MySQL workbench on my machine.
• My disk is 3% used.
• Memory is not a problem, 0k swap is used.
• This machine only runs openfire, nothing else.
• Netstat shows only the openfire process listening on 9090 and 9091, the admin ports.
• Running ps only shows the admin process.
• No updates had been installed.
• No configs had been changed.
• This machine is not accessible from the internet, so hacking seems unlikely. Also there are no other signs of it.

EDIT: Log clippings showing a connection refused, but which apparently only happened earlier in time:

    2012.05.16 09:27:47 org.jivesoftware.database.DbConnectionManager - Unable to get a connection from the database pool (attempt 10 out of 10).
com.mysql.jdbc.CommunicationsException: Communications link failure due to underlying exception: 

** BEGIN NESTED EXCEPTION ** 

java.net.ConnectException
MESSAGE: Connection refused

STACKTRACE:

java.net.ConnectException: Connection refused
    at java.net.PlainSocketImpl.socketConnect(Native Method)
    at java.net.PlainSocketImpl.doConnect(Unknown Source)
    at java.net.PlainSocketImpl.connectToAddress(Unknown Source)
    at java.net.PlainSocketImpl.connect(Unknown Source)
    at java.net.SocksSocketImpl.connect(Unknown Source)
    at java.net.Socket.connect(Unknown Source)
    at java.net.Socket.connect(Unknown Source)
    at java.net.Socket.<init>(Unknown Source)
    at java.net.Socket.<init>(Unknown Source)
    at com.mysql.jdbc.StandardSocketFactory.connect(StandardSocketFactory.java:256)
    at com.mysql.jdbc.MysqlIO.<init>(MysqlIO.java:271)
    at com.mysql.jdbc.Connection.createNewIO(Connection.java:2771)
    at com.mysql.jdbc.Connection.<init>(Connection.java:1555)
    at com.mysql.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:285)
    at java.sql.DriverManager.getConnection(Unknown Source)
    at java.sql.DriverManager.getConnection(Unknown Source)
    at org.logicalcobwebs.proxool.DefaultConnectionBuilder.buildConnection(DefaultConnectionBuilder.java:39)
    at org.logicalcobwebs.proxool.Prototyper.buildConnection(Prototyper.java:159)
    at org.logicalcobwebs.proxool.ConnectionPool.getConnection(ConnectionPool.java:211)
    at org.logicalcobwebs.proxool.ProxoolDriver.connect(ProxoolDriver.java:89)
    at java.sql.DriverManager.getConnection(Unknown Source)
    at java.sql.DriverManager.getConnection(Unknown Source)
    at org.jivesoftware.database.DefaultConnectionProvider.getConnection(DefaultConnectionProvider.java:86)
    at org.jivesoftware.database.DbConnectionManager.getConnection(DbConnectionManager.java:124)
    at org.jivesoftware.openfire.XMPPServer.verifyDataSource(XMPPServer.java:754)
    at org.jivesoftware.openfire.XMPPServer.start(XMPPServer.java:480)
    at org.jivesoftware.openfire.XMPPServer.<init>(XMPPServer.java:212)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(Unknown Source)
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown Source)
    at java.lang.reflect.Constructor.newInstance(Unknown Source)
    at java.lang.Class.newInstance0(Unknown Source)
    at java.lang.Class.newInstance(Unknown Source)
    at org.jivesoftware.openfire.starter.ServerStarter.start(ServerStarter.java:113)
    at org.jivesoftware.openfire.starter.ServerStarter.main(ServerStarter.java:58)


** END NESTED EXCEPTION **



Last packet sent to the server was 1 ms ago.
    at com.mysql.jdbc.Connection.createNewIO(Connection.java:2847)
    at com.mysql.jdbc.Connection.<init>(Connection.java:1555)
    at com.mysql.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:285)
    at java.sql.DriverManager.getConnection(Unknown Source)
    at java.sql.DriverManager.getConnection(Unknown Source)
    at org.logicalcobwebs.proxool.DefaultConnectionBuilder.buildConnection(DefaultConnectionBuilder.java:39)
    at org.logicalcobwebs.proxool.Prototyper.buildConnection(Prototyper.java:159)
    at org.logicalcobwebs.proxool.ConnectionPool.getConnection(ConnectionPool.java:211)
    at org.logicalcobwebs.proxool.ProxoolDriver.connect(ProxoolDriver.java:89)
    at java.sql.DriverManager.getConnection(Unknown Source)
    at java.sql.DriverManager.getConnection(Unknown Source)
    at org.jivesoftware.database.DefaultConnectionProvider.getConnection(DefaultConnectionProvider.java:86)
    at org.jivesoftware.database.DbConnectionManager.getConnection(DbConnectionManager.java:124)
    at org.jivesoftware.openfire.XMPPServer.verifyDataSource(XMPPServer.java:754)
    at org.jivesoftware.openfire.XMPPServer.start(XMPPServer.java:480)
    at org.jivesoftware.openfire.XMPPServer.<init>(XMPPServer.java:212)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(Unknown Source)
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown Source)
    at java.lang.reflect.Constructor.newInstance(Unknown Source)
    at java.lang.Class.newInstance0(Unknown Source)
    at java.lang.Class.newInstance(Unknown Source)
    at org.jivesoftware.openfire.starter.ServerStarter.start(ServerStarter.java:113)
    at org.jivesoftware.openfire.starter.ServerStarter.main(ServerStarter.java:58)
2012.05.16 09:27:48 org.jivesoftware.openfire.XMPPServer - Server halted
2012.05.16 09:36:34 org.jivesoftware.openfire.pubsub.PubSubModule - Publish-Subscribe domain: pubsub.cc.server.local
2012.05.16 09:36:36 org.jivesoftware.openfire.muc.spi.MultiUserChatServiceImpl - Multi User Chat domain: conference.cc.server.local
2012.05.16 09:37:48 org.jivesoftware.openfire.XMPPServer - Openfire 3.7.1 [May 16, 2012 9:37:48 AM]
2012.05.16 09:38:01 org.jivesoftware.openfire.container.AdminConsolePlugin - Admin console listening at:
  http://127.0.0.1:9090
  https://127.0.0.1:9091
2012.05.16 09:54:21 org.jivesoftware.openfire.XMPPServer - Server halted
2012.05.16 09:54:33 org.jivesoftware.openfire.XMPPServer - Openfire 3.7.1 [May 16, 2012 9:54:33 AM]
2012.05.16 09:54:46 org.jivesoftware.openfire.container.AdminConsolePlugin - Admin console listening at:
  http://127.0.0.1:9090
  https://127.0.0.1:9091
2012.05.16 10:07:24 org.jivesoftware.openfire.XMPPServer - Server halted
2012.05.16 10:07:37 org.jivesoftware.openfire.XMPPServer - Openfire 3.7.1 [May 16, 2012 10:07:37 AM]
2012.05.16 10:07:49 org.jivesoftware.openfire.container.AdminConsolePlugin - Admin console listening at:
  http://127.0.0.1:9090
  https://127.0.0.1:9091
2012.05.16 10:26:39 org.jivesoftware.openfire.XMPPServer - Server halted
2012.05.16 10:28:54 org.jivesoftware.openfire.XMPPServer - Openfire 3.7.1 [May 16, 2012 10:28:54 AM]
2012.05.16 10:29:10 org.jivesoftware.openfire.container.AdminConsolePlugin - Admin console listening at:
  http://127.0.0.1:9090
  https://127.0.0.1:9091
2012.05.16 10:33:25 org.jivesoftware.openfire.XMPPServer - Server halted
2012.05.16 10:33:38 org.jivesoftware.openfire.XMPPServer - Openfire 3.7.1 [May 16, 2012 10:33:38 AM]
2012.05.16 10:33:51 org.jivesoftware.openfire.container.AdminConsolePlugin - Admin console listening at:
  http://127.0.0.1:9090
  https://127.0.0.1:9091
2012.05.16 10:51:38 org.jivesoftware.openfire.XMPPServer - Server halted
2012.05.16 10:51:51 org.jivesoftware.openfire.XMPPServer - Openfire 3.7.1 [May 16, 2012 10:51:51 AM]
2012.05.16 10:52:04 org.jivesoftware.openfire.container.AdminConsolePlugin - Admin console listening at:
  http://127.0.0.1:9090
  https://127.0.0.1:9091
2012.05.16 10:53:01 org.jivesoftware.openfire.XMPPServer - Server halted
2012.05.16 10:53:35 org.jivesoftware.openfire.XMPPServer - Openfire 3.7.1 [May 16, 2012 10:53:35 AM]
2012.05.16 10:53:49 org.jivesoftware.openfire.container.AdminConsolePlugin - Admin console listening at:
  http://127.0.0.1:9090
  https://127.0.0.1:9091

After that one failure, I had attempted variously to reconfigure and restart the service, as you can see the admin console comes up.

Solution

Long story short: Got a look at another (working) installation elsewhere. Turns out there is a value called <setup> that resides in the file openfire.xml of a working installation as follows:

...(XML snipped)
      <minConnections>5</minConnections>  
      <maxConnections>25</maxConnections>  
      <connectionTimeout>1.0</connectionTimeout> 
    </defaultProvider> 
  </database>  
  <setup>true</setup> 
</jive>

The setup line was missing from my openfire.xml for some reason.

Interestingly, the file is owned and only writable by root, so some privileged process deleted the line (somehow???) when the setup process ran, while apparently leaving the connection parameters alone. But, since the file was only writable by root, the setup script run by the JRE wasn't able to write to the file. Also, I'm a moron, and missed this line among the Java exceptions:

java.io.IOException: XML properties file must be writable: openfire.xml

Apparently it's writable to some parts of the setup process but not others? I don't know. But if anyone else has this problem, hopefully they'll find the answer here. I was able to correct the problem and get Openfire to load immediately by adding the <setup> value to openfire.xml.

Comments

[ravi yarlagadda]

It sounds like OpenFire is failing miserably at actually loading its settings from the database. Can you check its logs (probably in /opt/openfire/logs) for anything interesting/significant?

[atroon]

Odd, I see a lot of 'Connection refused' and then java exceptions in the log from the DB server. I'll append them to the post.

[ravi yarlagadda]

Hmm.. connection refusal doesn't make much sense. Was it configured to use 127.0.0.1 for the database in the XML file? Is MySQL maybe set to listen on a non-standard port, or only a specific address? Verify that TCP is working by forcing a command line MySQL client to use TCP instead of the Unix socket - mysql -h 127.0.0.1 -u openfireuser -p.

[atroon]

I know it doesn't. It is configured to use 127.0.0.1, and I was able to connect both by socket and by MySQL workbench on my desktop machine, which connects via TCP.

[ravi yarlagadda]

Check TCP against 127.0.0.1, though - MySQL might not be listening on that address.

[atroon]

TCP on 127.0.0.1 works. It looks like netstat also reports java has a connection to mysql, and now the admin console seems to "remember" certain values, like the password for the LDAP user I bind with, and the domain name I used. But completing it still rolls me back to the startup screen.

[Admin]

atroon: This seems to be a common issue. Any idea how this occured? I had this issue and it made me want to move away from openfire but I am revisiting to see if i run into issue again. I fail to see how this can happen. In my case, I lost all my settings, and was unable to get into the admin console; frustrated I reinstalled entirely. OpenFire needs to get their stuff together and start coding for the enterprise if they want to be taken seriously. A bug like this is ridiculous.