JBoss JDBC warning - "Unable to fill pool"

Question

We're getting this random warning from JBoss.. any idea why?

It happens at random times when there are no active threads. Everything works when any processing resumes.

13:49:31,764 WARN  [JBossManagedConnectionPool] [ ] Unable to fill pool
org.jboss.resource.JBossResourceException: Could not create connection; - nested throwable: (java.sql.SQLException: Listener ref
used the connection with the following error:
ORA-12516, TNS:listener could not find available handler with matching protocol stack
The Connection descriptor used by the client was:
//localhost:1521/orcl
)
        at org.jboss.resource.adapter.jdbc.xa.XAManagedConnectionFactory.createManagedConnection(XAManagedConnectionFactory.java
:144)
        at org.jboss.resource.connectionmanager.InternalManagedConnectionPool.createConnectionEventListener(InternalManagedConne
ctionPool.java:577)
        at org.jboss.resource.connectionmanager.InternalManagedConnectionPool.fillToMin(InternalManagedConnectionPool.java:524)
        at org.jboss.resource.connectionmanager.PoolFiller.run(PoolFiller.java:74)
        at java.lang.Thread.run(Thread.java:619)
Caused by: java.sql.SQLException: Listener refused the connection with the following error:
ORA-12516, TNS:listener could not find available handler with matching protocol stack
The Connection descriptor used by the client was:
//localhost:1521/orcl

---

Update: As per richj's post, here is the format of one of the four data sources we are using:

  <xa-datasource>
        <jndi-name>ABCOracleDS</jndi-name>
        <track-connection-by-tx/>
        <isSameRM-override-value>false</isSameRM-override-value>
        <xa-datasource-class>oracle.jdbc.xa.client.OracleXADataSource</xa-datasource-class>
        <xa-datasource-property name="URL">jdbc:oracle:thin:@//localhost:1521/orcl</xa-datasource-property>
        <xa-datasource-property name="User">myuser</xa-datasource-property>
        <xa-datasource-property name="Password">mypw</xa-datasource-property>
        <min-pool-size>20</min-pool-size>
        <max-pool-size>200</max-pool-size>
        <valid-connection-checker-class-name>org.jboss.resource.adapter.jdbc.vendor.OracleValidConnectionChecker
        </valid-connection-checker-class-name>
        <exception-sorter-class-name>org.jboss.resource.adapter.jdbc.vendor.OracleExceptionSorter
        </exception-sorter-class-name>
        <no-tx-separate-pools/>

        <metadata>
            <type-mapping>Oracle10g</type-mapping>
        </metadata>
    </xa-datasource> 

Solution

The Connection descriptor used by the client was:
//localhost:1521/orcl

I think the problem is likely to be somewhere in the datasource definitions. Do you have more than one datasource definition in your JBoss hot deploy directory? Are your datasource definitions correct and complete - particularly the properties used to build the connection URL?

Update 1

I'm not sure about the slashes, I think maybe the URL:

jdbc:oracle:thin:@//localhost:1521/orcl

should look like this:

jdbc:oracle:thin:@localhost:1521:orcl

Update 2

Here is a reference from Oracle that supports the syntax with slashes.
Here is a reference from Orafaq that supports the syntax with colons.

Perhaps it's worth a try even if the slashes are legal?

Update 3

Just a few more ideas from the "getting desperate" box of tricks:

ORA-12516: TNS:listener could not find available handler with matching protocol stack

This message looks like a TNS configuration issue. The JDBC thin driver shouldn't need a TNS configuration, but I've found that sometimes it just refuses to work without one.

You could also try the fully qualified host name instead of localhost. Sometimes machine.company.com works when machine on its own doesn't.

Comments

[Marcus Leon]

Yes we are using 4 data sources. I'll update the post with one of them.

[Marcus Leon]

Also, note that the database / datasource functionality all works.. we just get these random warning/errors.

[Darrell Teague]

This is not a relevant/correct answer. The WARNings are random. Otherwise connections are being made. If the URL was wrong then NO connections could be made and nothing database related would work. We have also seen these random JBoss warnings. Will research and see where real answer lies (somewhere in how DB is handling stale connections).