JBoss: Deployment fails with ClassNotFoundException even though the class is there - JaxrsScanningProcessor

Question

I have a WAR file with REST services. It deploys just fine on JBoss EAP 6.2 (corresponding to AS 7.something) in the standalone mode but it fails sometimes - but often - in the domain mode with a ClassNotFoundException for my subclass of the the JAX-RS Application class, even though it is in the war (well, it runs in standalone). Since it sometimes works, I suspect there is some concurrency issue that leads to JBoss trying to load the class before it can see it.

This is the error:

ERROR [org.jboss.msc.service.fail] (MSC service thread 1-1) MSC000001: Failed to start
 service jboss.deployment.unit.myapp.POST_MODULE: org.jboss.msc.service.StartException in
 service jboss.deployment.unit.myapp.POST_MODULE: JBAS018733: Failed to process phase POST_MODULE of deployment "myapp"
        at org.jboss.as.server.deployment.DeploymentUnitPhaseService.start(DeploymentUnitPhaseService.java:127)
        ...
Caused by: org.jboss.as.server.deployment.DeploymentUnitProcessingException: JBAS011230: Could not load JAX-RS Application class
        at org.jboss.as.jaxrs.deployment.JaxrsScanningProcessor.scan(JaxrsScanningProcessor.java:218)
        at org.jboss.as.jaxrs.deployment.JaxrsScanningProcessor.deploy(JaxrsScanningProcessor.java:100)
        at org.jboss.as.server.deployment.DeploymentUnitPhaseService.start(DeploymentUnitPhaseService.java:120)
           [jboss-as-server-7.3.0.Final-redhat-14.jar:7.3.0.Final-redhat-14]
        ... 5 more
Caused by: java.lang.ClassNotFoundException: myapp.rs.RestApplication from [Module "deployment.myapp:main" from Service Module Loader]
        at org.jboss.modules.ModuleClassLoader.findClass(ModuleClassLoader.java:197)
           [jboss-modules.jar:1.3.0.Final-redhat-2]
        ...
        at org.jboss.modules.ConcurrentClassLoader.loadClass(ConcurrentClassLoader.java:118) [jboss-modules.jar:1.3.0.Final-redhat-2]
        at org.jboss.as.jaxrs.deployment.JaxrsScanningProcessor.scan(JaxrsScanningProcessor.java:214)
        ... 7 more

The class is there:

[jboss/domain]$ unzip -l data/content/6c/0ffc675ff1c2254540b6e8caffc0d2605ed549/content | grep RestApp
     2262  02-13-14 09:05   WEB-INF/classes/myapp/rs/RestApplication.class

This is how I deployed it (my user != the user running jboss, if that makes a difference):

me$ /opt/jboss/bin/jboss-cli.sh -c --controller=0.0.0.0:49999 --user=admin --password=***
[[email protected]:49999 /] deploy /vagrant/myapp.war --all-server-groups

The deployment fails in the same way when executed via the web Admin Console. The deployment to standalone is done by copying the .war to the standalone/deployments/ directory.

Any tips?

Config info

1. JBoss EAP 6.2
2. Rest Easy 3.0.6 (replacing the default 2.x module)
3. We have no JAX-RS stuff in web.xml aside of the resteasy.servlet.mapping.prefix context param; we use the resteasy-servlet-initializer library instead of manually configuring a servlet or similar thing in there

Solution

It seems I have fixed the problem by removing all resteasy/jaxrs/jboss libraries from the WAR file's WEB-INF/lib/.

I have noticed the war contained many libraries it should not, such as resteasy-jaxrs-3.0.6.Final.jar, jboss-jaxr-api_1.0_spec-1.0.0.Final.jar (which, BTW, conflicts with Rest Easy 3.0.6), javax.servlet-api-3.1.0.jar etc.

So I guess the classloader did not see/find the class because of classpath hell, the server and the webapp using different "instances" of the same libraries. (No idea why it worked under JBoss standalone; but standalone has the out-of-the-box configuration while domain has been adjusted a lot).

I have also sometimes included --runtime-name=myapp in the deployment command, which seems to be wrong, I should have used --runtime-name=myapp.war (including the ending).