mod_jk produces Error smh errno=13

tomcat load-balancing cluster mod-jk

14,348

Solution 1

Are you using selinux? If you are, you might want to check on it being an selinux context problem.

Solution 2

The correct answer is online:

http://www.laliluna.de/articles/2008/09/14/apache_tomcat_mod_jk_selinux.html

The part you need is near the bottom, which I will copy and paste here in case that page goes away:

# mkdir /var/run/mod_jk
# semanage fcontext -a -t httpd_var_run_t "/var/run/mod_jk(/.*)?"

This apparently configures SELinux to allow mod_jk to work. It worked for me anyway.

Solution 3

Your OS might prevent access to the shared memory file. Try adding

# Set this to a location allowed to the 'Apache' user 
JkShmFile     /var/log/httpd/mod_jk.shm

in your mode-jk.conf (I am using the filename spelling from your post but it might be mod_jk.conf or similar)

14,348

Author by

HOHOOH

Updated on September 18, 2022

Comments

HOHOOH over 1 year

I am trying to use mod_jk for load balancing.

Loadbalancer: 172.17.8.189 (HTTPD+mod_jk)
Node1: 172.17.8.193 (apache tomcat instance 1)
Node2: 172.17.8.196 (apache tomcat instance 2)

I checked my tomcat instances and both of them are listening at 8009. However, I keep getting these errors.

[Tue Jun 18 17:49:14 2013][18057:47118810657648] [error] init_jk::mod_jk.c (3348): Initializing shm:/etc/httpd/logs/jk.shm.18057 errno=13. Load balancing workers will not function properly.
[Tue Jun 18 17:49:14 2013][18057:47118810657648] [info] init_jk::mod_jk.c (3365): mod_jk/1.2.37 initialized

Debug log is

    [Tue Jun 18 18:14:00 2013][18778:47687756850032] [debug] jk_set_time_fmt::jk_util.c (461): Pre-processed log time stamp format is '[%a %b %d %H:%M:%S %Y]'
[Tue Jun 18 18:14:00 2013][18778:47687756850032] [debug] uri_worker_map_open::jk_uri_worker_map.c (865): rule map size is 2
[Tue Jun 18 18:14:00 2013][18778:47687756850032] [debug] uri_worker_map_add::jk_uri_worker_map.c (815): wildchar rule '/jasperserver-pro/*=loadbalancer' source 'JkMount' was added
[Tue Jun 18 18:14:00 2013][18778:47687756850032] [debug] uri_worker_map_add::jk_uri_worker_map.c (824): exact rule '/jkstatus/=status' source 'JkMount' was added
[Tue Jun 18 18:14:00 2013][18778:47687756850032] [debug] uri_worker_map_dump::jk_uri_worker_map.c (176): uri map dump after map open: id=0, index=0 file='(null)' reject_unsafe=0 reload=60 modified=0 checked=0
[Tue Jun 18 18:14:00 2013][18778:47687756850032] [debug] uri_worker_map_dump::jk_uri_worker_map.c (182): generation 0: size=0 nosize=0 capacity=0
[Tue Jun 18 18:14:00 2013][18778:47687756850032] [debug] uri_worker_map_dump::jk_uri_worker_map.c (182): generation 1: size=2 nosize=0 capacity=4
[Tue Jun 18 18:14:00 2013][18778:47687756850032] [debug] uri_worker_map_dump::jk_uri_worker_map.c (195): NEXT (1) map #0: uri=/jasperserver-pro/* worker=loadbalancer context=/jasperserver-pro/* source=JkMount type=Wildchar len=19
[Tue Jun 18 18:14:00 2013][18778:47687756850032] [debug] uri_worker_map_dump::jk_uri_worker_map.c (195): NEXT (1) map #1: uri=/jkstatus/ worker=status context=/jkstatus/ source=JkMount type=Exact len=10
[Tue Jun 18 18:14:00 2013][18778:47687756850032] [debug] init_jk::mod_jk.c (3303): Setting default connection pool max size to 1
[Tue Jun 18 18:14:00 2013][18778:47687756850032] [debug] jk_map_read_property::jk_map.c (482): Adding property 'worker.list' with value 'loadbalancer,status' to map.
[Tue Jun 18 18:14:00 2013][18778:47687756850032] [debug] jk_map_read_property::jk_map.c (482): Adding property 'worker.node1.port' with value '8009' to map.
[Tue Jun 18 18:14:00 2013][18778:47687756850032] [debug] jk_map_read_property::jk_map.c (482): Adding property 'worker.node1.host' with value '172.17.8.193' to map.
[Tue Jun 18 18:14:00 2013][18778:47687756850032] [debug] jk_map_read_property::jk_map.c (482): Adding property 'worker.node1.type' with value 'ajp13' to map.
[Tue Jun 18 18:14:00 2013][18778:47687756850032] [debug] jk_map_read_property::jk_map.c (482): Adding property 'worker.node1.lbfactor' with value '1' to map.
[Tue Jun 18 18:14:00 2013][18778:47687756850032] [debug] jk_map_read_property::jk_map.c (482): Adding property 'worker.node2.port' with value '8009' to map.
[Tue Jun 18 18:14:00 2013][18778:47687756850032] [debug] jk_map_read_property::jk_map.c (482): Adding property 'worker.node2.host' with value '172.17.8.196' to map.
[Tue Jun 18 18:14:00 2013][18778:47687756850032] [debug] jk_map_read_property::jk_map.c (482): Adding property 'worker.node2.type' with value 'ajp13' to map.
[Tue Jun 18 18:14:00 2013][18778:47687756850032] [debug] jk_map_read_property::jk_map.c (482): Adding property 'worker.node2.lbfactor' with value '1' to map.
[Tue Jun 18 18:14:00 2013][18778:47687756850032] [debug] jk_map_read_property::jk_map.c (482): Adding property 'worker.loadbalancer.type' with value 'lb' to map.
[Tue Jun 18 18:14:00 2013][18778:47687756850032] [debug] jk_map_read_property::jk_map.c (482): Adding property 'worker.loadbalancer.balance_workers' with value 'node1,node2' to map.
[Tue Jun 18 18:14:00 2013][18778:47687756850032] [debug] jk_map_read_property::jk_map.c (482): Adding property 'worker.loadbalancer.sticky_session' with value '1' to map.
[Tue Jun 18 18:14:00 2013][18778:47687756850032] [debug] jk_map_read_property::jk_map.c (482): Adding property 'worker.status.type' with value 'status' to map.
[Tue Jun 18 18:14:00 2013][18778:47687756850032] [debug] jk_map_resolve_references::jk_map.c (764): Checking for references with prefix worker. with wildcard (recursion 1)
[Tue Jun 18 18:14:00 2013][18778:47687756850032] [debug] jk_shm_calculate_size::jk_shm.c (129): worker loadbalancer of type lb has 2 members
[Tue Jun 18 18:14:00 2013][18778:47687756850032] [debug] jk_shm_calculate_size::jk_shm.c (136): shared memory will contain 0 ajp workers and 1 lb workers with 2 members
[Tue Jun 18 18:14:00 2013][18778:47687756850032] [error] init_jk::mod_jk.c (3348): Initializing shm:/etc/httpd/logs/jk.shm.18778 errno=13. Load balancing workers will not function properly.

As the result the request arent being properly rerouted.

My worker.properties

# Define list of workers that will be used
# for mapping requests
worker.list=loadbalancer,status

# Define Node1
# modify the host as your host IP or DNS name.
worker.node1.port=8009
worker.node1.host=172.17.8.193 
worker.node1.type=ajp13
worker.node1.lbfactor=1

# Define Node2
# modify the host as your host IP or DNS name.
worker.node2.port=8009
worker.node2.host=172.17.8.196
worker.node2.type=ajp13
worker.node2.lbfactor=1

# Load-balancing behaviour
worker.loadbalancer.type=lb
worker.loadbalancer.balance_workers=node1,node2
worker.loadbalancer.sticky_session=1
#worker.list=loadbalancer

# Status worker for managing load balancer
worker.status.type=status

My mode-jk.conf

# Load mod_jk module
# Specify the filename of the mod_jk lib
LoadModule jk_module modules/mod_jk.so
 
# Where to find workers.properties
JkWorkersFile conf/workers.properties

# Where to put jk logs
JkLogFile logs/mod_jk.log
 
# Set the jk log level [debug/error/info]
JkLogLevel info 
 
# Select the log format
JkLogStampFormat  "[%a %b %d %H:%M:%S %Y]"
 
# JkOptions indicates to send SSK KEY SIZE
JkOptions +ForwardKeySize +ForwardURICompat -ForwardDirectories
 
# JkRequestLogFormat
JkRequestLogFormat "%w %V %T"
               
# Mount your applications
JkMount /myapp/* loadbalancer

 
# You can use external file for mount points.
# It will be checked for updates each 60 seconds.
# The format of the file is: /url=worker
# /examples/*=loadbalancer
#JkMountFile conf/uriworkermap.properties               

# Add shared memory.
# This directive is present with 1.2.10 and
# later versions of mod_jk, and is needed for
# for load balancing to work properly
JkShmFile logs/jk.shm 
#JkShmFile /var/log/httpd/mod_jk.shm
             
# Add jkstatus for managing runtime data
<Location /jkstatus/>
    JkMount status
    Order deny,allow
    Deny from all
    Allow from localhost
</Location>

Server.xml

<?xml version='1.0' encoding='utf-8'?>
<!--
  Licensed to the Apache Software Foundation (ASF) under one or more
  contributor license agreements.  See the NOTICE file distributed with
  this work for additional information regarding copyright ownership.
  The ASF licenses this file to You under the Apache License, Version 2.0
  (the "License"); you may not use this file except in compliance with
  the License.  You may obtain a copy of the License at

      http://www.apache.org/licenses/LICENSE-2.0

  Unless required by applicable law or agreed to in writing, software
  distributed under the License is distributed on an "AS IS" BASIS,
  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  See the License for the specific language governing permissions and
  limitations under the License.
-->
<!-- Note:  A "Server" is not itself a "Container", so you may not
     define subcomponents such as "Valves" at this level.
     Documentation at /docs/config/server.html
 -->
<Server port="8005" shutdown="SHUTDOWN">

  <!--APR library loader. Documentation at /docs/apr.html -->
  <Listener className="org.apache.catalina.core.AprLifecycleListener" SSLEngine="on" />
  <!--Initialize Jasper prior to webapps are loaded. Documentation at /docs/jasper-howto.html -->
  <Listener className="org.apache.catalina.core.JasperListener" />
  <!-- Prevent memory leaks due to use of particular java/javax APIs-->
  <Listener className="org.apache.catalina.core.JreMemoryLeakPreventionListener" />
  <!-- JMX Support for the Tomcat server. Documentation at /docs/non-existent.html -->
  <Listener className="org.apache.catalina.mbeans.ServerLifecycleListener" />
  <Listener className="org.apache.catalina.mbeans.GlobalResourcesLifecycleListener" />

  <!-- Global JNDI resources
       Documentation at /docs/jndi-resources-howto.html
  -->
  <GlobalNamingResources>
    <!-- Editable user database that can also be used by
         UserDatabaseRealm to authenticate users
    -->
    <Resource name="UserDatabase" auth="Container"
              type="org.apache.catalina.UserDatabase"
              description="User database that can be updated and saved"
              factory="org.apache.catalina.users.MemoryUserDatabaseFactory"
              pathname="conf/tomcat-users.xml" />
  </GlobalNamingResources>

  <!-- A "Service" is a collection of one or more "Connectors" that share
       a single "Container" Note:  A "Service" is not itself a "Container", 
       so you may not define subcomponents such as "Valves" at this level.
       Documentation at /docs/config/service.html
   -->
  <Service name="Catalina">
  
    <!--The connectors can use a shared executor, you can define one or more named thread pools-->
    <!--
    <Executor name="tomcatThreadPool" namePrefix="catalina-exec-" 
        maxThreads="150" minSpareThreads="4"/>
    -->
    
    
    <!-- A "Connector" represents an endpoint by which requests are received
         and responses are returned. Documentation at :
         Java HTTP Connector: /docs/config/http.html (blocking & non-blocking)
         Java AJP  Connector: /docs/config/ajp.html
         APR (HTTP/AJP) Connector: /docs/apr.html
         Define a non-SSL HTTP/1.1 Connector on port 8080
    -->
    <Connector port="8081" protocol="HTTP/1.1" 
               connectionTimeout="20000" 
               redirectPort="8443" />
    <!-- A "Connector" using the shared thread pool-->
    <!--
    <Connector executor="tomcatThreadPool"
               port="8080" protocol="HTTP/1.1" 
               connectionTimeout="20000" 
               redirectPort="8443" />
    -->           
    <!-- Define a SSL HTTP/1.1 Connector on port 8443
         This connector uses the JSSE configuration, when using APR, the 
         connector should be using the OpenSSL style configuration
         described in the APR documentation -->
    <!--
    <Connector port="8443" protocol="HTTP/1.1" SSLEnabled="true"
               maxThreads="150" scheme="https" secure="true"
               clientAuth="false" sslProtocol="TLS" />
    -->

    <!-- Define an AJP 1.3 Connector on port 8009 -->
    <Connector port="8009" protocol="AJP/1.3" redirectPort="8443" />


    <!-- An Engine represents the entry point (within Catalina) that processes
         every request.  The Engine implementation for Tomcat stand alone
         analyzes the HTTP headers included with the request, and passes them
         on to the appropriate Host (virtual host).
         Documentation at /docs/config/engine.html -->

    <!-- You should set jvmRoute to support load-balancing via AJP ie :
    <Engine name="Catalina" defaultHost="localhost" jvmRoute="jvm1">         
    --> 
       <Engine name="Catalina" defaultHost="localhost" jvmRoute="node2">

      <!--For clustering, please take a look at documentation at:
          /docs/cluster-howto.html  (simple how to)
          /docs/config/cluster.html (reference documentation) -->
      <!--
      <Cluster className="org.apache.catalina.ha.tcp.SimpleTcpCluster"/>
      -->        
      
      <Cluster className="org.apache.catalina.ha.tcp.SimpleTcpCluster" channelSendOptions="8">

<Manager className="org.apache.catalina.ha.session.DeltaManager"

expireSessionsOnShutdown="false"

notifyListenersOnReplication="true"/>

 

<Channel className="org.apache.catalina.tribes.group.GroupChannel">

<Membership className="org.apache.catalina.tribes.membership.McastService"

address="228.0.0.4"

port="45564" frequency="500"

dropTime="3000"/>

<Sender className="org.apache.catalina.tribes.transport.ReplicationTransmitter">

<Transport className="org.apache.catalina.tribes.transport.nio.PooledParallelSender"/>

</Sender>

<Receiver className="org.apache.catalina.tribes.transport.nio.NioReceiver"

address="auto" port="4000" autoBind="100"

selectorTimeout="5000" maxThreads="6"/>

<Interceptor className="org.apache.catalina.tribes.group.interceptors.TcpFailureDetector"/>

<Interceptor className="org.apache.catalina.tribes.group.interceptors.MessageDispatch15Interceptor"/>

</Channel>

<Valve className="org.apache.catalina.ha.tcp.ReplicationValve" filter=""/>

<Valve className="org.apache.catalina.ha.session.JvmRouteBinderValve"/>

<ClusterListener className="org.apache.catalina.ha.session.JvmRouteSessionIDBinderListener"/>

<ClusterListener className="org.apache.catalina.ha.session.ClusterSessionListener"/>

</Cluster>
        

      <!-- The request dumper valve dumps useful debugging information about
           the request and response data received and sent by Tomcat.
           Documentation at: /docs/config/valve.html -->
      <!--
      <Valve className="org.apache.catalina.valves.RequestDumperValve"/>
      -->

      <!-- This Realm uses the UserDatabase configured in the global JNDI
           resources under the key "UserDatabase".  Any edits
           that are performed against this UserDatabase are immediately
           available for use by the Realm.  -->
      <Realm className="org.apache.catalina.realm.UserDatabaseRealm"
             resourceName="UserDatabase"/>

      <!-- Define the default virtual host
           Note: XML Schema validation will not work with Xerces 2.2.
       -->
      <Host name="localhost"  appBase="webapps"
            unpackWARs="true" autoDeploy="true"
            xmlValidation="false" xmlNamespaceAware="false">

        <!-- SingleSignOn valve, share authentication between web applications
             Documentation at: /docs/config/valve.html -->
        <!--
        <Valve className="org.apache.catalina.authenticator.SingleSignOn" />
        -->

        <!-- Access log processes all example.
             Documentation at: /docs/config/valve.html -->
        <!--
        <Valve className="org.apache.catalina.valves.AccessLogValve" directory="logs"  
               prefix="localhost_access_log." suffix=".txt" pattern="common" resolveHosts="false"/>
        -->

      </Host>
    </Engine>
  </Service>
</Server>

Would appreciate any feedback.

JScoobyCed almost 11 years

What OS are you using? CentOS or RedHat based maybe?
HOHOOH almost 11 years

I am using Redhat.

HOHOOH almost 11 years

I am using Redhat linux. I tried storing that in the shared memory as well but the problem still persists.
HOHOOH almost 11 years

Thank you everyone for the response. Karthir is right, I had selinux and had to disable it to get rid of the problem. Details can be found at. access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Lin‌ux/…
Michael Hampton about 8 years

Disabling SELinux does not fix anything. It only makes you less secure.
Debopam Mitra almost 8 years

@Hampton - Do you have any other workaround without disabling SELinux?
mwangi over 7 years

Disabling SELinux seemed to work for me. I want to know whether there is a different way to do it.
Philip Kearns over 5 years

For me disabling SELinux worked, so enabled it and told it to allow the port (I was using 8010): semanage port -a -t http_port_t -p tcp 8010
Evan Hu about 5 years

Not work for me. BTW, how to revert the effect of the command semanage fcontext -a -t httpd_var_run_t "/var/run/mod_jk(/.*)?"?