Cannot get Jedis connection; Could not get a resource from the pool

java spring redis jedis spring-data-redis

21,655

Solution 1

1.Try to use Jedis connection pool to get Jedis instance instead of using new Jedis(), for that you will have to configure JedisPool in config Since you are using it as bean, you should use JedisPool bean and then get Jedis from it wherever you are supposed to perform an operation and close it after finishing.

2.Otherwise use spring's redisTemplate with config as below

@Bean
public JedisConnectionFactory redisConnectionFactory() {
    JedisConnectionFactory factory = new JedisConnectionFactory();
    factory.setHostName(redisHostName);
    factory.setPort(redisPort);
    factory.setUsePool(true);

    return factory;
}

@Bean
public RedisTemplate<Object, Object> redisTemplate(RedisConnectionFactory cf) {
    RedisTemplate<Object, Object> redisTemplate = new RedisTemplate<Object, Object>();
    redisTemplate.setDefaultSerializer(new StringRedisSerializer());
    redisTemplate.setConnectionFactory(cf);

    return redisTemplate;
}

Solution 2

Fetch Jedis instance from spring-managed redis connection pool:

Jedis jedis = (Jedis) redisTemplate.getConnectionFactory().getConnection().getNativeConnection();

Then u can use it to instantiate JedisLock.

Solution 3

Change bind 127.0.0.1 to bind {replace with machine ip} in your redis.{os}.conf file of Redis

21,655

Author by

pampisthanu

Updated on March 27, 2020

Comments

pampisthanu about 4 years

I am running a batch job for every 5 minutes and I don't wanna other nodes to run the same job hence I am using Jedis lock to lock an object for 5 minutes. So that other node won't get the lock if they try to run the same job. Job started after acquiring the lock and when I try to read it from Redis I am getting the following exception saying

'Caused by: redis.clients.jedis.exceptions.JedisConnectionException: Could not get a resource from the pool
    at redis.clients.util.Pool.getResource(Pool.java:53)
    at redis.clients.jedis.JedisPool.getResource(JedisPool.java:226)
    at redis.clients.jedis.JedisPool.getResource(JedisPool.java:16)
    at org.springframework.data.redis.connection.jedis.JedisConnectionFactory.fetchJedisConnector(JedisConnectionFactory.java:194)
    ... 40 more
Caused by: redis.clients.jedis.exceptions.JedisConnectionException: java.net.SocketTimeoutException: Read timed out
    at redis.clients.jedis.Connection.disconnect(Connection.java:224)
    at redis.clients.jedis.BinaryClient.disconnect(BinaryClient.java:941)
    at redis.clients.jedis.Connection.close(Connection.java:214)
    at redis.clients.jedis.BinaryClient.close(BinaryClient.java:947)
    at redis.clients.jedis.Jedis.close(Jedis.java:3412)
    at redis.clients.jedis.JedisFactory.makeObject(JedisFactory.java:117)
    at org.apache.commons.pool2.impl.GenericObjectPool.create(GenericObjectPool.java:836)
    at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:434)
    at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:361)
    at redis.clients.util.Pool.getResource(Pool.java:49)'

This is what the code I have

@Bean
public Jedis getJedis()
{
    Jedis jedis = new Jedis(this.redisHost, nteger.valueOf(this.redisPort));
    jedis.auth(this.redisPassword);
    return jedis;
}

spring-boot Application.properties file

# DATA REDIS
spring.data.redis.repositories.enabled=true
# REDIS (RedisProperties)
spring.redis.host=10.160.49.22
spring.redis.password=qweqewqw
spring.redis.ssl=true 
#spring.redis.pool.max-active=10
#spring.redis.pool.max-idle=10
# spring.redis.pool.max-wait=30000
spring.redis.port=6379

Job executes the following code at the start to get the lock

JedisLock jedisLock = new JedisLock(jedis, getLockName(), getTimeInMillis());
jedisLoc.acquire()

After that redis repository class is trying to read values for a particular pattern..

List<String> hotelCodes = redisTemplate.execute(new RedisCallback<List<String>>() {
            /**
             * Gets called by {@link RedisTemplate} with an active Redis connection. Does not need to care about activating or
             * closing the connection or handling exceptions.
             *
             * @param connection active Redis connection
             * @return a result object or {@code null} if none
             * @throws DataAccessException
             */
            @Override
            public List<String> doInRedis(RedisConnection connection) throws DataAccessException {
                ScanOptions options = ScanOptions.scanOptions().match(pattern).count(1).build();
                Cursor<Map.Entry<byte[], byte[]>> entries = connection.hScan(HASH_KEY.getBytes(), options);
                List<String> result = new ArrayList<>();
                if (entries != null)
                    while (entries.hasNext()) {
                        Map.Entry<byte[], byte[]> entry = entries.next();
                        byte[] actualValue = entry.getValue();
                        result.add(new String(actualValue));
                    }

                return result;
            }
        });
        return hotelCodes;

Then this is the complete exception I see in log.

org.springframework.data.redis.RedisConnectionFailureException: Cannot get Jedis connection; nested exception is redis.clients.jedis.exceptions.JedisConnectionException: Could not get a resource from the pool
    at org.springframework.data.redis.connection.jedis.JedisConnectionFactory.fetchJedisConnector(JedisConnectionFactory.java:204)
    at org.springframework.data.redis.connection.jedis.JedisConnectionFactory.getConnection(JedisConnectionFactory.java:348)
    at org.springframework.data.redis.core.RedisConnectionUtils.doGetConnection(RedisConnectionUtils.java:129)
    at org.springframework.data.redis.core.RedisConnectionUtils.getConnection(RedisConnectionUtils.java:92)
    at org.springframework.data.redis.core.RedisConnectionUtils.getConnection(RedisConnectionUtils.java:79)
    at org.springframework.data.redis.core.RedisTemplate.execute(RedisTemplate.java:194)
    at org.springframework.data.redis.core.RedisTemplate.execute(RedisTemplate.java:169)
    at org.springframework.data.redis.core.RedisTemplate.execute(RedisTemplate.java:157)
    at com.hyatt.pms.jobs.dao.impl.HeartbeatRepositoryImpl.findAll(HeartbeatRepositoryImpl.java:62)
    at com.hyatt.pms.jobs.dao.impl.HeartbeatRepositoryImpl$$FastClassBySpringCGLIB$$e3fe6169.invoke(<generated>)
    at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)
    at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:738)
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
    at org.springframework.dao.support.PersistenceExceptionTranslationInterceptor.invoke(PersistenceExceptionTranslationInterceptor.java:136)
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
    at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:673)
    at com.hyatt.pms.jobs.dao.impl.HeartbeatRepositoryImpl$$EnhancerBySpringCGLIB$$68719252.findAll(<generated>)
    at com.hyatt.pms.jobs.processors.HeartbeatTestProcessor.processCluster(HeartbeatTestProcessor.java:54)
    at com.hyatt.pms.jobs.processors.TestProcessor.process(TestProcessor.java:61)
    at com.hyatt.pms.jobs.processors.HeartbeatTestProcessor.process(HeartbeatTestProcessor.java:39)
    at com.hyatt.pms.jobs.processors.HeartbeatTestProcessor$$FastClassBySpringCGLIB$$99fdfbdc.invoke(<generated>)
    at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)
    at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:669)
    at com.hyatt.pms.jobs.processors.HeartbeatTestProcessor$$EnhancerBySpringCGLIB$$860eb7e4.process(<generated>)
    at com.hyatt.pms.jobs.domain.jobs.HeartbeatJob.runHealthCheck(HeartbeatJob.java:34)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.springframework.scheduling.support.ScheduledMethodRunnable.run(ScheduledMethodRunnable.java:65)
    at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54)
    at org.springframework.scheduling.concurrent.ReschedulingRunnable.run(ReschedulingRunnable.java:81)
    at java.util.concurrent.Executors$RunnableAdapter.call$$$capture(Executors.java:511)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java)
    at java.util.concurrent.FutureTask.run$$$capture(FutureTask.java:266)
    at java.util.concurrent.FutureTask.run(FutureTask.java)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)
Caused by: redis.clients.jedis.exceptions.JedisConnectionException: Could not get a resource from the pool
    at redis.clients.util.Pool.getResource(Pool.java:53)
    at redis.clients.jedis.JedisPool.getResource(JedisPool.java:226)
    at redis.clients.jedis.JedisPool.getResource(JedisPool.java:16)
    at org.springframework.data.redis.connection.jedis.JedisConnectionFactory.fetchJedisConnector(JedisConnectionFactory.java:194)
    ... 40 more
Caused by: redis.clients.jedis.exceptions.JedisConnectionException: java.net.SocketTimeoutException: Read timed out
    at redis.clients.jedis.Connection.disconnect(Connection.java:224)
    at redis.clients.jedis.BinaryClient.disconnect(BinaryClient.java:941)
    at redis.clients.jedis.Connection.close(Connection.java:214)
    at redis.clients.jedis.BinaryClient.close(BinaryClient.java:947)
    at redis.clients.jedis.Jedis.close(Jedis.java:3412)
    at redis.clients.jedis.JedisFactory.makeObject(JedisFactory.java:117)
    at org.apache.commons.pool2.impl.GenericObjectPool.create(GenericObjectPool.java:836)
    at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:434)
    at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:361)
    at redis.clients.util.Pool.getResource(Pool.java:49)
    ... 43 more
Caused by: java.net.SocketTimeoutException: Read timed out
    at java.net.SocketInputStream.socketRead0(Native Method)
    at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
    at java.net.SocketInputStream.read(SocketInputStream.java:171)
    at java.net.SocketInputStream.read(SocketInputStream.java:141)
    at sun.security.ssl.InputRecord.readFully(InputRecord.java:465)
    at sun.security.ssl.InputRecord.read(InputRecord.java:503)
    at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:983)
    at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1385)
    at sun.security.ssl.SSLSocketImpl.writeRecord(SSLSocketImpl.java:757)
    at sun.security.ssl.AppOutputStream.write(AppOutputStream.java:123)
    at redis.clients.util.RedisOutputStream.flushBuffer(RedisOutputStream.java:52)
    at redis.clients.util.RedisOutputStream.flush(RedisOutputStream.java:216)
    at redis.clients.jedis.Connection.disconnect(Connection.java:220)
    ... 52 more

Anyone knows why I get this exception and how to fix it?