Spring Boot - Redis Cache Connection Pool Exhausted After Few Requests

Question

I am using Redis for cache in my application. I am using cache frequently to fetch the data. I am using spring-boot version 1.5.9, spring-data-redis 1.8.9, jedis 2.9.0 and commons-pool 1.6.

I am not able to understand

1. why it is not closing connections ?
2. why it is not able to fetch the connections from pool ?

This is the configurations for Redis which I am using:

import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.data.redis.cache.RedisCacheManager;
import org.springframework.data.redis.connection.jedis.JedisConnectionFactory;
import org.springframework.data.redis.core.StringRedisTemplate;
import redis.clients.jedis.JedisPoolConfig;

import java.time.Duration;

@Configuration
public class ApplicationConfiguration {

    @Value("${spring.redis.host}")
    private String REDIS_HOST;

    @Value("${spring.redis.port}")
    private int REDIS_PORT;

    @Value("${spring.redis.database}")
    private int REDIS_DATABASE;

    @Value("${spring.redis.pool.max-active}")
    private int REDIS_POOL_MAX_ACTIVE;

    @Value("${spring.redis.pool.max-idle}")
    private int REDIS_POOL_MAX_IDLE;

    @Value("${spring.redis.pool.min-idle}")
    private int REDIS_POOL_MIN_IDLE;

    @Value("${spring.redis.pool.max-wait}")
    private long REDIS_POOL_TIMEOUT;

    @Value("${spring.redis.timeout}")
    private int REDIS_TIMEOUT;

    @Bean
    public JedisConnectionFactory jedisConnectionFactory() {
        JedisPoolConfig poolConfig = new JedisPoolConfig();
        //Maximum number of active connections that can be allocated from this pool at the same time
        poolConfig.setMaxTotal(REDIS_POOL_MAX_ACTIVE);
        //Number of connections to Redis that just sit there and do nothing
        poolConfig.setMaxIdle(REDIS_POOL_MAX_IDLE);
        //Minimum number of idle connections to Redis - these can be seen as always open and ready to serve
        poolConfig.setMinIdle(REDIS_POOL_MIN_IDLE);
        //The maximum number of milliseconds that the pool will wait (when there are no available connections) for a connection to be returned before throwing an exception
        poolConfig.setMaxWaitMillis(REDIS_POOL_TIMEOUT);
        //The minimum amount of time an object may sit idle in the pool before it is eligible for eviction by the idle object evictor
        poolConfig.setMinEvictableIdleTimeMillis(Duration.ofSeconds(60).toMillis());
        //The minimum amount of time a connection may sit idle in the pool before it is eligible for eviction by the idle connection evictor
        poolConfig.setSoftMinEvictableIdleTimeMillis(Duration.ofSeconds(10).toMillis());
        //Idle connection checking period
        poolConfig.setTimeBetweenEvictionRunsMillis(Duration.ofSeconds(5).toMillis());
        //Maximum number of connections to test in each idle check
        poolConfig.setNumTestsPerEvictionRun(3);
        //Tests whether connection is dead when connection retrieval method is called
        poolConfig.setTestOnBorrow(true);
        //Tests whether connection is dead when returning a connection to the pool
        poolConfig.setTestOnReturn(true);
        //Tests whether connections are dead during idle periods
        poolConfig.setTestWhileIdle(true);
        poolConfig.setBlockWhenExhausted(true);

        JedisConnectionFactory connectionFactory = new JedisConnectionFactory(poolConfig);
        connectionFactory.setUsePool(true);
        connectionFactory.setHostName(REDIS_HOST);
        connectionFactory.setPort(REDIS_PORT);
        connectionFactory.setDatabase(REDIS_DATABASE);
        connectionFactory.setTimeout(REDIS_TIMEOUT);

        return connectionFactory;
    }

    @Bean
    public LoggingRedisTemplate stringRedisTemplate(@Autowired JedisConnectionFactory jedisConnectionFactory) {
        LoggingRedisTemplate stringRedisTemplate = new LoggingRedisTemplate(jedisConnectionFactory);
        stringRedisTemplate.setEnableTransactionSupport(true);
        return stringRedisTemplate;
    }

    @Bean
    public RedisCacheManager cacheManager(@Autowired StringRedisTemplate stringRedisTemplate) {
        RedisCacheManager cacheManager = new RedisCacheManager(stringRedisTemplate);
        cacheManager.setUsePrefix(true);
        return cacheManager;
    }

}

Then I am using service to access the data:

import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.dao.DataAccessException;
import org.springframework.data.redis.connection.RedisConnection;
import org.springframework.data.redis.connection.StringRedisConnection;
import org.springframework.data.redis.core.RedisCallback;
import org.springframework.data.redis.core.types.RedisClientInfo;
import org.springframework.stereotype.Component;

import java.util.List;

@Component
public class RedisService {

    @Autowired
    private LoggingRedisTemplate stringRedisTemplate;

    public String getStringValue(final String key) {
//        return stringRedisTemplate.opsForValue().get(key);
        return readValueWithCallBack(key);
    }

    public void setStringValue(final String key, final String value) {
//        stringRedisTemplate.opsForValue().setIfAbsent(key, value);
        writeValueWithCallBack(key,value);
    }

    public void removeStringValue(final String key) {
//        stringRedisTemplate.delete(key);
        removeValueWithCallback(key);
    }


    public Long removeValueWithCallback(final String key){

        return (Long) stringRedisTemplate.execute( new RedisCallback<Object>() {
            @Override
            public Object doInRedis(RedisConnection connection) throws DataAccessException {
                StringRedisConnection stringRedisConnection = (StringRedisConnection) connection;
                stringRedisConnection.multi();
                Long deletedKeysCount = stringRedisConnection.del(key);
                stringRedisConnection.exec();
                stringRedisConnection.close();
                return deletedKeysCount;
            }
        });
    }

    public String readValueWithCallBack(final String key){

        return (String) stringRedisTemplate.execute( new RedisCallback<Object>() {
            @Override
            public Object doInRedis(RedisConnection connection) throws DataAccessException {
                StringRedisConnection stringRedisConnection = (StringRedisConnection) connection;
                String value = stringRedisConnection.get(key);
                List<RedisClientInfo> redisClientInfos = stringRedisConnection.getClientList();
                stringRedisConnection.close();
                return value;
            }
        });
    }

    public void writeValueWithCallBack(final String key, final String value){

        stringRedisTemplate.execute( new RedisCallback<Object>() {
            @Override
            public Object doInRedis(RedisConnection connection) throws DataAccessException {
                StringRedisConnection stringRedisConnection = (StringRedisConnection) connection;
                stringRedisConnection.multi();
                stringRedisConnection.set(key,value);
                stringRedisConnection.exec();
                stringRedisConnection.close();
                return null;
            }
        });
    }
}

and this is the Redis Template I created to avoid exception and move ahead with next step normally after exception :

import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.data.redis.connection.RedisConnectionFactory;
import org.springframework.data.redis.core.RedisCallback;
import org.springframework.data.redis.core.SessionCallback;
import org.springframework.data.redis.core.StringRedisTemplate;
import org.springframework.data.redis.core.script.RedisScript;
import org.springframework.data.redis.serializer.RedisSerializer;
import org.springframework.stereotype.Component;

import java.util.List;

/**
 *
 * An extension of RedisTemplate that logs exceptions instead of letting them propagate.
 * If the Redis server is unavailable, cache operations are always a "miss" and data is fetched from the database.
 */
@Component
public class LoggingRedisTemplate extends StringRedisTemplate {

    private static final Logger logger = LoggerFactory.getLogger(LoggingRedisTemplate.class);

    public LoggingRedisTemplate(RedisConnectionFactory connectionFactory) {
        super(connectionFactory);
    }

    @Override
    public <T> T execute(final RedisCallback<T> action, final boolean exposeConnection, final boolean pipeline) {
        try {
            return super.execute(action, exposeConnection, pipeline);
        }
        catch(final Throwable t) {
            logger.warn("Error executing cache operation: {}", t.getMessage());
            return null;
        }
    }


    @Override
    public <T> T execute(final RedisScript<T> script, final List<String> keys, final Object... args) {
        try {
            return super.execute(script, keys, args);
        }
        catch(final Throwable t) {
            logger.warn("Error executing cache operation: {}", t.getMessage());
            return null;
        }
    }


    @Override
    public <T> T execute(final RedisScript<T> script, final RedisSerializer<?> argsSerializer, final RedisSerializer<T> resultSerializer, final List<String> keys, final Object... args) {
        try {
            return super.execute(script, argsSerializer, resultSerializer, keys, args);
        }
        catch(final Throwable t) {
            logger.warn("Error executing cache operation: {}", t.getMessage());
            return null;
        }
    }


    @Override
    public <T> T execute(final SessionCallback<T> session) {
        try {
            return super.execute(session);
        }
        catch(final Throwable t) {
            logger.warn("Error executing cache operation: {}", t.getMessage());
            return null;
        }
    }
}

I have taken reference for this logging template from here : https://stackoverflow.com/a/26666102/8499307

And I am using this configuration in application.properties

spring.data.redis.repositories.enabled=false

spring.cache.type=redis
spring.redis.host=localhost
spring.redis.port=6379
spring.redis.database=0
spring.redis.pool.max-active=256
spring.redis.pool.max-idle=12
spring.redis.pool.max-wait=100
spring.redis.pool.min-idle=6
spring.redis.timeout=100

I was using earlier stringRedisTemplate.opsForValue().get(key) to fetch the data but in few posts they suggested to use callback to close the connections properly but that also did not worked.

Please Comment if anything else is required.

Solution

Problem Solved :)

I have made changes only in RedisService and now it is working like a charm. I have done changes to close connections, once read/write/delete operation is done.

You can checkout the code below:

import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.data.redis.RedisConnectionFailureException;
import org.springframework.data.redis.connection.jedis.JedisConnectionFactory;
import org.springframework.data.redis.core.StringRedisTemplate;
import org.springframework.stereotype.Component;

@Component
public class RedisService {

    @Autowired
    private LoggingRedisTemplate stringRedisTemplate;

    private static final Logger logger = LoggerFactory.getLogger(RedisService.class);

    public String getStringValue(final String key) {
        String data = stringRedisTemplate.opsForValue().get(key);

        try {
            closeConnection(stringRedisTemplate);
        }catch (RedisConnectionFailureException e){
            closeClients(stringRedisTemplate);
        }finally {
            closeConnection(stringRedisTemplate);
        }
        return data;
    }

    public void setStringValue(final String key, final String value) {
        stringRedisTemplate.opsForValue().setIfAbsent(key, value);

        try {
            closeConnection(stringRedisTemplate);
        }catch (RedisConnectionFailureException e){
            closeClients(stringRedisTemplate);
        }finally {
            closeConnection(stringRedisTemplate);
        }
    }

    public void removeStringValue(final String key) {
        stringRedisTemplate.delete(key);

        try {
            closeConnection(stringRedisTemplate);
        }catch (RedisConnectionFailureException e){
            closeClients(stringRedisTemplate);
        }finally {
            closeConnection(stringRedisTemplate);
        }
    }

    private void closeConnection(StringRedisTemplate stringRedisTemplate){
        try {
            JedisConnectionFactory connectionFactory = (JedisConnectionFactory) stringRedisTemplate.getConnectionFactory();
            connectionFactory.getConnection().close();
            connectionFactory.destroy();
        }catch (RedisConnectionFailureException e){
            logger.info("Connection closed already");
        }
    }

    private void closeClients(LoggingRedisTemplate stringRedisTemplate){
        try {
            if(null != stringRedisTemplate.getClientList()){
                stringRedisTemplate.getClientList().remove(0);
                stringRedisTemplate.getClientList().remove(1);
                stringRedisTemplate.getClientList().forEach(redisClientInfo -> {
                    String address = redisClientInfo.getAddressPort();
                    if(null != address){
                        String [] addressList = address.split(":");
                        stringRedisTemplate.killClient(addressList[0],Integer.parseInt(addressList[1]));
                    }
                });
            }
        }catch (Exception e){
            logger.warn("Unable to close client connections, ", e);
        }
    }

}

I wish, it will help others also :)

Comments

[Denys]

I would suggest to try with commons-pool2-2 (if you really need to use that). Version 1.6 is very old, not sure how Jedis 2.9 and Spring Data Redis will work with it. There is a chance that connections are not returning back and exhausting as a result.

[ramkishorbajpai]

Thanks for the response. I will try using commons-pool2-2. If you can look into code and tell me what I am missing and why connections are not returned to the pool.

[Nobita]

On prod, we are having a redis which lot of apps are using, So will this only close connection/clients from my app or for all the apps?

[ramkishorbajpai]

@Nobita Yes, this should close connections only for your app