Understanding Spring Cloud Eureka Server self preservation and renew threshold

Question

I am new to developing microservices, although I have been researching about it for a while, reading both Spring's docs and Netflix's.

I have started a simple project available on Github. It is basically a Eureka server (Archimedes) and three Eureka client microservices (one public API and two private). Check github's readme for a detailed description.

The point is that when everything is running I would like that if one of the private microservices is killed, the Eureka server realizes and removes it from the registry.

I found this question on Stackoverflow, and the solution passes by using enableSelfPreservation:false in the Eureka Server config. Doing this after a while the killed service disappears as expected.

However I can see the following message:

THE SELF PRESERVATION MODE IS TURNED OFF.THIS MAY NOT PROTECT INSTANCE EXPIRY IN CASE OF NETWORK/OTHER PROBLEMS.

1. What is the purpose of the self preservation? The doc states that with self preservation on "clients can get the instances that do not exist anymore". So when is it advisable to have it on/off?

Furthermore, when self preservation is on, you may get an outstanding message in the Eureka Server console warning:

EMERGENCY! EUREKA MAY BE INCORRECTLY CLAIMING INSTANCES ARE UP WHEN THEY'RE NOT. RENEWALS ARE LESSER THAN THRESHOLD AND HENCE THE INSTANCES ARE NOT BEING EXPIRED JUST TO BE SAFE.

Now, going on with the Spring Eureka Console.

Lease expiration enabled    true/false
Renews threshold    5
Renews (last min)   4

I have come across a weird behaviour of the threshold count: when I start the Eureka Server alone, the threshold is 1.

2. I have a single Eureka server and is configured with registerWithEureka: false to prevent it from registering on another server. Then, why does it show up in the threshold count?

3. For every client I start the threshold count increases by +2. I guess it is because they send 2 renew messages per min, am I right?

4. The Eureka server never sends a renew so the last min renews is always below the threshold. Is this normal?

renew threshold 5
rewnews last min: (client1) +2 + (client2) +2 -> 4

Server cfg:

server:
  port: ${PORT:8761}

eureka:
  instance:
    hostname: localhost
  client:
    registerWithEureka: false
    fetchRegistry: false
    serviceUrl:
      defaultZone: http://${eureka.instance.hostname}:${server.port}/eureka/
  server:
    enableSelfPreservation: false
#   waitTimeInMsWhenSyncEmpty: 0

Client 1 cfg:

spring:
  application:
    name: random-image-microservice

server:
  port: 9999

eureka:
  client:
    serviceUrl:
      defaultZone: http://localhost:8761/eureka/
    healthcheck:
      enabled: true

Solution 1

I got the same question as @codependent met, I googled a lot and did some experiment, here I come to contribute some knowledge about how Eureka server and instance work.

Every instance needs to renew its lease to Eureka Server with frequency of one time per 30 seconds, which can be define in eureka.instance.leaseRenewalIntervalInSeconds.

Renews (last min): represents how many renews received from Eureka instance in last minute

Renews threshold: the renews that Eureka server expects received from Eureka instance per minute.

For example, if registerWithEureka is set to false, eureka.instance.leaseRenewalIntervalInSeconds is set to 30 and run 2 Eureka instance. Two Eureka instance will send 4 renews to Eureka server per minutes, Eureka server minimal threshold is 1 (written in code), so the threshold is 5 (this number will be multiply a factor eureka.server.renewalPercentThreshold which will be discussed later).

SELF PRESERVATION MODE: if Renews (last min) is less than Renews threshold, self preservation mode will be activated.

So in upper example, the SELF PRESERVATION MODE is activated, because threshold is 5, but Eureka server can only receive 4 renews/min.

1. Question 1:

The SELF PRESERVATION MODE is design to avoid poor network connectivity failure. Connectivity between Eureka instance A and B is good, but B is failed to renew its lease to Eureka server in a short period due to connectivity hiccups, at this time Eureka server can't simply just kick out instance B. If it does, instance A will not get available registered service from Eureka server despite B is available. So this is the purpose of SELF PRESERVATION MODE, and it's better to turn it on.

2. Question 2:

The minimal threshold 1 is written in the code. registerWithEureka is set to false so there will be no Eureka instance registers, the threshold will be 1.

In production environment, generally we deploy two Eureka server and registerWithEureka will be set to true. So the threshold will be 2, and Eureka server will renew lease to itself twice/minute, so RENEWALS ARE LESSER THAN THRESHOLD won't be a problem.

3. Question 3:

Yes, you are right. eureka.instance.leaseRenewalIntervalInSeconds defines how many renews sent to server per minute, but it will multiply a factor eureka.server.renewalPercentThreshold mentioned above, the default value is 0.85.

4. Question 4:

Yes, it's normal, because the threshold initial value is set to 1. So if registerWithEureka is set to false, renews is always below threshold.

I have two suggestions for this:

1. Deploy two Eureka server and enable registerWithEureka.
2. If you just want to deploy in demo/dev environment, you can set eureka.server.renewalPercentThreshold to 0.49, so when you start up a Eureka server alone, threshold will be 0.

Solution 2

I've created a blog post with the details of Eureka here, that fills in some missing detail from Spring doc or Netflix blog. It is the result of several days of debugging and digging through source code. I understand it's preferable to copy-paste rather than linking to an external URL, but the content is too big for an SO answer.

Comments

[codependent]

Thank you for linking such a clarifying post. A recommended reading!!

[jabrena]

Hi Nie Xing, do you know what is the meaning of the Label: "Lease expiration enabled:"

[Nie Xing]

Hi @jabrena, I think it means lease will expire if instance doesn't renew its lease to eureka server. It might be related to eureka.instance.leaseExpirationDurationInSeconds which value is 90 seconds by default. If you set this value to 0 or a negative number, the label might turn to "Lease expiration enabled: False". You can try it out.

[jabrena]

Hi @Abhijit Sarkar, the article is amazing. I solved many configuration doubts.

[Ankit Bansal]

can you answer this stackoverflow.com/questions/48437752/…

[SledgeHammer]

Doesn't work. Threshold=0, Renews=0 still triggers the warning.

[Dhruv]

I disabled the selfPreservationMode but still I get this message on Eureka - EUREKA MAY BE INCORRECTLY CLAIMING INSTANCES ARE UP WHEN THEY'RE NOT. RENEWALS ARE LESSER THAN THRESHOLD AND HENCE THE INSTANCES ARE NOT BEING EXPIRED JUST TO BE SAFE. It should not happen right?

[breakline]

good info thanks