Spring Boot + Eureka Server + Hystrix with Turbine: empty turbine.stream

Question

I'm trying to run Spring Boot (with Spring Cloud) + Eureka Server + Hystrix Dashboard and Turbine stream, but I run into a problem I couldn't find any solution so far. I use Spring Boot 1.2.1.RELEASE and Spring Cloud 1.0.0.RC2. Here is what I have:

The first instance is running Eureka server and Hystrix dashboard:

@Configuration
@EnableAutoConfiguration
@EnableEurekaServer
@EnableHystrixDashboard
@EnableDiscoveryClient
class Application {

    public static void main(String[] args) {
        SpringApplication.run Application, args
    }
}

Here you can find build.gradle for that instance - https://gist.github.com/wololock/570272ad7cf2d14a4d3c

Eureka server is running ok, I can see registered instances on eureka server dashboard, I can also use LoadBalancer to get the URL to registered instance using its id. So far everything is ok.

I have a few instances that are run with @EnableHystrix annotation and use @HystrixCommand to define which methods have to be monitored by Hystrix. When I pass URL to hystrix.stream of single instance to Hystrix dashboard, I can see it running with no problem.

I have also separate Turbine server, not complicated one:

@EnableAutoConfiguration
@EnableTurbine
@Configuration
@EnableDiscoveryClient
class Application {

    public static void main(String[] args) {
        SpringApplication.run Application, args
    }
}

Here you can find build.gradle for Turbine server instance - https://gist.github.com/wololock/ff0d855b8a890232851e

It uses very simple configuration, build mostly on the one provided by sample turbine app - https://github.com/spring-cloud-samples/turbine

info:
  component: Turbine

endpoints:
  restart:
    enabled: true
  shutdown:
    enabled: true

turbine:
  appConfig: pdf-creator-service

InstanceDiscovery:
  impl: io.spring.platform.netflix.turbine.EurekaInstanceDiscovery

server:
  port: 8989

management:
  port: 8990

eureka:
  instance:
    leaseRenewalIntervalInSeconds: 10
  client:
    serviceUrl:
      defaultZone: ${vcap.services.${PREFIX:}eureka.credentials.uri:http://user:password@localhost:8761}/eureka/

After running those instances in order:

1. eureka server
2. turbine server
3. discover client instance,

I have 2nd and 3rd instance registered in eureka server, turbine server log says, that there is one instance up:

[2015-02-06 12:35:04.162] boot - 20495  INFO [Timer-0] --- EurekaInstanceDiscovery: Fetching instance list for apps: [pdf-creator-service]
[2015-02-06 12:35:04.162] boot - 20495  INFO [Timer-0] --- EurekaInstanceDiscovery: Fetching instances for app: pdf-creator-service
[2015-02-06 12:35:04.162] boot - 20495  INFO [Timer-0] --- EurekaInstanceDiscovery: Received instance list for app: pdf-creator-service = 1
[2015-02-06 12:35:04.162] boot - 20495  INFO [Timer-0] --- InstanceObservable: Retrieved hosts from InstanceDiscovery: 1
[2015-02-06 12:35:04.162] boot - 20495  INFO [Timer-0] --- InstanceObservable: Found hosts that have been previously terminated: 0
[2015-02-06 12:35:04.162] boot - 20495  INFO [Timer-0] --- InstanceObservable: Hosts up:1, hosts down: 0
[2015-02-06 12:36:04.162] boot - 20495  INFO [Timer-0] --- EurekaInstanceDiscovery: Fetching instance list for apps: [pdf-creator-service]
[2015-02-06 12:36:04.162] boot - 20495  INFO [Timer-0] --- EurekaInstanceDiscovery: Fetching instances for app: pdf-creator-service
[2015-02-06 12:36:04.162] boot - 20495  INFO [Timer-0] --- EurekaInstanceDiscovery: Received instance list for app: pdf-creator-service = 1
[2015-02-06 12:36:04.162] boot - 20495  INFO [Timer-0] --- InstanceObservable: Retrieved hosts from InstanceDiscovery: 1
[2015-02-06 12:36:04.162] boot - 20495  INFO [Timer-0] --- InstanceObservable: Found hosts that have been previously terminated: 0
[2015-02-06 12:36:04.162] boot - 20495  INFO [Timer-0] --- InstanceObservable: Hosts up:1, hosts down: 0

Calling hystrix.stream from a single instance works, e.g. curl http://localhost:8885/hystrix.stream returns:

data: {"type":"HystrixCommand","name":"post","group":"PdfController","currentTime":1423223614259,"isCircuitBreakerOpen":false,"errorPercentage":0,"errorCount":0,"requestCount":0,"rollingCountCollapsedRequests":0,"rollingCountExceptionsThrown":0,"rollingCountFailure":0,"rollingCountFallbackFailure":0,"rollingCountFallbackRejection":0,"rollingCountFallbackSuccess":0,"rollingCountResponsesFromCache":0,"rollingCountSemaphoreRejected":0,"rollingCountShortCircuited":0,"rollingCountSuccess":0,"rollingCountThreadPoolRejected":0,"rollingCountTimeout":0,"currentConcurrentExecutionCount":0,"latencyExecute_mean":0,"latencyExecute":{"0":0,"25":0,"50":0,"75":0,"90":0,"95":0,"99":0,"99.5":0,"100":0},"latencyTotal_mean":0,"latencyTotal":{"0":0,"25":0,"50":0,"75":0,"90":0,"95":0,"99":0,"99.5":0,"100":0},"propertyValue_circuitBreakerRequestVolumeThreshold":20,"propertyValue_circuitBreakerSleepWindowInMilliseconds":5000,"propertyValue_circuitBreakerErrorThresholdPercentage":50,"propertyValue_circuitBreakerForceOpen":false,"propertyValue_circuitBreakerForceClosed":false,"propertyValue_circuitBreakerEnabled":true,"propertyValue_executionIsolationStrategy":"THREAD","propertyValue_executionIsolationThreadTimeoutInMilliseconds":8000,"propertyValue_executionIsolationThreadInterruptOnTimeout":true,"propertyValue_executionIsolationThreadPoolKeyOverride":null,"propertyValue_executionIsolationSemaphoreMaxConcurrentRequests":10,"propertyValue_fallbackIsolationSemaphoreMaxConcurrentRequests":10,"propertyValue_metricsRollingStatisticalWindowInMilliseconds":10000,"propertyValue_requestCacheEnabled":true,"propertyValue_requestLogEnabled":true,"reportingHosts":1}

data: {"type":"HystrixCommand","name":"generate","group":"WkHtmlToPdfGenerator","currentTime":1423223614259,"isCircuitBreakerOpen":false,"errorPercentage":0,"errorCount":0,"requestCount":0,"rollingCountCollapsedRequests":0,"rollingCountExceptionsThrown":0,"rollingCountFailure":0,"rollingCountFallbackFailure":0,"rollingCountFallbackRejection":0,"rollingCountFallbackSuccess":0,"rollingCountResponsesFromCache":0,"rollingCountSemaphoreRejected":0,"rollingCountShortCircuited":0,"rollingCountSuccess":0,"rollingCountThreadPoolRejected":0,"rollingCountTimeout":0,"currentConcurrentExecutionCount":0,"latencyExecute_mean":0,"latencyExecute":{"0":0,"25":0,"50":0,"75":0,"90":0,"95":0,"99":0,"99.5":0,"100":0},"latencyTotal_mean":0,"latencyTotal":{"0":0,"25":0,"50":0,"75":0,"90":0,"95":0,"99":0,"99.5":0,"100":0},"propertyValue_circuitBreakerRequestVolumeThreshold":20,"propertyValue_circuitBreakerSleepWindowInMilliseconds":5000,"propertyValue_circuitBreakerErrorThresholdPercentage":50,"propertyValue_circuitBreakerForceOpen":false,"propertyValue_circuitBreakerForceClosed":false,"propertyValue_circuitBreakerEnabled":true,"propertyValue_executionIsolationStrategy":"THREAD","propertyValue_executionIsolationThreadTimeoutInMilliseconds":8000,"propertyValue_executionIsolationThreadInterruptOnTimeout":true,"propertyValue_executionIsolationThreadPoolKeyOverride":null,"propertyValue_executionIsolationSemaphoreMaxConcurrentRequests":10,"propertyValue_fallbackIsolationSemaphoreMaxConcurrentRequests":10,"propertyValue_metricsRollingStatisticalWindowInMilliseconds":10000,"propertyValue_requestCacheEnabled":true,"propertyValue_requestLogEnabled":true,"reportingHosts":1}

data: {"type":"HystrixThreadPool","name":"PdfController","currentTime":1423223614259,"currentActiveCount":0,"currentCompletedTaskCount":4,"currentCorePoolSize":10,"currentLargestPoolSize":4,"currentMaximumPoolSize":10,"currentPoolSize":4,"currentQueueSize":0,"currentTaskCount":4,"rollingCountThreadsExecuted":0,"rollingMaxActiveThreads":0,"propertyValue_queueSizeRejectionThreshold":5,"propertyValue_metricsRollingStatisticalWindowInMilliseconds":10000,"reportingHosts":1}

But when I attach turbine.stream to hystrix dashboard, I get nothing. Logs say:

[2015-02-06 12:42:48.922] boot - 24816  INFO [Timer-0] --- EurekaInstanceDiscovery: Received instance list for app: pdf-creator-service = 1
[2015-02-06 12:42:48.922] boot - 24816  INFO [Timer-0] --- InstanceObservable: Retrieved hosts from InstanceDiscovery: 1
[2015-02-06 12:42:48.922] boot - 24816  INFO [Timer-0] --- InstanceObservable: Found hosts that have been previously terminated: 0
[2015-02-06 12:42:48.922] boot - 24816  INFO [Timer-0] --- InstanceObservable: Hosts up:1, hosts down: 0
[2015-02-06 12:43:26.237] boot - 24816  INFO [XNIO-2 task-4] --- TurbineStreamServlet: FilterCriteria: []
[2015-02-06 12:43:26.237] boot - 24816  INFO [XNIO-2 task-4] --- TurbineStreamServlet: StatsType filters: []
[2015-02-06 12:43:26.237] boot - 24816  INFO [XNIO-2 task-4] --- TurbineStreamingConnection: Relevance config: []
[2015-02-06 12:43:26.237] boot - 24816  INFO [XNIO-2 task-4] --- TurbineStreamingConnection: Relevance metrics config: {}
[2015-02-06 12:43:26.237] boot - 24816  INFO [XNIO-2 task-4] --- ClusterMonitor: Registering event handler for cluster monitor: StreamingHandler_f1308dda-58c5-47a5-b1e2-5a0bea32226b
[2015-02-06 12:43:26.237] boot - 24816  INFO [XNIO-2 task-4] --- TurbineDataDispatcher: 

Just added and starting handler tuple: StreamingHandler_f1308dda-58c5-47a5-b1e2-5a0bea32226b
[2015-02-06 12:43:26.238] boot - 24816  INFO [XNIO-2 task-4] --- AggDataFromCluster: Per handler dispacher started for: StreamingHandler_f1308dda-58c5-47a5-b1e2-5a0bea32226b
[2015-02-06 12:43:26.238] boot - 24816  INFO [XNIO-2 task-4] --- ClusterMonitor: All event handlers for cluster monitor: [StreamingHandler_637572ab-acda-4bf4-81cd-6a658adb73eb, StreamingHandler_f1308dda-58c5-47a5-b1e2-5a0bea32226b, StaticListener_For_Aggregator, StreamingHandler_5ec12ee8-3fcd-4a6f-9006-d2a6ecc309d0, StreamingHandler_72d7b9e2-ad98-42a0-9ac3-abe4aa57cc7a]
[2015-02-06 12:43:26.238] boot - 24816  INFO [XNIO-2 task-4] --- ClusterMonitor: Starting up the cluster monitor for default_agg

If I do curl http://localhost:8989/turbine.stream I get only:

: ping
data: {"reportingHostsLast10Seconds":0,"name":"meta","type":"meta","timestamp":1423223006935}

: ping
data: {"reportingHostsLast10Seconds":0,"name":"meta","type":"meta","timestamp":1423223010935}

: ping
data: {"reportingHostsLast10Seconds":0,"name":"meta","type":"meta","timestamp":1423223013936}

: ping
data: {"reportingHostsLast10Seconds":0,"name":"meta","type":"meta","timestamp":1423223017936}

: ping
data: {"reportingHostsLast10Seconds":0,"name":"meta","type":"meta","timestamp":1423223020937}

: ping
: ping
data: {"reportingHostsLast10Seconds":0,"name":"meta","type":"meta","timestamp":1423223024937}

: ping
data: {"reportingHostsLast10Seconds":0,"name":"meta","type":"meta","timestamp":1423223028938}

: ping
data: {"reportingHostsLast10Seconds":0,"name":"meta","type":"meta","timestamp":1423223032938}

: ping
: ping
data: {"reportingHostsLast10Seconds":0,"name":"meta","type":"meta","timestamp":1423223036938}

: ping
data: {"reportingHostsLast10Seconds":0,"name":"meta","type":"meta","timestamp":1423223039939}

: ping
data: {"reportingHostsLast10Seconds":0,"name":"meta","type":"meta","timestamp":1423223043939}

: ping
data: {"reportingHostsLast10Seconds":0,"name":"meta","type":"meta","timestamp":1423223046940}

: ping
data: {"reportingHostsLast10Seconds":0,"name":"meta","type":"meta","timestamp":1423223050940}

: ping
: ping
data: {"reportingHostsLast10Seconds":0,"name":"meta","type":"meta","timestamp":1423223054941}

My question is: do I missed something in setting it up and running? Previously I tried to manage this by having eureka and turbine server in single instance, but in that case turbine couldn't even find the registered application in eureka using proper application name. I made a progress after splitting eureka and turbine, but it still does not work correctly.

I will be grateful for any suggestion. If you need more information, just let me know, I might miss something important.

UPDATE 20150209

Following Dave's suggestion I applied small changes in application.yml file of turbine-server. Now the file contains only:

info:
  component: Turbine

turbine:
  appConfig: pdf-creator-service
  clusterNameExpression: 'default'

server:
  port: 8989

management:
  port: 8990

eureka:
  instance:
    leaseRenewalIntervalInSeconds: 10
  client:
    serviceUrl:
      defaultZone: ${vcap.services.${PREFIX:}eureka.credentials.uri:http://user:password@localhost:8761}/eureka/

But it doesn't make turbine.stream working. After turbine-server gets aware of registered client in eureka server, it fails with given exception:

[2015-02-09 21:25:03.516] boot - 4808  INFO [Timer-0] --- EurekaInstanceDiscovery: Fetching instance list for apps: [pdf-creator-service]
[2015-02-09 21:25:03.516] boot - 4808  INFO [Timer-0] --- EurekaInstanceDiscovery: Fetching instances for app: pdf-creator-service
[2015-02-09 21:25:03.516] boot - 4808  INFO [Timer-0] --- EurekaInstanceDiscovery: Received instance list for app: pdf-creator-service = 1
[2015-02-09 21:25:03.520] boot - 4808 ERROR [Timer-0] --- EurekaInstanceDiscovery: Failed to fetch instances for app: pdf-creator-service, retrying once more
org.springframework.expression.spel.SpelEvaluationException: EL1008E:(pos 0): Property or field 'default' cannot be found on object of type 'com.netflix.appinfo.InstanceInfo' - maybe not public?
    at org.springframework.expression.spel.ast.PropertyOrFieldReference.readProperty(PropertyOrFieldReference.java:226)
    at org.springframework.expression.spel.ast.PropertyOrFieldReference.getValueInternal(PropertyOrFieldReference.java:93)
    at org.springframework.expression.spel.ast.PropertyOrFieldReference.getValueInternal(PropertyOrFieldReference.java:81)
    at org.springframework.expression.spel.ast.SpelNodeImpl.getValue(SpelNodeImpl.java:120)
    at org.springframework.expression.spel.standard.SpelExpression.getValue(SpelExpression.java:242)
    at org.springframework.cloud.netflix.turbine.EurekaInstanceDiscovery.getClusterName(EurekaInstanceDiscovery.java:183)
    at org.springframework.cloud.netflix.turbine.EurekaInstanceDiscovery.marshallInstanceInfo(EurekaInstanceDiscovery.java:141)
    at org.springframework.cloud.netflix.turbine.EurekaInstanceDiscovery.getInstancesForApp(EurekaInstanceDiscovery.java:123)
    at org.springframework.cloud.netflix.turbine.EurekaInstanceDiscovery.getInstanceList(EurekaInstanceDiscovery.java:88)
    at com.netflix.turbine.discovery.InstanceObservable.getInstanceList(InstanceObservable.java:327)
    at com.netflix.turbine.discovery.InstanceObservable.access$500(InstanceObservable.java:66)
    at com.netflix.turbine.discovery.InstanceObservable$1.run(InstanceObservable.java:258)
    at java.util.TimerThread.mainLoop(Timer.java:555)
    at java.util.TimerThread.run(Timer.java:505)

I tried writting 'default' as well as default, but the result is unfortunately the same.

Solution

Thanks Dave Syer for finding the proper solution. Basically what did the trick was adding:

turbine:
    clusterNameExpression: new String("default")

to application.yml file in turbine server application instance. It might look weird, I didn't believe it will work, but it does. Now when I call my hystrix client application I get the proper information in hystrix.stream that is served by this application and in turbine.stream of Turbine server as well. My current application.yml in turbine server looks as follows:

info:
  component: Turbine

turbine:
  clusterNameExpression: new String("default")
  appConfig: pdf-creator-service

server:
  port: 8989

management:
  port: 8990

eureka:
  instance:
    leaseRenewalIntervalInSeconds: 10
  client:
    serviceUrl:
      defaultZone: ${vcap.services.${PREFIX:}eureka.credentials.uri:http://user:password@localhost:8761}/eureka/

turbine.appConfig holds the information about the hystrix clients (by their IDs). To add another client to your turbine server you will have to simply put another instance id, coma separated with the previous one. And that's all folks :)

Solution 1

It works for me if I add some configuration for the cluster, e.g.

turbine:
  appConfig: customers,stores
  clusterNameExpression: new String('default')

Turbine has to know how to construct the "cluster" name (an aggregation key for sets of applications). The default is to use the appname, so if you don't set the clusterNameExpression you need to use a query param in the stream URL, e.g. /turbine.stream?cluster=CUSTOMERS (uppercased appname).

Solution 2

In my case I had :7979/mock.stream as client for testing turbine app. The turbine checks time of event (timeOfEvent) so it will show only the current date time events.

It can be turned off using

turbine.InstanceMonitor.eventStream.skipLineLogic.enabled = false

Comments

[Dave Syer]

I can reproduce the problem at least (see simple sample at github.com/spring-cloud-samples/tests/tree/master/turbine in the test source). I'll have a closer look next week. Maybe Spencer will be able to comment before then.

[Szymon Stepniak]

Thanks for your interest Dave. I will also try to dig into and find something more to help in finding solution for that issue.

[spencergibb]

I don't think 'default' will work. The SPEL expression must evaluate to a property on a Netflix InstanceInfo. Looking now.

[spencergibb]

I see why 'default' works. You don't have turbine.aggregator.clusterConfig defined. Define it to be the uppercase name of you app (what you see in eureka) and remove clusterNameExpression.

[Dave Syer]

That would work too, but then you need "cluster=APPNAME" in the stream url.

[Szymon Stepniak]

Hey Spencer, thanks for your suggestion. I tried the approach with turbine.aggregator.clusterConfig, but the result was the same (I did query the url with additional ?cluster=[CLUSTERNAME] parameter, it failed without it anyway).

[Marcin Grzejszczak]

I've managed to make it work also with the new String('default') but it puzzles me that providing explicit clusters doesn't seem to work at all.

[Yonatan]

Please note that the documentation says you should double quote the expression. That is, you should write clusterNameExpression: "'default'". cloud.spring.io/spring-cloud-netflix/spring-cloud-netflix.ht‌​ml. Please consider to re-edit the answer...

[PAA]

Could you pls guide me here stackoverflow.com/questions/61444872/… ?

[Szymon Stepniak]

Hey Dave, I tried out adding clusterNameExpression: 'default' to the application.yml in turbine server, but it didnt help and cause an exception. I updated initial post with all details.

[Dave Syer]

Escaping the quotes seems to be a problem. It worked for me. Maybe try new String('default')?

[Szymon Stepniak]

I must admit, that I didn't believe that using new String('default') might change, but it actually did the trick :) Thank you @DaveSyer for your help, I can now move forward and put myself into new troubles :)

[Christian]

finding this answer after months. ;) new String('default') works for me as well. Please consider editing the original answer!!

[Praveen_Shukla]

@SzymonStepniak: where should we place the application.yml file. In the application code or turbine server ?

[Szymon Stepniak]

@Praveen_Shukla In standard path - src/main/resources

[Praveen_Shukla]

@SzymonStepniak: thanks for quick rply. one more question should It be on turbine project or my application project ?

[Szymon Stepniak]

@Praveen_Shukla In Turbine project

[PAA]

@DaveSyer - Could you please guide me here: stackoverflow.com/questions/61431036/… ?