• Progen
    September 28, 2021

    Fault Tolerance in Spring Cloud Applications using Netflix Hystrix Circuit Breaker

    In this article, I will illustrate how you could build a fault tolerant Spring cloud application using Hystrix circuit breaker pattern. Many people have struggled to setup a standalone Hystrix dashboard for their spring apps. Especially with the new spring cloud (Hoxton.SR8) and Spring boot 2.3.4.RELEASE the stream URLs have changed and break the older version dashboard.

    This article will demonstrate how to build a standalone Hystrix dashboard and a central hystrix stream aggregator which collects events via RabbitMQ. In the end of the document, there is a link to the working sample.

    What is needed

    • RabbitMQ, Spring Cloud Hoxton.SR8 or higher, Spring Boot 2.3.4 Release or higher
    • Eureka server application
    • Turbine stream monitoring application
    • Turbine stream aggregator application using RabbitMQ
    • Spring application to test the circuit breaker

    1. Setup Eureka Server

    • Create a spring boot application with below dependencies
    <dependency>
        <groupId>org.springframework.cloud</groupId>
        <artifactId>spring-cloud-starter-netflix-eureka-server</artifactId>
    </dependency>
    • Add required config in bootstrap.yml. In below sample, EUREKA_SERVER_ADDRESS (in this case it is http://localhost:8761) is an environment variable been passed to the application during runtime
    server:
      port: 8761eureka:
      instance:
        prefer-ip-address: true
      client:
        registerWithEureka: false
        fetchRegistry: false
        serviceUrl:
          defaultZone: ${EUREKA_SERVER_ADDRESS}/eureka/
      server:
        waitTimeInMsWhenSyncEmpty: 0
    • Enable standalone Eureka server by annotating the main class with @EnableEurekaServer
    package com.example.eureka;
    import org.springframework.boot.SpringApplication;
    import org.springframework.boot.autoconfigure.SpringBootApplication;
    import org.springframework.cloud.netflix.eureka.server.EnableEurekaServer;@SpringBootApplication
    @EnableEurekaServer
    public class EurekaApplication {    public static void main(String[] args) {
            SpringApplication.run(EurekaApplication.class, args);
        }
    }

    2. Setup the hystrix stream monitoring application

    • Create a spring boot application with below dependencies. This app will enable a standlone Hystrix dashboard
    <dependency>
        <groupId>org.springframework.cloud</groupId>
        <artifactId>spring-cloud-starter-netflix-hystrix-dashboard</artifactId>
    </dependency>
    <dependency>
        <groupId>org.springframework.cloud</groupId>
        <artifactId>spring-cloud-starter-netflix-eureka-client</artifactId>
    </dependency>
    • Configure bootstrap.yml with eureka client config, server port and most importantly allow to proxy stream to all endpoints (*) or to specific stream URL’s

    Note: The context-path is optional to set

    server:
      servlet:
        context-path: /monitoring
      port: 9000eureka:
      client:
        registerWithEureka: true
        serviceUrl:
          defaultZone: ${EUREKA_SERVER_ADDRESS}
    
    
    hystrix:
      dashboard:
        proxy-stream-allow-list: "*"
    • Enable Hystrix Dashboard by annotating the main class with @EnableHystrixDashboard annotation. Also add @EnableDiscoveryClient annotation to enable eureka client
    package com.example.monitoring;
    
    import org.springframework.boot.SpringApplication;
    import org.springframework.boot.autoconfigure.SpringBootApplication;
    import org.springframework.cloud.client.discovery.EnableDiscoveryClient;
    import org.springframework.cloud.netflix.hystrix.dashboard.EnableHystrixDashboard;
    
    @SpringBootApplication
    @EnableHystrixDashboard
    @EnableDiscoveryClient
    public class MonitoringApplication {
    
        public static void main(String[] args) {
            SpringApplication.run(MonitoringApplication.class, args);
        }
    }

    3. Setup the turbine stream application

    • Create a spring boot application with below dependencies
    <dependency>
        <groupId>org.springframework.cloud</groupId>
        <artifactId>spring-cloud-starter-netflix-eureka-client</artifactId>
    </dependency>
    <dependency>
        <groupId>org.springframework.cloud</groupId>
        <artifactId>spring-cloud-starter-netflix-turbine-stream</artifactId>
    </dependency>
    <dependency>
        <groupId>org.springframework.cloud</groupId>
        <artifactId>spring-cloud-starter-stream-rabbit</artifactId>
    </dependency>
    • Configure bootstrap.yml with eureka client config. One of the Spring upgrades has broken the turbine stream exchange name. So make sure to specify the stream destination name explicitly in config and expose all the management api’s for the application
    server:
      port: 8989
    
    eureka:
      client:
        registerWithEureka: true
        serviceUrl:
          defaultZone: ${EUREKA_SERVER_ADDRESS}spring:
      rabbitmq:
        host: localhost
        port: 5672turbine:
      stream:
        destination: hystrixStreamOutput
    
    
    management:
      endpoints:
        web:
          exposure:
            include: "*"
    • Enable turbine stream by annotating the application with @EnableTurbineStream. This application will act as an aggregator for the streams from multiple applications
    package com.example.turbine;
    
    import org.springframework.boot.SpringApplication;
    import org.springframework.boot.autoconfigure.SpringBootApplication;
    import org.springframework.cloud.client.discovery.EnableDiscoveryClient;
    import org.springframework.cloud.netflix.turbine.stream.EnableTurbineStream;
    
    @SpringBootApplication
    @EnableTurbineStream
    @EnableDiscoveryClient
    public class TurbineStreamServiceApplication {
    
       public static void main(String[] args) {
          SpringApplication.run(TurbineStreamServiceApplication.class, args);
       }
    }

    4. Spring cloud test application

    • Create a Spring boot application with below dependencies
    <dependency>
        <groupId>org.springframework.cloud</groupId>
        <artifactId>spring-cloud-starter-netflix-hystrix</artifactId>
    </dependency>
    <dependency>
        <groupId>org.springframework.cloud</groupId>
        <artifactId>spring-cloud-starter-stream-rabbit</artifactId>
    </dependency>
    <dependency>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-web</artifactId>
    </dependency>
    <dependency>
        <groupId>org.springframework.cloud</groupId>
        <artifactId>spring-cloud-starter-netflix-eureka-client</artifactId>
    </dependency>
    • Add RabbitMQ config and eureka client config in bootstrap.yml
    spring:
      application:
        name: test-service
      rabbitmq:
        host: localhost
        port: 5672
    server:
      servlet:
        context-path: /test
      port: 9001
    eureka:
      client:
        registerWithEureka: true
        serviceUrl:
          defaultZone: ${EUREKA_SERVER_ADDRESS}
            • Add a Spring service class to simulate the circuit breaker
    package com.example.test.service;
    
    import com.netflix.hystrix.contrib.javanica.annotation.HystrixCommand;
    import org.springframework.beans.factory.annotation.Autowired;
    import org.springframework.context.annotation.Bean;
    import org.springframework.core.ParameterizedTypeReference;
    import org.springframework.http.HttpMethod;
    import org.springframework.stereotype.Service;
    import org.springframework.web.client.RestTemplate;
    
    import java.util.Date;
    
    @Service
    public class TestService {
        public static final String COMMAND_KEY = "TestCommandKey";
    
        @Bean
        public RestTemplate restTemplate() {
            return new RestTemplate();
        }
    
        @Autowired
        RestTemplate restTemplate;
    
        @HystrixCommand(commandKey = COMMAND_KEY, fallbackMethod = "processPageFallback")
        public String processPage(String url) {
    
            String response = restTemplate.exchange(url
                    , HttpMethod.GET
                    , null
                    , new ParameterizedTypeReference<String>() {
                    }).getBody();
    
            System.out.println("Response Received as " + response + " -  " + new Date());
    
            return "All ok we are in " + url;
        }
    
        public String processPageFallback(String url) {
            url = "www.yahoo.com";
            System.out.println("Falling back to " + url);
            String response = restTemplate.exchange("http://www.yahoo.com"
                    , HttpMethod.GET
                    , null
                    , new ParameterizedTypeReference<String>() {
                    }).getBody();
    
            System.out.println("Response Received as " + response + " -  " + new Date());
    
            return "Failed falling back to " + url;
        }
    
    
    }
    • Expose the service via Spring Controller
    package com.example.test.controller;
    
    import com.example.test.service.TestService;
    import org.springframework.beans.factory.annotation.Autowired;
    import org.springframework.web.bind.annotation.GetMapping;
    import org.springframework.web.bind.annotation.RequestParam;
    import org.springframework.web.bind.annotation.RestController;
    
    @RestController("/test")
    public class TestController {
        @Autowired
        TestService testService;
    
        @GetMapping
        public String testService(@RequestParam("url") String url) {
            return testService.processPage(url);
        }
    
    }

    Testing Time

    1. Start RabbitMQ Service
    2. Bring up the Eureka server application. Access the eureka web console via http://localhost:8761. You will notice that the monitoring, stream aggregator and test service applications are registered with eureka

    3. Bring up the turbine stream aggregator application. Note that the turbine stream URL will be http://localhost:8989/. If we specify a context name, then make sure to append the context name to the end of the URL

    4. Bring up the monitoring application. You will be able to access the monitoring dashboard by navigating to http://localhost:9000/monitoring/hystrix fill in the turbine stream from above step and click Monitor Stream button

    5. Call the test REST api via browser http://localhost:9001/test/?url=http://www.google.com. Try giving a wrong URL in the argument and you will see that the fallback method is triggered

    Success
    Failure

    That’s it, hope you like this article. A working version of the sample is available here https://github.com/negorp/hystrix_fault_tolerance.git

    Hope this article helps!. Remember to like and share 🙂