A Minimalistic Circuit Breaker Pattern Implementation For Java EE

The Circuit Breakr Pattern aims to open the "circuit" after hitting a configurable threshold of erroneous method calls.

A natural fit is the use of an interceptor to measure the performance and monitor the exceptions. After reaching the threshold the interceptor returns without calling the target:


public class Breakr {

    private long errorCounter;
    private static final int FAILURES_COUNT = 5;
    private static final int TIMEOUT_IN_MS = 1000;

    @AroundInvoke
    public Object guard(InvocationContext ic) throws Exception {
        long maxNbrOfFailures = FAILURES_COUNT;
        long timeout = TIMEOUT_IN_MS;

        IgnoreCallsWhen configuration = ic.
                getMethod().
                getAnnotation(IgnoreCallsWhen.class);

        if (configuration != null) {
            maxNbrOfFailures = configuration.failures();
            timeout = configuration.slowerThanMillis();
        }

        long start = System.currentTimeMillis();
        try {
            if (errorCounter >= maxNbrOfFailures) {
                return null;
            }
            return ic.proceed();
        } catch (Exception ex) {
            errorCounter++;
            throw ex;
        } finally {
            long duration = System.currentTimeMillis() - start;
            if (duration >= timeout) {
                errorCounter++;
            }
        }
    }

}

The threshold can be easily configured with an annotation:


@Target(ElementType.METHOD)
@Retention(RetentionPolicy.RUNTIME)
public @interface IgnoreCallsWhen {

    long slowerThanMillis() default 1000;

    long failures() default 3;
}

Usage example:


@Singleton
@Interceptors(Breakr.class)
public class Slow {

    @IgnoreCallsWhen(slowerThanMillis = 10)
    public String tooSlow() {
        try {
            Thread.sleep(100);
        } catch (InterruptedException ex) {}
        return "Slow: " + System.currentTimeMillis();
    }

}

The source is available from github.com/AdamBien/breakr.

Use the following maven dependency to install breakr:


<dependency>
	<groupId>com.airhacks</groupId>
	<artifactId>breakr</artifactId>
	<version>[CURRENT_VERSION]</version>
</dependency>

See you at Java EE Workshops at Munich Airport, Terminal 2 particularly in Java EE Microservices. Is Munich's airport too far? Learn from home: airhacks.io.

Comments:

While idea seems sound, the specifics of this implementation seem a bit unusual. An exception will cause errorCounter to increment and the method will not be called; a long invocation will also cause the counter to increment, but the method will still be called no matter how long it takes: there is no interruption. I realize this is just a sample and it must be modified to suit implementation, it just seems very unlikely that this implementation would be suitable without interrupting the long running or possibly frozen method invocation. I guess that's out of scope though :-)
As usual, thanks for the helpful article.

Posted by David on November 04, 2015 at 05:31 AM CET #

@David - Isn't the circuit break pattern supposed to prevent *subsequent* calls from being made in the event of a call taking too long or failing?

What Adam's implementation doesn't appear to do that the Circuit Breaker does prescribe is retrying periodically and closing the circuit when a call succeeds. It's a great start though, especially as an example of using interceptors for cross-cutting concerns.

Posted by David on November 05, 2015 at 12:11 PM CET #

@David circuit breaker isn't about to interrupt and clean up frozen invocations. It's pattern used to detect unavailability of parts of distributed system and preventing new requests to be send to this broken parts.
This example implementation is missing one very important part. There should be scheduler or some other solution to reset errorcount.
In this example code when system once detect failure and open circuit breaker, it will never come back to healthy behaviour.
Note that this sample code is not thread-safe (comparison and incrementation of errorcount should be done on atomicinteger).
I can't remember how singleton EJB behave on RuntimeException. SSB is destroyed when one is thrown and new instance is created. If the same applies to singleton EJB, circuit breaker won't work when code underneath throws RuntimeEx.
Sorry for typos. I'm writing on tablet with polish spellchecker ;)

Posted by Marek on November 05, 2015 at 07:08 PM CET #

Hello. I really enjoy reading your blog. And i'm happy that you spread the knowledge about important patterns and solutions like CircuitBreaker. In times of microservices, it's really relevant how we think about securing remote communication between multiple applications and how we react on network/application failure as a client/caller.

I have some concerns about your solution :

1. Main thing that does not fit for me is lack of possibility to reset errorCounter. I do not know if I understood it correctly, but if circuit once is opened the only option for closing it again is restart of the application. Of course we can always throw RuntimeException to destroy our SSB and at the same time Interceptor instance but this is not the best idea. There should be some easier solution for the possibility of retry/close. The simplest idea is to run scheduled task from ScheduledExecutorService with some defined configurable delay - like 5 minutes. The purpose of this task should be reseting state of errorCounter which is equivalent to closing circuit.

2. Singleton with @ConcurrencyManagement(ConcurrencyManagementType.BEAN) allow full concurrent access to all methods. This means that the solution is not thread-safe. Incrementing and comparing errorCount variable is not safe. Maybe you should consider using AtomicLong.

3. In normal SSB (not Singleton) if the result of the method call is always RuntimeException than CircuitBreaker interceptor instance will be always destroyed and it will never increment error counter.

4. Your solution depends on JAVA EE technology. It works only in EE enviroment. Solutions as this should be independent of frameworks.

I want to share with you my implementation of CircuitBreaker https://github.com/nikom1337/simple-circuitbreaker based on Hystrix library from Netflix. It's not full CircuitBreaker due to missing HALF-OPEN state. Also my implementation depends on Hystrix Sliding Window algorithm for measuring circuit health. If i find some spare time, I will implement some simpler solution to get rid of Hystrix dependence and decrease coupling. But this is simple, thread-safe and technology agnostic example which you can use in standard JAVA application and also in JAVA EE application.

PS. You have typo in pattern name - it should be circuit not cirquit :)

Posted by Korneliusz on November 05, 2015 at 07:21 PM CET #

@David,

this implementation is already working for me perfectly :-). I'm just sharing the draft. Version 0.0.2 is waiting to be released. I'm just thinking whether I could remove anything.

thanks for the comment,

adam

Posted by 47.68.108.224 on November 06, 2015 at 05:38 AM CET #

@Korneliusz,

"1. Main thing that does not fit for me is lack of possibility to reset errorCounter.(...)"

+1 However - it works for me now, because in an error case I'm just restarting the docker container. Version 0.0.2 is on the way :-)

"Singleton with @ConcurrencyManagement(ConcurrencyManagementType.BEAN) allow full concurrent access to all methods.(...)"

Yes. This is only a usage example. A restricted resource is usually a singleton or injected proxy of a singleton (e.g. DataSource, EntityManager).

" This means that the solution is not thread-safe"
Accessing int should be threadsafe. However, I'm going to use AtomicLong in the future version as you suggested.

" In normal SSB (not Singleton) if the result of the method call is always RuntimeException"

A Stateless Session Bean is not the right abstraction. Usually we used Circuit Breaker between WARs / Apps. In such case @Singleton would be the way to go.

"Your solution depends on JAVA EE technology. It works only in EE enviroment. Solutions as this should be independent of frameworks."

The first time I do not agree at all with you :-). Because of Java EE, this pattern comprises one class and one annotation. The version 0.0.2 will introduce another class. The solution is going to be another "kb" library. A generic solution would be far more bloated than this. For me a small library / deployment size is crucial for fast deployment cycles and productivity.

I spend 99% of my time in Java EE projects right now :-)

" You have typo in pattern name - it should be circuit not cirquit"

Thanks - I fixed that. Interestingly - I mistyped the pattern only in the post and not at github ..."

thanks for sharing your implementation!,

cheers,

adam

Posted by 47.68.108.224 on November 06, 2015 at 05:54 AM CET #

Have you used any market solution for circuit breaking like hystrix and found it bad or just wanted to have really compact implementation?

Posted by Damian on November 16, 2015 at 11:56 AM CET #

You probably don't want the finally block. You want to move the code inside the finally block after the ic.proceed().

Currently, if there is an exception thrown AND the method took too long to execute, the counter will be incremented twice since the finally block will be executed even if you throw an exception.

Posted by Robert Saulnier on December 08, 2016 at 06:48 PM CET #

Post a Comment:
  • HTML Syntax: NOT allowed
...the last 150 posts
...the last 10 comments
License