1. Imagine You’re a Dog Who Wants Biscuits 🐕

Hey, buddy! Imagine you’re a dog living in a house.

Every morning, you run to your owner and ask for biscuits.

But today, something unexpected happens.

Your owner is busy and doesn’t hear you.

You bark once.

Nothing happens.

You bark again after a few seconds.

Still nothing!

Finally, your owner hears you and gives you a biscuit.

What did you do?

You tried again after a temporary failure.

Congratulations! You have just understood the basic idea of the Retry Pattern.

In software engineering, a Retry mechanism automatically attempts an operation again when it fails for a reason that might be temporary.

For example:

• A network connection temporarily fails.
• A downstream service times out.
• A server temporarily becomes unavailable.
• A service returns an error that might be resolved by trying again.

In microservices, these temporary problems are common because services communicate over networks.

A retry can help an operation succeed without requiring the user to submit the entire request again.

But remember, buddy: barking again is useful only if your owner can hear you the next time!

Similarly, retrying is useful only when another attempt has a reasonable chance of succeeding.

2. Why Do We Need Retry in Microservices?

Imagine we have two microservices:

• Order Service: Receives a customer’s order.
• Product Service: Returns product information.

The communication flow looks like this:

Client
   |
   v
Order Service
   |
   v
Product Service
   |
   v
Product Details

Now imagine the customer requests product details.

The Order Service performs several operations:

1. Receives the request.
2. Validates the input.
3. Performs business logic.
4. Prepares the downstream API call.
5. Calls Product Service.

Suppose 90% of the work is completed, but a temporary network issue occurs while calling Product Service.

The entire operation fails.

Now the client could submit the request again. However, the Order Service might need to repeat much of the processing it already completed.

That can waste time and resources.

Instead, the Order Service can retry the downstream call if the failure is temporary and retrying is safe.

Order Service
      |
      v
 Product Service
      |
      X  Temporary failure
      |
      v
   Wait briefly
      |
      v
 Product Service
      |
      v
    Success!

This can reduce unnecessary repeated work and improve the user experience.

Important: Retry should normally be applied to the operation that needs to be repeated, rather than blindly repeating an entire business workflow.

3. When Should We Retry?

Now imagine your owner gives you three different situations.

Situation 1: Your owner didn’t hear you

You bark again after a short delay.

This is similar to a temporary network failure. Another attempt might succeed.

Situation 2: Your owner says, “No biscuits today!”

You bark again, but your owner still refuses.

Repeating the same action does not fix the underlying problem.

This is similar to invalid input or another permanent failure.

Situation 3: Your owner already gave you a biscuit, but you didn’t see it

You bark again, and your owner gives you another biscuit.

Now you have two biscuits even though you wanted only one!

This is similar to retrying a non-idempotent operation when the first attempt actually succeeded but its response was lost.

Let’s understand these situations in software.

Failures that are usually not worth retrying

Examples include:

• 400 Bad Request: The request contains invalid input.
• 401 Unauthorized: Authentication is missing or invalid.
• 403 Forbidden: The caller is not permitted to perform the operation.
• 404 Not Found: The requested resource does not exist.

Repeating the same request without changing anything usually won’t solve these problems.

Failures that might be worth retrying

Examples include:

• 500 Internal Server Error
• 502 Bad Gateway
• 503 Service Unavailable
• Temporary network failures
• Timeouts, when the operation is safe to retry

These failures may be temporary, but not every server error is transient. Configure retries according to the failure type and the behavior of the downstream service.

For 429 Too Many Requests, retrying after the indicated delay, such as a Retry-After value when provided, may be appropriate.

Here is a practical decision table:

HTTP status or failure	Retry?
200 OK	No; the operation succeeded
400 Bad Request	Usually no
401 Unauthorized	Usually no; resolve authentication
403 Forbidden	Usually no
404 Not Found	Usually no
409 Conflict	Depends on the conflict and operation
429 Too Many Requests	Possibly, after an appropriate delay
500 Internal Server Error	Possibly, depending on the cause
502 Bad Gateway	Often worth considering
503 Service Unavailable	Often worth considering
Network error or timeout	Only when retrying is safe

The most important rule is this:

Retry transient failures, not every failure.

4. What Is Idempotency, and Why Does It Matter?

Now imagine you ask your owner for one biscuit.

Your owner puts a biscuit in your bowl, but you are distracted and don’t notice it.

You bark again.

Your owner puts another biscuit in the bowl.

You now have two biscuits!

The first attempt actually succeeded. You just didn’t receive or notice the confirmation.

This can happen in distributed systems.

Imagine Service A calls Service B to create a new order.

Service A
    |
    | Create order
    v
Service B
    |
    v
Database: Order ID 101 created
    |
    X Response lost or timed out
    |
Service A retries
    |
    v
Database: Another order created

If the operation is not protected against duplicate processing, the retry might create a second order.

This is why idempotency is important.

An operation is idempotent when repeating the same operation has the same intended effect as performing it once.

For example, an API can use an idempotency key to recognize repeated requests representing the same logical operation.

Request 1:
Idempotency-Key: ORDER-123

Request 2:
Idempotency-Key: ORDER-123

If the first request already created the order, the server can recognize the repeated request and return the original result rather than creating another order.

This is particularly important for payments, order creation, and other operations that cause business side effects.

Remember: A timeout does not prove that the downstream operation failed. It may have succeeded even though the response never reached the caller.

For more details, idempotency is an important companion topic to Retry when designing reliable APIs.

5. What Are the Different Retry Strategies?

Imagine you barked for biscuits and your owner didn’t hear you.

You can try again after different amounts of time.

You might wait the same amount of time every time, increase the waiting time after each failure, or introduce some randomness.

These are different retry strategies.

We will cover four approaches:

1. Fixed Interval Retry
2. Exponential Backoff
3. Exponential Backoff with Jitter
4. Custom Retry

6. Fixed Interval Retry

Imagine your owner tells you:

“If I don’t hear you, wait exactly 2 seconds before barking again.”

You follow the same rule every time.

That is Fixed Interval Retry.

Suppose the initial request starts at second 2 and the delay is 2 seconds.

Original attempt: 2 seconds
First retry:      4 seconds
Second retry:     6 seconds
Third retry:      8 seconds

The waiting period between attempts remains constant.

Advantages

• Simple to understand.
• Easy to configure.
• Easy to debug.

Disadvantage: The Thundering Herd Problem

Imagine 1,000 dogs are waiting for biscuits.

Their owners all miss the first bark.

Every dog waits exactly 2 seconds and barks again at the same time.

Suddenly, 1,000 dogs bark together!

In microservices, this is similar to the Thundering Herd Problem.

Many clients retry simultaneously, creating a new traffic spike precisely when the downstream service is already struggling.

Fixed intervals can synchronize retries and increase the load on an unhealthy service.

Let’s see how to configure Fixed Interval Retry in Spring Boot.

Step 1: Add the dependency

Add Resilience4j to your Maven pom.xml:

<dependency>
    <groupId>io.github.resilience4j</groupId>
    <artifactId>resilience4j-spring-boot3</artifactId>
    <version>2.1.0</version>
</dependency>

Use a compatible version for your Spring Boot application.

Step 2: Create the Product Feign Client

@FeignClient(name = "product-service")
public interface ProductClient {

    @GetMapping("/products/{id}")
    String getProductById(
        @PathVariable("id") String id
    );
}

This client calls the Product Service.

Step 3: Apply the Retry annotation

import io.github.resilience4j.retry.annotation.Retry;
import org.springframework.stereotype.Service;

@Service
public class OrderService {

    private final ProductClient productClient;

    public OrderService(ProductClient productClient) {
        this.productClient = productClient;
    }

    @Retry(
        name = "productService",
        fallbackMethod = "productFallback"
    )
    public String invokeProductAPI(String id) {

        return productClient.getProductById(id);
    }

    public String productFallback(
            String id,
            Throwable t) {

        return "Product Service is busy. Please try again later.";
    }
}

Let’s understand the annotation.

• @Retry enables retry behavior on the method.
• name = "productService" identifies the retry configuration.
• fallbackMethod specifies an alternative response if the operation still fails and the failure is handled by the fallback mechanism.

Important: Keep the annotated method focused on the operation you want to retry. If the method contains several business operations, retries could repeat work you did not intend to repeat.

Step 4: Configure the retry interval

Add these properties to application.properties:

server.port=8081
spring.application.name=order-service

resilience4j.retry.instances.productService.maxAttempts=3
resilience4j.retry.instances.productService.waitDuration=2s

Here:

• maxAttempts=3 means one original attempt plus two retries.
• waitDuration=2s specifies the fixed waiting period between failed attempts.

For example:

Attempt 1 -> Failure
Wait 2 seconds
Attempt 2 -> Failure
Wait 2 seconds
Attempt 3 -> Success

The retry stops when an attempt succeeds or the configured attempts are exhausted.

If all attempts fail, the fallback method can provide an alternative response.

7. Exponential Backoff

Now imagine your owner realizes that barking every 2 seconds is causing too much noise.

Your owner gives you a smarter rule:

“After the first failure, wait 1 second. After the second failure, wait 2 seconds. After the third failure, wait 4 seconds.”

You now wait longer after each failed attempt.

This is called Exponential Backoff.

The delay increases exponentially rather than remaining constant.

A simplified formula is:

[
\text{Delay}=\text{Base}\times 2^n
]

Here, (n) starts at zero for the first retry delay.

Suppose the base delay is 100 milliseconds.

Attempt	Waiting time
Original attempt	No retry delay
First retry	100 ms
Second retry	200 ms
Third retry	400 ms

Why is this useful?

If the downstream service is temporarily overloaded, retrying every second may increase its workload.

Exponential Backoff gradually spreads retries over a longer period, giving the service more time to recover.

Advantages

• Reduces retry pressure during outages.
• Helps reduce synchronized retry traffic compared with fixed intervals in some situations.
• Gives a struggling downstream service more time to recover.

Disadvantages

• If the service recovers quickly, the caller might wait longer than necessary.
• If every client follows the same schedule, retries can still become synchronized.

Let’s configure Exponential Backoff using Resilience4j.

Configuration

resilience4j.retry.instances.productService.maxAttempts=4
resilience4j.retry.instances.productService.waitDuration=1s
resilience4j.retry.instances.productService.enableExponentialBackoff=true
resilience4j.retry.instances.productService.exponentialBackoffMultiplier=2

This configuration uses four total attempts: one original attempt and up to three retries.

With a 1-second initial delay and a multiplier of 2, the approximate delays are:

First retry  -> 1 second
Second retry -> 2 seconds
Third retry  -> 4 seconds

Resilience4j’s exponential backoff configuration can also use a maximum interval, depending on the configuration approach and version.

8. Exponential Backoff With Jitter

Let’s return to our dogs.

Imagine 1,000 dogs follow the same Exponential Backoff rule.

They all wait 1 second, then 2 seconds, then 4 seconds.

Even though the waiting period increases, all the dogs might still bark together.

How can we solve this?

Your owner gives every dog a slightly different waiting time.

One dog waits 1.2 seconds.

Another waits 1.7 seconds.

Another waits 1.4 seconds.

Now the dogs bark at different times instead of all barking together.

This is the basic idea of Jitter.

Jitter introduces randomness into retry delays.

A simplified full-jitter formula is:

[
\text{Delay}=\operatorname{random}(0,\min(\text{MaxDelay},\text{BaseDelay}\times2^n))
]

The actual range depends on the chosen jitter strategy.

Advantages

• Spreads retry attempts over time.
• Reduces the likelihood of a Thundering Herd.
• Helps reduce peak retry traffic in distributed systems.

Disadvantages

• More complicated to reason about.
• Retry timing becomes less predictable.
• Logs and tests may be harder to interpret because delays vary.

Configuration in Resilience4j

Your source material uses the following properties:

resilience4j.retry.instances.productService.maxAttempts=4
resilience4j.retry.instances.productService.waitDuration=1s
resilience4j.retry.instances.productService.enableExponentialBackoff=true
resilience4j.retry.instances.productService.exponentialBackoffMultiplier=2
resilience4j.retry.instances.productService.enableRandomizedWait=true

These properties illustrate the intended exponential-backoff-plus-randomized-wait setup in your notes. Verify the supported property names and behavior against the Resilience4j version used by your application; some versions use different configuration names for exponential random backoff.

9. Fixed Interval vs. Exponential Backoff vs. Jitter

Let’s compare the strategies.

Strategy	How it works	Main benefit	Main drawback
Fixed Interval	Wait the same amount of time	Simple configuration	Can synchronize retries
Exponential Backoff	Increase delay after each failure	Gives an unhealthy service more time	Can still synchronize clients
Exponential Backoff + Jitter	Increase delay with randomness	Spreads retries over time	Less predictable timing
Custom Retry	Define your own delay logic	Full flexibility	More code to maintain

For distributed microservices, Exponential Backoff with Jitter is often a strong choice when many clients might retry during the same outage.

However, the right strategy depends on your latency requirements, retry budget, and downstream service behavior.

10. Can We Create a Custom Retry Strategy?

Yes!

Imagine your owner uses a special rule:

“Wait 2 seconds after every failed attempt.”

Or perhaps your owner wants a Fibonacci-style sequence or another application-specific delay.

Resilience4j supports custom retry configuration through Java code.

Step 1: Create the custom Retry bean

import io.github.resilience4j.retry.Retry;
import io.github.resilience4j.retry.RetryConfig;
import io.github.resilience4j.core.IntervalFunction;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;

@Configuration
public class RetryConfiguration {

    @Bean
    public Retry customRetry() {

        IntervalFunction customInterval = attempt -> 2000L;

        RetryConfig config = RetryConfig.custom()
            .maxAttempts(6)
            .intervalFunction(customInterval)
            .retryExceptions(Exception.class)
            .build();

        return Retry.of("customRetry", config);
    }
}

This example defines a constant 2-second delay for each retry. It demonstrates how to provide a custom interval function.

Here, maxAttempts(6) means one original attempt and up to five retries.

Important: Retrying every Exception is intentionally broad for demonstration purposes. In a real application, configure only the exceptions that are safe and useful to retry.

Step 2: Use the custom Retry

import io.github.resilience4j.retry.Retry;
import org.springframework.stereotype.Service;

@Service
public class OrderService {

    private final ProductClient productClient;
    private final Retry customRetry;

    public OrderService(
            ProductClient productClient,
            Retry customRetry) {

        this.productClient = productClient;
        this.customRetry = customRetry;
    }

    public String invokeProductAPI(String id) {

        return customRetry.executeSupplier(() ->
            productClient.getProductById(id)
        );
    }
}

The executeSupplier method executes the operation using the configured Retry instance.

If the operation fails with an exception selected by the retry configuration, another attempt may occur according to the configured retry policy.

If all attempts fail, the exception can propagate to the caller, where you should handle it appropriately.

Advantages

• Full control over retry timing.
• Can implement application-specific delay strategies.
• Useful when standard configuration is not sufficient.

Disadvantages

• More code to maintain.
• Easier to introduce incorrect retry behavior.
• Requires careful testing of delays and failure conditions.

11. Important Best Practices for Retry

Before using Retry in production, remember these rules.

1. Retry only transient failures

Don’t repeatedly retry invalid input or permanent business errors.

2. Check idempotency

For operations that create records, charge payments, or produce other side effects, ensure repeated attempts cannot accidentally duplicate the operation.

3. Limit the number of attempts

Unlimited retries can keep consuming resources and prevent the application from recovering.

4. Use an appropriate delay

Fixed delays are simple, while exponential backoff and jitter can reduce repeated pressure during outages.

5. Configure timeouts

A Retry policy does not replace connection or read timeouts. Each attempt should have sensible time limits.

6. Avoid retrying everything at every layer

If the API Gateway, Order Service, and Feign client all retry independently, a single request can produce many downstream attempts.

For example, three attempts at one layer and three at another can produce up to nine downstream calls in some designs.

Understand where retries happen and set a reasonable overall retry budget.

7. Be careful with fallback responses

If all attempts fail, return a meaningful error or fallback response. Don’t report a failed payment or order as successful merely to avoid an error.

12. Where Does Retry Fit in Fault-Tolerant Microservices?

Retry is one of several resilience patterns.

Let’s quickly revisit the patterns we have covered in this series.

• RateLimiter: Controls how many requests are admitted during a period.
• Bulkhead: Limits concurrency or isolates resources.
• TimeLimiter: Limits how long a supported asynchronous operation may wait.
• CircuitBreaker: Helps stop repeated calls to a failing dependency.
• Retry: Attempts an operation again when a suitable failure occurs.

These patterns solve different problems.

For example, imagine Product Service becomes temporarily unavailable.

A Retry policy might make another attempt after a delay. If the service continues failing, a CircuitBreaker can help prevent repeated calls. A Bulkhead can limit the resources consumed by the operation.

The patterns can work together, but their order and configuration must be designed carefully.

13. Key Takeaways 🐶

Let’s finish our dog lesson!

Imagine you’re waiting for biscuits.

If your owner doesn’t hear you, you can try again. But you shouldn’t bark endlessly, and you shouldn’t bark at exactly the same moment as 1,000 other dogs!

Remember these five things:

1. Retry helps recover from temporary failures.
2. Don’t blindly retry permanent errors or unsafe operations.
3. Fixed Interval uses the same delay between attempts.
4. Exponential Backoff increases the delay after failures.
5. Jitter adds randomness to help spread retries across clients.

The easiest way to remember it:

A dog tries again when the owner doesn’t hear the first bark.

A microservice retries when a temporary failure may be recoverable.

But a smart dog knows when to wait, and a smart microservice knows when to stop retrying!

In the next article, we’ll explore CircuitBreaker in Spring Boot Microservices, using the same simple dog-friendly approach.

Until then, keep coding and keep learning! 🐶💻