I was tasked with integrating a really old, internal API which had a particular quirk. This API seemed to fail, at random. After asking around, it seemed that handling random errors was going to be a part of this task.

We spent some time testing the API by sending the same exact request multiple times until we could reliably predict the failure rate. Our most conservative estimate was that the API would fail for no apparent reason at least one out of every ten requests.

We decided that the simplest solution was to retry the request up to three times if we encountered an Internal Server Error. Any request which failed more than three times would be considered invalid, and the app would rely on the user to fix the request by altering their inputs.

The app had to query a few different endpoints from the unreliable API so our solution had to work on every request. We chose to house the solution in an interceptor as this is Angular's way of modifying HTTP requests/responses for the app as a whole.

Setup

I've created a demo application with a mock server that emulates the failure rate of the unreliable API we integrated. The repository also has a Cypress e2e specification which tests the app agains the mock server. Below the environment setup steps:

git clone https://github.com/LySofDev/retry-on-internal-server-error-demo.git demo
cd demo
npm i
cd ./server
npm i
cd ..
npm run dev

You should now have the Angular application listening on port 4200, the Cypress test runner open and displaying two spec files, and the mock server listening on port 3000. I recommend using the Cypress test runner to experiment with the application since we have to fill in a form for each request.

Integration Test Failures

Run the internal_server_error_spec in Cypress to see the app interacting with the server without the random error handler. We should see at least one or two test runs failing.

We can manipulate the failure rate of the server in the server/src/index.ts file by changing the value of the RANDOM_INTERNAL_SERVER_ERROR_CHANCE variable. See the inline documentation for details on how this affects the rate of failure.

Unit Test Failures

Let's add a specification file for the interceptor that we'll be developing. Create a file src/app/retry-on-internal-server-error.interceptor.ts and add the following boilerplate code.

import {
  HttpInterceptor,
  HttpRequest,
  HttpHandler,
  HttpEvent
} from '@angular/common/http';
import { Observable } from 'rxjs';
import { Injectable } from '@angular/core';

@Injectable()
export class RetryOnInternalServerErrorInterceptor implements HttpInterceptor {
  intercept(
    request: HttpRequest<any>,
    next: HttpHandler
  ): Observable<HttpEvent<any>> {
    return next.handle(request);
  }
}

The above implementation essentially does nothing. It receives every request returns the observable result of the request being called with the handler. That is the minimum definition of the Angular interceptor so we are all set. Let's add it to our src/app/app.module.ts

import { BrowserModule } from '@angular/platform-browser';
import { NgModule } from '@angular/core';
import { HttpClientModule, HTTP_INTERCEPTORS } from '@angular/common/http';

import { AppComponent } from './app.component';
import { BrowserAnimationsModule } from '@angular/platform-browser/animations';
import { MatToolbarModule } from '@angular/material/toolbar';
import { MatFormFieldModule } from '@angular/material/form-field';
import { MatInputModule } from '@angular/material/input';
import { ReactiveFormsModule } from '@angular/forms';
import { MatButtonModule } from '@angular/material/button';
import { MatCardModule } from '@angular/material/card';
import { MatProgressSpinnerModule } from '@angular/material/progress-spinner';
import { MatSnackBarModule } from '@angular/material/snack-bar';

import { RetryOnInternalServerErrorInterceptor } from './retry-on-internal-server-errror.interceptor';

@NgModule({
  declarations: [AppComponent],
  imports: [
    BrowserModule,
    BrowserAnimationsModule,
    MatToolbarModule,
    MatFormFieldModule,
    MatInputModule,
    ReactiveFormsModule,
    MatButtonModule,
    MatCardModule,
    MatProgressSpinnerModule,
    MatSnackBarModule,
    HttpClientModule,
  ],
  providers: [
    {
      provide: HTTP_INTERCEPTORS,
      useClass: RetryOnInternalServerErrorInterceptor,
      multi: true
    }
  ],
  bootstrap: [AppComponent],
})
export class AppModule {}

The new interceptor is now part of the stack of HTTP interceptors that every request/response will go through. Given the amazing developers that we are, we're going to go ahead and add a spec file with some tests for our interceptor. Create a file src/app/retry-on-internal-server-error.interceptor.spec.ts and add the following:

import { Injectable } from '@angular/core';
import {
  HttpClientTestingModule,
  HttpTestingController,
  TestRequest,
} from '@angular/common/http/testing';
import {
  HttpClient,
  HTTP_INTERCEPTORS,
  HttpErrorResponse,
} from '@angular/common/http';
import { Observable, Subject } from 'rxjs';
import { TestBed, async, fakeAsync, inject } from '@angular/core/testing';

import { RetryOnInternalServerErrorInterceptor } from './retry-on-internal-server-error.interceptor';

@Injectable()
class MockService {
  constructor(private http: HttpClient) {}

  mockRequest(): Observable<any> {
    return this.http.get('/mock');
  }
}

describe('RetryOnInternalServerErrorInterceptor', () => {
  let testRequest: TestRequest;
  let testNext: jest.Mock;
  let testError: jest.Mock;
  let testComplete: jest.Mock;

  beforeEach(async(() => {
    testNext = jest.fn();
    testError = jest.fn();
    testComplete = jest.fn();
    TestBed.configureTestingModule({
      imports: [HttpClientTestingModule],
      providers: [
        {
          provide: HTTP_INTERCEPTORS,
          useClass: RetryOnInternalServerErrorInterceptor,
          multi: true,
        },
        MockService,
      ],
    });
  }));

  beforeEach(inject(
    [MockService, HttpTestingController],
    (mockService: MockService, http: HttpTestingController) => {
      mockService.mockRequest().subscribe({
        next: testNext,
        error: testError,
        complete: testComplete,
      });
      testRequest = http.expectOne('/mock');
    }
  ));

  describe('when receiving a 200 response', () => {
    beforeEach(() => {
      testRequest.flush(null);
    });

    it('forwards the response', () => {
      expect(testNext).toHaveBeenCalledWith(null);
    });

    it('completes', () => {
      expect(testComplete).toHaveBeenCalled();
    });

    it('doesnt throw an error', () => {
      expect(testError).not.toHaveBeenCalled();
    });
  });

  describe('when receiving a 400 response', () => {
    beforeEach(() => {
      testRequest.error(new ErrorEvent('Bad Request'), {
        status: 400,
        statusText: 'Bad Request',
      });
    });

    it('doesnt forward any response', () => {
      expect(testNext).not.toHaveBeenCalled();
    });

    it('doesnt complete', () => {
      expect(testComplete).not.toHaveBeenCalled();
    });

    it('throws an error', () => {
      expect(testError).toHaveBeenCalled();
    });
  });

  describe('when receiving a 401 response', () => {
    beforeEach(() => {
      testRequest.error(new ErrorEvent('Unauthorized'), {
        status: 401,
        statusText: 'Unauthorized',
      });
    });

    it('doesnt forward any response', () => {
      expect(testNext).not.toHaveBeenCalled();
    });

    it('doesnt complete', () => {
      expect(testComplete).not.toHaveBeenCalled();
    });

    it('throws an error', () => {
      expect(testError).toHaveBeenCalled();
    });
  });

  describe('when receiving a 500 error', () => {
    beforeEach(() => {
      testRequest.error(new ErrorEvent('Internal Server Error'), {
        status: 500,
        statusText: 'Internal Server Error',
      });
    });

    it('retries the request', inject(
      [HttpTestingController],
      (http: HttpTestingController) => {
        http.expectOne('/mock');
      }
    ));

    describe('when the retry succeeds', () => {
      beforeEach(inject(
        [HttpTestingController],
        (http: HttpTestingController) => {
          testRequest = http.expectOne('/mock');
          testRequest.flush(null);
        }
      ));

      it('forwards the response', () => {
        expect(testNext).toHaveBeenCalledWith(null);
      });

      it('completes', () => {
        expect(testComplete).toHaveBeenCalled();
      });

      it('doesnt throw an error', () => {
        expect(testError).not.toHaveBeenCalled();
      });
    });

    describe('when the retry fails', () => {
      beforeEach(inject(
        [HttpTestingController],
        (http: HttpTestingController) => {
          testRequest = http.expectOne('/mock');
          testRequest.error(new ErrorEvent('Internal Server Error'), {
            status: 500,
            statusText: 'Internal Server Error',
          });
        }
      ));

      it('retries the request again', inject(
        [HttpTestingController],
        (http: HttpTestingController) => {
          http.expectOne('/mock');
        }
      ));

      describe('when the second retry succeeds', () => {
        beforeEach(inject(
          [HttpTestingController],
          (http: HttpTestingController) => {
            testRequest = http.expectOne('/mock');
            testRequest.flush(null);
          }
        ));

        it('forwards the response', () => {
          expect(testNext).toHaveBeenCalledWith(null);
        });

        it('completes', () => {
          expect(testComplete).toHaveBeenCalled();
        });

        it('doesnt throw an error', () => {
          expect(testError).not.toHaveBeenCalled();
        });
      });

      describe('when the second retry fails', () => {
        beforeEach(inject(
          [HttpTestingController],
          (http: HttpTestingController) => {
            testRequest = http.expectOne('/mock');
            testRequest.error(new ErrorEvent('Internal Server Error'), {
              status: 500,
              statusText: 'Internal Server Error',
            });
          }
        ));

        it('retries the request again', inject(
          [HttpTestingController],
          (http: HttpTestingController) => {
            http.expectOne('/mock');
          }
        ));

        describe('when the third retry succeeds', () => {
          beforeEach(inject(
            [HttpTestingController],
            (http: HttpTestingController) => {
              testRequest = http.expectOne('/mock');
              testRequest.flush(null);
            }
          ));

          it('forwards the response', () => {
            expect(testNext).toHaveBeenCalledWith(null);
          });

          it('completes', () => {
            expect(testComplete).toHaveBeenCalled();
          });

          it('doesnt throw an error', () => {
            expect(testError).not.toHaveBeenCalled();
          });
        });

        describe('when the third retry fails', () => {
          beforeEach(inject(
            [HttpTestingController],
            (http: HttpTestingController) => {
              testRequest = http.expectOne('/mock');
              testRequest.error(new ErrorEvent('Internal Server Error'), {
                status: 500,
                statusText: 'Internal Server Error',
              });
            }
          ));

          it('doesnt forward any response', () => {
            expect(testNext).not.toHaveBeenCalled();
          });

          it('doesnt complete', () => {
            expect(testComplete).not.toHaveBeenCalled();
          });

          it('throws an error', () => {
            expect(testError).toHaveBeenCalled();
          });
        });
      });
    });
  });
});

Take a moment to run the above spec file with the following command:

npm run test -- retry-on-internal-server-error.interceptor

The first few tests should pass as we don't want to modify the behavior of the request/response chain if the error is not an Internal Server Error. The only failures we should see are with the last few tests focused on the 500 error codes.

Our test will attempt to make several requests which will be mocked with responses containing a 500 error code. We will test that the interceptor retries the request up to three times before passing the failure on down the request/response chain.

Solution

Let's just look at the solution since it's only a few lines of code.

import {
  HttpInterceptor,
  HttpRequest,
  HttpHandler,
  HttpEvent,
  HttpErrorResponse,
} from '@angular/common/http';
import { Observable, of, throwError, isObservable } from 'rxjs';
import { catchError, flatMap, retry } from 'rxjs/operators';
import { Inject, InjectionToken, Injectable } from '@angular/core';
/**
 * Upper limit of retry attempts for a request with an Internal Server Error response.
 */
export const INTERNAL_SERVER_ERROR_RETRY_LIMIT = new InjectionToken<number>(
  'INTERNAL_SERVER_ERROR_RETRY_LIMIT',
  { factory: () => 3 }
);
/**
 * Retries a request up to [INTERNAL_SERVER_ERROR_RETRY_LIMIT] times
 * if the response contained an Internal Server Error with status code 500.
 * Otherwise, it forwards the response.
 */
@Injectable()
export class RetryOnInternalServerErrorInterceptor implements HttpInterceptor {
  constructor(
    @Inject(INTERNAL_SERVER_ERROR_RETRY_LIMIT)
    private readonly retryLimit: number
  ) {}

  intercept(
    request: HttpRequest<any>,
    next: HttpHandler
  ): Observable<HttpEvent<any>> {
    return next.handle(request).pipe(
      catchError((error: any) => {
        const error$ = throwError(error);
        if (error instanceof HttpErrorResponse && error.status === 500) {
          return error$;
        }
        return of(error$);
      }),
      retry(this.retryLimit),
      flatMap((value: any) => (isObservable(value) ? value : of(value)))
    );
  }
}

Let's break it down. We added a dependency to our interceptor with the token INTERNAL_SERVER_ERROR_RETRY_LIMIT which will be the number of times we want to retry any particular request. This dependency will automatically be resolved to three, as per our earlier specification. But we can change it to another number in the app.module.ts if we find that three isn't quite the perfect value.

Next, the actual interceptor. We immediately pass the request to the handler so that it can be transacted over the network. We'll use a pipe to listen for the response notifications. Here's where it gets a bit complicated but bare with me.

If the response is emitted in the form of an error notification, the catchError operator will receive the notification. Within the operator's projection function, we identify whether the error is an HTTP Error and not some other runtime error, and we validate that the status code is in fact 500. We also wrap the error in a new observable which will immediately emit the error again. Why? Hold on.

If the previously mentioned conditions are true, then the catchError operator will emit the error notification containing the 500 error. This will trigger the next operator in the pipe, the retry operator. The retry operator is very simple, given an error notification, it will resubscribe to the source up to N times. In our case, N will be the retryLimit. So, there's the retry mechanic in action but we have to take a few extra steps to prevent other errors from being retried. After all, we're only interested in retrying Internal Server Errors.

With that in mind, if the error notification is not an Internal Server Error, we will wrap the error in a throwError observable and then an of observable. Essentially, it's an error notification inside an error observable inside a normal, high-order observable. Wait, wait, for what?

This is how we skip the retry operator with the other error notifications. The high-order observable containing the error observable, will be ignored by the retry operator. It will then activate the flatMap operator whose projector function will receive the error observable and verify that it is, in fact, an observable, and not a scalar notification. It will then flatten the observable into the top-level observable stream. What is it flattening? Well, the error observable, which will cause the error to flow down the stream to the observers, as we would expect.

Ok, but what about normal notifications which just need to go through? Simple, the flatMap operator will also pass these on by flattening them into the top-level observable stream as well.

Conclusion

So, that's it. Some might say that retryWhen is a better operator for this case but I couldn't quite get it to work the same way. Not with the same level of simplicity, at least.

These Observables may seem complicated at first but think about all that we achieved in some fairly concise code. Could we really achieve the same result without taking advantage of RxJs?

Blog

Unreliable API? Not a problem with Angular and RxJs

Esteban Hernández

Setup

Integration Test Failures

Unit Test Failures

Solution

Conclusion

Join Our Newsletter. No Spam, Only the good stuff.

Related