Service Fabric is a distributed systems platform for packaging, deploying, and managing stateless and stateful distributed applications and containers at large scale. Service Fabric runs on Windows and Linux, on any cloud, any datacenter, across geographic regions, or on your laptop.

Introduction

In this post we will learn two things that can make life easier to support advanced scenarios when using Service Remoting for communication between services and actors. We will do this by making use of a simple library I created. It provides you with the ability to:

Intercept service fabric remoting messages so you can take action when the call is about to be made and after it has been delivered.
Add custom headers to the remoting messages. This can be used to, for example, add a trace identifier for call tracing & logging purposes.

Message Interception

Messages can be intercepted on both the sending side and the receiving side

Client-side message interception

On the receiving side messages can be intercepted using the BeforeHandleRequestResponseAsync and AfterHandleRequestResponseAsync extension points when creating a service listener:

protected override IEnumerable<ServiceInstanceListener> CreateServiceInstanceListeners()
{
    yield return new ServiceInstanceListener(context =>
        new FabricTransportServiceRemotingListener(context,
            new ExtendedServiceRemotingMessageDispatcher(context, this)
            {
                // Optional, log the call before being handled
                BeforeHandleRequestResponseAsync = reqInf =>
                {
                    var sw = new Stopwatch();
                    sw.Start();
                    ServiceEventSource.Log.ServiceMessage(Context, 
                            $"BeforeHandleRequest {reqInf.Service} {reqInf.Method}");
                    return Task.FromResult<object>(sw);
                },
                // Optional, log the call after being handled
                AfterHandleRequestResponseAsync = respInf =>
                {
                    var sw = (Stopwatch) respInf.State;
                    ServiceEventSource.Log.ServiceMessage(Context, 
                            $"AfterHandleRequest {respInf.Method} took {sw.ElapsedMilliseconds}ms");
                    return Task.CompletedTask;
                }
            }));
}

Server-side message interception

On the sending side messages can be intercepted using the BeforeSendRequestResponseAsync and AfterSendRequestResponseAsync extension points when creating the ExtendedServiceRemotingClientFactory on constructor of the ServiceProxyFactory:

var proxyFactory = new ServiceProxyFactory(handler => // or ActorProxyFactory in case of actors
    new ExtendedServiceRemotingClientFactory(
        new FabricTransportServiceRemotingClientFactory(remotingCallbackMessageHandler: handler), 
            customHeadersProvider)
    {
        // Optional, log the call before being handled
        BeforeSendRequestResponseAsync = reqInf =>
        {
            var sw = new Stopwatch();
            sw.Start();
            Console.WriteLine($"BeforeSendRequest {reqInf.Method}");
            return Task.FromResult<object>(sw);
        },
        // Optional, log the call after being handled
        AfterSendRequestResponseAsync = respInf =>
        {
            var sw = (Stopwatch)respInf.State;
            var duration = sw.ElapsedMilliseconds;
            Console.WriteLine($"AfterSendRequest {respInf.Method} took {duration}ms");
            return Task.CompletedTask;
        }
    });

Custom Headers

Custom headers can be used to pass data between the sender and the receiver like tracing information or security context data. Using the BeforeHandleRequestResponseAsync and AfterHandleRequestResponseAsync actions additional logging can be applied monitor the flow between remoting calls.

Prepare the Reliable Service

Modify the service and create a listener that can handle the requests:

protected override IEnumerable<ServiceInstanceListener> CreateServiceInstanceListeners()
{
    yield return new ServiceInstanceListener(context =>
        new FabricTransportServiceRemotingListener(context,
            new ExtendedServiceRemotingMessageDispatcher(context, this)));
}

Adding custom headers (the caller)

var customHeaders = new CustomHeaders
{
    {"Header1", DateTime.Now},
    {"Header2", Guid.NewGuid()}
};

var serviceUri = new Uri("fabric:/ServiceFabric.Remoting.CustomHeaders.DemoApplication/DemoService");
var proxyFactory = new ServiceProxyFactory(handler => 
                    new ExtendedServiceRemotingClientFactory(
                        new FabricTransportServiceRemotingClientFactory(remotingCallbackMessageHandler: handler), 
                            customHeaders));
var proxy = proxyFactory.CreateServiceProxy<IDemoService>(serviceUri);

There is an overload of the Create method that accepts a Func. This is useful in scenarios where the created proxy factory or proxy is reused. Since creating a proxy factory is expensive this is the preferred way if you need dynamic header values. The func is invoked on every request made using the proxy.

Reading custom headers (the callee)

The receiving service or actor can extract the values in the custom headers using the RemotingContext class:


public async Task<string> SayHello()
{
    var remotingContext =
        string.Join(", ", RemotingContext.Keys.Select(k => $"{k}: {RemotingContext.GetData(k)}"));

    ServiceEventSource.Log.ServiceMessage(Context, 
            $"SayHelloToActor got context: {remotingContext}");
    return Task.FromResult($"Got the following message headers: {remotingContext}")
}

What is next?

More documentation and a demo Service Fabric application can be found at this repository:

Expecho / ServiceFabric-Remoting-CustomHeaders

This package allows injecting custom message headers into remoting messages (Actors and Reliable Services, V2 remoting only) at runtime. The headers are available client side to read. It also provides message interception using BeforeHandleRequestResponseAsync and AfterHandleRequestResponseAsync to act on remoting events.

ServiceFabric.Remoting.CustomHeaders

This package allows injecting custom headers into remoting messages (Actors and Reliable Services, V2 remoting only) at runtime. The headers are available client side to read It also provides message interception using BeforeHandleRequestResponseAsync and AfterHandleRequestResponseAsync to act on remoting events.

Common used classes:

NuGet

Download the NuGet package

Examples

This repository includes a Service Fabric application for demonstration purposes. A Console Application is used to access the application and shows the usage of the package.

Usage scenarios

Custom headers can be used to pass data between the sender and the receiver like tracing information or security context data. Using the BeforeHandleRequestResponseAsync and AfterHandleRequestResponseAsync actions additional logging can be applied monitor the flow between remoting calls.

How to use

Prepare Reliable Services

Modify the service and create a listener that can handle the requests

protected override IEnumerable<ServiceInstanceListener> CreateServiceInstanceListeners()
{
    yield return new ServiceInstanceListener

…

View on GitHub

Blog

Service Fabric Remoting: Interception & Custom Headers

Peter Bons