TheLazy.Dev
Portfolio GitHub

Blog

aspnetcore |docker |serilog |elasticsearch

Structured logging with Serilog and Seq and ElasticSearch under Docker

hasdrubal

Ali MSELMI

Posted on April 20, 2020

Structured logging with Serilog and Seq and ElasticSearch under Docker

This blog post demonstrates Structured Logging with Serilog, Seq, ElasticSearch and kibana under Docker containers.
This post is a follow up on the beginner post I wrote on Serilog.

If you are a newbie on Serilog then I will recommend that you give a quick read to official Serilog page and this blog post.

Example application

The complete below example shows serilog structured logging in a containerized web application with microservices style using docker, with events sent to the Seq and elastic search as well as a date-stamped rolling log file with the use of available below sinks:

  • Serilog.Sinks.File
  • Serilog.Sinks.Http
  • Serilog.Sinks.Seq
  • Serilog.Sinks.ElasticSearch

Create a new empty solution

To start, we need create a new solution using Visual studio or your favorite IDE.
Create a new empty solution

Create a new web Application project

Create new Web application project like below:
Create a new web Application project

Install the core Serilog package and the File, Seq and ElasticSearch sinks

In Visual Studio, open the Package Manager Console and type:

Install-Package Autofac
Install-Package Autofac.Extensions.DependencyInjection
Install-Package Microsoft.AspNetCore.Mvc.NewtonsoftJson
Install-Package Microsoft.Extensions.DependencyInjection
Install-Package Microsoft.Extensions.Hosting
Install-Package Microsoft.Extensions.Logging
Install-Package Microsoft.Extensions.Options
Install-Package Newtonsoft.Json
Install-Package Serilog
Install-Package Serilog.AspNetCore
Install-Package Serilog.Extensions.Hosting
Install-Package Serilog.Extensions.Logging
Install-Package Serilog.Settings.AppSettings
Install-Package Serilog.Settings.Configuration
Install-Package Serilog.Sinks.ElasticSearch
Install-Package Serilog.Sinks.File
Install-Package Serilog.Sinks.Http
Install-Package Serilog.Sinks.Seq

Add the following code to Program.cs

Create GetConfigurationmethod 



private static IConfiguration GetConfiguration()
{
    var builder = new ConfigurationBuilder()
        .SetBasePath(Directory.GetCurrentDirectory())
        .AddJsonFile("appsettings.json", optional: false, reloadOnChange: true)
        .AddJsonFile($"appsettings.Development.json", optional: true)
        .AddEnvironmentVariables();

    return builder.Build();
}
Enter fullscreen mode Exit fullscreen mode

Create CreateSerilogLoggermethod 



private static ILogger CreateSerilogLogger(IConfiguration configuration)
{
    var seqServerUrl = configuration["Serilog:SeqServerUrl"];
    return new LoggerConfiguration()
        .MinimumLevel.Verbose()
        .Enrich.WithProperty("ApplicationContext", AppName)
        .Enrich.FromLogContext()
        .WriteTo.File("catalog.api.log.txt", rollingInterval: RollingInterval.Day)
        .WriteTo.Elasticsearch().WriteTo.Elasticsearch(ConfigureElasticSink(configuration, "Development"))
        .WriteTo.Seq(string.IsNullOrWhiteSpace(seqServerUrl) ? "http://seq" : seqServerUrl)
        .CreateLogger();
}
Enter fullscreen mode Exit fullscreen mode

Create ConfigureElasticSinkmethod 



private static ElasticsearchSinkOptions ConfigureElasticSink(IConfiguration configuration, string environment)
{
    return new ElasticsearchSinkOptions(new Uri(configuration["Serilog:ElasticConfiguration"]))
    {
        BufferCleanPayload = (failingEvent, statuscode, exception) =>
        {
            dynamic e = JObject.Parse(failingEvent);
            return JsonConvert.SerializeObject(new Dictionary<string, object>()
                {
                    { "@timestamp",e["@timestamp"]},
                    { "level","Error"},
                    { "message","Error: "+e.message},
                    { "messageTemplate",e.messageTemplate},
                    { "failingStatusCode", statuscode},
                    { "failingException", exception}
                });
        },
        MinimumLogEventLevel = LogEventLevel.Verbose,
        AutoRegisterTemplate = true,
        AutoRegisterTemplateVersion = AutoRegisterTemplateVersion.ESv7,
        CustomFormatter = new ExceptionAsObjectJsonFormatter(renderMessage: true),
        IndexFormat = $"{Assembly.GetExecutingAssembly().GetName().Name.ToLower().Replace(".", "-")}-{environment?.ToLower().Replace(".", "-")}-{DateTime.UtcNow:yyyy-MM}",
        EmitEventFailure = EmitEventFailureHandling.WriteToSelfLog | 
                               EmitEventFailureHandling.WriteToFailureSink |
                               EmitEventFailureHandling.RaiseCallback
    };
}
Enter fullscreen mode Exit fullscreen mode

Create GetDefinedPorthelper method 



private static int GetDefinedPort(IConfiguration config)
{
    var port = config.GetValue("PORT", 80);
    return port;
}
Enter fullscreen mode Exit fullscreen mode

Create CreateHostBuildermethod 



private static IWebHost CreateHostBuilder(IConfiguration configuration, string[] args) =>
    WebHost.CreateDefaultBuilder(args)
        .UseConfiguration(configuration)
        .CaptureStartupErrors(false)
        .ConfigureKestrel(options =>
        {
            var httpPort = GetDefinedPort(configuration);
            options.Listen(IPAddress.Any, httpPort, listenOptions =>
            {
                listenOptions.Protocols = HttpProtocols.Http1AndHttp2;
            });
        })
        .UseStartup<Startup>()
        .UseContentRoot(Directory.GetCurrentDirectory())
        .UseSerilog()
        .Build();
Enter fullscreen mode Exit fullscreen mode

Create Main method 



public static readonly string Namespace = typeof(Program).Namespace;
public static readonly string AppName = Namespace.Substring(Namespace.LastIndexOf('.', Namespace.LastIndexOf('.') - 1) + 1);

public static int Main(string[] args)
{
    var configuration = GetConfiguration();
    Log.Logger = CreateSerilogLogger(configuration);
    try
    {
        Log.Information("Configuring web host ({ApplicationContext})", AppName);
        var host = CreateHostBuilder(configuration, args);
        Log.Information("Starting web host ({ApplicationContext})", AppName);
        host.Run();
        return 0;
    }
    catch (Exception ex)
    {
        Log.Fatal(ex, "Program terminated unexpectedly ({ApplicationContext})!", AppName);
        return 1;
    }
    finally
    {
        Log.CloseAndFlush();
    }
}
Enter fullscreen mode Exit fullscreen mode

Add the following code to Startup.cs

Inject IConfigurationinto the constructure 



public Startup(IConfiguration configuration)
{
    Configuration = configuration;
}
public IConfiguration Configuration { get; }
Enter fullscreen mode Exit fullscreen mode

Create ConfigureServices method 



public IServiceProvider ConfigureServices(IServiceCollection services)
{
    services.AddCustomMVC(Configuration);

    var container = new ContainerBuilder();
    container.Populate(services);

    return new AutofacServiceProvider(container.Build());
}
Enter fullscreen mode Exit fullscreen mode

Create Configuremethod 



public void Configure(IApplicationBuilder app, IWebHostEnvironment env, ILoggerFactory loggerFactory)
{
    var pathBase = Configuration["PATH_BASE"];

    if (!string.IsNullOrEmpty(pathBase))
    {
        loggerFactory.CreateLogger<Startup>().LogDebug("Using PATH BASE '{pathBase}'", pathBase);
        app.UsePathBase(pathBase);
    }

    app.UseCors("CorsPolicy");
    app.UseRouting();
    app.UseEndpoints(endpoints =>
    {
        endpoints.MapDefaultControllerRoute();
        endpoints.MapControllers();
    });
}
Enter fullscreen mode Exit fullscreen mode

Create the CustomExtensionMethodsextension method 



public static class CustomExtensionMethods
{
    public static IServiceCollection AddCustomMVC(this IServiceCollection services, IConfiguration configuration)
    {
        services.AddControllers(options =>
        {

        }).AddNewtonsoftJson();

        services.AddCors(options =>
        {
            options.AddPolicy("CorsPolicy",
                builder => builder
                .SetIsOriginAllowed((host) => true)
                .AllowAnyMethod()
                .AllowAnyHeader()
                .AllowCredentials());
        });

        return services;
    }
}
Enter fullscreen mode Exit fullscreen mode

Add appsettings.json / appsettings.Development.json

It's very important to use the container name as ElasticSearchurl and not http://localhost:9200



{
  "Serilog": {
    "SeqServerUrl": "http://seq",
    "LogstashgUrl": "http://locahost:8080",
    "ElasticConfiguration": "http://elasticsearch:9200",
    "MinimumLevel": {
          "Default": "Debug",
          "Override": {
            "Microsoft": "Debug",
            "CatalogAPI": "Debug",
            "MYSHOP": "Debug",
            "System": "Warning"
          }
        }
      }
  }
Enter fullscreen mode Exit fullscreen mode

Add Docker support to our solution

Add Docker compose project to our solution

Add Docker compose project to our solution

Add Dockerfile to the project 



FROM mcr.microsoft.com/dotnet/core/aspnet:3.1-buster-slim AS base
WORKDIR /app
EXPOSE 80
EXPOSE 443

FROM mcr.microsoft.com/dotnet/core/sdk:aspnet:3.1-buster AS build
WORKDIR /src

COPY "Services/Catalog/Catalog.API/CatalogAPI.csproj" "Services/Catalog/Catalog.API/CatalogAPI.csproj"
COPY "docker-compose.dcproj" "docker-compose.dcproj"

COPY "NuGet.config" "NuGet.config"

RUN dotnet restore "MyShop.sln"

COPY . .
WORKDIR /src/Services/Catalog/Catalog.API
RUN dotnet publish --no-restore -c Release -o /app

FROM build AS publish

FROM base AS final
WORKDIR /app
COPY --from=publish /app .
ENTRYPOINT ["dotnet", "CatalogAPI.dll"]
Enter fullscreen mode Exit fullscreen mode

Add docker-compose.yml to the docker project 



version: '3.4'

services:

  seq:
    image: datalust/seq:latest

  catalog.api:
    image: ${REGISTRY:-myshop}/catalogapi
    build:
      context: .
      dockerfile: Services/Catalog/CatalogAPI/Dockerfile

networks:
  elastic:
    driver: bridge

volumes:
  elasticsearchdata:
    external: true
Enter fullscreen mode Exit fullscreen mode

Add Docker docker-compose.overrides.yml to the docker project 



version: '3.4'

services:

  seq:
    environment:
      - ACCEPT_EULA=Y
    ports:
      - "5340:80"

  elasticsearch:
    build:
      context: elk/elasticsearch/
    volumes:
      - ./elk/elasticsearch/config/elasticsearch.yml:/usr/share/elasticsearch/config/elasticsearch.yml:ro
    ports:
      - "9200:9200"
      - "9300:9300"
    environment:
      ES_JAVA_OPTS: "-Xmx256m -Xms256m"

  logstash:
    build:
      context: elk/logstash/
    volumes:
      - ./elk/logstash/config/logstash.yml:/usr/share/logstash/config/logstash.yml:ro
      - ./elk/logstash/pipeline:/usr/share/logstash/pipeline:ro
    ports:
      - "8080:8080"
    environment:
      LS_JAVA_OPTS: "-Xmx256m -Xms256m"
    depends_on:
      - elasticsearch

  kibana:
    build:
      context: elk/kibana/
    volumes:
      - ./elk/kibana/config/:/usr/share/kibana/config:ro
    ports:
      - "5601:5601"
    depends_on:
      - elasticsearch

  catalog.api:
    environment:
      - ASPNETCORE_ENVIRONMENT=Development
      - ASPNETCORE_URLS=http://0.0.0.0:80
      - PORT=80
      - PATH_BASE=/catalog-api
    ports:
      - "4201:80"
Enter fullscreen mode Exit fullscreen mode

Create Configuration for ElasticSearchand Kibana

Create the folder structure

Create "elk" folder in the root folder of the solution with following structure
Create the folder structure

Create elasticsearch configuration

Under elk/elasticsearch create new Dockerfileand new config folder like bellow:
Create elasticsearch configuration
Add the following code to the Dockerfile



# https://github.com/elastic/elasticsearch-docker
FROM docker.elastic.co/elasticsearch/elasticsearch-oss:7.6.2

# Add your elasticsearch plugins setup here
# Example: RUN elasticsearch-plugin install analysis-icu
Enter fullscreen mode Exit fullscreen mode

Create new elasticsearch.yml under the config folder







  
  
  Default Elasticsearch configuration from elasticsearch-docker.




  
  
  from https://github.com/elastic/elasticsearch-docker/blob/master/build/elasticsearch/elasticsearch.yml




#

cluster.name: "docker-cluster"

network.host: 0.0.0.0



# minimum_master_nodes need to be explicitly set when bound on a public IP

# set to 1 to allow single node clusters

# Details: https://github.com/elastic/elasticsearch/pull/17288

discovery.zen.minimum_master_nodes: 1



## Use single node discovery in order to disable production mode and avoid bootstrap checks

## see https://www.elastic.co/guide/en/elasticsearch/reference/current/bootstrap-checks.html

#

discovery.type: single-node
Enter fullscreen mode Exit fullscreen mode




Create kibana configuration

Under elk/kibana create new Dockerfileand new config folder like bellow:

Add the following code to the Dockerfile




# https://github.com/elastic/kibana-docker

FROM docker.elastic.co/kibana/kibana-oss:7.6.2



# Add your kibana plugins setup here

# Example: RUN kibana-plugin install <name|url>

Create new kibana.yml under the config folder



## Default Kibana configuration from kibana-docker.

## from https://github.com/elastic/kibana-docker/blob/master/build/kibana/config/kibana.yml

#

server.name: kibana

server.host: "0"

elasticsearch.hosts: "http://elasticsearch:9200"
Enter fullscreen mode Exit fullscreen mode




Conclusion

We have seen how to configure .net core web application to log in Elasticsearch, Kibanaand Sequsing Serilog. This solution offers many advantages and I invite you to discover the range of possibilities on the Elastic and Seq websites.
The sources are here feel free to clone the solution and play with it.

💖 💪 🙅 🚩
hasdrubal
Ali MSELMI

Posted on April 20, 2020

Join Our Newsletter. No Spam, Only the good stuff.

Sign up to receive the latest update from our blog.

Related

Structured logging with Serilog and Seq and ElasticSearch under Docker
aspnetcore Structured logging with Serilog and Seq and ElasticSearch under Docker

April 20, 2020