In this story, we will look at a log capturing app using the following technologies:

• MongoDB
• Java + Spring Boot
• Swagger UI

This demo application showcases the ability of MongoDB to store data without schema. All log entries can have arbitrary fields and we can search logs via those arbitrary fields. To make the app easy to test, we’ll visualize its REST API via Swagger UI.

The Challenge of Schemaless Data

In this story we’ll build an app which captures log data from 3rd party apps. The main challenge here: we don’t know how the log data from these 3rd party apps looks like. Still, we want to store the data in a compact format, plus we want to support queries for this log data.

Let’s look at a concrete example. Here’s two sample logs we want to store, in JSON format:

We have one microservice PAYMENT which creates logs during a payment process. On the right hand, we have a log from another app called LOGIN. Some of the log fields are common like severity, but some are unique to an app like creditCardProvider which only exists in the PAYMENT app.

The SQL Approach

If we wanted to store this data in an SQL store, we’d have a hard time to figure out a neat solution. Either we’d create a huge table which has a column for every log field we ever see. But this would result in many null entries and we would waste storage. Another solution is to store all data in JSON format, so you’d only have two rows in your table: the primary key and the JSON document. However, with this solution, you wouldn’t be able to search by JSON fields in a fast way.

The NoSQL Approach

NoSQL Document Stores come to our rescue: they store data in JSON format without any fix schema. Basically, a document store contains JSON documents – and nothing else. Even the ID of a given document is contained in the JSON itself.

Now, we can do really fast querying by JSON field, so you can do something like this:

> db.logEvents.find({_appType:"PAYMENT"})

This would return the first log above. More interestingly, you can perform the following query:

> db.logEvents.find({creditCardProvider:"VISA"})

The document store is robust to the fact that not every log contains the field creditCardProver!

Implement the Log Capturing App

Enough talk - let’s get started implementing our app!

MongoCollection Configuration

We use MongoCollection from the official MongoDB Java Driver for DB access. Let’s go ahead and create a Bean with a properly configured MongoCollection:

@Configuration
public class MongoConfiguration {
  @Bean
  public MongoCollection<Document> getMongoCollection() {
    MongoClient mongoClient = MongoClients.create();
    MongoDatabase db = mongoClient.getDatabase("mydb");
    return db.getCollection("logEvents");
  }
}

MongoDB Queries

Let’s use the above declared Bean to search entries in the MongoDB by id:

public class MongoDbService {
  @Autowired
  MongoCollection<Document> collection;

  public Optional<Document> findById(String id) {
    Document doc = collection.find(eq("_id", new ObjectId(id))).first();
    return Optional.ofNullable(doc);
  }
}

Using the @Autowired annotation, we can pull in our Bean of type MongoCollection. This class offers many query capabilities like searching by id, by key-value pair, and even more complex queries like greater-than etc. On my Github repo, you can see other query samples.

One more note: all DB data is represented as Documents, which are described in the official documentation as follows:

The Document class can represent dynamically structured documents of any complexity ... Document implements Map<String, Object>

Java Date Model

Inside our logging app, we don’t want to use Documents, but use a custom POJO instead:

import lombok.Data;
@Data
public final class LogEvent {
  private Map<String, String> logEvent;
}

We represent each log entry as a map with String keys and values.

Side note: the code snippet above uses Lombok which offers a neat way to create setter, getters, and constructors.

Document to LogEvent Mapper

Our DB layer represents logs as Document, but our service layer uses LogEvent. Therefore, we need a mapper method:

public LogEvent toLog(Document document) {
  Map<String, String> map = new HashMap<>();
  document.forEach((key, val) -> map.put(key, val.toString()));
  return new LogEvent(map);
}

REST Controller

Now it’s time to create the controller layer. We’ll use REST Controllers, which can be nicely created using Spring Boot’s @RequestMapping annotation:

@RequestMapping(method = RequestMethod.GET, path = "/api/log/{id}")
public ResponseEntity<LogEvent> getLogById(
    @PathVariable(value = "id") String id) {
  return mongoDbService.findById(id)
      .map(documentToLogMapper::toLog)
      .map(ResponseEntity::ok)
      .orElse(ResponseEntity.notFound().build());
}

Everything we’ve coded so far comes together now!😎 So let’s go through the above code snippet step by step! First, our REST controller will serve HTTP GET requests to the URI "api/log/{id}", id is the identifier of the log event. Second, the REST call will return a LogEvent in JSON format. Third, we use functional programming in the method body. We start with the DB query, pass the result to our mapper, and finally check if everything went fine – and if not: return a 404 NOT_FOUND result.

Spring Boot Service

Only one thing is missing to make our app executable:

@SpringBootApplication
public class LogCaptureApp {
  public static void main(String[] args) {
    new SpringApplication(LogCaptureApp.class).run();
  }
}

That’s it, that’s all necessary code😊 Now your logging app can serve REST requests to read logs!

Swagger UI Documentation

One more thing – let’s add a simple GUI to our log capturing app. We use Swagger UI for this purpose. Swagger UI allows to visualize your API without any further hustle. All you need is configure Swagger:

@Configuration
@EnableSwagger2
public class SwaggerUIConfiguration {
  @Bean
  public Docket apiDocu() {
    return new Docket(DocumentationType.SWAGGER_2)
        .apiInfo(apiInfo())
        .select()
        .apis(RequestHandlerSelectors.basePackage("com.example.controller"))
        .paths(PathSelectors.any())
        .build();
  }
  private ApiInfo apiInfo() {
    return new ApiInfoBuilder()
        .title("Log capturing app with MongoDB + SpringBoot")
        .build();
  }
}

Run the App

The complete code can be found on Github:

My Github repo contains more functionality than what we saw in this story, like searching log events by date range and storing log events from JSON data.

So, if you have implemented the log app as you read through this story, go ahead and run it now. If not, no problem, you can use my fully working example from Github.

Also, make sure you have a MongoDB Server running locally on the standard port. Then, compile and run the app:

$ git clone https://github.com/pmgysel/mongodb-logging-app.git
$ cd mongodb-logging-app
$ mvn clean package
$ java -jar target/mongodb-logging-app-0.0.1-SNAPSHOT.jar

Go to the Swagger UI page in your favorite web browser: http://localhost:8080/swagger-ui.html

You can use the log capturing app as follows:

• createLogRadom: Create random log entries to get some data into the DB
• Alternatively, create arbitrary log entries using the createLog endpoint. Swagger UI will show sample request to make your life easy 😊
• Now search for logs by key-value pair (getLogByOneField) or via date range (getLogByDateRange)
• Sample search:

Conclusion

So, in this story, we built a fully working log capturing app in Java. Since our requirements were to support flexible log entries, we chose MongoDB as database solution.

This scenario is very well suited to NoSQL Document Stores. Note that every application calls for another database – if your data has a fix schema, you might be better off with a traditional SQL store.

Feel free to drop a comment with your ideas! Also, like ❤️ this story if it was helpful for you!