Logging is one of the activities that is commonly overlooked by developers, occasionally in some relatively small projects. As soon as the service goes live in production, monitoring production environment becomes relatively difficult; you can only assume it has similar behavior to staging environment. Furthermore, some developers also rarely have direct access to the application in production. This is where logging comes into play.

Why logging is important ?

Suppose your team has already successfully deployed to production after passing multiple testing scenarios (such as unit tests, integration, etc). The next day, one of the users encounters an issue and reports the problem to your product manager. Unfortunately, they cannot provide clear information regarding the steps or behaviors that caused the problem to arise.

Your team will probably have a headache trying to figure out the causes and how to replicate the issue. So, instead of relying on information from the user, one of your team members comes up with an idea to store the user's historical activities and behaviors in some kind of structured data. With that, you can easily trace and replicate the issue by analyzing historical logs, and this is what logging is meant to be.

Disclaimer

Perhaps you are wondering what kinds of logs need to be stored. Definitely, there are a lot of important things that might be on that list, apart from user activity, such as server resource consumption and database changes. However, in this article, it's focused on user activity only, so that doesn't mean the others are not important

Background

During my initial period as a developer, it took me a long time to find good references about log formats. Right now, I have come up with an idea to share my common log format, which is quite concise and understandable. Hopefully, this article can provide you with guidance or serve as a starting point to explore more advanced formats later. I would like to say that this article is beginner-friendly.

Format and Structure

Actually, there are a lot of formats you can use to store the data, such as JSON, CEF, and CLF (see more here). But from my perspective, the important thing is to ensure that your log data is accessible later for querying and filtering. This will help you find specific references, like a particular user with their corresponding ID.

In this article, we will use JSON format. The reason is pretty simple, right now almost entire level of developers already experienced a lot with this format.

{
    "level": "String",
    "error": "String",
    "request": { "method": "String", "url": "String" },
    "response": { "message": "String", "status": Number },
    "timestamp": "String",
    "user": { "id": "String" }
}

List of Attributes and Explanations

All of these attributes have different characteristics; some attributes can be labeled as optional, and some cannot, which means they are mandatory. Let's break them down below.

1. Timestamp (Mandatory)
There's no debate about this attribute; it's necessary to include it as it will help you find when the error event occurred. Let's say a user reported encountering the issue in the morning. You can then filter to find logs that occurred during the morning. Lastly, don't forget to synchronize or localize your timestamps according to the users' timezones. Otherwise, it will lead to confusion later.

2. Level (Mandatory)
Since logging produces a lot of data, it's essential to label each piece of data according to its urgency level. This helps to filter or automatically trigger alert messages when a specific level occurs. Even though these level labels are customizable, you can follow common labels such as 'fatal,' error, warn, info, debug, and trace. Additionally, those labels are already set as defaults in several logging libraries such as Pino, Winston, and Loguru.

3. Request (Mandatory)
For this attribute, it consists of several sub-attributes such as HTTP method and URL path. These will help you identify the API, as similar URL endpoints could have different HTTP methods (e.g., GET and POST sometimes refer to the same endpoint, /resource).

4. Response (Mandatory)
This also consists of several sub-attributes, namely message response and status code. Later, they will help you identify what the problems are related to. Is it about validation error 400, a not found resource 404, or even an internal service error 500

5. Error (Optional)
As you know, for this logging mechanism, almost every successful and failed response will be stored. To reduce the size of the log file, it's necessary to make this error attribute optional, which means including it only when an error is raised.

6. User (Optional or Mandatory)
This attribute stores user identity data, which is mainly used to find specific user behavior. In terms of urgency level, this attribute is at the top. However, you need to be concerned about personal and privacy security. That's the reason why I labeled it as optional—just make sure your log file is not easily targeted by attackers since it could compromise user personal data. From my perspective, I would prefer to store only the identifier (ID) rather than the entire user data. This keeps it concise and is sufficient for later identification. Additionally, don't forget to comply with regulations about personal data, such as GDPR.

Finally, this is what the actual JSON would look like when it's filled with data.

{
    "level": "info",
    "error": "The trip is already booked by another user",
    "request": { "method": "POST", "url": "/api/v1/trip" },
    "response": { "message": "Unprocessable Entity", "status": 422 },
    "timestamp": "27/10/2023 09.25.20.114 WIB",
    "user": { "id": null }
}

What's Next ?

You can proceed to implement a logging mechanism in your backend services. The actual implementation may vary slightly depending on the framework or programming language you are using. However, it will typically involve the use of middleware or interceptors. Afterward, you can store your logs in a file and monitor them with tools such as Grafana.

Tutorial

In case you're wondering how to create logs with that structure and need a technical guide, I am planning to write the tutorial later, which will probably be available soon. So, stay tuned!