In the world of IT and web services, outages and system failures are inevitable. When they occur, a detailed postmortem is crucial for understanding what went wrong and preventing similar issues in the future. This blog post will guide you through the process of writing an effective postmortem using a real-life use case example.

Why Write a Postmortem?

A postmortem helps teams:

• Understand the root cause of the issue.
• Document the timeline of events and actions taken.
• Identify areas for improvement and implement preventative measures.
• Communicate transparently with stakeholders about what happened and what will be done to prevent recurrence.

Structure of a Postmortem

A well-structured postmortem includes the following sections:

• Issue Summary
• Timeline
• Root Cause and Resolution
• Corrective and Preventative Measures

Let’s dive into each section with a use case example.

Use Case Example

Scenario: An e-commerce website experienced an outage on June 12, 2024. Here’s how the postmortem was structured and written.

Issue Summary

Duration of the Outage:
Start: June 12, 2024, 09:00 AM (WAT)
End: June 12, 2024, 11:30 AM (WAT)

Impact:
The e-commerce website was completely inaccessible, affecting approximately 95% of users. This resulted in lost sales and numerous customer complaints. Over 200 complaints were received within the first hour.

Root Cause:
The root cause was a misconfigured database connection pool that led to the exhaustion of available connections, preventing the web application from accessing the database.

Timeline

09:00 AM (WAT): Issue detected through a monitoring alert indicating high database connection usage.
09:05 AM (WAT): Engineering team notified via pager duty.
09:10 AM (WAT): Initial investigation focused on the web server load and potential DDoS attack.
09:30 AM (WAT): Misleading path: assumed high traffic causing server overload, but server metrics were normal.
09:45 AM (WAT): Database team brought in for further investigation.
10:00 AM (WAT): Identified issue with the database connection pool limits.
10:15 AM (WAT): Escalated to the senior database administrator.
10:45 AM (WAT): Senior DBA confirmed connection pool misconfiguration.
11:00 AM (WAT): Connection pool configuration updated and increased.
11:15 AM (WAT): Web application restarted, and database connections restored.
11:30 AM (WAT): Service fully restored and confirmed stable.

Root Cause and Resolution

Root Cause:
The outage was caused by a configuration error in the database connection pool settings. The connection pool was set to a maximum of 50 connections, which was insufficient for handling peak traffic loads. As a result, the application exhausted all available connections, leading to timeouts and an inability to process any database queries.

Resolution:
The database connection pool settings were reviewed and updated. The maximum number of connections was increased to 200, providing enough capacity to handle peak loads. After updating the configuration, the web application was restarted to apply the changes. Monitoring tools confirmed the restoration of normal operations.

Corrective and Preventative Measures

Improvements:

1. Review and Adjust Connection Pool Settings: Regularly review and adjust database connection pool settings based on traffic patterns and load testing results.
2. Enhanced Monitoring: Implement more granular monitoring for database connection usage to detect issues before they lead to outages.
3. Automated Scaling: Explore the implementation of automated scaling solutions for the database connection pool based on real-time demand.

Tasks:

1. Increase Connection Pool Limit:
   Update the database configuration to set a higher default connection pool limit.

2. Implement Connection Pool Monitoring:
   Add detailed monitoring for connection pool usage and set up alerts for unusual patterns.

3. Conduct Load Testing:
   Perform load testing to determine optimal connection pool settings for peak traffic.

4. Automate Scaling Solutions:
   Research and implement an automated scaling solution for the database connection pool to dynamically adjust based on load.

5. Review Configuration Management:
   Establish a regular review process for all configuration settings related to the database and web application to ensure they meet current traffic demands.

6. Update Documentation:
   Document the configuration changes and update the runbooks to include steps for adjusting the connection pool settings.

Writing a detailed postmortem helps your team understand the root cause of an outage, improve your processes, and communicate effectively with stakeholders. By following the structured approach outlined in this post and our use case example, you can ensure your postmortems are thorough and actionable, leading to a more resilient and reliable service.