When a major incident occurs in an IT environment, the immediate priority is to restore services as quickly as possible.
However, the real value lies in the Post-Major Incident Review (PMIR), also known as a Post-Incident Review (PIR) or Post-Mortem. This review is crucial for identifying the root cause, learning from the event, and preventing similar issues from happening again. In this guide, we'll explore the importance of post-major incident reviews, how to conduct them effectively, and best practices for ensuring continuous improvement.
What is a Post-Major Incident Review (PMIR)?
A Post-Major Incident Review (PMIR) is a formal meeting or process that takes place after a major incident has been resolved. The goal is to analyse the incident, understand its causes, and assess the incident management process itself. It provides an opportunity for key stakeholders to review the incident, share insights, and document lessons learned.
The PMIR is critical for organizations because it:
1. Uncovers Root Causes:
Identifies the underlying technical and process failures that contributed to the incident. Or even the root cause for Problem Management to manage/investigate.
2. Promotes Learning:
Provides a structured environment for teams to learn from mistakes or missteps.
3. Improves Future Response:
Leads to actionable recommendations that enhance the organization’s ability to respond to and prevent future incidents.
4. Ensures Accountability:
Ensures that action items are assigned to responsible teams or individuals to prevent recurrence.
Key Components of a Successful Post-Major Incident Review
To ensure a productive PMIR, follow these key components:
1. Incident Recap
Begin by providing a clear and concise recap of the major incident. This should include:
- Incident Description: What happened and when it started.
- Impact Summary: Affected systems, services, or users, along with the severity of the impact.
- Timeline: A chronological timeline from when the incident was detected to when it was resolved.Include important milestones such as when key decisions were made and actions were taken.
2. Root Cause Analysis
Use structured methods such as the 5 Whys, Fishbone Diagrams, or Fault Tree Analysis to uncover the true root cause(s) of the incident. This should go beyond the immediate issue (e.g., "server failure") and identify why thefailure occurred (e.g., "misconfigured server settings" or "inadequate testing before a software release").
Key questions to ask during the analysis:
- What were the contributing factors that led to the incident?
- Was there a process breakdown that allowed the issue to escalate?
- Could this issue have been prevented? How?
The Root Cause Analysis may not be covered in some organisations. This is often covered within the Problem Management process. And, the Problem Management Practice will use the PMIR to begina Root Cause Analysis investigation. However, some organisations practice dual role for Major Incident Managers and Problem Managers. This is at your discretion to suit your specific environment.
3. Incident Response Evaluation
Evaluate the efficiency and effectiveness of the incident response process. This includes:
- Response Time: How long did it take to detect and respond to the incident?
- Communication: Was communication between teams and stakeholders effective? Were the right people informed at the right time?
- Escalation: Was the escalation process followed properly? Were there any delays in engaging the right teams?
Actionable Recommendations
Example 1: The value of Major Incident Management for large organisations
Take a major retailer as an example.
Many large, well-known retailers rely heavily on self-service point-of-sale devices and payment terminals to take payment for goods. During busy trading periods, they may process around £4,000 of transactions per second.
That equates to:
- £240,000 per minute
- £14,400,000 per hour
That is a substantial loss of earnings for every hour that point-of-sale devices and payment terminals are unavailable.
But the impact of a Major Incident is rarely limited to lost sales.
When clients or consumers become frustrated, the situation can quickly escalate into negative press coverage, social media criticism and wider reputational damage. The organisation may also experience longer-term commercial consequences if valued clients lose confidence and decide to take their business elsewhere.
A one-hour outage can create significant impact across multiple areas, including:
- Productivity loss: colleagues are unable to do their jobs effectively
- Financial loss: direct revenue loss, potentially reaching millions per hour
- Reputational damage: immediate frustration from clients, consumers, shareholders and the wider market
- Regulatory exposure: potential failure to maintain required standards, records or service obligations
This is why Major Incident Management is not simply an IT process. It is a business-critical capability.
Example 2: The value of Major Incident Management for smaller organisations
The value of Major Incident Management is just as important for smaller organisations.
Consider a professional services organisation with three office locations across Europe. The business experiences a complete loss of network access. As a result, 1,000 employees across the three locations are unable to log in, access systems, communicate effectively or progress client work.
If each employee costs the business around £145 per day, the direct productivity cost of a full-day outage is approximately £145,000.
Again, this only reflects the most visible cost.
The wider impact could include missed project deadlines, delayed client deliverables, reduced confidence from partners and suppliers, and increased pressure on internal teams trying to manage the disruption.
For smaller organisations, the financial loss may not reach the same figures as a major retailer, but the relative impact can be just as serious. A prolonged outage can affect cashflow, client relationships, service delivery and the organisation’s ability to operate with confidence.
Best Practices for Conducting a Post-Major Incident Review
To ensure that your PMIR is thorough and effective, consider these best practices:
1. Create a Blame-Free Environment
The purpose of the PMIR is to learn and improve, not to assign blame. Encourage transparency and open communication during the review, making it clear that the focus is on understanding what went wrong and preventing future incidents, not on pointing fingers.
2. Involve All Relevant Stakeholders
Ensure that everyone involved in the incident or affected by it participates in the review. This includes technical teams (IT, DevOps, network engineers), management, and possibly businessstakeholders. Including a diverse range of perspectives helps uncover insights that may be missed by a single team.
3. Keep it Structured
Use a formal agenda to guide the PMIR process. This ensures that the discussion stays focused and covers all key aspects of the incident. It’s important to allocate enough time for each section (e.g., incident recap, root cause analysis) without rushing.
4. Document the Review
Document all findings, discussions, and actionitems from the PMIR. This documentation should be stored in a centralized location where it can be accessed by relevant teams in the future. Keeping a well-organized log of incident reviews helps identify recurring patterns and long-term issues.
5. Turn Insights into Action
A PMIR is only valuable if it leads to actionable improvements. Be diligent about tracking action items and holding teams accountable for implementing them. Use project management or incident tracking tools to ensure that recommendations are executed and monitored.
Common Challenges in Post-Major Incident Reviews
While PMIRs are an invaluable tool for improvement, they can come with challenges:
1. Time Constraints:
In fast-paced environments, it can be difficult to find time for an in-depth review. However, skipping a PMIR can result in missed opportunities for
improvement.
2. Incomplete Information:
Sometimes, teams may not have all the data needed for a thorough analysis. Ensure proper logging and monitoring are in place during incidents to gather sufficient data for the review.
3. Blame Culture:
If a blame culture exists within an organization, team members may be reluctant to share insights or admit mistakes. Leaders must foster a safe environment that encourages openness.
Conclusion
Post-Major Incident Reviews are a critical component of Major Incident Management. They provide organizations with a structured way to analyse incidents, learn from them, and improve their response capabilities.
By fostering a culture of continuous improvement, conducting thorough root cause analysis, and implementing actionable recommendations, organizations can reduce the likelihood of future incidents and enhance overall system reliability.
