Incident response runbook for Egyptian online stores

Incident response runbook covering containment, communication, and postmortem learning.

Compliance and scale

An incident is not a slow afternoon or a feature you wish worked better; it is the moment money or trust is actively bleeding right now, and a calm written plan beats a panicked group chat every single time. A payment gateway that stops authorising, a storefront that returns errors to every shopper, a sudden wave of clearly fake cash-on-delivery orders, or a sign that a customer list has leaked are all the same kind of emergency: you must stop the damage first, then tell the people it touched, then learn enough that it does not happen the same way twice. Treating these like ordinary support tickets is how a ten-minute glitch turns into a day of lost orders and a week of broken trust.

This lesson is the runbook your team opens when the store is on fire, not the slip you fill in for a small bug or a missing report. It assumes the problem is live and costing you, so the order of moves matters more than getting every detail perfect. What follows is a sequence you can pin above the desk and hand to whoever is on shift when it breaks: detect what is really happening, contain it fast, communicate honestly with the buyers it affects, and run a short postmortem so the same fire never lights twice.

First, confirm what is actually broken

Panic makes people fix the wrong thing. Before you touch anything, spend five minutes proving what the incident really is, because a "payments are down" report can turn out to be one courier's webhook or a single browser caching an old page.

  • Check the status of the affected system. Is the storefront down for everyone or just you on one device? Try a second phone on mobile data, not your office Wi-Fi. If only one checkout method fails, the blast radius is far smaller than "the whole store is down."
  • Read the audit log to see who changed what. Your store keeps an append-only record of every meaningful change — who edited a setting, who altered a price, who touched a permission. If the timeline shows a config change minutes before things broke, you have likely found the cause without guessing.
  • Name the incident in one honest sentence. "Card payments have failed for all customers since roughly 2pm" or "We received forty COD orders to fake numbers in ten minutes." A precise sentence stops three people debugging three different theories.
  • Assign one owner. One person decides and speaks; everyone else feeds them facts. Incidents go sideways when two people change settings at the same time.

Contain the damage before you explain it

The goal of containment is to stop the bleeding, even if the fix is temporary and ugly. A clean rollback or a paused feature today beats a perfect explanation tomorrow.

  1. Pause the affected operation. If a payment method is charging twice or failing silently, switch it off and leave cash on delivery running so orders still flow. If fake orders are flooding in, pause new-order acceptance or tighten it rather than shipping to phantom addresses in the governorates.
  2. Rotate any exposed credentials. If a password, API key, or webhook secret may have leaked, change it immediately — assume the worst rather than hoping. Turn on two-factor authentication on every admin account the same hour, so a stolen password alone can no longer get in.
  3. Reverse the bad change. If the audit log points to a setting someone flipped, set it back. Note the time you did it so the postmortem has a clean before-and-after.
  4. Freeze risky money movements. Hold refunds and payouts on suspicious orders until a human verifies them, so a fraud spike or a double-charge bug does not drain your wallet while you work.

Communicate honestly with the people it touched

Silence during an incident does more damage than the incident itself, because buyers fill the gap with worse assumptions. Tell only the people actually affected, tell them early, and tell them the truth.

  • Reach affected buyers directly on the channels they use — a short WhatsApp message or email naming the problem, what you are doing, and when you will update them. "Your payment did not go through because of a technical issue on our side; your order is held safely and we will confirm within the hour" turns anger into patience.
  • Never hide a charge or a delay. If a buyer was charged twice, say so and promise the InstaPay or wallet refund with a clear timeframe in EGP. An honest refund offered first earns more loyalty than one extracted after a fight.
  • If personal data may have leaked, say what and to whom in plain language. Egyptian buyers care that their phone and address are safe; a calm, specific notice that you have already rotated credentials and closed the hole respects that.
  • Keep one public line consistent. If you must post anything to your store or social channels, match it to what you told buyers privately so the story never contradicts itself.

Learn from it with a short postmortem

The incident is not over when the store works again; it is over when you have changed something so it cannot recur the same way. Within a day, write half a page — no blame, just facts.

  1. What happened and when, pulled from the audit log timeline and your own notes.
  2. The root cause in one line — not "the system broke" but "a webhook secret was committed publicly and someone replayed it."
  3. One concrete fix you will ship, with an owner and a date, such as enabling two-factor everywhere or adding a check that flags ten orders from one number.

File the postmortem where your team can find it, and review the pattern monthly. An incident handled cleanly and explained honestly often leaves customers trusting you more than if it had never happened.

Related lessons

This Storix Academy lesson is published as "Incident response runbook for Egyptian online stores". Every step below is written for merchants selling online in Egypt, so apply one change at a time and measure the result before moving on.