Deleted my live production database - Lesson learned

tehaisperlis

Hazmi Irfan

Posted on March 7, 2021

Deleted my live production database - Lesson learned

You can read more on how I accidentally deleted my live production database for my website https://www.logname.dev here.

The TLDR version is

  • I wanted to know how many user registered.
  • I don't have admin page so I want to access the database directly
  • Instead of connecting to the database I run the delete command.

Anyway, I decided to compile a list of things that I learn from every comment that I receive. Hopefully you won't repeat the same mistake that I did.

1. You should never connect to a live production database.

I guess this one is pretty obvious and yet this is the main mistake I did. I only want to know how many people registered and I didn't have an admin page back then.

Instead of creating a simple admin page which only took less than 20 minutes, I instead tried to connect to the live production database and run a query there.

2. Backup backup backup.

Too be fair to myself, the website had only run for just a day so even if I add an automated backup it might not yet run at that time.

Still, I only need to click 1 button in the Heroku website to manually back it up and I didn't.

3. Adding a space before your command will not save it in history.

As far as I know it works in Linux system. If you run a dangerous command, you can start with a space first. This will make sure your command will not be saved in history and you might not accidentally run it again like I did!

4. Avoid using confirm flag for dangerous command.

When I deleted my database on purpose the first time, Heroku actually trigger a prompt for confirmation. I can either reply yes or I need to stop the script and run it again with --confirm flag. I did the latter. If I deleted it with the first option, when I accidentally run the command again, Heroku will prompt the confirmation flow again, this will give me time to stop the script. Alas, I choose the 2nd flow and the rest is history(literally).

When I look back, I realize that it wasn't just 1 big mistake but a couple of small mistake that adds up. I may have made some of this mistake in the past but never all of it in the same time.

Example, I knew it's a mistake connecting to live production database but I did it on my previous project and it was fine back then.

I also knew you should never perform deletion of database on a command line but I did it before and I was still fine.

But combining these 2 mistake result in 1 of the biggest mistake I have made so far as a developer. One that I wish I won't repeat in the future.

💖 💪 🙅 🚩
tehaisperlis
Hazmi Irfan

Posted on March 7, 2021

Join Our Newsletter. No Spam, Only the good stuff.

Sign up to receive the latest update from our blog.

Related