Deleted my live production database - Lesson learned
Hazmi Irfan
Posted on March 7, 2021
You can read more on how I accidentally deleted my live production database for my website https://www.logname.dev here.
The TLDR version is
- I wanted to know how many user registered.
- I don't have admin page so I want to access the database directly
- Instead of connecting to the database I run the delete command.
Anyway, I decided to compile a list of things that I learn from every comment that I receive. Hopefully you won't repeat the same mistake that I did.
1. You should never connect to a live production database.
I guess this one is pretty obvious and yet this is the main mistake I did. I only want to know how many people registered and I didn't have an admin page back then.
Instead of creating a simple admin page which only took less than 20 minutes, I instead tried to connect to the live production database and run a query there.
2. Backup backup backup.
Too be fair to myself, the website had only run for just a day so even if I add an automated backup it might not yet run at that time.
Still, I only need to click 1 button in the Heroku website to manually back it up and I didn't.
3. Adding a space before your command will not save it in history.
As far as I know it works in Linux system. If you run a dangerous command, you can start with a space first. This will make sure your command will not be saved in history and you might not accidentally run it again like I did!
4. Avoid using confirm flag for dangerous command.
When I deleted my database on purpose the first time, Heroku actually trigger a prompt for confirmation. I can either reply yes
or I need to stop the script and run it again with --confirm
flag. I did the latter. If I deleted it with the first option, when I accidentally run the command again, Heroku will prompt the confirmation flow again, this will give me time to stop the script. Alas, I choose the 2nd flow and the rest is history(literally).
When I look back, I realize that it wasn't just 1 big mistake but a couple of small mistake that adds up. I may have made some of this mistake in the past but never all of it in the same time.
Example, I knew it's a mistake connecting to live production database but I did it on my previous project and it was fine back then.
I also knew you should never perform deletion of database on a command line but I did it before and I was still fine.
But combining these 2 mistake result in 1 of the biggest mistake I have made so far as a developer. One that I wish I won't repeat in the future.
Posted on March 7, 2021
Join Our Newsletter. No Spam, Only the good stuff.
Sign up to receive the latest update from our blog.