How often do you test your backups and restores?
Regular backups are good practise but if you haven’t tested restoring a backup then you can’t know for sure your backups are doing what you think they are. Regularly restoring backups to a test environment is a good way to make sure they are working as you expect.
One setup is to have a nightly or weekly production database backup restored into your staging environment. That way you can check the latest code against a recent database with real data. This only works when you have a database that is small enough to restore in a reasonable time. It can end up being a bottleneck when teams get larger and everyone wants to check their work on staging before approving it for release.
Once you have multiple databases, sharding, etc. then backing up and restoring becomes much more complicated. You’re still going to end up with individual databases you can backup and restore though. This means you can still run checks on each of the backups to make sure they’re good.
When was the last time you did a test restore? What other strategies have you used for checking backups?