GitLab is having some problems today after experiencing data loss due to a failure to properly back up data.
The service is backed by YCombinator and Khosla Ventures and it has become an incredibly popular tool for developers who are happy to use it because it offers all-around solutions for their needs. The service pretty much includes everything developers could need from the beginning to the end of the projects they’re working on.
A lot of companies depend on it, whether they’re small startups, individual developers hoping to sell whatever they’re working on, or large enterprises, such as Intel, for instance.
Well, they’re all currently waiting for the database to be completely copied and for the site to once more be accessible. GitLab announced that it was performing an emergency database maintenance, which would take GitLab.com offline.
Following this announcement, which made everything wonder whether they were being under cyber attack and trying to cover up the issues, they explained that it was actually a problem with their production database which they were working to recover.
What did GitLab lose?
“We accidentally deleted production data and might have to restore from backup,” reads a tweet sent out by GitLab.
So, what data was lost? It seems that no Git commits had been lost, just things like merge requests and issue posts and other live production data. About 300GB of data that is. The Register reports that it was all because of a tired sysadmin who accidentally deleted a directory on the wrong server. Before he canceled the command, 300GB of data had gone down the drain and only about 4.5 GB remained. The last viable backup was taken six hours beforehand so everything in between was pretty much gone.
For its part, GitLab tried about everything to get everything back. They’ve tried 5 backup and replication techniques to regain the data, but nothing worked. This is quite ironic, especially after GitLab decided to operate its own Ceph clusters last year, claiming it was going to make it all more efficient, and reliable.
GitHub will apparently start looking into alternative cloud hosting providers now, rolling back on the decision to leave the cloud in the first place.
According to the last tweet from GitLab, they’re 78% done with the database copy so it might be ust a little while longer before everything is up and running.
