As I have mentioned before, I am a big fan of Dropbox. I have been using the file replication service for two or three years and now depend upon it daily. However, my timing is obviously terrible. On Saturday, January 11, I published an article describing how I use Dropbox to keep copies of my old family photographs on multiple computers. I actually wrote the story on Friday evening and editor Pam improved my words on Saturday morning. I published the completed article Saturday afternoon.
What I didn't know at the time was that Dropbox went down Friday night after some normally routine upgrades went awry. This appears to be about the same time I finished writing the article. While the company restored most functionality within three hours, problems for some users persisted until Sunday.
Talk about timing. Arrgghh! Quite a few newsletter readers tried to follow my recommendations Saturday evening or Sunday and found that things didn't work.
Here is an excerpt from Sunday's announcement from Dropbox head of infrastructure Akhil Gupta:
We use thousands of databases to run Dropbox. Each database has one master and two slave machines for redundancy. In addition, we perform full and incremental data backups and store them in a separate environment.
On Friday at 5:30 p.m. PT, we had a planned maintenance scheduled to upgrade the OS on some of our machines. During this process, the upgrade script checks to make sure there is no active data on the machine before installing the new OS.
A subtle bug in the script caused the command to reinstall a small number of active machines. Unfortunately, some master-slave pairs were impacted which resulted in the site going down.
User files were never at risk during the outage, the company said.
Dropbox restored most services Friday night by performing recovery from backups. Problems persisted, though. On Sunday at 1:59pm PT, the company reported that "[a]bout 5% of our users are still experiencing problems syncing from the desktop client, and about 20% of users are having issues accessing Dropbox through our mobile apps." By 4:40pm PT on Sunday, "core service" was fully restored although the company was still "working through a few last issues with the Dropbox photos tab." Everything apparently returned to normal in the wee hours of Monday morning.
I have looked at my files on Dropbox. I have more than 28 gigabytes of data stored there so I didn't check every file. However, all the files I spot checked are exactly where I expect them to be and all of them are the latest versions as stored on my computers' hard drives. I added quite a few new files on Sunday afternoon and a few more on Monday. All of the new files were replicated (copied) to the other computers and to Dropbox's web servers within seconds.
In short, Dropbox appears to now be operating normally.