Magic Pocket

Magic Pocket. It is a massive-scale, custom-built cold storage system developed by Dropbox to reliably and cost-effectively store exabytes of user data. The system represents a fundamental shift from reliance on third-party cloud infrastructure to a vertically integrated, in-house storage architecture. Its development was a major engineering undertaking aimed at ensuring data durability, operational efficiency, and long-term scalability for one of the world's largest file hosting services.

Overview

Magic Pocket serves as the foundational object storage layer for all data uploaded to the Dropbox platform, handling hundreds of petabytes and eventually exabytes of information. The system was designed to achieve exceptional levels of fault tolerance and data integrity, targeting durability metrics that far exceed typical commercial cloud storage offerings. By building this proprietary system, Dropbox gained direct control over its entire data center stack, from the hard disk drive hardware to the management software. This move was strategically significant in the competitive landscape against other major providers like AWS S3 and Google Cloud Storage.

Technology

The architecture of Magic Pocket is built around custom-designed storage servers and a sophisticated software stack that prioritizes erasure coding over traditional data replication. This erasure coding scheme, similar to techniques used in other large-scale systems like Facebook's f4 and Microsoft Azure, allows for significant reductions in storage overhead while maintaining high durability. The system employs a distributed system model where data is sharded across thousands of machines within multiple data centers, ensuring resilience against hardware failures and data center outages. Key components include automated processes for data integrity verification, load balancing, and failure detection, all managed through a global namespace that tracks every stored block of data.

History and development

The project to develop Magic Pocket was initiated by Dropbox engineers around 2013, culminating in a full migration of user data from Amazon S3 to the new system by 2016. This migration was one of the largest of its kind, involving the movement of over 500 petabytes of data without service interruption for millions of users. The development effort required close collaboration with hardware manufacturers to design cost-optimized storage servers and involved pioneering work on the software's consensus algorithm and metadata management. The success of the project was highlighted in technical presentations at venues like USENIX and influenced infrastructure strategies at other technology companies.

Impact and reception

The deployment of Magic Pocket had a profound impact on Dropbox's operational economics, reportedly saving the company tens of millions of dollars annually in storage costs. It was widely recognized within the software engineering and distributed systems communities as a landmark achievement in infrastructure engineering. The technical details, shared through blog posts and conference talks, contributed valuable knowledge to the field of large-scale storage systems. The system's reliability and scale cemented Dropbox's capability to operate as a top-tier infrastructure provider, influencing its subsequent offerings like Dropbox Paper and Smart Sync.

See also

* Dropbox (service) * Cold storage * Erasure coding * Object storage * Amazon S3 * Distributed data store * Data center * Petabyte

Category:Cloud storage Category:Data management Category:Dropbox