Mercury (project)

Mercury (project)
AI-generated (Stable Diffusion 3.5) · CC BY 4.0 · source
Name	Mercury
Developer	MIT Media Lab, DARPA, IBM, Google
Released	2017
Latest release	2021
Programming language	C++, Python
License	Apache License, MIT License

Mercury (project) is an experimental software and hardware initiative that targets high-performance data processing, low-latency networking, and secure computation across heterogeneous environments. Launched as a multi-institutional collaboration, Mercury sought to combine innovations from academic laboratories, corporate research groups, and government agencies to advance capabilities in distributed systems, compiler toolchains, and cryptographic protocols. The project emphasized modularity, portability, and measurable metrics to influence standards and production deployments.

Background and objectives

Mercury originated from research agendas at MIT Media Lab, Massachusetts Institute of Technology, and funding calls from DARPA and the National Science Foundation to address challenges seen in projects like MapReduce, Apache Hadoop, TensorFlow, GraphLab, and Spark (software). Goals included achieving deterministic performance comparable to High-Performance Computing clusters used at Lawrence Berkeley National Laboratory and Oak Ridge National Laboratory while supporting cloud environments such as Amazon Web Services, Google Cloud Platform, and Microsoft Azure. Objectives explicitly targeted integration with standards bodies including the IETF, IEEE, and the W3C to influence networking, serialization, and security practices used by initiatives like Kubernetes, Docker, and OpenStack.

Architecture and design

The Mercury architecture combined concepts from x86-64 architecture, ARM architecture, and research in RISC-V to support heterogeneous execution on servers and edge devices like Raspberry Pi and NVIDIA Jetson. Its design included a microkernel-inspired runtime influenced by seL4, a scheduling model comparable to Apache Mesos, and a data plane reminiscent of Data Plane Development Kit innovations used by Cisco Systems. Persistent storage models drew on techniques from Ceph, Amazon S3, and Google File System, while the messaging layer referenced designs from ZeroMQ, RabbitMQ, and Apache Kafka.

Development history and timeline

Initial research began in 2015 at the Computer Science and Artificial Intelligence Laboratory and industry labs at IBM Research and Google Research. A prototype was demonstrated at the USENIX conference in 2017, followed by a community sprint at Open Source Summit in 2018. Major releases tracked milestones similar to those in Linux kernel development and incorporated contributions from projects such as LLVM, GCC, and GitHub. Reviews and peer commentary appeared in venues including ACM SIGCOMM, IEEE INFOCOM, and Usenix ATC.

Implementation and technologies

Implementation used toolchains from LLVM Project and runtime libraries influenced by glibc and Boost (C++ libraries). Networking stacks leveraged DPDK and eBPF integrations inspired by Linux Foundation efforts. Containerization support aligned with OCI and Cloud Native Computing Foundation projects like Kubernetes and Prometheus for observability. Cryptographic primitives were implemented using libraries such as OpenSSL, libsodium, and advances from National Institute of Standards and Technology research into post-quantum schemes. Build systems relied on Bazel (software), CMake, and continuous integration through Travis CI and Jenkins.

Use cases and deployments

Early adopters included research teams at Lawrence Livermore National Laboratory for scientific workflows, startups in Silicon Valley leveraging real-time analytics comparable to Apache Storm, and media platforms using transcoding pipelines akin to FFmpeg. Edge computing pilots integrated Mercury with EdgeX Foundry and OpenFog Consortium reference architectures for Internet of Things scenarios seen in Smart Cities projects coordinated with municipalities like Barcelona and Singapore. Financial services firms used Mercury prototypes for low-latency trading experiments inspired by technologies from Nasdaq and Bloomberg L.P..

Performance and evaluation

Benchmarking compared Mercury against systems such as Apache Spark, Dask (software), MPI, and proprietary stacks from NVIDIA and Intel Corporation. Metrics focused on throughput, tail latency, and energy efficiency measured in cooperation with laboratories at Argonne National Laboratory and Sandia National Laboratories. Published evaluations appeared in ACM Digital Library proceedings and compared favorably on microbenchmarks and end-to-end workloads including stream processing, batch analytics, and machine learning training similar to experiments run with PyTorch and Keras.

Security and privacy considerations

Security design incorporated concepts from TLS, IPsec, and authenticated encryption modes promoted by IETF standards. Privacy-preserving computation leveraged secure multi-party computation research associated with Yao's Garbled Circuits and homomorphic encryption advances discussed at CRYPTO and EUROCRYPT conferences. Threat modeling referenced frameworks used by NIST and OWASP; penetration testing used tooling popularized by Metasploit Framework and static analysis from Coverity and Clang Static Analyzer.

Community, governance, and licensing

The Mercury project adopted an open governance model influenced by Apache Software Foundation and Linux Foundation practices, with contributor agreements akin to those used by Mozilla Foundation and Eclipse Foundation. Source code and documentation were hosted on GitHub with continuous contribution guidelines similar to Contributor Covenant. Licensing choices combined permissive terms from the Apache License and MIT License to facilitate adoption by corporations like Google and Amazon while enabling academic reuse at institutions such as Stanford University and University of California, Berkeley.

Category:Distributed computing projects