This article was accepted into the corpus but its outbound wikilinks were never NER-processed — typical at the deepest BFS hop or when the run's entity cap was reached. No expansion funnel to show.
| ACM Artifact Review and Badging | |
|---|---|
| Name | ACM Artifact Review and Badging |
| Formation | 2017 |
| Headquarters | New York City |
| Parent organization | Association for Computing Machinery |
ACM Artifact Review and Badging
ACM Artifact Review and Badging is a program administered by the Association for Computing Machinery that formalizes the evaluation and recognition of research artifacts submitted with publications at conferences and journals. It interfaces with publication processes at venues such as SIGCOMM, SIGGRAPH, STOC, and CHI and aligns with initiatives from organizations like the National Science Foundation, IEEE, and the Computing Research Association to improve reproducibility and transparency in computational research.
The program originated within the Association for Computing Machinery and was influenced by practices at venues including NeurIPS, ICML, OOPSLA, and SOSP and by policy developments at institutions such as the National Institutes of Health and the European Research Council. It establishes a workflow linking artifact evaluation to editorial processes used by ACM Digital Library venues, and it interacts with registries and infrastructures exemplified by Zenodo, GitHub, Dataverse, and Figshare. The initiative draws on methodological debates seen in journals like Nature, Science, and Communications of the ACM and on reproducibility efforts at conferences such as KDD, ACL, and EMNLP.
The stated purpose is to provide badges—visible endorsements—that indicate the availability, usability, and replication status of artifacts associated with published work at venues including SIGMOD, SIGPLAN, SIGARCH, and SIGMETRICS. Scope covers software, datasets, workflows, and experimental protocols used in submissions to conferences like CVPR, ICSE, and CHI and to journals such as Transactions on Graphics and Transactions on Computer Systems. The program aims to complement peer review practices at editorial boards like IEEE Transactions, Elsevier journals, and Springer venues and to coordinate with funding agency mandates from bodies such as the National Science Foundation and Horizon Europe.
Badges are awarded across categories such as Artifact Available, Artifact Functional, Artifact Reusable, and Results Reproduced, analogous to recognition schemes in venues like PLOS, F1000Research, and Open Science Framework. Specific criteria reference documentation standards used by organizations like the Software Sustainability Institute, the Research Data Alliance, and FAIR principles promoted by the European Commission. Evaluation criteria include archival stability (via Zenodo or Dryad), licensing clarity (informed by Creative Commons and GNU), build and test reproducibility (following practices from Docker, Ansible, and Singularity), and benchmark comparability (as used by SPEC, MLPerf, and TPC).
Artifact submission typically occurs alongside paper submission or during camera-ready stages for conferences including NeurIPS, ICML, and SC; reviewers include artifact evaluators drawn from SIG committees such as SIGCOMM, SIGGRAPH, and SIGMOD. The workflow uses platforms like EasyChair, HotCRP, and CMT and may integrate continuous integration services like Travis CI, GitHub Actions, and Jenkins for automated checks. Reviewers follow checklists inspired by procedures at ACM SIGPLAN PC, IEEE Peer Review, and editorial policies at journals like ACM Transactions and IEEE Transactions. Decisions about badges are coordinated with program chairs and editors from venues such as KDD, AAAI, and IJCAI.
Artifacts evaluated include source code, binary releases, datasets, virtual environments, and experiment scripts analogous to artifacts curated by repositories at CERN, EMBL-EBI, and ICPSR. Guidelines require metadata schemas comparable to schema.org, DataCite, and Dublin Core and emphasize provenance records used by projects at the Library of Congress, National Archives, and ORCID. Evaluators assess build instructions referencing containerization tools from Docker, Kubernetes, and Vagrant and test reproducibility using platforms like Binder and ReproZip. Ethical and privacy considerations reflect policies from Institutional Review Boards, HIPAA frameworks, and GDPR interpretations when datasets involve human subjects.
When conferred, badges appear in ACM Digital Library entries and in conference proceedings, similar to badges or icons used by venues such as PLOS, bioRxiv, and arXiv. Policies govern retroactive assignment, appeals, and revocation akin to procedures at journals like Nature Methods, Science Advances, and Proceedings of the National Academy of Sciences. Licensing and attribution requirements reference Creative Commons, MIT License, and Apache License conventions and coordinate with archiving strategies at LOCKSS, CLOCKSS, and Portico to ensure long-term access. Display rules are aligned with metadata practices adopted by CrossRef, DataCite, and ORCID.
Adoption has spread across ACM conferences including SIGCOMM, SIGGRAPH, and CHI and has been cited in policy discussions involving the National Science Foundation, Wellcome Trust, and European Commission. Proponents compare its effects to transparency measures at NeurIPS, ICML, and ACL, noting improvements in reuse and citation similar to those observed with open data mandates at Dryad and Zenodo. Criticism echoes debates in literature involving Science, Nature, and Communications of the ACM about administrative burden, reviewer workload, and the limitations of badges as incentives, referenced in community discussions at workshops hosted by CRA, USENIX, and FORCE11. Concerns also cite potential mismatches with licensing regimes used by GNU, Apache, and proprietary vendors and with reproducibility challenges highlighted by reproducibility studies in computational biology, machine learning, and systems research.