syslog-ng

syslog-ng
Name	syslog-ng
Developer	Balabit (now One Identity)
Released	1998
Operating system	Unix-like
License	Proprietary and open source editions

syslog-ng syslog-ng is a log management and message routing application for Unix-like systems. It provides centralized logging, log parsing, and delivery to diverse storage and analysis backends. syslog-ng is used alongside tools from projects and organizations such as Red Hat, SUSE, Canonical (company), Debian, and Amazon Web Services in environments managed by teams from IBM, Microsoft, Google, Oracle Corporation, and VMware. Administrators integrate syslog-ng with platforms including Splunk, Elastic NV, Graylog, Azure Monitor, and Datadog.

Overview

syslog-ng originated in the late 1990s and evolved within a landscape influenced by standards and implementations like RFC 3164, RFC 5424, and the original syslog daemon implementations on UNIX System V and Berkeley Software Distribution. Its development intersected with companies and projects such as Balabit, SUSE Linux Enterprise Server, and contributors from academic institutions and vendors including SUN Microsystems, HP, and Cisco Systems. The project has both open source and commercial editions, used in enterprises such as Goldman Sachs, Bank of America, AT&T, Verizon, Comcast, and public sector deployments in jurisdictions like European Union agencies and United States Department of Defense integrations.

Architecture and Components

The architecture relies on modules and daemons interacting with subsystems like systemd, inetd, and network stacks from vendors such as Intel Corporation and Broadcom. Core components include the main daemon, parsers, filters, destinations, and transport modules that integrate with protocols and projects like TLS, OpenSSL, GnuTLS, and TCP/IP. syslog-ng uses plugin interfaces similar in concept to extension mechanisms found in Apache HTTP Server, Postfix, and NGINX. It interoperates with directory and identity services such as LDAP, Active Directory, and Kerberos for authentication and access control, and with time synchronization services like NTP and Chrony.

Configuration

Configuration uses a declarative language for sources, parsers, filters, and destinations, analogous to configuration approaches in BIND, PostgreSQL, and MongoDB. Administrators craft files often managed by configuration tools and automation systems like Ansible, Puppet (software), Chef (company), SaltStack, and Terraform. Version control of configurations is commonly done in Git, with CI/CD pipelines involving Jenkins, GitLab CI, and Travis CI. Configurations may reference schemas and patterns informed by projects like Logstash, Beats, and Fluentd.

Features and Capabilities

syslog-ng supports structured logging formats and protocols, integrating with ecosystems around JSON, CSV, Avro, and Protocol Buffers. It offers message parsing using pattern libraries similar to PCRE, and enrichment via lookups against databases such as MySQL, PostgreSQL, SQLite, and Redis. For delivery and storage it plugs into systems like Hadoop, Apache Kafka, Cassandra, and Amazon S3, and works with analysis products from Splunk, Elastic Stack, and SAS Institute. Features include reliable transport over TCP, encrypted channels using TLS and OpenSSL, high-availability patterns used by HAProxy and Keepalived, and compatibility with container platforms such as Docker and Kubernetes.

Deployment and Use Cases

Typical deployments range from single-host logging on distributions like Red Hat Enterprise Linux and Ubuntu to large-scale log aggregation in cloud environments operated by Amazon Web Services, Google Cloud Platform, and Microsoft Azure. Use cases include security information and event management (SIEM) in stacks built with QRadar, ArcSight, and OSSIM; compliance reporting for standards such as PCI DSS, ISO/IEC 27001, and HIPAA; and operational monitoring for services from Netflix, Airbnb, and Spotify. Integrations often involve orchestration and monitoring tools like Prometheus, Nagios, Zabbix, and New Relic.

Performance and Scalability

Performance tuning draws upon practices used in high-throughput projects like Apache Kafka, Nginx, and HAProxy. syslog-ng supports multithreaded processing and batching, and scales horizontally with message brokers and storage backends such as Apache Cassandra, MongoDB, and Elasticsearch. Large-scale deployments have been described in case studies from F5 Networks, Akamai Technologies, and financial institutions like JPMorgan Chase and Citigroup. Benchmarking and profiling often reference tools and methodologies from perf (Linux tool), sysstat, and dstat.

Security and Compliance

Security features include transport encryption, authentication via TLS certificates and X.509 infrastructures, audit logging aligned with recommendations from NIST, and integration with identity providers such as Okta and Ping Identity. Compliance-driven deployments align syslog-ng outputs with controls for SOX, GDPR, and sector-specific guidance from agencies like FINRA and SEC. Security operations teams combine syslog-ng with threat intelligence feeds from vendors like Mandiant and CrowdStrike, and incident response frameworks described by CERT Coordination Center and MITRE such as MITRE ATT&CK.

Category:Logging software