Syslog pipeline customization

Download PDF

Extend the LogZilla syslog-ng pipeline via config.yaml, conf.d includes, SYSLOG_EXTRA_LOG_RULES, custom sources, and raw stream forwarding to another receiver

Syslog pipeline customization (advanced)

Administrators who need to customize the syslog-ng pipeline beyond what is available in the UI can use config.yaml, the conf.d/ include directory, dedicated sources, and safe ways to inject extra rules into the main pipeline.

For common changes such as ports, batching, buffering, and debug toggles, use Syslog Settings.

Configuration locations

  • /etc/logzilla/syslog-ng/config.yaml
    • Primary YAML used to render the syslog-ng configuration.
  • /etc/logzilla/syslog-ng/conf.d/
    • Additional *.conf files included by the main template.

Important:

  • Avoid creating custom top-level log statements. Use SYSLOG_EXTRA_LOG_RULES or the patterns shown below to keep the main pipeline intact.

When to customize

  • Add a dedicated listener or transport not covered by UI.
  • Tag a source via source_type for specialized parsing.
  • Insert targeted filters/rewrites into the main pipeline.
  • Create raw TCP/UDP inputs for non-syslog data.

Sources (standard)

Standard sources are generated from settings and usually require no changes:

  • bsd --- TCP on SYSLOG_BSD_TCP_PORT (BSD syslog)
  • bsd_udp --- UDP on SYSLOG_BSD_UDP_PORT (BSD syslog)
  • rfc5424 --- TCP on SYSLOG_RFC5424_PORT (RFC 5424 syslog)
  • json --- TCP on SYSLOG_JSON_PORT (newline JSON)
  • tls --- TCP on SYSLOG_TLS_PORT (TLS RFC 5424)
  • raw --- TCP on SYSLOG_RAW_PORT (no parsing)
  • raw_udp --- UDP on SYSLOG_RAW_UDP_PORT (no parsing)

Dedicated sources with source_type

To target a subset of events for specific parsing rules:

  1. Set source_type on the syslog-ng source in config.yaml.
  2. In the relevant Lua rule, set SOURCE_FILTER = "<tag>".
  3. Ensure the tag is included in Parser settings as required.

Only events with the matching source_type are processed by rules that declare that tag.

Example: add a TLS source with a dedicated tag

yaml
sources:
  - name: tls_west
    enabled: true
    type: network
    port: 6514
    transport: tls
    tls_cert_file: /etc/ssl/logzilla/server.crt
    tls_key_file: /etc/ssl/logzilla/server.key
    flags: ["syslog-protocol"]
    program_override: "tls-wf"
    extra_fields:
      site: "west-dc"
    source_type: "west"

If rules declare SOURCE_FILTER = "west", ensure the tag is permitted in Parser settings.

Example: add a raw UDP source for unparsed logs

yaml
sources:
  - name: raw_udp_1516
    enabled: true
    type: network
    port: 1516
    transport: udp
    flags: ["no-parse"]
    program_override: "raw-udp"
    extra_fields:
      log_type: "raw"
    source_type: "devices"

Injecting extra log rules (advanced)

Use SYSLOG_EXTRA_LOG_RULES in settings or set extra_log_rules in config.yaml to inject statements into the main log {}. Keep injected logic minimal and well-scoped.

yaml
extra_log_rules: "filter(f_only_host);"

When more complex filters are needed, place reusable building blocks in conf.d/ and reference them from extra_log_rules.

Example: forward the raw source stream to another receiver

To mirror the incoming event stream to another log receiver before any LogZilla processing (deduplication, parsing rules, rewrites), add a syslog-ng destination in conf.d/. A typical use case is forwarding from a production LogZilla server to a dev or lab LogZilla server so the second system receives the exact original messages and parses them independently.

This operates at the syslog-ng level, before the LogZilla pipeline. To forward events after deduplication and rewrite rules instead, use the Dedup Forwarder.

Create the forwarding config file (requires root/sudo). Replace YOUR_SERVER_NAME_OR_IP, then run:

bash
sudo mkdir -p /etc/logzilla/syslog-ng/conf.d && sudo tee /etc/logzilla/syslog-ng/conf.d/fwd_to_dev.conf > /dev/null << 'EOF'
destination d_dev_server {
    network("YOUR_SERVER_NAME_OR_IP"
        port(514)
        transport(udp)
    );
};
EOF

Verify the file was created correctly:

bash
cat /etc/logzilla/syslog-ng/conf.d/fwd_to_dev.conf

Then restart the syslog container to pick up the new config:

bash
sudo logzilla restart -c syslog

On restart, LogZilla detects the destination in conf.d/ and adds it to the main pipeline automatically, so all sources are mirrored to the new receiver. No SYSLOG_EXTRA_LOG_RULES entry is needed.

Messages are forwarded in syslog-ng's standard output format: the header (priority, timestamp, host) is re-serialized in normalized form, and the program and message content are unchanged. A receiving LogZilla server parses these exactly as it would the originals.

Important:

  • Keep the destination name and its opening brace on the same line (destination d_dev_server {). LogZilla scans conf.d/ for this pattern to register the destination into the main pipeline.
  • UDP transport is deliberate: if the receiving server goes down, UDP packets are simply dropped and production intake is unaffected. A TCP destination combined with flow control can throttle the main pipeline when the receiver is unreachable.

Restarting after changes

bash
logzilla restart -c syslog

Related topics

Syslog pipeline customization | LogZilla Documentation