Can Filebeat use multiple config files?

Question

I have several applications running on a single server. I'd like to use filebeat to ship the logs of each of them to logstash. However, for the sake of configuration management, I'd like to be able to add configuration to filebeat for each app separately.

Logstash reads its config from a conf.d directory. It's my understanding that one can add files there and they get combined when logstash loads them. Is there any similar feature for filebeat? Or am I stuck with maintaining a single filebeat.yml file per server?

I'm running both filebeat and logstash as services on CentOS 7, using the yum/rpm packages from elastic's repositories. Filebeat is version 1.3.1 and logstash is version 2.4.0.

Solution

Yes, Filebeat has a conf.d like feature, but it is not enabled by default. Filebeat will look inside of the declared directory for additional *.yml files that contain prospector configurations. The configuration varies by Filebeat major version.

Filebeat 7.x:

The behavior is the same as 6.x, but the config option is filebeat.config.inputs instead of filebeat.config.prospectors.

# /etc/filebeat/filebeat.yml
filebeat.config.inputs:
  enabled: true
  path: inputs.d/*.yml

Then create individual config files for each app that's generating logs.

# /etc/filebeat/inputs.d/someapp.yml
- paths:
  - /var/log/someapp/stdout.log
  fields:
    app: someapp

Filebeat 6.x:

You specify a path option in the filebeat.config.prospectors section of the filebeat.yml file.

filebeat.config.prospectors:
  enabled: true
  path: /etc/filebeat/conf.d/*.yml

/etc/filebeat/conf.d/someapp.yml

Note that this file does not contain filebeat.prospectors like it did in earlier versions.

- paths:
    - /var/log/someapp/stdout.log
  fields:
    app: someapp

Filebeat 1.x and 5.x:

You declare the directory inside of the main filebeat.yml using the config_dir option.

filebeat:
  config_dir: /etc/filebeat/conf.d

/etc/filebeat/conf.d/someapp.yml

filebeat:
  prospectors:
    - paths:
        - /var/log/someapp/stdout.log
      fields:
        app: someapp

Comments

[izrik]

Excellent. That's exactly what I was looking for. I don't know how I missed it in the docs.

[Arun Mohan]

Can I specify different outputs for different configuration files? for example, filebeat-config-1.yml must ship data to logstash but filebeat-config-2.yml must ship data to elasticsearch. Is this possible too?

[A J]

No, you cannot use multiple outputs in any Beat. Let your Logstash servers do the routing.

[Joao Vitorino]

How to do in filebeat 7.x elastic.co/guide/en/beats/filebeat/current/…