The examples in this section cover the following use cases:

Java stack traces consist of multiple lines, with each line after the initial line beginning with whitespace, as in this example:

To consolidate these lines into a single event in Filebeat, use the following multiline configuration with filestream:

Using log input:

This configuration merges any line that begins with whitespace up to the previous line.

Here’s a Java stack trace that presents a slightly more complex example:

To consolidate these lines into a single event in Filebeat, use the following multiline configuration with filestream:

Using log input:

In this example, the pattern matches the following lines:

Several programming languages use the backslash (\) character at the end of a line to denote that the line continues, as in this example:

To consolidate these lines into a single event in Filebeat, use the following multiline configuration with filestream:

Using log input:

This configuration merges any line that ends with the \ character with the line that follows.

Activity logs from services such as Elasticsearch typically begin with a timestamp, followed by information on the specific activity, as in this example:

To consolidate these lines into a single event in Filebeat, use the following multiline configuration with filestream:

Using log input:

This configuration uses the negate: true and match: after settings to specify that any line that does not match the specified pattern belongs to the previous line.

Sometimes your application logs contain events, that begin and end with custom markers, such as the following example:

To consolidate this as a single event in Filebeat, use the following multiline configuration with filestream:

Using log input:

The flush_pattern option, specifies a regex at which the current multiline will be flushed. If you think of the pattern option specifying the beginning of an event, the flush_pattern option will specify the end or last line of the event.

To make it easier for you to test the regexp patterns in your multiline config, we’ve created a Go Playground. You can simply plug in the regexp pattern along with the multiline.negate setting that you plan to use, and paste a sample message between the content backticks (` `). Then click Run, and you’ll see which lines in the message match your specified configuration. For example:

Unresolved directive in filebeat-options.asciidoc - include::../../x-pack/filebeat/docs/inputs/input-aws-cloudwatch.asciidoc[]

Unresolved directive in filebeat-options.asciidoc - include::../../x-pack/filebeat/docs/inputs/input-aws-s3.asciidoc[]

Unresolved directive in filebeat-options.asciidoc - include::../../x-pack/filebeat/docs/inputs/input-azure-eventhub.asciidoc[]

Unresolved directive in filebeat-options.asciidoc - include::../../x-pack/filebeat/docs/inputs/input-azure-blob-storage.asciidoc[]

Unresolved directive in filebeat-options.asciidoc - include::../../x-pack/filebeat/docs/inputs/input-cel.asciidoc[]

Unresolved directive in filebeat-options.asciidoc - include::../../x-pack/filebeat/docs/inputs/input-cloudfoundry.asciidoc[]

Unresolved directive in filebeat-options.asciidoc - include::../../x-pack/filebeat/docs/inputs/input-cometd.asciidoc[]

The container input supports the following configuration options plus the Common options described later.

The following configuration options are supported by all inputs.

Use the enabled option to enable and disable inputs. By default, enabled is set to true.

A list of tags that Filebeat includes in the tags field of each published event. Tags make it easy to select specific events in Kibana or apply conditional filtering in Logstash. These tags will be appended to the list of tags specified in the general configuration.

Example:

Optional fields that you can specify to add additional information to the output. For example, you might add fields that you can use for filtering log data. Fields can be scalar values, arrays, dictionaries, or any nested combination of these. By default, the fields that you specify here will be grouped under a fields sub-dictionary in the output document. To store the custom fields as top-level fields, set the fields_under_root option to true. If a duplicate field is declared in the general configuration, then its value will be overwritten by the value declared here.

If this option is set to true, the custom fields are stored as top-level fields in the output document instead of being grouped under a fields sub-dictionary. If the custom field names conflict with other field names added by Filebeat, then the custom fields overwrite the other fields.

A list of processors to apply to the input data.

See Filter and enhance data with processors for information about specifying processors in your config.

The ingest pipeline ID to set for the events generated by this input.

If this option is set to true, fields with null values will be published in the output document. By default, keep_null is set to false.

If present, this formatted string overrides the index for events from this input (for elasticsearch outputs), or sets the raw_index field of the event’s metadata (for other outputs). This string can only refer to the agent name and version and the event timestamp; for access to dynamic fields, use output.elasticsearch.index or a processor.

Example value: "%{[agent.name]}-myindex-%{+yyyy.MM.dd}" might expand to "filebeat-myindex-2019.11.01".

By default, all events contain host.name. This option can be set to true to disable the addition of this field to all events. The default value is false.

Unresolved directive in filebeat-options.asciidoc - include::../../x-pack/filebeat/docs/inputs/input-entity-analytics.asciidoc[]

However, one of the limitations of these data sources can be mitigated if you configure Filebeat adequately.

By default, Filebeat identifies files based on their inodes and device IDs. However, on network shares and cloud providers these values might change during the lifetime of the file. If this happens Filebeat thinks that file is new and resends the whole content of the file. To solve this problem you can configure file_identity option. Possible values besides the default inode_deviceid are path, inode_marker and fingerprint.

Selecting path instructs Filebeat to identify files based on their paths. This is a quick way to avoid rereading files if inode and device ids might change. However, keep in mind if the files are rotated (renamed), they will be reread and resubmitted.

The option inode_marker can be used if the inodes stay the same even if the device id is changed. You should choose this method if your files are rotated instead of path if possible. You have to configure a marker file readable by Filebeat and set the path in the option path of inode_marker.

The content of this file must be unique to the device. You can put the UUID of the device or mountpoint where the input is stored. The following example oneliner generates a hidden marker file for the selected mountpoint /logs: Please note that you should not use this option on Windows as file identifiers might be more volatile.

Selecting fingerprint instructs Filebeat to identify files based on their content byte range.

Please refer to the fingerprint configuration for details.

To set the generated file as a marker for file_identity you should configure the input the following way:

When dealing with file rotation, avoid harvesting symlinks. Instead use the paths setting to point to the original file, and specify a pattern that matches the file you want to harvest and all of its rotated files. Also make sure your log rotation strategy prevents lost or duplicate messages. For more information, see Log rotation results in lost or duplicate events.

Furthermore, to avoid duplicate of rotated log messages, do not use the path method for file_identity. Or exclude the rotated files with exclude_files option.

The prospector is running a file system watcher which looks for files specified in the paths option. At the moment only simple file system scanning is supported.

A unique identifier for this filestream input. Each filestream input must have a unique ID.

A list of glob-based paths that will be crawled and fetched. All patterns supported by Go Glob are also supported here. For example, to fetch all files from a predefined level of subdirectories, the following pattern can be used: /var/log//.log. This fetches all .log files from the subfolders of /var/log. It does not fetch log files from the /var/log folder itself. It is possible to recursively fetch all files in all subdirectories of a directory using the optional recursive_glob settings.

Filebeat starts a harvester for each file that it finds under the specified paths. You can specify one path per line. Each line begins with a dash (-).

The scanner watches the configured paths. It scans the file system periodically and returns the file system events to the Prospector.

Enable expanding into recursive glob patterns. With this feature enabled, the rightmost in each path is expanded into a fixed number of glob patterns. For example: /foo/ expands to /foo, /foo/, /foo//, and so on. If enabled it expands a single into a 8-level deep pattern.

This feature is enabled by default. Set prospector.scanner.recursive_glob to false to disable it.

A list of regular expressions to match the files that you want Filebeat to ignore. By default no files are excluded.

The following example configures Filebeat to ignore all the files that have a gz extension:

See [regexp-support] for a list of supported regexp patterns.

If your rotated files have an incrementing index appended to the end of the filename, e.g. active file apache.log and the rotated files are named apache.log.1, apache.log.2, etc, use the following configuration.

If the rotation date is appended to the end of the filename, e.g. active file apache.log and the rotated files are named apache.log-20210526, apache.log-20210527, etc. use the following configuration:

The file encoding to use for reading data that contains international characters. See the encoding names recommended by the W3C for use in HTML5.

Valid encodings:

The plain encoding is special, because it does not validate or transform any input.

A list of regular expressions to match the lines that you want Filebeat to exclude. Filebeat drops any lines that match a regular expression in the list. By default, no lines are dropped. Empty lines are ignored.

The following example configures Filebeat to drop any lines that start with DBG.

See [regexp-support] for a list of supported regexp patterns.

A list of regular expressions to match the lines that you want Filebeat to include. Filebeat exports only the lines that match a regular expression in the list. By default, all lines are exported. Empty lines are ignored.

The following example configures Filebeat to export any lines that start with ERR or WARN:

The following example exports all log lines that contain sometext, except for lines that begin with DBG (debug messages):

See [regexp-support] for a list of supported regexp patterns.

The size in bytes of the buffer that each harvester uses when fetching a file. The default is 16384.

The maximum number of bytes that a single log message can have. All bytes after message_max_bytes are discarded and not sent. The default is 10MB (10485760).

This option expects a list of parsers that the log line has to go through.

Available parsers:

In this example, Filebeat is reading multiline messages that consist of 3 lines and are encapsulated in single-line JSON objects. The multiline message is stored under the key msg.

See the available parser settings in detail below.

Options that control how Filebeat deals with log messages that span multiple lines. See Manage multiline messages for more information about configuring multiline options.

These options make it possible for Filebeat to decode logs structured as JSON messages. Filebeat processes the logs line by line, so the JSON decoding only works if there is one JSON object per message.

The decoding happens before line filtering. You can combine JSON decoding with filtering if you set the message_key option. This can be helpful in situations where the application logs are wrapped in JSON objects, like when using Docker.

Example configuration:

Use the container parser to extract information from containers log files. It parses lines into common message lines, extracting timestamps too.

The following snippet configures Filebeat to read the stdout stream from all containers under the default Kubernetes logs path:

The syslog parser parses RFC 3146 and/or RFC 5424 formatted syslog messages.

The supported configuration options are:

Example configuration:

Timestamps

The RFC 3164 format accepts the following forms of timestamps:

Note: The local timestamp (for example, Jan 23 14:09:01) that accompanies an RFC 3164 message lacks year and time zone information. The time zone will be enriched using the timezone configuration option, and the year will be enriched using the Filebeat system’s local time (accounting for time zones). Because of this, it is possible for messages to appear in the future. An example of when this might happen is logs generated on December 31 2021 are ingested on January 1 2022. The logs would be enriched with the year 2022 instead of 2021.

The RFC 5424 format accepts the following forms of timestamps:

Formats with an asterisk (*) are a non-standard allowance.

Use the include_message parser to filter messages in the parsers pipeline. Messages that match the provided pattern are passed to the next parser, the others are dropped.

You should use include_message instead of include_lines if you would like to control when the filtering happens. include_lines runs after the parsers, include_message runs in the parsers pipeline.

This example shows you how to include messages that start with the string ERR or WARN:

This input exposes metrics under the HTTP monitoring endpoint. These metrics are exposed under the /inputs path. They can be used to observe the activity of the input.

Note:

The following configuration options are supported by all inputs.

Use the enabled option to enable and disable inputs. By default, enabled is set to true.

A list of tags that Filebeat includes in the tags field of each published event. Tags make it easy to select specific events in Kibana or apply conditional filtering in Logstash. These tags will be appended to the list of tags specified in the general configuration.

Example:

Optional fields that you can specify to add additional information to the output. For example, you might add fields that you can use for filtering log data. Fields can be scalar values, arrays, dictionaries, or any nested combination of these. By default, the fields that you specify here will be grouped under a fields sub-dictionary in the output document. To store the custom fields as top-level fields, set the fields_under_root option to true. If a duplicate field is declared in the general configuration, then its value will be overwritten by the value declared here.

If this option is set to true, the custom fields are stored as top-level fields in the output document instead of being grouped under a fields sub-dictionary. If the custom field names conflict with other field names added by Filebeat, then the custom fields overwrite the other fields.

A list of processors to apply to the input data.

See Filter and enhance data with processors for information about specifying processors in your config.

The ingest pipeline ID to set for the events generated by this input.

If this option is set to true, fields with null values will be published in the output document. By default, keep_null is set to false.

If present, this formatted string overrides the index for events from this input (for elasticsearch outputs), or sets the raw_index field of the event’s metadata (for other outputs). This string can only refer to the agent name and version and the event timestamp; for access to dynamic fields, use output.elasticsearch.index or a processor.

Example value: "%{[agent.name]}-myindex-%{+yyyy.MM.dd}" might expand to "filebeat-myindex-2019.11.01".

By default, all events contain host.name. This option can be set to true to disable the addition of this field to all events. The default value is false.

Unresolved directive in filebeat-options.asciidoc - include::../../x-pack/filebeat/docs/inputs/input-gcp-pubsub.asciidoc[]

Unresolved directive in filebeat-options.asciidoc - include::../../x-pack/filebeat/docs/inputs/input-http-endpoint.asciidoc[]

Unresolved directive in filebeat-options.asciidoc - include::../../x-pack/filebeat/docs/inputs/input-httpjson.asciidoc[]

The journald input supports the following configuration options plus the Common options described later.

An optional unique identifier for the input. By providing a unique id you can operate multiple inputs on the same journal. This allows each input’s cursor to be persisted independently in the registry file.

A list of paths that will be crawled and fetched. Each path can be a directory path (to collect events from all journals in a directory), or a file path. If you specify a directory, Filebeat merges all journals under the directory into a single journal and reads them.

If no paths are specified, Filebeat reads from the default journal.

The number of seconds to wait before trying to read again from journals. The default is 1s.

The maximum number of seconds to wait before attempting to read again from journals. The default is 60s.

The position to start reading the journal from. Valid settings are:

If you have old log files and want to skip lines, start Filebeat with seek: tail specified. Then stop Filebeat, set seek: cursor, and restart Filebeat.

The position to start reading the journal from if no cursor information is available. Valid options are head, tail and since.

A time offset from the current time to start reading from. To use since, either the seek option must be set to since, or the seek mode must be set to cursor and the cursor_seek_fallback set to since.

This example demonstrates how to resume from the persisted cursor when it exists, or otherwise begin reading logs from the last 24 hours.

Iterate only the entries of the units specified in this option. The iterated entries include messages from the units, messages about the units by authorized daemons and coredumps. However, it does not match systemd user units.

Read only the entries with the selected syslog identifiers.

Collect the messages using the specified transports. Example: syslog.

Valid transports:

A collection of filter expressions used to match fields. The format of the expression is field=value. Filebeat fetches all events that exactly match the expressions. Pattern matching is not supported.

If you configured a filter expression, only entries with this field set will be iterated by the journald reader of Filebeat. If the filter expressions apply to different fields, only entries with all fields set will be iterated. If they apply to the same fields, only entries where the field takes one of the specified values will be iterated.

match: List of filter expressions to match fields. or: The filter expressions listed under or are connected with a disjunction (or). and: The filter expressions listed under and are connected with a conjunction (and).

Please note that these expressions are limited. You can build complex filtering, but full logical expressions are not supported.

The following include matches configuration reads all systemd syslog entries:

To reference fields, use one of the following:

You can use the following translated names in filter expressions to reference journald fields:

The following translated fields for Docker are also available:

The following configuration options are supported by all inputs.

Use the enabled option to enable and disable inputs. By default, enabled is set to true.

A list of tags that Filebeat includes in the tags field of each published event. Tags make it easy to select specific events in Kibana or apply conditional filtering in Logstash. These tags will be appended to the list of tags specified in the general configuration.

Example:

Optional fields that you can specify to add additional information to the output. For example, you might add fields that you can use for filtering log data. Fields can be scalar values, arrays, dictionaries, or any nested combination of these. By default, the fields that you specify here will be grouped under a fields sub-dictionary in the output document. To store the custom fields as top-level fields, set the fields_under_root option to true. If a duplicate field is declared in the general configuration, then its value will be overwritten by the value declared here.

If this option is set to true, the custom fields are stored as top-level fields in the output document instead of being grouped under a fields sub-dictionary. If the custom field names conflict with other field names added by Filebeat, then the custom fields overwrite the other fields.

A list of processors to apply to the input data.

See Filter and enhance data with processors for information about specifying processors in your config.

The ingest pipeline ID to set for the events generated by this input.

If this option is set to true, fields with null values will be published in the output document. By default, keep_null is set to false.

If present, this formatted string overrides the index for events from this input (for elasticsearch outputs), or sets the raw_index field of the event’s metadata (for other outputs). This string can only refer to the agent name and version and the event timestamp; for access to dynamic fields, use output.elasticsearch.index or a processor.

Example value: "%{[agent.name]}-myindex-%{+yyyy.MM.dd}" might expand to "filebeat-myindex-2019.11.01".

By default, all events contain host.name. This option can be set to true to disable the addition of this field to all events. The default value is false.

This input works with all Kafka versions in between 0.11 and 2.8.0. Older versions might work as well, but are not supported.

The kafka input supports the following configuration options plus the Common options described later.

A list of Kafka bootstrapping hosts (brokers) for this cluster.

A list of topics to read from.

The Kafka consumer group id.

The Kafka client id (optional).

The version of the Kafka protocol to use (defaults to "1.0.0").

The initial offset to start reading, either "oldest" or "newest". Defaults to "oldest".

The following configuration options are supported by all inputs.

Use the enabled option to enable and disable inputs. By default, enabled is set to true.

A list of tags that Filebeat includes in the tags field of each published event. Tags make it easy to select specific events in Kibana or apply conditional filtering in Logstash. These tags will be appended to the list of tags specified in the general configuration.

Example:

Optional fields that you can specify to add additional information to the output. For example, you might add fields that you can use for filtering log data. Fields can be scalar values, arrays, dictionaries, or any nested combination of these. By default, the fields that you specify here will be grouped under a fields sub-dictionary in the output document. To store the custom fields as top-level fields, set the fields_under_root option to true. If a duplicate field is declared in the general configuration, then its value will be overwritten by the value declared here.

If this option is set to true, the custom fields are stored as top-level fields in the output document instead of being grouped under a fields sub-dictionary. If the custom field names conflict with other field names added by Filebeat, then the custom fields overwrite the other fields.

A list of processors to apply to the input data.

See Filter and enhance data with processors for information about specifying processors in your config.

The ingest pipeline ID to set for the events generated by this input.

If this option is set to true, fields with null values will be published in the output document. By default, keep_null is set to false.

If present, this formatted string overrides the index for events from this input (for elasticsearch outputs), or sets the raw_index field of the event’s metadata (for other outputs). This string can only refer to the agent name and version and the event timestamp; for access to dynamic fields, use output.elasticsearch.index or a processor.

Example value: "%{[agent.name]}-myindex-%{+yyyy.MM.dd}" might expand to "filebeat-myindex-2019.11.01".

By default, all events contain host.name. This option can be set to true to disable the addition of this field to all events. The default value is false.

However, one of the limitations of these data sources can be mitigated if you configure Filebeat adequately.

By default, Filebeat identifies files based on their inodes and device IDs. However, on network shares and cloud providers these values might change during the lifetime of the file. If this happens Filebeat thinks that file is new and resends the whole content of the file. To solve this problem you can configure file_identity option. Possible values besides the default inode_deviceid are path and inode_marker.

Selecting path instructs Filebeat to identify files based on their paths. This is a quick way to avoid rereading files if inode and device ids might change. However, keep in mind if the files are rotated (renamed), they will be reread and resubmitted.

The option inode_marker can be used if the inodes stay the same even if the device id is changed. You should choose this method if your files are rotated instead of path if possible. You have to configure a marker file readable by Filebeat and set the path in the option path of inode_marker.

The content of this file must be unique to the device. You can put the UUID of the device or mountpoint where the input is stored. The following example oneliner generates a hidden marker file for the selected mountpoint /logs: Please note that you should not use this option on Windows as file identifiers might be more volatile.

To set the generated file as a marker for file_identity you should configure the input the following way:

When dealing with file rotation, avoid harvesting symlinks. Instead use the paths setting to point to the original file, and specify a pattern that matches the file you want to harvest and all of its rotated files. Also make sure your log rotation strategy prevents lost or duplicate messages. For more information, see Log rotation results in lost or duplicate events.

Furthermore, to avoid duplicate of rotated log messages, do not use the path method for file_identity. Or exclude the rotated files with exclude_files option.

The log input supports the following configuration options plus the Common options described later.

A list of glob-based paths that will be crawled and fetched. All patterns supported by Go Glob are also supported here. For example, to fetch all files from a predefined level of subdirectories, the following pattern can be used: /var/log//.log. This fetches all .log files from the subfolders of /var/log. It does not fetch log files from the /var/log folder itself. It is possible to recursively fetch all files in all subdirectories of a directory using the optional recursive_glob settings.

Filebeat starts a harvester for each file that it finds under the specified paths. You can specify one path per line. Each line begins with a dash (-).

Enable expanding into recursive glob patterns. With this feature enabled, the rightmost in each path is expanded into a fixed number of glob patterns. For example: /foo/ expands to /foo, /foo/, /foo//, and so on. If enabled it expands a single into a 8-level deep pattern.

This feature is enabled by default. Set recursive_glob.enabled to false to disable it.

The file encoding to use for reading data that contains international characters. See the encoding names recommended by the W3C for use in HTML5.

Valid encodings:

The plain encoding is special, because it does not validate or transform any input.

A list of regular expressions to match the lines that you want Filebeat to exclude. Filebeat drops any lines that match a regular expression in the list. By default, no lines are dropped. Empty lines are ignored.

If multiline settings are also specified, each multiline message is combined into a single line before the lines are filtered by exclude_lines.

The following example configures Filebeat to drop any lines that start with DBG.

See [regexp-support] for a list of supported regexp patterns.

A list of regular expressions to match the lines that you want Filebeat to include. Filebeat exports only the lines that match a regular expression in the list. By default, all lines are exported. Empty lines are ignored.

If multiline settings also specified, each multiline message is combined into a single line before the lines are filtered by include_lines.

The following example configures Filebeat to export any lines that start with ERR or WARN:

The following example exports all log lines that contain sometext, except for lines that begin with DBG (debug messages):

See [regexp-support] for a list of supported regexp patterns.

The size in bytes of the buffer that each harvester uses when fetching a file. The default is 16384.

The maximum number of bytes that a single log message can have. All bytes after max_bytes are discarded and not sent. This setting is especially useful for multiline log messages, which can get large. The default is 10MB (10485760).

These options make it possible for Filebeat to decode logs structured as JSON messages. Filebeat processes the logs line by line, so the JSON decoding only works if there is one JSON object per line.

The decoding happens before line filtering and multiline. You can combine JSON decoding with filtering and multiline if you set the message_key option. This can be helpful in situations where the application logs are wrapped in JSON objects, as with like it happens for example with Docker.

Example configuration:

You must specify at least one of the following settings to enable JSON parsing mode:

Options that control how Filebeat deals with log messages that span multiple lines. See Manage multiline messages for more information about configuring multiline options.

A list of regular expressions to match the files that you want Filebeat to ignore. By default no files are excluded.

The following example configures Filebeat to ignore all the files that have a gz extension:

See [regexp-support] for a list of supported regexp patterns.

If this option is enabled, Filebeat ignores any files that were modified before the specified timespan. Configuring ignore_older can be especially useful if you keep log files for a long time. For example, if you want to start Filebeat, but only want to send the newest files and files from last week, you can configure this option.

You can use time strings like 2h (2 hours) and 5m (5 minutes). The default is 0, which disables the setting. Commenting out the config has the same effect as setting it to 0.

The files affected by this setting fall into two categories:

For files which were never seen before, the offset state is set to the end of the file. If a state already exist, the offset is not changed. In case a file is updated again later, reading continues at the set offset position.

The ignore_older setting relies on the modification time of the file to determine if a file is ignored. If the modification time of the file is not updated when lines are written to a file (which can happen on Windows), the ignore_older setting may cause Filebeat to ignore files even though content was added at a later time.

To remove the state of previously harvested files from the registry file, use the clean_inactive configuration option.

Before a file can be ignored by Filebeat, the file must be closed. To ensure a file is no longer being harvested when it is ignored, you must set ignore_older to a longer duration than close_inactive.

If a file that’s currently being harvested falls under ignore_older, the harvester will first finish reading the file and close it after close_inactive is reached. Then, after that, the file will be ignored.

The close_* configuration options are used to close the harvester after a certain criteria or time. Closing the harvester means closing the file handler. If a file is updated after the harvester is closed, the file will be picked up again after scan_frequency has elapsed. However, if the file is moved or deleted while the harvester is closed, Filebeat will not be able to pick up the file again, and any data that the harvester hasn’t read will be lost. The close_* settings are applied synchronously when Filebeat attempts to read from a file, meaning that if Filebeat is in a blocked state due to blocked output, full queue or other issue, a file that would otherwise be closed remains open until Filebeat once again attempts to read from the file.

When this option is enabled, Filebeat closes the file handle if a file has not been harvested for the specified duration. The counter for the defined period starts when the last log line was read by the harvester. It is not based on the modification time of the file. If the closed file changes again, a new harvester is started and the latest changes will be picked up after scan_frequency has elapsed.

We recommended that you set close_inactive to a value that is larger than the least frequent updates to your log files. For example, if your log files get updated every few seconds, you can safely set close_inactive to 1m. If there are log files with very different update rates, you can use multiple configurations with different values.

Setting close_inactive to a lower value means that file handles are closed sooner. However this has the side effect that new log lines are not sent in near real time if the harvester is closed.

The timestamp for closing a file does not depend on the modification time of the file. Instead, Filebeat uses an internal timestamp that reflects when the file was last harvested. For example, if close_inactive is set to 5 minutes, the countdown for the 5 minutes starts after the harvester reads the last line of the file.

You can use time strings like 2h (2 hours) and 5m (5 minutes). The default is 5m.

When this option is enabled, Filebeat closes the file handler when a file is renamed. This happens, for example, when rotating files. By default, the harvester stays open and keeps reading the file because the file handler does not depend on the file name. If the close_renamed option is enabled and the file is renamed or moved in such a way that it’s no longer matched by the file patterns specified for the path, the file will not be picked up again. Filebeat will not finish reading the file.

Do not use this option when path based file_identity is configured. It does not make sense to enable the option, as Filebeat cannot detect renames using path names as unique identifiers.

WINDOWS: If your Windows log rotation system shows errors because it can’t rotate the files, you should enable this option.

When this option is enabled, Filebeat closes the harvester when a file is removed. Normally a file should only be removed after it’s inactive for the duration specified by close_inactive. However, if a file is removed early and you don’t enable close_removed, Filebeat keeps the file open to make sure the harvester has completed. If this setting results in files that are not completely read because they are removed from disk too early, disable this option.

This option is enabled by default. If you disable this option, you must also disable clean_removed.

WINDOWS: If your Windows log rotation system shows errors because it can’t rotate files, make sure this option is enabled.

When this option is enabled, Filebeat closes a file as soon as the end of a file is reached. This is useful when your files are only written once and not updated from time to time. For example, this happens when you are writing every single log event to a new file. This option is disabled by default.

When this option is enabled, Filebeat gives every harvester a predefined lifetime. Regardless of where the reader is in the file, reading will stop after the close_timeout period has elapsed. This option can be useful for older log files when you want to spend only a predefined amount of time on the files. While close_timeout will close the file after the predefined timeout, if the file is still being updated, Filebeat will start a new harvester again per the defined scan_frequency. And the close_timeout for this harvester will start again with the countdown for the timeout.

This option is particularly useful in case the output is blocked, which makes Filebeat keep open file handlers even for files that were deleted from the disk. Setting close_timeout to 5m ensures that the files are periodically closed so they can be freed up by the operating system.

If you set close_timeout to equal ignore_older, the file will not be picked up if it’s modified while the harvester is closed. This combination of settings normally leads to data loss, and the complete file is not sent.

When you use close_timeout for logs that contain multiline events, the harvester might stop in the middle of a multiline event, which means that only parts of the event will be sent. If the harvester is started again and the file still exists, only the second part of the event will be sent.

This option is set to 0 by default which means it is disabled.

The clean_* options are used to clean up the state entries in the registry file. These settings help to reduce the size of the registry file and can prevent a potential inode reuse issue.

When this option is enabled, Filebeat removes the state of a file after the specified period of inactivity has elapsed. The state can only be removed if the file is already ignored by Filebeat (the file is older than ignore_older). The clean_inactive setting must be greater than ignore_older
scan_frequency to make sure that no states are removed while a file is still being harvested. Otherwise, the setting could result in Filebeat resending the full content constantly because clean_inactive removes state for files that are still detected by Filebeat. If a file is updated or appears again, the file is read from the beginning.

The clean_inactive configuration option is useful to reduce the size of the registry file, especially if a large amount of new files are generated every day.

This config option is also useful to prevent Filebeat problems resulting from inode reuse on Linux. For more information, see Inode reuse causes Filebeat to skip lines.

When this option is enabled, Filebeat cleans files from the registry if they cannot be found on disk anymore under the last known name. This means also files which were renamed after the harvester was finished will be removed. This option is enabled by default.

If a shared drive disappears for a short period and appears again, all files will be read again from the beginning because the states were removed from the registry file. In such cases, we recommend that you disable the clean_removed option.

You must disable this option if you also disable close_removed.

How often Filebeat checks for new files in the paths that are specified for harvesting. For example, if you specify a glob like /var/log/*, the directory is scanned for files using the frequency specified by scan_frequency. Specify 1s to scan the directory as frequently as possible without causing Filebeat to scan too frequently. We do not recommend to set this value <1s.

If you require log lines to be sent in near real time do not use a very low scan_frequency but adjust close_inactive so the file handler stays open and constantly polls your files.

The default setting is 10s.

experimental[]

If you specify a value other than the empty string for this setting you can determine whether to use ascending or descending order using scan.order. Possible values are modtime and filename. To sort by file modification time, use modtime, otherwise use filename. Leave this option empty to disable it.

If you specify a value for this setting, you can use scan.order to configure whether files are scanned in ascending or descending order.

The default setting is disabled.

experimental[]

Specifies whether to use ascending or descending order when scan.sort is set to a value other than none. Possible values are asc or desc.

The default setting is asc.

If this option is set to true, Filebeat starts reading new files at the end of each file instead of the beginning. When this option is used in combination with log rotation, it’s possible that the first log entries in a new file might be skipped. The default setting is false.

This option applies to files that Filebeat has not already processed. If you ran Filebeat previously and the state of the file was already persisted, tail_files will not apply. Harvesting will continue at the previous offset. To apply tail_files to all files, you must stop Filebeat and remove the registry file. Be aware that doing this removes ALL previous states.

The symlinks option allows Filebeat to harvest symlinks in addition to regular files. When harvesting symlinks, Filebeat opens and reads the original file even though it reports the path of the symlink.

When you configure a symlink for harvesting, make sure the original path is excluded. If a single input is configured to harvest both the symlink and the original file, Filebeat will detect the problem and only process the first file it finds. However, if two different inputs are configured (one to read the symlink and the other the original path), both paths will be harvested, causing Filebeat to send duplicate data and the inputs to overwrite each other’s state.

The symlinks option can be useful if symlinks to the log files have additional metadata in the file name, and you want to process the metadata in Logstash. This is, for example, the case for Kubernetes log files.

Because this option may lead to data loss, it is disabled by default.

The backoff options specify how aggressively Filebeat crawls open files for updates. You can use the default values in most cases.

The backoff option defines how long Filebeat waits before checking a file again after EOF is reached. The default is 1s, which means the file is checked every second if new lines were added. This enables near real-time crawling. Every time a new line appears in the file, the backoff value is reset to the initial value. The default is 1s.

The maximum time for Filebeat to wait before checking a file again after EOF is reached. After having backed off multiple times from checking the file, the wait time will never exceed max_backoff regardless of what is specified for backoff_factor. Because it takes a maximum of 10s to read a new line, specifying 10s for max_backoff means that, at the worst, a new line could be added to the log file if Filebeat has backed off multiple times. The default is 10s.

Requirement: Set max_backoff to be greater than or equal to backoff and less than or equal to scan_frequency (backoff ⇐ max_backoff ⇐ scan_frequency). If max_backoff needs to be higher, it is recommended to close the file handler instead and let Filebeat pick up the file again.

This option specifies how fast the waiting time is increased. The bigger the backoff factor, the faster the max_backoff value is reached. The backoff factor increments exponentially. The minimum value allowed is 1. If this value is set to 1, the backoff algorithm is disabled, and the backoff value is used for waiting for new lines. The backoff value will be multiplied each time with the backoff_factor until max_backoff is reached. The default is 2.

The harvester_limit option limits the number of harvesters that are started in parallel for one input. This directly relates to the maximum number of file handlers that are opened. The default for harvester_limit is 0, which means there is no limit. This configuration is useful if the number of files to be harvested exceeds the open file handler limit of the operating system.

Setting a limit on the number of harvesters means that potentially not all files are opened in parallel. Therefore we recommended that you use this option in combination with the close_* options to make sure harvesters are stopped more often so that new files can be picked up.

Currently if a new harvester can be started again, the harvester is picked randomly. This means it’s possible that the harvester for a file that was just closed and then updated again might be started instead of the harvester for a file that hasn’t been harvested for a longer period of time.

This configuration option applies per input. You can use this option to indirectly set higher priorities on certain inputs by assigning a higher limit of harvesters.

Different file_identity methods can be configured to suit the environment where you are collecting log messages.

Set the location of the marker file the following way:

The following configuration options are supported by all inputs.

Use the enabled option to enable and disable inputs. By default, enabled is set to true.

A list of tags that Filebeat includes in the tags field of each published event. Tags make it easy to select specific events in Kibana or apply conditional filtering in Logstash. These tags will be appended to the list of tags specified in the general configuration.

Example:

Optional fields that you can specify to add additional information to the output. For example, you might add fields that you can use for filtering log data. Fields can be scalar values, arrays, dictionaries, or any nested combination of these. By default, the fields that you specify here will be grouped under a fields sub-dictionary in the output document. To store the custom fields as top-level fields, set the fields_under_root option to true. If a duplicate field is declared in the general configuration, then its value will be overwritten by the value declared here.

If this option is set to true, the custom fields are stored as top-level fields in the output document instead of being grouped under a fields sub-dictionary. If the custom field names conflict with other field names added by Filebeat, then the custom fields overwrite the other fields.

A list of processors to apply to the input data.

See Filter and enhance data with processors for information about specifying processors in your config.

The ingest pipeline ID to set for the events generated by this input.

If this option is set to true, fields with null values will be published in the output document. By default, keep_null is set to false.

If present, this formatted string overrides the index for events from this input (for elasticsearch outputs), or sets the raw_index field of the event’s metadata (for other outputs). This string can only refer to the agent name and version and the event timestamp; for access to dynamic fields, use output.elasticsearch.index or a processor.

Example value: "%{[agent.name]}-myindex-%{+yyyy.MM.dd}" might expand to "filebeat-myindex-2019.11.01".

By default, all events contain host.name. This option can be set to true to disable the addition of this field to all events. The default value is false.

The mqtt input supports the following configuration options plus the Common options described later.

The following configuration options are supported by all inputs.

Use the enabled option to enable and disable inputs. By default, enabled is set to true.

A list of tags that Filebeat includes in the tags field of each published event. Tags make it easy to select specific events in Kibana or apply conditional filtering in Logstash. These tags will be appended to the list of tags specified in the general configuration.

Example:

Optional fields that you can specify to add additional information to the output. For example, you might add fields that you can use for filtering log data. Fields can be scalar values, arrays, dictionaries, or any nested combination of these. By default, the fields that you specify here will be grouped under a fields sub-dictionary in the output document. To store the custom fields as top-level fields, set the fields_under_root option to true. If a duplicate field is declared in the general configuration, then its value will be overwritten by the value declared here.

If this option is set to true, the custom fields are stored as top-level fields in the output document instead of being grouped under a fields sub-dictionary. If the custom field names conflict with other field names added by Filebeat, then the custom fields overwrite the other fields.

A list of processors to apply to the input data.

See Filter and enhance data with processors for information about specifying processors in your config.

The ingest pipeline ID to set for the events generated by this input.

If this option is set to true, fields with null values will be published in the output document. By default, keep_null is set to false.

If present, this formatted string overrides the index for events from this input (for elasticsearch outputs), or sets the raw_index field of the event’s metadata (for other outputs). This string can only refer to the agent name and version and the event timestamp; for access to dynamic fields, use output.elasticsearch.index or a processor.

Example value: "%{[agent.name]}-myindex-%{+yyyy.MM.dd}" might expand to "filebeat-myindex-2019.11.01".

By default, all events contain host.name. This option can be set to true to disable the addition of this field to all events. The default value is false.

Unresolved directive in filebeat-options.asciidoc - include::../../x-pack/filebeat/docs/inputs/input-netflow.asciidoc[]

Unresolved directive in filebeat-options.asciidoc - include::../../x-pack/filebeat/docs/inputs/input-o365audit.asciidoc[]

The redis input supports the following configuration options plus the Common options described later.

The list of Redis hosts to connect to.

The password to use when connecting to Redis.

How often Filebeat reads entries from Redis slowlogs. Specify 1s to scan Redis as frequently as possible without causing Filebeat to scan too frequently. Do not set this value to less than 1s.

The default is 10s.

How long to wait for a response from Redis before the input returns an error. The default is 1s.

The network type to use for the Redis connection. Valid settings include: tcp, tcp4, tcp6, and unix. The default is tcp.

The maximum number of concurrent connections. The default is 10.

The following configuration options are supported by all inputs.

Use the enabled option to enable and disable inputs. By default, enabled is set to true.

A list of tags that Filebeat includes in the tags field of each published event. Tags make it easy to select specific events in Kibana or apply conditional filtering in Logstash. These tags will be appended to the list of tags specified in the general configuration.

Example:

Optional fields that you can specify to add additional information to the output. For example, you might add fields that you can use for filtering log data. Fields can be scalar values, arrays, dictionaries, or any nested combination of these. By default, the fields that you specify here will be grouped under a fields sub-dictionary in the output document. To store the custom fields as top-level fields, set the fields_under_root option to true. If a duplicate field is declared in the general configuration, then its value will be overwritten by the value declared here.

If this option is set to true, the custom fields are stored as top-level fields in the output document instead of being grouped under a fields sub-dictionary. If the custom field names conflict with other field names added by Filebeat, then the custom fields overwrite the other fields.

A list of processors to apply to the input data.

See Filter and enhance data with processors for information about specifying processors in your config.

The ingest pipeline ID to set for the events generated by this input.

If this option is set to true, fields with null values will be published in the output document. By default, keep_null is set to false.

If present, this formatted string overrides the index for events from this input (for elasticsearch outputs), or sets the raw_index field of the event’s metadata (for other outputs). This string can only refer to the agent name and version and the event timestamp; for access to dynamic fields, use output.elasticsearch.index or a processor.

Example value: "%{[agent.name]}-myindex-%{+yyyy.MM.dd}" might expand to "filebeat-myindex-2019.11.01".

By default, all events contain host.name. This option can be set to true to disable the addition of this field to all events. The default value is false.

The stdin input supports the following configuration options plus the Common options described later.

The file encoding to use for reading data that contains international characters. See the encoding names recommended by the W3C for use in HTML5.

Valid encodings:

The plain encoding is special, because it does not validate or transform any input.

A list of regular expressions to match the lines that you want Filebeat to exclude. Filebeat drops any lines that match a regular expression in the list. By default, no lines are dropped. Empty lines are ignored.

If multiline settings are also specified, each multiline message is combined into a single line before the lines are filtered by exclude_lines.

The following example configures Filebeat to drop any lines that start with DBG.

See [regexp-support] for a list of supported regexp patterns.

A list of regular expressions to match the lines that you want Filebeat to include. Filebeat exports only the lines that match a regular expression in the list. By default, all lines are exported. Empty lines are ignored.

If multiline settings also specified, each multiline message is combined into a single line before the lines are filtered by include_lines.

The following example configures Filebeat to export any lines that start with ERR or WARN:

The following example exports all log lines that contain sometext, except for lines that begin with DBG (debug messages):

See [regexp-support] for a list of supported regexp patterns.

The size in bytes of the buffer that each harvester uses when fetching a file. The default is 16384.

The maximum number of bytes that a single log message can have. All bytes after max_bytes are discarded and not sent. This setting is especially useful for multiline log messages, which can get large. The default is 10MB (10485760).

These options make it possible for Filebeat to decode logs structured as JSON messages. Filebeat processes the logs line by line, so the JSON decoding only works if there is one JSON object per line.

The decoding happens before line filtering and multiline. You can combine JSON decoding with filtering and multiline if you set the message_key option. This can be helpful in situations where the application logs are wrapped in JSON objects, as with like it happens for example with Docker.

Example configuration:

You must specify at least one of the following settings to enable JSON parsing mode:

Options that control how Filebeat deals with log messages that span multiple lines. See Manage multiline messages for more information about configuring multiline options.

The following configuration options are supported by all inputs.

Use the enabled option to enable and disable inputs. By default, enabled is set to true.

A list of tags that Filebeat includes in the tags field of each published event. Tags make it easy to select specific events in Kibana or apply conditional filtering in Logstash. These tags will be appended to the list of tags specified in the general configuration.

Example:

Optional fields that you can specify to add additional information to the output. For example, you might add fields that you can use for filtering log data. Fields can be scalar values, arrays, dictionaries, or any nested combination of these. By default, the fields that you specify here will be grouped under a fields sub-dictionary in the output document. To store the custom fields as top-level fields, set the fields_under_root option to true. If a duplicate field is declared in the general configuration, then its value will be overwritten by the value declared here.

If this option is set to true, the custom fields are stored as top-level fields in the output document instead of being grouped under a fields sub-dictionary. If the custom field names conflict with other field names added by Filebeat, then the custom fields overwrite the other fields.

A list of processors to apply to the input data.

See Filter and enhance data with processors for information about specifying processors in your config.

The ingest pipeline ID to set for the events generated by this input.

If this option is set to true, fields with null values will be published in the output document. By default, keep_null is set to false.

If present, this formatted string overrides the index for events from this input (for elasticsearch outputs), or sets the raw_index field of the event’s metadata (for other outputs). This string can only refer to the agent name and version and the event timestamp; for access to dynamic fields, use output.elasticsearch.index or a processor.

Example value: "%{[agent.name]}-myindex-%{+yyyy.MM.dd}" might expand to "filebeat-myindex-2019.11.01".

By default, all events contain host.name. This option can be set to true to disable the addition of this field to all events. The default value is false.

The syslog input configuration includes format, protocol specific options, and the Common options described later.

The following configuration options are supported by all inputs.

Use the enabled option to enable and disable inputs. By default, enabled is set to true.

A list of tags that Filebeat includes in the tags field of each published event. Tags make it easy to select specific events in Kibana or apply conditional filtering in Logstash. These tags will be appended to the list of tags specified in the general configuration.

Example:

Optional fields that you can specify to add additional information to the output. For example, you might add fields that you can use for filtering log data. Fields can be scalar values, arrays, dictionaries, or any nested combination of these. By default, the fields that you specify here will be grouped under a fields sub-dictionary in the output document. To store the custom fields as top-level fields, set the fields_under_root option to true. If a duplicate field is declared in the general configuration, then its value will be overwritten by the value declared here.

If this option is set to true, the custom fields are stored as top-level fields in the output document instead of being grouped under a fields sub-dictionary. If the custom field names conflict with other field names added by Filebeat, then the custom fields overwrite the other fields.

A list of processors to apply to the input data.

See Filter and enhance data with processors for information about specifying processors in your config.

The ingest pipeline ID to set for the events generated by this input.

If this option is set to true, fields with null values will be published in the output document. By default, keep_null is set to false.

If present, this formatted string overrides the index for events from this input (for elasticsearch outputs), or sets the raw_index field of the event’s metadata (for other outputs). This string can only refer to the agent name and version and the event timestamp; for access to dynamic fields, use output.elasticsearch.index or a processor.

Example value: "%{[agent.name]}-myindex-%{+yyyy.MM.dd}" might expand to "filebeat-myindex-2019.11.01".

By default, all events contain host.name. This option can be set to true to disable the addition of this field to all events. The default value is false.

The tcp input supports the following configuration options plus the Common options described later.

The maximum size of the message received over TCP. The default is 20MiB.

The host and TCP port to listen on for event streams.

Specify the framing used to split incoming events. Can be one of delimiter or rfc6587. delimiter uses the characters specified in line_delimiter to split the incoming events. rfc6587 supports octet counting and non-transparent framing as described in RFC6587. line_delimiter is used to split the events in non-transparent framing. The default is delimiter.

Specify the characters used to split the incoming events. The default is '\n'.

The at most number of connections to accept at any given point in time.

The number of seconds of inactivity before a remote connection is closed. The default is 300s.

Configuration options for SSL parameters like the certificate, key and the certificate authorities to use.

See [configuration-ssl] for more information.

This input exposes metrics under the HTTP monitoring endpoint. These metrics are exposed under the /inputs path. They can be used to observe the activity of the input.

The following configuration options are supported by all inputs.

Use the enabled option to enable and disable inputs. By default, enabled is set to true.

A list of tags that Filebeat includes in the tags field of each published event. Tags make it easy to select specific events in Kibana or apply conditional filtering in Logstash. These tags will be appended to the list of tags specified in the general configuration.

Example:

Optional fields that you can specify to add additional information to the output. For example, you might add fields that you can use for filtering log data. Fields can be scalar values, arrays, dictionaries, or any nested combination of these. By default, the fields that you specify here will be grouped under a fields sub-dictionary in the output document. To store the custom fields as top-level fields, set the fields_under_root option to true. If a duplicate field is declared in the general configuration, then its value will be overwritten by the value declared here.

If this option is set to true, the custom fields are stored as top-level fields in the output document instead of being grouped under a fields sub-dictionary. If the custom field names conflict with other field names added by Filebeat, then the custom fields overwrite the other fields.

A list of processors to apply to the input data.

See Filter and enhance data with processors for information about specifying processors in your config.

The ingest pipeline ID to set for the events generated by this input.

If this option is set to true, fields with null values will be published in the output document. By default, keep_null is set to false.

If present, this formatted string overrides the index for events from this input (for elasticsearch outputs), or sets the raw_index field of the event’s metadata (for other outputs). This string can only refer to the agent name and version and the event timestamp; for access to dynamic fields, use output.elasticsearch.index or a processor.

Example value: "%{[agent.name]}-myindex-%{+yyyy.MM.dd}" might expand to "filebeat-myindex-2019.11.01".

By default, all events contain host.name. This option can be set to true to disable the addition of this field to all events. The default value is false.

The udp input supports the following configuration options plus the Common options described later.

The maximum size of the message received over UDP. The default is 10KiB.

The host and UDP port to listen on for event streams.

The size of the read buffer on the UDP socket. If not specified the default from the operating system will be used.

The read and write timeout for socket operations. The default is 5m.

This input exposes metrics under the HTTP monitoring endpoint. These metrics are exposed under the /inputs path. They can be used to observe the activity of the input.

The following configuration options are supported by all inputs.

Use the enabled option to enable and disable inputs. By default, enabled is set to true.

A list of tags that Filebeat includes in the tags field of each published event. Tags make it easy to select specific events in Kibana or apply conditional filtering in Logstash. These tags will be appended to the list of tags specified in the general configuration.

Example:

Optional fields that you can specify to add additional information to the output. For example, you might add fields that you can use for filtering log data. Fields can be scalar values, arrays, dictionaries, or any nested combination of these. By default, the fields that you specify here will be grouped under a fields sub-dictionary in the output document. To store the custom fields as top-level fields, set the fields_under_root option to true. If a duplicate field is declared in the general configuration, then its value will be overwritten by the value declared here.

If this option is set to true, the custom fields are stored as top-level fields in the output document instead of being grouped under a fields sub-dictionary. If the custom field names conflict with other field names added by Filebeat, then the custom fields overwrite the other fields.

A list of processors to apply to the input data.

See Filter and enhance data with processors for information about specifying processors in your config.

The ingest pipeline ID to set for the events generated by this input.

If this option is set to true, fields with null values will be published in the output document. By default, keep_null is set to false.

If present, this formatted string overrides the index for events from this input (for elasticsearch outputs), or sets the raw_index field of the event’s metadata (for other outputs). This string can only refer to the agent name and version and the event timestamp; for access to dynamic fields, use output.elasticsearch.index or a processor.

Example value: "%{[agent.name]}-myindex-%{+yyyy.MM.dd}" might expand to "filebeat-myindex-2019.11.01".

By default, all events contain host.name. This option can be set to true to disable the addition of this field to all events. The default value is false.

The unix input supports the following configuration options plus the Common options described later.

The maximum size of the message received over the socket. The default is 20MiB.

The path to the Unix socket that will receive events.

The type to of the Unix socket that will receive events. Valid values are stream and datagram. The default is stream.

The group ownership of the Unix socket that will be created by Filebeat. The default is the primary group name for the user Filebeat is running as. This option is ignored on Windows.

The file mode of the Unix socket that will be created by Filebeat. This is expected to be a file mode as an octal string. The default value is the system default (generally 0755).

Specify the framing used to split incoming events. Can be one of delimiter or rfc6587. delimiter uses the characters specified in line_delimiter to split the incoming events. rfc6587 supports octet counting and non-transparent framing as described in RFC6587. line_delimiter is used to split the events in non-transparent framing. The default is delimiter.

Specify the characters used to split the incoming events. The default is '\n'.

The at most number of connections to accept at any given point in time.

The number of seconds of inactivity before a connection is closed. The default is 300s.

See [configuration-ssl] for more information.

This input exposes metrics under the HTTP monitoring endpoint. These metrics are exposed under the /inputs path. They can be used to observe the activity of the input.

The following configuration options are supported by all inputs.

Use the enabled option to enable and disable inputs. By default, enabled is set to true.

A list of tags that Filebeat includes in the tags field of each published event. Tags make it easy to select specific events in Kibana or apply conditional filtering in Logstash. These tags will be appended to the list of tags specified in the general configuration.

Example:

Optional fields that you can specify to add additional information to the output. For example, you might add fields that you can use for filtering log data. Fields can be scalar values, arrays, dictionaries, or any nested combination of these. By default, the fields that you specify here will be grouped under a fields sub-dictionary in the output document. To store the custom fields as top-level fields, set the fields_under_root option to true. If a duplicate field is declared in the general configuration, then its value will be overwritten by the value declared here.

If this option is set to true, the custom fields are stored as top-level fields in the output document instead of being grouped under a fields sub-dictionary. If the custom field names conflict with other field names added by Filebeat, then the custom fields overwrite the other fields.

A list of processors to apply to the input data.

See Filter and enhance data with processors for information about specifying processors in your config.

The ingest pipeline ID to set for the events generated by this input.

If this option is set to true, fields with null values will be published in the output document. By default, keep_null is set to false.

If present, this formatted string overrides the index for events from this input (for elasticsearch outputs), or sets the raw_index field of the event’s metadata (for other outputs). This string can only refer to the agent name and version and the event timestamp; for access to dynamic fields, use output.elasticsearch.index or a processor.

Example value: "%{[agent.name]}-myindex-%{+yyyy.MM.dd}" might expand to "filebeat-myindex-2019.11.01".

By default, all events contain host.name. This option can be set to true to disable the addition of this field to all events. The default value is false.

Unresolved directive in filebeat-options.asciidoc - include::../../x-pack/filebeat/docs/inputs/input-gcs.asciidoc[]

Unresolved directive in configuring-howto.asciidoc - include::/github/workspace/../docs/filebeat-modules-options.asciidoc[]