-
Notifications
You must be signed in to change notification settings - Fork 1.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
record_transformer remove_keys not working as expected #2109
Comments
You can use |
I also meet the bug ,to simplify the step to reproduce
the second filter try to remove the key 'foo.bar'
according to record_accessor syntax , they should be same Syntax
This is simple syntax. For example,
Useful for special characters, ., and etc: $['dot.key'][0]['space key'] for record["dot.key"][0]["space key"] |
@grpubr How about using |
remove_keys $['foo']['bar'] means that the nested key should be removed. it works as design eg.
however, we are trying to remove key containing dot, which does not work as expected eg.
they are not same. |
I asked @okkez to investigate that |
In previous version, following configuration does not work properly: ``` <source> @type dummy dummy [ {"foo.bar": "test1234", "message": "Hello"} ] tag dummy </source> <filter dummy> @type record_transformer remove_keys "$['foo.bar']" </filter> <match dummy> @type stdout </match> ``` This shows like following: ``` 2018-11-27 15:19:18 +0900 [info]: #0 fluentd worker is now running worker=0 2018-11-27 15:19:19.008586045 +0900 dummy: {"foo.bar":"test1234","message":"Hello"} 2018-11-27 15:19:20.009721132 +0900 dummy: {"foo.bar":"test1234","message":"Hello"} 2018-11-27 15:19:21.010784035 +0900 dummy: {"foo.bar":"test1234","message":"Hello"} ``` In this version, it works well. See also fluent#2109 Signed-off-by: Kenji Okimoto <okimoto@clear-code.com>
@grpubr @aleks-mariusz Try #2192 please. |
Released v1.3.1. |
I have a pipeline set up to get kubernetes based log messages into elasticsearch using fluentd (which originate from docker containers using the systemd logging driver, most examples online have the separate json files in /var/log/containers which docker seems to have moved away from recently).
The messages arrive in elasticsearch fine, but along the path the log takes, i am trying to get fluentd to delete some keys that are excessive and unnecessary. After using field_map in the systemd_entry block, I am using the record_transformer's remove_keys option inside a block, however certain keys do not get deleted and i'm wondering if this is a bug or am i just using this functionality incorrectly.
fluentd version: 1.2.4 running inside docker (1.13.1) container deployed by kubernetes (1.11.2):
the Gemfile used to install Fluentd
the Dockerfile used to create the container:
The configuration question:
And finally, what the entry looks like arriving in elasticsearch:
Basically, my goal is to get rid of the system.argv and system.process-name keys, this is attempt in the next-to-last block (right before the <match **> above).
The text was updated successfully, but these errors were encountered: