in_tail: Ensure to discard TailWatcher with missing target when follow_inodes #4239
Conversation
I think this is just a list order problem. I will improve the test. |
|
It seems that log duplication and handle leak still can occur on macOS... https://github.com/fluent/fluentd/actions/runs/5539804708/jobs/10111141849?pr=4239 |
It seems that this can happen only on macOS. |
It can happen on Windows and macOS because StatWatcher does not work on those platforms (specification). |
daipom
force-pushed
the
in_tail-ensure-discard-watcher-with-missing-target
branch
2 times, most recently
from
9e812a5
to
5a4c9b3
Compare
|
The above problems are fixed. |
daipom
force-pushed
the
in_tail-ensure-discard-watcher-with-missing-target
branch
2 times, most recently
from
0c13c9f
to
7ff9dd3
Compare
|
Rebased to the latest master. |
daipom
force-pushed
the
in_tail-ensure-discard-watcher-with-missing-target
branch
from
7ff9dd3
to
f87ab7b
Compare
…w_inodes For example, when a rotation process is slow, there is a small time lag between moving and adding files. There is a possibility that StatWatcher notifies too quickly to the TailWatcher before the new file is moved to that target path. From the TailWatcher, it appears as if the file is resurrected once it disappeared. In this case, `refresh_watcher` can't recognize it, so TailWatcher needs to discard self correctly. In the previous implementation, it was not done. So it caused the handle leak and log duplication. (Please check the added test-case) Signed-off-by: Daijiro Fukuda <fukuda@clear-code.com>
daipom
force-pushed
the
in_tail-ensure-discard-watcher-with-missing-target
branch
from
f87ab7b
to
781e313
Compare
|
Thanks! |
|
Thanks for your review! |
Which issue(s) this PR fixes:
Not related to a particular issue, but it can have something to do with reported log duplication or resource leak phenomenon about
in_tail.What this PR does / why we need it:
Ensure to discard TailWatcher with missing target when follow_inodes.
For example, when a rotation process is slow, there is a small time lag between moving and adding files.
There is a possibility that StatWatcher notifies too quickly to the TailWatcher before the new file is moved to that target path. From the TailWatcher, it appears as if the file is resurrected once it disappeared.
In this case,
refresh_watchercan't recognize it, so TailWatcher needs to discard self correctly.In the previous implementation, it was not done.
So it caused the handle leak and log duplication.
(Please check the added test case)
Because of the problem #4237 (comment), log duplication can still occur during
rotate_wait.We need to set
rotate_wait 0for a workaround.(Duplication will occur if
refresh_watcheris called during therotate_wait. We need the mechanism not to add duplicated TailWathcer with detaching TailWatcher.)Docs Changes:
Not needed.
Release Note:
Same as the title.
How to Reproduce:
rerfresh_watcherTWICEtest.log(new inode)@tails[test.log]so old inode information is lost from@tailsat this timingtest.log1(old inode)test.log1test.log1, log duplication occurs.TailWatcher(test.log, old inode)(orphan) (This actually opens the file handle oftest.log1)TailWatcher(test.log1, old inode)When StatWatcher (or TimeTrigger) notifies between B and C, then the TailWatcher becomes an orphan, and a handle leak happens.