telemetry worker: flush data after stops #515
Open
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
cataphract
requested review from
pawelchcki,
bwoebi and
bantonsson
and removed request for
a team and
pawelchcki
BenchmarksComparisonBenchmark execution time: 2024-10-10 12:22:42 Comparing candidate commit 9c34541 in PR branch Found 1 performance improvements and 0 performance regressions! Performance is the same for 50 metrics, 2 unstable metrics. scenario:benching deserializing traces from msgpack to their internal representation
CandidateCandidate benchmark detailsGroup 1
Group 2
Group 3
Group 4
Group 5
Group 6
Group 7
Group 8
Group 9
Group 10
Group 11
Group 12
BaselineOmitted due to size. |
bantonsson
reviewed
Jul 4, 2024
There was a problem hiding this comment.
These booleans are woefully undocumented and I'm not completely sure about the expected life cycle, but from reading the code it looks like your interpretation is correct.
I think that it might be better to leave the worker as started if it's restartable and leave the checks in place.
| @@ -296,7 +293,9 @@ impl TelemetryWorker { | |||
| self.log_err(&e); | |||
| } | |||
| self.data.started = false; | |||
| self.deadlines.clear_pending(); | |||
| if !self.config.restartable { | |||
There was a problem hiding this comment.
Wouldn't it be enough to only include self.data.started = false; inside the if statement as well, and leave the exit early checks?
There was a problem hiding this comment.
I guess it depends. Do we want Start, Stop, Stop to generate two stops? Because that's what PHP ends up generating. The second stop is a noop, but if I moved the assignment self.data.started = false under the condition if !self.config.restartable, then the stops would be effective
There was a problem hiding this comment.
If the exit early checks are still in the code, then how would the second Stop be effective?
There was a problem hiding this comment.
From what I understand, what you're proposing is that I reset started = false only if !restartable. In that case started would stay true forever once there is a start. So the early check
if !self.data.started {
return BREAK;
}would never be hit.
There was a problem hiding this comment.
Yes, you're right. I was confusing the early checks.
There is something in the logic that feels a bit broken. The FlushData is also protected by this !self.data.started check. Should it work after a restartable stop?
The things that Stop does, should they happen when the first request ends, or when the worker stops?
There was a problem hiding this comment.
Should [FlushData] work after a restartable stop?
My guess is yes, otherwise there is no way to send the data that's collected after the stop (build_observability_batch is called only from the handlers of Stop and FlushData), at least not without an intervening start+stop.
The things that Stop does, should they happen when the first request ends, or when the worker stops?
That is a good point. The final flush of the metrics should happen when the worker stops, not when handling Stop. I guess at some point they were the same, but then the restart thing was introduced. But regardless, once we have a way to send metrics after a Stop, for that happen periodic flushes should still happen. So FlushData shouldn't be skipped or unscheduled after a Stop.
| @@ -458,7 +454,9 @@ impl TelemetryWorker { | |||
| .await; | |||
|
|
|||
| self.data.started = false; | |||
| self.deadlines.clear_pending(); | |||
| if !self.config.restartable { | |||
cataphract
force-pushed
the
glopes/flush-data-after-stop
branch
from
4f63a34
to
251bafa
Compare
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #515 +/- ##
==========================================
+ Coverage 70.10% 70.24% +0.13%
==========================================
Files 214 214
Lines 28801 28805 +4
==========================================
+ Hits 20192 20235 +43
+ Misses 8609 8570 -39
|
cataphract
force-pushed
the
glopes/flush-data-after-stop
branch
4 times, most recently
from
447c409
to
a7d11b0
Compare
Telemetry workers are functionally dead after a Stop lifecycle action, provided there's no intervening Start. While AddPoint actions are still processed, their data is never flushed, since the Stop action handler unschedules FlushMetrics and FlushData actions. PHP sends a Stop action at the end of every request via ddog_sidecar_telemetry_end(), but a Start action is only generated just after a telemetry worker is spawned. It is not clear to me whether the intention is to a Start/Stop pair on every PHP requests (where Stop flushes the metrics) or if the intention is to to have only such a pair in the first request, with the Stop event generated by ddog_sidecar_telemetry_end() effectively a noop. It would appear, judging by [this comment](#391): > Also allow the telemetry worker to have a mode where it's continuing execution after a start-stop cycle, otherwise it won't send any more metrics afterwards. that the intention is to keep sending metrics after a Start/Stop pair. In that case: * The Stop action handler should not unschedule FlushData and FlushMetrics events and * FlushData, if called outside a Start-Stop pair, should not be a noop. Finally: swap the order in which FlushData and FlushMetrics are scheduled so that FlushMetrics runs first and therefore its generated data can be sent by the next FlushData.
cataphract
force-pushed
the
glopes/flush-data-after-stop
branch
from
e0747f5
to
9c34541
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Telemetry workers are functionally dead after a Stop lifecycle action, provided there's no intervening Start. While AddPoint actions are still processed, their data is never flushed, since the Stop action handler unschedules FlushMetrics and FlushData actions.
PHP sends a Stop action at the end of every request via ddog_sidecar_telemetry_end(), but a Start action is only generated just after a telemetry worker is spawned. With no more Start actions generated, no metrics can effectively be sent after the first Stop.
It is not clear to me whether the intention is to have a Start/Stop pair on every PHP request (where Stop flushes the metrics) or if the intention is to to have only such a pair in the first request, with the Stop event generated by ddog_sidecar_telemetry_end() effectively a noop. It would appear, judging by this
comment:
that the intention is to keep sending metrics after a Start/Stop pair. It also makes more sense, insofar as data is flushed only on the interval, rather than after every request via Stop. In that case:
Finally: swap the order in which FlushData and FlushMetrics are scheduled so that FlushMetrics runs first and therefore its generated data can be sent by the next FlushData.