Enhance http-kafka idempotency key

[jfallows]

When idempotency-key http header is specified by the client, the intention is to detect and prevent duplication of work for repeated requests.

Therefore, we check the reply-to topic for existing responses, correlated by the value of idempotency-key in zilla:correlation-id header.

However, it is possible that the same idempotency-key could be used with a different http request, making the correlated response inappropriate for that distinct request.

Additionally, we need to capture the md5 hash of the request payload and include it in both the request message sent to the request topic and in the filter on the reply-to topic for the correlated response.

This md5 hash can be appended to the http idempotency-key request header value and sent in the zilla:correlation-id header, so the requirements on the kafka service remain unchanged, i.e. copy the opaque value of zilla:correlation-id header on the kafka request message to a corresponding zilla:correlation-id header on the kafka response message.

Then if the same request is repeated, the correlated response will be sent back to the client immediately and the kafka service can safely detect and ignore the duplicate request due to the already processed response.

However, if a different request is sent using the same idempotency-key http header, the md5 hash will differ and the zilla:correlation-id will therefore also differ, causing the new request to be processed by the kafka service as desired.

[akrambek]

Instead, we need to capture the md5 hash of the request payload and include it in both the request message sent to the request topic and in the filter on the reply-to topic for the correlated response.

On the response side of the message, do we take care of that in zilla or it should be done by producer of the message?

[jfallows]

@akrambek yes, zilla should take care of that automatically, so the http client just sends a regular idempotency-key header, and zilla constructs the zilla:correlation-id header by incorporating the md5 hash of the http request payload and appending it to value of idempotency-key http header.

Given that kafka encodes headers after the message payload, we should be able to compute the md5 hash of the request payload on the fly, and include it as a header in the kafka extension, at the end of the DATA frame with fin bit set.

Inside the kafka consumer + producer, the zilla:correlation-id header is copied verbatim from request message to response message to provide the correlated response at zilla.

[jfallows]

@akrambek btw, using this approach, i think we might be able to safely ignore requests with duplicate zilla:correlation-id values at the service because the response has already been produced to kafka and is being used by zilla as the correlated response.

More precisely, we would potentially be able to ignore requests with zilla:correlation-id that match existing responses, making the service response a no-op. That logic should help with the case where the correlated response has expired from the responses topic even when the duplicated request has not yet expired.