http response status code 404 when using go version >= 1.17

[lianato3]

I have the following code sample:

package main

import (
	"fmt"
	"net/http"

	"github.com/valyala/fasthttp"
)

func main() {
	req := fasthttp.AcquireRequest()
	defer fasthttp.ReleaseRequest(req)

	req.SetRequestURI("https://ads-api.twitter.com/10/conversion_event")
	req.Header.SetMethod(http.MethodPost)

	resp := fasthttp.AcquireResponse()
	defer fasthttp.ReleaseResponse(resp)

	client := &fasthttp.Client{}

	_ = client.Do(req, resp)

	fmt.Println(resp.StatusCode())
}

fasthttp version is v1.36.0

when I ran the code with go 1.15/ 1.16 I'm getting 401 status code as expected.

But when I ran it with go 1.17/1.18 I'm am getting 404, I'm not sure where is the issue here, whether its in golang or fasthttp? will appreciate your help please

[erikdubbelboer]

fasthttp doesn't do anything specific based on the Go version. But nothing in the Go version should make the response change as it would result in exactly the same request going out. I'm also getting 404 with curl. Why do you expect it to return 401?

% curl -v 'https://ads-api.twitter.com/10/conversion_event'
*   Trying 104.244.42.3:443...
* Connected to ads-api.twitter.com (104.244.42.3) port 443 (#0)
...
> GET /10/conversion_event HTTP/1.1
> Host: ads-api.twitter.com
> User-Agent: curl/7.79.1
> Accept: */*
> 

< HTTP/1.1 404 Not Found
< date: Tue, 10 May 2022 15:49:53 GMT
< server: tsa_o
< set-cookie: guest_id=v1%3A165219779441443467; Max-Age=34214400; Expires=Sat, 10 Jun 2023 15:49:54 GMT; Path=/; Domain=.twitter.com; Secure; SameSite=None
< content-type: application/json; charset=utf-8
< cache-control: no-cache, no-store, max-age=0
< content-length: 143
< timing-allow-origin: https://twitter.com, https://mobile.twitter.com
< strict-transport-security: max-age=631138519
< x-response-time: 121
< x-connection-hash: d822072afd507850d79e0495178bfa42f7165d61d68a1d4a3fedff188ea93115
< 
* Connection #0 to host ads-api.twitter.com left intact
{"errors":[{"code":"ROUTE_NOT_FOUND","message":"The requested resource could not be found"}],"request":{"params":{"*":"10\/conversion_event"}}

[lianato3]

it should be used with POST method , and when running curl -X POST -v 'https://ads-api.twitter.com/10/conversion_event' getting 401 as well

[erikdubbelboer]

They seem to have some kind of "spam" detection. Even if you add this:

req.Header.SetUserAgent("curl/7.79.1")
req.Header.Set("Accept", "*/*")
req.Header.SetContentType("application/x-www-form-urlencoded")

Which makes fasthttp send:

POST /10/conversion_event HTTP/1.1
Host: ads-api.twitter.com
User-Agent: curl/7.79.1
Accept: */*
Content-Length: 0
Content-Type: application/x-www-form-urlencoded

You will get a 404. Even though curl sends the same request (just the order of the headers is different) and gets a 401 with this:

curl -v -X POST --http1.1 -d '' 'https://ads-api.twitter.com/10/conversion_event'
> POST /10/conversion_event HTTP/1.1
> Host: ads-api.twitter.com
> User-Agent: curl/7.79.1
> Accept: */*
> Content-Length: 0
> Content-Type: application/x-www-form-urlencoded

Nothing you can do about this.

[lianato3]

The issue is that I'm not really expecting to get 401 instead of 404, but I upgraded our prod environment to use go 1.18 and all those requests to twitter started to return 404, where we expect 200.

And when I ran some local tests, it seems to consistently failing with 404 using go 1.18/ 1.17 and work correctly with go 1.15/1.16

[erikdubbelboer]

That could be the same "spam" detection on twitters end. The only thing I can imagine changing between those Go versions is is maybe some TCP timing or settings. Have you tried downgrading again to see if it works again? Was the Go version really the only thing you changed?

[lianato3]

Yes, the only thing that changed is go version :)
I'm running the go piece of code all day with different versions and keep getting the same response status 404 only with 1.18/ 1.17

when I downgrade the version its working again :)

[erikdubbelboer]

I'm afraid all I can say is that fasthttp does nothing different based on the Go version.

[lianato3]

when using net/http with go 1.18 it returns 401 :

resp2, _ := http.PostForm("https://ads-api.twitter.com/10/conversion_event", url.Values{})
fmt.Println(resp2.StatusCode)

perhaps it can give a direction, whats the main difference between net/https vs fasthttp in terms of using the golang core lib?

[erikdubbelboer]

The main difference is the order of headers and some different values of headers. But they are all valid requests which again shows me that it's some twitter spam filter.

[lianato3]

After debugging it more, the 404 happens when ClientSessionCache is not nil in the tls.Config:

&tls.Config{
    ClientSessionCache: tls.NewLRUClientSessionCache(0),
})

However I don't have enough understanding of how the ClientSessionCache changes the request in the tls later, this only happens in go >= 17 versions.

Any idea/direction of what may it be? I've also tried setting the ClientSessionCache with fakeSessionCache like cache and same 404 result.

[erikdubbelboer]

It could be that something in the session cache changed in Go >= 17, I'm not aware of any changes. It would be weird if this has any impact on the request to twitter, but I guess it could somehow trigger their spam filter which looks at things like tls properties.

I don't know how ClientSessionCache is used internally in the tls library but I can imagine that even setting it to a fakeSessionCache will still trigger the tls properties send to twitter to be different.

If this really fixes it then it's probably best to fork fasthttp for your use and remove this line:

fasthttp/client.go

Line 1904 in b40b5a4

c.ClientSessionCache = tls.NewLRUClientSessionCache(0)

Of course the best solution would be for Twitter to fix this issue, but I don't see that happening easily.