[webconnectivity]: Support user-agent strings in request headers to bypass scrapper security

[DecFox]

We observed that several e-commerce sites in Portugal had failing measurements:
https://explorer.ooni.org/domain/www.asics.com
https://explorer.ooni.org/domain/www.elcorteingles.pt

although the websites are accessible through the browser. On further inspection, we found that this is also the case with other http clients like curl and we require a minimum number of user-agent request headers in order to access the website:

curl 'https://www.elcorteingles.pt/' \
  -H 'sec-ch-ua: "Not;A=Brand";v="99", "Brave";v="139", "Chromium";v="139"' \
  -H 'sec-ch-ua-platform: "Android"' \
  -H 'user-agent: Mozilla/5.0 (Linux; Android 6.0; Nexus 5 Build/MRA58N) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/139.0.0.0 Mobile Safari/537.36'

is successful while:

curl -v 'https://www.elcorteingles.pt/' \
  -H 'sec-ch-ua: "Not;A=Brand";v="99", "Brave";v="139", "Chromium";v="139"' \
  -H 'user-agent: Mozilla/5.0 (Linux; Android 6.0; Nexus 5 Build/MRA58N) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/139.0.0.0 Mobile Safari/537.36'

errors out with:

...
* Request completely sent off
* HTTP/2 stream 1 was not closed cleanly: INTERNAL_ERROR (err 2)

To this end, we should support using user-agent strings in request headers so we can measure domains with webconnectivity more confidently

[DecFox]

@hellais pointed out that some work on the scrapper invasion strategies in the past reflected that WAFs will also match the TLS stack to the user-agent strings and consider the ordering of HTTP headers in determining what to do. The go stdlib does not support maintaining the order of request headers but we should in my opinion still support user-agent strings in the experiment as a start. These are the headers that should be used for Chrome 132:

// Updated headers for Chrome 132
	headers.Add("sec-ch-device-memory", "8")
	headers.Add("sec-ch-ua", `"Not A(Brand";v="8", "Chromium";v="132", "Google Chrome";v="132"`)
	headers.Add("sec-ch-ua-mobile", "?0")
	headers.Add("sec-ch-ua-arch", `"x86"`)
	headers.Add("sec-ch-ua-platform", `"Linux"`)
	headers.Add("sec-ch-ua-model", `""`)
	headers.Add("sec-ch-ua-full-version-list", `"Not A(Brand";v="8.0.0.0", "Chromium";v="132.0.6834.159", "Google Chrome";v="132.0.6834.159"`)
	headers.Add("dnt", "1")
	headers.Add("upgrade-insecure-requests", "1")
	headers.Add("user-agent", "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/132.0.0.0 Safari/537.36")
	headers.Add("accept", "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7")
	headers.Add("sec-fetch-site", "none")
	headers.Add("sec-fetch-mode", "navigate")
	headers.Add("sec-fetch-user", "?1")
	headers.Add("sec-fetch-dest", "document")
	headers.Add("accept-encoding", "gzip, deflate, br, zstd")
	headers.Add("accept-language", "it-IT,it;q=0.9,en-US;q=0.8,en;q=0.7,sv;q=0.6")
	headers.Add("priority", "u=0, i")