Document how to handle timeouts and retries

[decentral1se]

Radar is heavily rate limiting due to bot scraping. We could help API consumers with this.

#28 is a first start.

I am testing https://github.com/hashicorp/go-retryablehttp atm and have a patch:

package web

import (
	"net/http"
	"time"

	"github.com/hashicorp/go-retryablehttp"
)

func GetHttpClient() *http.Client {
	retryClient := retryablehttp.NewClient()
	retryClient.RetryMax = 10

	httpClient := retryClient.StandardClient()
	httpClient.Timeout = 30 * time.Second

	return httpClient
}

Which I use like so:

radarClient := radarPkg.NewRadarClient(
  radarPkg.WithLogger(slogHandler),
  radarPkg.WithClient(httpClient),
)

I am not sure if we should make the decision for people what client they should use but it might save work? OTOH, they'll probably want to configure the retryablehttp client also. So, I am not sure what is best. Documenting this approach could also be fine?

[goapunk]

hmm, good question, I will think about it a bit longer. In general I think documenting it with an example is definitely a good approach since it is easy enough with your pr (#28). This way we also don't need to add any dependency to this library directly. On the other hand, if you run into rate limiting regularly I guess having an http client which actually works also might be nice...

[decentral1se]

I have asked what limits / retry-after / policies are in place via the radar IRC channel. It might be nice to tweak a default to match those. Otherwise, yeh, I'm not sure. You get rate limited like almost instantly... so it is probably necessary to take some sort of action here.

I'm seeing stability with the following (it recovers/continues):

func GetHttpClient(logger *slog.Logger) *http.Client {
	retryClient := retryablehttp.NewClient()
	retryClient.RetryMax = 10
	retryClient.RetryWaitMin = 10 * time.Second
	retryClient.RetryWaitMax = 300 * time.Second
	retryClient.Logger = logger

	httpClient := retryClient.StandardClient()
	httpClient.Timeout = 300 * time.Second

	return httpClient
}

[goapunk]

We could set those as default If they work well for you! I wonder if we should add an authentication option, as that should help with the rate limiting, right?

[decentral1se]

I will ask the Radar people again on IRC for some guidance. Didn't get a response.

#31 seems like a good first step in the meantime.

[decentral1se]

I will say that practice has shown that the config in #29 (comment) is woefully slow. It will certainly not fail you but it will take you hours to get information out > 30/40 events 😅 retryClient.RetryMax = 10 does seem to be important tho.

[goapunk]

Hmm, that sounds not very feasible :/ how does your current flow look like for these events? Do you make an api call for each event?

[decentral1se]

Code is here: https://git.vvvvvvaria.org/prikbord.page/libspeciaal/src/branch/main/cli/radar.go

I end up having to make multiple requests indeed, querying directly the event and then making a request to the relevant location also. I'd love to know how to cut down the number of requests if you have any tips! 🤔

[goapunk]

I guess it depends on whether you need all fields from the location or just some, if you just need a few you could try using ResolvedFields to get the location data in one query with the event. To be honest I don't remember how well it works and whether it can only be used for search results or for direct event/entity queries

[decentral1se]

Great advice! Thanks! That did indeed speed things up massively, allowing me to retrieve everything I need in one go: https://git.vvvvvvaria.org/prikbord.page/libspeciaal/src/commit/952b400fc2c7cdcef6d0f9e135527eeeb46a1d41/cli/radar.go#L123-L139

[decentral1se]

So, just to finish the rabbit hole on this one... if you're facing rate limiting from the Radar API - a guarantee if you make many requests - then you can try the following 2 things.

1. Use a custom retry enabled http client lib like so:

func GetHttpClient() *http.Client {
	retryClient := retryablehttp.NewClient()
	retryClient.RetryMax = 10
	retryClient.Logger = log.Default()
	httpClient := retryClient.StandardClient()
	return httpClient
}

2. Use the Field machinery to make less requests, e.g. Document how to handle timeouts and retries #29 (comment)

I think this is just a documentation thing then. I'm not sure where it should end up.