Feature Request: Discuss Retry-After header in Generic Request

MCroft

MCroft

Developer
Staff member
Looking at Generic Request, I see we don't have any throttling logic. I know this can cause failures for scripts that are checking versions via GitHub APIs (such as garbo).

The straightforward fix for them is to just fail if they get rate limited, but I wonder if there are more robust options we could build out, and if so what they should do.

This is a discussion and not a feature I'm going to put together yet; I'm not even sure what the "right" behavior is.

GitHub (and AWS and other services implementing rate limits) implement the RFC-6585 Retry-After header.

We could honor that, but what should we do if a request is rate limited?

My simplest thought is that if response header Retry-After is present and not null, then add "\nRate Limited Request: Retry-After: <value>" to the Response Text, so that the client can parse and deal with it (is it 5 seconds? 50? 3600?).

Can anyone think of ways this might break things? I want to do the least possible change and still be useful.

I also don't know if KoL supplies 503/429 messages, since that mechanism is designed to cover cases of scheduled maintenance with a known end-period. Which sounds like nightly maintenance, but I'd have to see the headers of maint.php to see if they're sending that header or what the values might be.
 
H

heeheehee

Developer
Staff member
> 503/429 messages, since that mechanism is designed to cover cases of scheduled maintenance with a known end-period

Incidentally, that's not how I view either of those status codes. 429 ("Too many requests") serves a purpose directly related to Retry-After in terms of throttling, and it seems that 503 was used in this way (along with many others, including scheduled maintenance) before 429 was standardized in that same RFC.

For internal usage, e.g. updating scripts via github, I don't see any reason why we shouldn't add internal retry logic. For scripts, it's more complicated because of script expectations. I think most scripts would prefer delays over errors (which will likely be swallowed silently since usually if you call visit_url, its return value will be captured for inspection). And I'd be surprised if many scripts consistently defensively check that visit_url(target) != "". But if the script is intended to be interactive, it may be useful to actually propagate the error and/or let the user decide how it should be handled.

There is some precedent here -- we will transparently follow redirects.

One of the limitations of visit_url is in its function signature: we don't have a principled way of passing options (currently we use function overloading for that purpose), and its return value is just a string representing the body (without a good way of returning errors -- to be clear: I really do not want us to introduce a script-visible equivalent of `errno`). I don't think anyone wants another overload of visit_url that takes a dozen more parameters, either.

(I think it may be interesting to explore an options map or record for built-in functions like this that can take many parameters.)
 
You must log in or register to reply here.
Top