This code is part of an object storage, users send requests to proxies, which then instruct users on which specific target holds or is responsible for retrieving an object. Occasionally, these target nodes can become unavailable due to rolling deployments or machine issues, leading to connection resets or connection errors until the proxies detect this failure and reroute requests to available targets.

I'm trying to implement an intelligent retry mechanism such that, upon encountering connection resets or errors from the target, subsequent retries would be redirected to the proxies or load balancers (LB) instead of repeatedly trying the same unavailable target.

Here's what I initially attempted using urllib3's Retry mechanism:

class ProxyAwareRetry(Retry):
def __init__(self, client_url, \*\*kwargs):
    super().__init__(\*\*kwargs)
    self.client_url = client_url
    self.parsed_client_url = urlparse(client_url)
    self.proxy_pool_manager = PoolManager()

    def new(self, **kwargs):
        return ProxyAwareRetry(client_url=self.client_url, **kwargs)
    
    def increment(
        self,
        method=None,
        url=None,
        response=None,
        error=None,
        _pool=None,
        _stacktrace=None,
    ):
        # check for conn error or resets
        if self._is_connection_error(error) or self._is_read_error(error):
            # Option 1 (problematic): Directly changing pool host/port affects future requests.
            # _pool.host = self.parsed_client_url.hostname
            # _pool.port = self.parsed_client_url.port
            # _pool.scheme = self.parsed_client_url.scheme
    
            # Option 2 (expensive and ineffective): Create a new connection pool.
            _pool = self.proxy_pool_manager.connection_from_url(self.client_url)
    
        return super().increment(
            method=method,
            url=url,
            response=response,
            error=error,
            _pool=_pool,
            _stacktrace=_stacktrace,
        )

Option 1: Modifying _pool.host/port/scheme directly caused all future requests to use the proxy (undesirable side effects).

Option 2: Creating a new pool didn’t seem effective, and also seems expensive.Note: The URL param is just the path, I want to change the host and port or underlying retry.

Is there a recommended approach or best practice to achieve this retry behavior—redirecting retries specifically after connection errors back through proxies, using either urllib3 or requests?