最新消息:雨落星辰是一个专注网站SEO优化、网站SEO诊断、搜索引擎研究、网络营销推广、网站策划运营及站长类的自媒体原创博客

python - Twitter(X) Filtered Stream (v2 API) - "TooManyConnections" - Stack Overflow

programmeradmin4浏览0评论

I have an intermittent issue when streaming the Twitter Filtered Stream endpoint.

I would say once a day, at different times I am hit with the response -

{
  "title": "ConnectionException",
  "detail": "This stream is currently at the maximum allowed connection limit.",
  "connection_issue": "TooManyConnections",
  "type": ";
}

I use the endpoint to stream filtered Tweets based on my published rules, and I believe (according to Twitters somewhat unreliable documentation) that this endpoint is not rate limited based on how many Tweets you download, rather how many times you reconnect to the stream.

Therein lies my puzzlement. The script connecting is the only one I have, i.e. no other scripts in dev or staging. Below is how I connect -

while True:
    try:
        response = requests.get(url, headers=headers, stream=True, timeout=(10, None))

        if response.status_code == 200:
            # Reset retry count on successful connection
            retry_count = 0

            for line in response.iter_lines():
                if line:
                    try:
                        tweet_data = json.loads(line.decode('utf-8'))
                        insert_tweet(tweet_data)
                    except json.JSONDecodeError as e:
                        logging.error(f"JSON Decode Error: {e}")

        elif response.status_code == # Here I handle other response code accordingly.

If I do hit an issue in the response, I use an exponential backoff strategy (jitter) so as to not smash the reconnects -

def backoff_with_jitter(retry_count):
    base_backoff = min(INITIAL_BACKOFF * (2 ** retry_count), MAX_BACKOFF)
    jitter = random.uniform(0, base_backoff)
    time.sleep(jitter)

I also log every time there is a reconnection, and this happens rarely. My question is why am I getting the above error response when, 1. only this script is connecting and 2. the reconnections are sparse?

In my research I did come across this thread on the official forum and this SO question, unfortunately it looks like I am not the only one to experience this. Note: I am using python which I believe handles garbage collection i.e. closes the response before reconnect, and I am on the PRO license for the api access.

发布评论

评论列表(0)

  1. 暂无评论