I am using Sonnet 3.5 via AWS Bedrock and facing the dreaded 'Throttling Exception' every single time.
Here are the CloudWatch log from my latest run.
In Summary: The first call works: It uses up close to 1800 tokens. All subsequent calls fail with 'Throttling Exception'.
I have read that the limit for Sonnet 3.5 is 400,000 tokens / minute. I am nowhere near breaching that.
First Call ( This one works)
{
"schemaType": "ModelInvocationLog",
"schemaVersion": "1.0",
"timestamp": "2025-02-07T08:11:59Z",
"accountId": "821052193763",
"identity": {
"arn": "arn:aws:iam::821052193763:user/aws_demos"
},
"region": "us-east-1",
"requestId": "ea303aba-9b49-4d9e-bd25-4d839878a1f8",
"operation": "ConverseStream",
"modelId": "anthropic.claude-3-5-sonnet-20240620-v1:0",
"input": {
"inputContentType": "application/json",
"inputBodyJson": {
"messages": [
{
"role": "user",
"content": [
{
"text": "Use all the tools at your disposal to find the right answer for the query below: \n\nWhat is the favorability rating on work-life balance and flexibility for Company1 in the year 2022??\n\n\n (reminder to respond in a JSON blob no matter what)"
}
]
}
],
"system": [
{
"text": "Respond to the human as helpfully and accurately as possible. You have access to the following tools:\n\nVectorStore(*args: Any, callbacks: Union[List[langchain_core.callbacks.base.BaseCallbackHandler], langchain_core.callbacks.base.BaseCallbackManager, NoneType] = None, tags: Optional[List[str]] = None, metadata: Optional[Dict[str, Any]] = None, **kwargs: Any) -> Any - Useful for searching information from the knowledge base. This should be given the first priority while searching for information. It has information about Company1, Company2 and Company3's ESG reports in detail. If the information is not found, then the database tools must be used to find the answer, args: {'tool_input': {'type': 'string'}}\nsql_db_query - Input to this tool is a detailed and correct SQL query, output is a result from the database. If the query is not correct, an error message will be returned. If an error is returned, rewrite the query, check the query, and try again. If you encounter an issue with Unknown column 'xxxx' in 'field list', use sql_db_schema to query the correct table fields., args: {'query': {'title': 'Query', 'description': 'A detailed and correct SQL query.', 'type': 'string'}}\nsql_db_schema - Input to this tool is a comma-separated list of tables, output is the schema and sample rows for those tables. Be sure that the tables actually exist by calling sql_db_list_tables first! Example Input: table1, table2, table3, args: {'table_names': {'title': 'Table Names', 'description': \"A comma-separated list of the table names for which to return the schema. Example input: 'table1, table2, table3'\", 'type': 'string'}}\nsql_db_list_tables - Input is an empty string, output is a comma-separated list of tables in the database., args: {'tool_input': {'title': 'Tool Input', 'description': 'An empty string', 'default': '', 'type': 'string'}}\nsql_db_query_checker - Use this tool to double check if your query is correct before executing it. Always use this tool before executing a query with sql_db_query!, args: {'query': {'title': 'Query', 'description': 'A detailed and SQL query to be checked.', 'type': 'string'}}\n\nUse a json blob to specify a tool by providing an action key (tool name) and an action_input key (tool input).\n\nValid \"action\" values: \"Final Answer\" or VectorStore, sql_db_query, sql_db_schema, sql_db_list_tables, sql_db_query_checker\n\nProvide only ONE action per $JSON_BLOB, as shown:\n\n```\n{\n \"action\": $TOOL_NAME,\n \"action_input\": $INPUT\n}\n```\n\nFollow this format:\n\nQuestion: input question to answer\nThought: consider previous and subsequent steps\nAction:\n```\n$JSON_BLOB\n```\nObservation: action result\n... (repeat Thought/Action/Observation N times)\nThought: I know what to respond\nAction:\n```\n{\n \"action\": \"Final Answer\",\n \"action_input\": \"Final response to human\"\n}\n\nBegin! Reminder to ALWAYS respond with a valid json blob of a single action. Use tools if necessary. Respond directly if appropriate. Format is Action:```$JSON_BLOB```then Observation"
}
],
"inferenceConfig": {
"temperature": 0,
"stopSequences": [
"\nObservation"
]
},
"toolConfig": {
"tools": [
{
"toolSpec": {
"name": "VectorStore",
"description": "Useful for searching information from the knowledge base. This should be given the first priority while searching for information. It has information about Company1, Company2 and Company3's ESG reports in detail. If the information is not found, then the database tools must be used to find the answer",
"inputSchema": {
"json": {
"type": "object",
"properties": {
"__arg1": {
"title": "__arg1",
"type": "string"
}
},
"required": [
"__arg1"
]
}
}
}
},
{
"toolSpec": {
"name": "sql_db_query",
"description": "Input to this tool is a detailed and correct SQL query, output is a result from the database. If the query is not correct, an error message will be returned. If an error is returned, rewrite the query, check the query, and try again. If you encounter an issue with Unknown column 'xxxx' in 'field list', use sql_db_schema to query the correct table fields.",
"inputSchema": {
"json": {
"type": "object",
"properties": {
"query": {
"type": "string",
"description": "A detailed and correct SQL query."
}
},
"required": [
"query"
]
}
}
}
},
{
"toolSpec": {
"name": "sql_db_schema",
"description": "Input to this tool is a comma-separated list of tables, output is the schema and sample rows for those tables. Be sure that the tables actually exist by calling sql_db_list_tables first! Example Input: table1, table2, table3",
"inputSchema": {
"json": {
"type": "object",
"properties": {
"table_names": {
"type": "string",
"description": "A comma-separated list of the table names for which to return the schema. Example input: 'table1, table2, table3'"
}
},
"required": [
"table_names"
]
}
}
}
},
{
"toolSpec": {
"name": "sql_db_list_tables",
"description": "Input is an empty string, output is a comma-separated list of tables in the database.",
"inputSchema": {
"json": {
"type": "object",
"properties": {
"tool_input": {
"type": "string",
"description": "An empty string",
"default": ""
}
}
}
}
}
},
{
"toolSpec": {
"name": "sql_db_query_checker",
"description": "Use this tool to double check if your query is correct before executing it. Always use this tool before executing a query with sql_db_query!",
"inputSchema": {
"json": {
"type": "object",
"properties": {
"query": {
"type": "string",
"description": "A detailed and SQL query to be checked."
}
},
"required": [
"query"
]
}
}
}
}
]
}
},
"inputTokenCount": 1760
},
"output": {
"outputContentType": "application/json",
"outputBodyJson": {
"output": {
"message": {
"role": "assistant",
"content": [
{
"text": "Certainly! I'll use the tools at my disposal to find the answer to your query about Company1's favorability rating on work-life balance and flexibility for the year 2022. Let's start by searching the knowledge base using the VectorStore tool.\n\nAction:\n```\n{\n \"action\": \"VectorStore\",\n \"action_input\": \"Company1 favorability rating work-life balance flexibility 2022\"\n}\n```"
}
]
}
},
"stopReason": "stop_sequence",
"metrics": {
"latencyMs": 3125
},
"usage": {
"inputTokens": 1760,
"outputTokens": 101,
"totalTokens": 1861
}
},
"outputTokenCount": 101
}
}
Subsequent call
{
"schemaType": "ModelInvocationLog",
"schemaVersion": "1.0",
"timestamp": "2025-02-07T08:12:04Z",
"accountId": "821052193763",
"identity": {
"arn": "arn:aws:iam::821052193763:user/aws_demos"
},
"region": "us-east-1",
"requestId": "fc1f2dfb-6e4f-46f4-af97-d122fd6afae1",
"operation": "Converse",
"modelId": "anthropic.claude-3-5-sonnet-20240620-v1:0",
"errorCode": "ThrottlingException"
}