I'm trying to run Elasticsearch API requests in a Python Lambda I've defined and deployed into AWS but I've been running into some issues with an API 503 error and the message is "the backend service does not exist".
The code and requests work locally when I use port forwarding against the service to localhost and I can query the service here either from a command terminal or via my Jupyter Notebook.
For additional information, the Elastic service is on a load balancer in a Kubernetes cluster in the same AWS account and my connection is via an encoded API key I've generated.
My suspicions are this is a vpc/subnet/security groups issue or an IAM role issue although the code does not fail when I create the Elasticsearch object for the connection. It fails when it calls things like es_client.security.get_user() or .perform_request() but works with es_client.info(), so I'm not 100% sure what needs changing, adding or removing or what I've missed.
Additional information:
The VPC is the one associated with the EKS cluster.
The Security group is open to the port I'm trying to connect to in the inbound rule (443).
Full error being returned is:
"errorMessage": "ApiError(503, 'Backend service does not exist')",
"errorType": "ApiError",
"requestId": "22568c22-7c9c-4452-a69e-f681f9e55a2d",
"stackTrace": [
" File \"/var/task/app.py\", line 91, in lambda_handler\n resp = es_client.security.get_user()\n",
" File \"/opt/python/lib/python3.12/site-packages/elasticsearch/_sync/client/utils.py\", line 455, in wrapped\n return api(*args, **kwargs)\n",
" File \"/opt/python/lib/python3.12/site-packages/elasticsearch/_sync/client/security.py\", line 1791, in get_user\n return self.perform_request( # type: ignore[return-value]\n",
" File \"/opt/python/lib/python3.12/site-packages/elasticsearch/_sync/client/_base.py\", line 423, in perform_request\n return self._client.perform_request(\n",
" File \"/opt/python/lib/python3.12/site-packages/elasticsearch/_sync/client/_base.py\", line 271, in perform_request\n response = self._perform_request(\n",
" File \"/opt/python/lib/python3.12/site-packages/elasticsearch/_sync/client/_base.py\", line 352, in _perform_request\n raise HTTP_EXCEPTIONS.get(meta.status, ApiError)(\n"
]
}
I'm trying to run Elasticsearch API requests in a Python Lambda I've defined and deployed into AWS but I've been running into some issues with an API 503 error and the message is "the backend service does not exist".
The code and requests work locally when I use port forwarding against the service to localhost and I can query the service here either from a command terminal or via my Jupyter Notebook.
For additional information, the Elastic service is on a load balancer in a Kubernetes cluster in the same AWS account and my connection is via an encoded API key I've generated.
My suspicions are this is a vpc/subnet/security groups issue or an IAM role issue although the code does not fail when I create the Elasticsearch object for the connection. It fails when it calls things like es_client.security.get_user() or .perform_request() but works with es_client.info(), so I'm not 100% sure what needs changing, adding or removing or what I've missed.
Additional information:
The VPC is the one associated with the EKS cluster.
The Security group is open to the port I'm trying to connect to in the inbound rule (443).
Full error being returned is:
"errorMessage": "ApiError(503, 'Backend service does not exist')",
"errorType": "ApiError",
"requestId": "22568c22-7c9c-4452-a69e-f681f9e55a2d",
"stackTrace": [
" File \"/var/task/app.py\", line 91, in lambda_handler\n resp = es_client.security.get_user()\n",
" File \"/opt/python/lib/python3.12/site-packages/elasticsearch/_sync/client/utils.py\", line 455, in wrapped\n return api(*args, **kwargs)\n",
" File \"/opt/python/lib/python3.12/site-packages/elasticsearch/_sync/client/security.py\", line 1791, in get_user\n return self.perform_request( # type: ignore[return-value]\n",
" File \"/opt/python/lib/python3.12/site-packages/elasticsearch/_sync/client/_base.py\", line 423, in perform_request\n return self._client.perform_request(\n",
" File \"/opt/python/lib/python3.12/site-packages/elasticsearch/_sync/client/_base.py\", line 271, in perform_request\n response = self._perform_request(\n",
" File \"/opt/python/lib/python3.12/site-packages/elasticsearch/_sync/client/_base.py\", line 352, in _perform_request\n raise HTTP_EXCEPTIONS.get(meta.status, ApiError)(\n"
]
}
- 1 It doesn't sound like the IAM role would have any relation to the issue here. Since you are running Elasticsearch in Kubernetes, you would not be using IAM authentication to connect to Elasticsearch. It is most likely a network connection issue, possibly due to the security groups or something. You need to edit the question to actually provide the information about how the Lambda is configured to run in the VPC, and the security group rules on the Elasticsearch instance, etc. Also including the full error message would be appropriate. – Mark B Commented Jan 30 at 16:49
- Understand on the IAM. I did have some IAM issues previously with ec2 permissions before running into this error which is why I mentioned it. I have added some additional information on the error and security group detail. The vpc is an existing one that I have attached to the lambda but I'm not 100% sure on what needs changing on this or if I create a new one from scratch, what the settings need to be. – bardsleyta Commented Jan 30 at 17:15
1 Answer
Reset to default 0Having done a bit more digging , I appear to have found the solution. The default port number was incorrect which is why the error was appearing and after changing it to the same port number I used locally, it worked. I would also suggest for others that encounter is to double check that your lambda is configured to the correct vpc.