I have a GCP Load Balancer connecting to GCP CloudRun which is running an express app. The express app has basic authentication and the CloudRun service only allows calls from the Load Balancer.

However, I am seeing a lot of spam requests like a lot of requests from servers all over the world hitting routes like /.env or /manifest.xml etc.

This looks like scraper activities or malicious.

Since I have implemented a GCP load balancer, I would like to prevent these requests at the load balancer level.

I have thought about allowing only certain routes, but that does not necessarily solve the problem.

I have also thought about blacklisting user agents, but that is not a good approach either as it requires me to keep a list of user agents, and most of them are from Chrome or Safari or even Go-http-client.

Any suggestions on how to do this effectively?