I have a critical FastAPI Python app that has one get endpoint /abc which gets certain data. When user requests this endpoint, it connects to the Microsoft SQL server to retrieve some info. The SQL Server sits on another VM, and in case if that VM is down, it should return some basic info instead of specific user_data.

Now problem is. when that VM is down, docker container starts flipping up & down, basic health endpoint is not reachable (as a whole app). Although I am catching exceptions, I cannot understand why it is happening the way it does. Prometheus shows container is down.

Below is just a high level code snippet (as it sits on company's PC, cannot get it)

from sqlalchemy import create_engine
@router.get(/abc)
async def serve_abc(request: Request):
    try:
        ip = request.client.host
        user_data = get_user_data(ip)   <---- **here I make a request to a db**
        return user_data
    
    except (HTTPException, Exception) as e:
        logger.error("an error occurred")
        return basic_data
         
def get_user_data(ip: str):
    sql_query="....."
    
    try:
        connection_string=f"mssql+pymssql://(creds)...."
        engine = create_engine(connection_string)
        with engine.connect() as conn:
             df = pd.read_sql(sql_query, conn)
             return df.to_dict(orient='records)
    except pymssql.OperationalError as e:       <-------this error happens when VM is down
        logger.error(blablabla)
        raise                  <--------- I have tried just returning [] as well

    except Exception as e:
        logger.error(blablalba)
        raise

There is a global exception handler as well.

Container is unreachable, load balancer is flipping up and down status all the time.