I am trying to budget for setting up a llm based RAG application which will serve users with dynamic size(Anything from 100 to 2000).

I am able to figure out the GPU requirement to host a certain llm[1], let's say LLAMA 70billion at half-precision will require 168 GB. But I am unable to figure out how to calculate the token speed for a single user and then for multiple concurrent users and how to look for appropriate hardware for.

How should I approach this problem?

Thanks for taking the time to read this. [1]:

I am trying to budget for setting up a llm based RAG application which will serve users with dynamic size(Anything from 100 to 2000).

I am able to figure out the GPU requirement to host a certain llm[1], let's say LLAMA 70billion at half-precision will require 168 GB. But I am unable to figure out how to calculate the token speed for a single user and then for multiple concurrent users and how to look for appropriate hardware for.

How should I approach this problem?

Thanks for taking the time to read this. [1]: https://www.substratus.ai/blog/calculating-gpu-memory-for-llm

• python-3.x
• large-language-model
• transformer-model
• inference
• llama

1 Answer 1

From experience - it is not so simple. You need to have take into account:

1. engine used for inference (TGI? pure transformers? llama-cpp)
2. card type (really it matters whether it is H100 or L40S or A100)
3. batch size
4. is it a chatbot like experience or maybe you need to process offline?
5. What is the maximum context you would like to process?

On basis of this you need to run some benchmark and generalize it