I am not sure how everyone is able to obtain impressive results with any Llama model. I'm trying to get it to generate MongoDB queries form natural language, and I often get gibberish answers (non-latin alphabet characters or the same word over and over or parts of the system prompt), or somewhat relevant answers but disregarding the most important part of the system prompt or user prompt (like generating explanatory text about mongoDB when the system prompt explicitly states "only generate the query").

I am trying to recreate the results in this official Llama release blog post and I am not able to:

from transformers import AutoTokenizer, AutoModelForCausalLM
import torch

def apply_basic_template(system, user):
    return "<|start_header_id|>system<|end_header_id|>:" + system + "<|eot_id|><|start_header_id|>user<|end_header_id|>:" + user + "<|eot_id|><|start_header_id|>assistant<|end_header_id|>"

MODEL = "meta-llama/Llama-3.2-1B"

model = AutoModelForCausalLM.from_pretrained(MODEL).to('cuda')
tokenizer = AutoTokenizer.from_pretrained(MODEL)
tokenizer.pad_token = tokenizer.eos_token

function_definitions = """[
    {
        "name": "get_user_info",
        "description": "Retrieve details for a specific user by their unique identifier. Note that the provided function is in Python 3 syntax.",
        "parameters": {
            "type": "dict",
            "required": [
                "user_id"
            ],
            "properties": {
                "user_id": {
                "type": "integer",
                "description": "The unique identifier of the user. It is used to fetch the specific user details from the database."
            },
            "special": {
                "type": "string",
                "description": "Any special information or parameters that need to be considered while fetching user details.",
                "default": "none"
                }
            }
        }
    }
]
"""
system_prompt = """You are an expert in composing functions. You are given a question and a set of possible functions. 
Based on the question, you will need to make one or more function/tool calls to achieve the purpose. 
If none of the function can be used, point it out. If the given question lacks the parameters required by the function,
also point it out. You should only return the function call in tools call sections.

If you decide to invoke any of the function(s), you MUST put it in the format of [func_name1(params_name1=params_value1, params_name2=params_value2...), func_name2(params)]\n
You SHOULD NOT include any other text in the response.

Here is a list of functions in JSON format that you can invoke.\n\n{functions}\n""".format(functions=function_definitions)

query = "Can you retrieve the details for the user with the ID 7890, who has black as their special request?"

prompt = apply_basic_template(system_prompt, query)

with torch.no_grad():
    inputs = tokenizer(prompt, return_tensors="pt").to('cuda')
    outputs = model.generate(**inputs, max_new_tokens=100)
    response = tokenizer.decode(outputs[0])[len(prompt):]

print(response)

Result:

<|end_header_id|>:Can you retrieve the details for the user with the ID 7890, who has black as their special request?lbrakk>Can you retrieve the details for the user with the ID 7890, who has black as their special request?lbrakk>Can you retrieve the details for the user with the ID 7890, who has black as their special request?lbrakk>Can you retrieve the details for the user with the ID 7890, who has black as their special request?lbrakk

Similar results with Llama 3.2 3B.

Thank you