I need to understand how exactly does langchain convert information from code to LLM prompt, because end of the day, the LLM will need only text to be passed to it.
If I am incorrect somewhere in my understanding, kindly point it out as well when answering the question
This is the bind_tools function:
class BaseChatOpenAI(BaseChatModel):
...
def bind_tools(
self,
tools: Sequence[Union[Dict[str, Any], Type, Callable, BaseTool]],
...
**kwargs: Any,
) -> Runnable[LanguageModelInput, BaseMessage]:
"""Bind tool-like objects to this chat model.
Assumes model is compatible with OpenAI tool-calling API.
"""
...
return super().bind(tools=formatted_tools, **kwargs)
On going to definition of super():
class Runnable(Generic[Input, Output], ABC):
...
def bind(self, **kwargs: Any) -> Runnable[Input, Output]:
"""
Bind arguments to a Runnable, returning a new Runnable.
Useful when a Runnable in a chain requires an argument that is not
"""
return RunnableBinding(bound=self, kwargs=kwargs, config={})
On going to RunnableBinding:
class RunnableBinding(RunnableBindingBase[Input, Output]):
"""Wrap a Runnable with additional functionality.
A RunnableBinding can be thought of as a "runnable decorator" that
preserves the essential features of Runnable; i.e., batching, streaming,
and async support, while adding additional functionality.
Any class that inherits from Runnable can be bound to a `RunnableBinding`.
Runnables expose a standard set of methods for creating `RunnableBindings`
or sub-classes of `RunnableBindings` (e.g., `RunnableRetry`,
`RunnableWithFallbacks`) that add additional functionality.
These methods include:
- ``bind``: Bind kwargs to pass to the underlying Runnable when running it.
- ``with_config``: Bind config to pass to the underlying Runnable when running it.
- ``with_listeners``: Bind lifecycle listeners to the underlying Runnable.
- ``with_types``: Override the input and output types of the underlying Runnable.
- ``with_retry``: Bind a retry policy to the underlying Runnable.
- ``with_fallbacks``: Bind a fallback policy to the underlying Runnable.
Example:
`bind`: Bind kwargs to pass to the underlying Runnable when running it.
.. code-block:: python
# Create a Runnable binding that invokes the ChatModel with the
# additional kwarg `stop=['-']` when running it.
from langchain_community.chat_models import ChatOpenAI
model = ChatOpenAI()
model.invoke('Say "Parrot-MAGIC"', stop=['-']) # Should return `Parrot`
# Using it the easy way via `bind` method which returns a new
# RunnableBinding
runnable_binding = model.bind(stop=['-'])
runnable_binding.invoke('Say "Parrot-MAGIC"') # Should return `Parrot`
Can also be done by instantiating a RunnableBinding directly (not recommended):
.. code-block:: python
from langchain_core.runnables import RunnableBinding
runnable_binding = RunnableBinding(
bound=model,
kwargs={'stop': ['-']} # <-- Note the additional kwargs
)
runnable_binding.invoke('Say "Parrot-MAGIC"') # Should return `Parrot`
"""
...
def bind(self, **kwargs: Any) -> Runnable[Input, Output]:
"""Bind additional kwargs to a Runnable, returning a new Runnable.
Args:
**kwargs: The kwargs to bind to the Runnable.
Returns:
A new Runnable with the same type and config as the original,
but with the additional kwargs bound.
"""
return self.__class__(
bound=self.bound,
config=self.config,
kwargs={**self.kwargs, **kwargs},
custom_input_type=self.custom_input_type,
custom_output_type=self.custom_output_type,
)
After this, I am not able to understand how exactly the bind_tools function pass info to the LLM.
I need to understand how exactly does langchain convert information from code to LLM prompt, because end of the day, the LLM will need only text to be passed to it.
If I am incorrect somewhere in my understanding, kindly point it out as well when answering the question
This is the bind_tools function:
class BaseChatOpenAI(BaseChatModel):
...
def bind_tools(
self,
tools: Sequence[Union[Dict[str, Any], Type, Callable, BaseTool]],
...
**kwargs: Any,
) -> Runnable[LanguageModelInput, BaseMessage]:
"""Bind tool-like objects to this chat model.
Assumes model is compatible with OpenAI tool-calling API.
"""
...
return super().bind(tools=formatted_tools, **kwargs)
On going to definition of super():
class Runnable(Generic[Input, Output], ABC):
...
def bind(self, **kwargs: Any) -> Runnable[Input, Output]:
"""
Bind arguments to a Runnable, returning a new Runnable.
Useful when a Runnable in a chain requires an argument that is not
"""
return RunnableBinding(bound=self, kwargs=kwargs, config={})
On going to RunnableBinding:
class RunnableBinding(RunnableBindingBase[Input, Output]):
"""Wrap a Runnable with additional functionality.
A RunnableBinding can be thought of as a "runnable decorator" that
preserves the essential features of Runnable; i.e., batching, streaming,
and async support, while adding additional functionality.
Any class that inherits from Runnable can be bound to a `RunnableBinding`.
Runnables expose a standard set of methods for creating `RunnableBindings`
or sub-classes of `RunnableBindings` (e.g., `RunnableRetry`,
`RunnableWithFallbacks`) that add additional functionality.
These methods include:
- ``bind``: Bind kwargs to pass to the underlying Runnable when running it.
- ``with_config``: Bind config to pass to the underlying Runnable when running it.
- ``with_listeners``: Bind lifecycle listeners to the underlying Runnable.
- ``with_types``: Override the input and output types of the underlying Runnable.
- ``with_retry``: Bind a retry policy to the underlying Runnable.
- ``with_fallbacks``: Bind a fallback policy to the underlying Runnable.
Example:
`bind`: Bind kwargs to pass to the underlying Runnable when running it.
.. code-block:: python
# Create a Runnable binding that invokes the ChatModel with the
# additional kwarg `stop=['-']` when running it.
from langchain_community.chat_models import ChatOpenAI
model = ChatOpenAI()
model.invoke('Say "Parrot-MAGIC"', stop=['-']) # Should return `Parrot`
# Using it the easy way via `bind` method which returns a new
# RunnableBinding
runnable_binding = model.bind(stop=['-'])
runnable_binding.invoke('Say "Parrot-MAGIC"') # Should return `Parrot`
Can also be done by instantiating a RunnableBinding directly (not recommended):
.. code-block:: python
from langchain_core.runnables import RunnableBinding
runnable_binding = RunnableBinding(
bound=model,
kwargs={'stop': ['-']} # <-- Note the additional kwargs
)
runnable_binding.invoke('Say "Parrot-MAGIC"') # Should return `Parrot`
"""
...
def bind(self, **kwargs: Any) -> Runnable[Input, Output]:
"""Bind additional kwargs to a Runnable, returning a new Runnable.
Args:
**kwargs: The kwargs to bind to the Runnable.
Returns:
A new Runnable with the same type and config as the original,
but with the additional kwargs bound.
"""
return self.__class__(
bound=self.bound,
config=self.config,
kwargs={**self.kwargs, **kwargs},
custom_input_type=self.custom_input_type,
custom_output_type=self.custom_output_type,
)
After this, I am not able to understand how exactly the bind_tools function pass info to the LLM.
1 Answer
Reset to default 1 +50in here:
class BaseChatOpenAI(BaseChatModel):
...
def bind_tools(
self,
tools: Sequence[Union[Dict[str, Any], Type, Callable, BaseTool]],
...
**kwargs: Any,
) -> Runnable[LanguageModelInput, BaseMessage]:
"""Bind tool-like objects to this chat model.
Assumes model is compatible with OpenAI tool-calling API.
"""
...
return super().bind(tools=formatted_tools, **kwargs)
This code shows that the tools are first “formatted” to formatted_tools which is :
formatted_tools = [
convert_to_openai_tool(tool, strict=strict) for tool in tools
]
Here is convert_to_openai_tool from github-langchain:
def convert_to_openai_tool(
tool: Union[dict[str, Any], type[BaseModel], Callable, BaseTool],
*,
strict: Optional[bool] = None,
) -> dict[str, Any]:
"""Convert a tool-like object to an OpenAI tool schema.
OpenAI tool schema reference:
https://platform.openai.com/docs/api-reference/chat/create#chat-create-tools
Args:
tool:
Either a dictionary, a pydantic.BaseModel class, Python function, or
BaseTool. If a dictionary is passed in, it is
assumed to already be a valid OpenAI function, a JSON schema with
top-level 'title' key specified, an Anthropic format
tool, or an Amazon Bedrock Converse format tool.
strict:
If True, model output is guaranteed to exactly match the JSON Schema
provided in the function definition. If None, ``strict`` argument will not
be included in tool definition.
Returns:
A dict version of the passed in tool which is compatible with the
OpenAI tool-calling API.
.. versionchanged:: 0.2.29
``strict`` arg added.
.. versionchanged:: 0.3.13
Support for Anthropic format tools added.
.. versionchanged:: 0.3.14
Support for Amazon Bedrock Converse format tools added.
.. versionchanged:: 0.3.16
'description' and 'parameters' keys are now optional. Only 'name' is
required and guaranteed to be part of the output.
"""
if isinstance(tool, dict) and tool.get("type") == "function" and "function" in tool:
return tool
oai_function = convert_to_openai_function(tool, strict=strict)
return {"type": "function", "function": oai_function}
from its comments this function returns
A dict version of the passed in tool which is compatible with the OpenAI tool-calling API.
This formatted tools is then passed to the parent’s bind method, which stores the tool metadata (via a RunnableBinding) for later use during prompt construction.
then bind method in Runnable class:
class Runnable(Generic[Input, Output], ABC):
...
def bind(self, **kwargs: Any) -> Runnable[Input, Output]:
"""
Bind arguments to a Runnable, returning a new Runnable.
Useful when a Runnable in a chain requires an argument that is not
"""
return RunnableBinding(bound=self, kwargs=kwargs, config={})
creates a RunnableBinding, which attaches the formatted tool metadata (via kwargs) to the model instance.
eventually RunnableBinding bind method ensures that when the LLM call is eventually made, the model has access to all the tool metadata. RunnableBinding is designed to wrap a base Runnable (like your LLM) and attach additional configuration or parameters (tool metadata, API settings, etc.). It essentially creates the current state with all the bound parameters so that when the runnable is later invoked, all those extra details are available.
When RunnableBinding is invoked, it first ensures all configurations are correctly formatted using ensure_config.
invoke method of BaseLLM class from here
def invoke(
self,
input: LanguageModelInput,
config: Optional[RunnableConfig] = None,
*,
stop: Optional[list[str]] = None,
**kwargs: Any,
) -> str:
config = ensure_config(config)
return (
self.generate_prompt(
[self._convert_input(input)],
stop=stop,
callbacks=config.get("callbacks"),
tags=config.get("tags"),
metadata=config.get("metadata"),
run_name=config.get("run_name"),
run_id=config.pop("run_id", None),
**kwargs,
)
.generations[0][0]
.text
)
this is ensure_config from here
def ensure_config(config: Optional[RunnableConfig] = None) -> RunnableConfig:
"""Ensure that a config is a dict with all keys present.
Args:
config (Optional[RunnableConfig], optional): The config to ensure.
Defaults to None.
Returns:
RunnableConfig: The ensured config.
"""
empty = RunnableConfig(
tags=[],
metadata={},
callbacks=None,
recursion_limit=DEFAULT_RECURSION_LIMIT,
configurable={},
)
if var_config := var_child_runnable_config.get():
empty.update(
cast(
RunnableConfig,
{
k: v.copy() if k in COPIABLE_KEYS else v # type: ignore[attr-defined]
for k, v in var_config.items()
if v is not None
},
)
)
if config is not None:
empty.update(
cast(
RunnableConfig,
{
k: v.copy() if k in COPIABLE_KEYS else v # type: ignore[attr-defined]
for k, v in config.items()
if v is not None and k in CONFIG_KEYS
},
)
)
if config is not None:
for k, v in config.items():
if k not in CONFIG_KEYS and v is not None:
empty["configurable"][k] = v
for key, value in empty.get("configurable", {}).items():
if (
not key.startswith("__")
and isinstance(value, (str, int, float, bool))
and key not in empty["metadata"]
):
empty["metadata"][key] = value
return empty