core[minor], integrations...[patch]: Support ToolCall as Tool input and ToolMessage as Tool output

[baskaryan]

Changes:

• ToolCall, InvalidToolCall and ToolCallChunk can all accept a "type" parameter now
• LLM integration packages add "type" to all the above
• Tool supports ToolCall inputs that have "type" specified
• Tool outputs ToolMessage when a ToolCall is passed as input
• Tools can separately specify ToolMessage.content and ToolMessage.raw_output
• Tools emit events for validation errors (using on_tool_error and on_tool_end)

Example:

@tool("structured_api", response_format="content_and_raw_output")
def _mock_structured_tool_with_raw_output(
    arg1: int, arg2: bool, arg3: Optional[dict] = None
) -> Tuple[str, dict]:
    """A Structured Tool"""
    return f"{arg1} {arg2}", {"arg1": arg1, "arg2": arg2, "arg3": arg3}


def test_tool_call_input_tool_message_with_raw_output() -> None:
    tool_call: Dict = {
        "name": "structured_api",
        "args": {"arg1": 1, "arg2": True, "arg3": {"img": "base64string..."}},
        "id": "123",
        "type": "tool_call",
    }
    expected = ToolMessage("1 True", raw_output=tool_call["args"], tool_call_id="123")
    tool = _mock_structured_tool_with_raw_output
    actual = tool.invoke(tool_call)
    assert actual == expected

    tool_call.pop("type")
    with pytest.raises(ValidationError):
        tool.invoke(tool_call)

    actual_content = tool.invoke(tool_call["args"])
    assert actual_content == expected.content

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

1 Skipped Deployment

Name	Status	Preview	Comments	Updated (UTC)
langchain	⬜️ Ignored (Inspect)	Visit Preview		Jul 11, 2024 8:56pm

[baskaryan]

corresponding ToolNode update langchain-ai/langgraph#977

[eyurtsev]

partial review

[eyurtsev]

why is this needed?

[baskaryan]

convenience / to make sure ppl don't mess up the type

[eyurtsev]

How is that easier than using ToolCallChunk? Is it validating that there are no extra arguments or typos in the arguments?

[baskaryan]

type is hard coded

[eyurtsev]

What can tool input be here? (if you know of the top of you head and have time could be good to update the doc-string)

[ccurme]

Integration tests look good: https://github.com/langchain-ai/langchain/actions/runs/9893304257

[ccurme]

why do we say they're by definition dicts (and can't be str)?

[baskaryan]

copy/pasted from elsewhere, cc @eyurtsev

[eyurtsev]

Comment is originally from observing this:

@tool
def foo(x: str):
   return x

foo.invoke({'x': 'hello'})

It's possible that str is still supported on its own. I don't know what we want to do in that case -- since we have pretty odd behavior when it comes to langsmith API requiring auto upgrades to dicts

[ccurme]

There are places we've duplicated logic (e.g., in astream) that may also need to be updated.

What will the behavior be if a tool call is missing a type? Possible there will be issues operating on old serialized tool calls (ok if so-- helps to know in case bugs pop up)? And do we need to update other integrations (e.g., community and other repos)?

[baskaryan]

only thing that depends on type atm is Tools when they're taking a ToolCall (which is a new feature, you couldn't pass in ToolCalls at all before)