Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

New design for Multimodal Messages #73

Open
HavenDV opened this issue Oct 14, 2024 · 0 comments
Open

New design for Multimodal Messages #73

HavenDV opened this issue Oct 14, 2024 · 0 comments
Labels
enhancement New feature or request

Comments

@HavenDV
Copy link
Contributor

HavenDV commented Oct 14, 2024

https://python.langchain.com/docs/how_to/multimodal_inputs/

There are currently several leading approaches to presenting a series of messages:

AnthropicMessage
  Content = IList<ContentBlock = OneOf<Text, Image, ToolUse, ToolResult>>
  Other non content properties

OllamaMessage
  Content = string
  Images = IList<string>
  ToolCalls = IList<ToolCall>

OpenAiMessage  // Each Role Has Different Content
  System
    Content = IList<ContentPart = OneOf<Text>>
  User
    Content = IList<ContentPart = OneOf<Text, Image>>
  Assistant
    Content = IList<ContentPart = OneOf<Text, Refusal>>
    ToolCalls
  Tool
    Content = IList<ContentPart = OneOf<Text>>

GoogleMessage
  Content = IList<ContentPart = OneOf<Text, Blob = (byte[], string MimeType)>>

I like the simplicity of Anthropic, but I would change Block to ContentPart

So far in LangChain I see it as:

Message
Content = IList<ContentPart = OneOf<Text, Image, ToolUse, ToolResult, Blob, Video>>

or as separate messages that don't allow parts inside

TextMessage
ImageMessage
ToolUseMessage
ToolResultMessage
BlobMessage // allow you to specify a MimeType 
VideoMessage

When the user returns multiple parts, we just use two messages in a row 

OpenAI also has changes to the message structure in the Realtime API
I will add to this taking into account the changes in the OpenAI Realtime API

@HavenDV HavenDV added the enhancement New feature or request label Oct 14, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

1 participant