[BUG] Instructor Instrumentator causing Pydantic ValidationErrors

[ikarth]

Describe the bug
I'm having repeated validation errors when I'm attempting to use the InstructorInstrumentor (via LiteLLM talking to vLLM).

To Reproduce

import instructor
from litellm import completion, drop_params
import litellm
litellm.drop_params = True

from typing import Iterable, List, Optional
from pydantic import BaseModel, ConfigDict, Field

from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import SimpleSpanProcessor

endpoint = "http://127.0.0.1:6006/v1/traces"
tracer_provider = TracerProvider()
tracer_provider.add_span_processor(SimpleSpanProcessor(OTLPSpanExporter(endpoint)))
from openinference.instrumentation.instructor import InstructorInstrumentor
from openinference.instrumentation.litellm import LiteLLMInstrumentor

InstructorInstrumentor().instrument(tracer_provider=tracer_provider)
LiteLLMInstrumentor().instrument(tracer_provider=tracer_provider)

client = instructor.from_litellm(completion)

class WeatherReport(BaseModel):
	"""The weather for today."""
	temperature: float
	precipitation: str

messages = [{"role": "user", "content": "Generate a weather report."}]

response = client.chat.completions.create(
		api_base=API_BASE,
        api_key=API_KEY,
        model=API_MODEL,
        messages=messages,
        response_model=WeatherReport
		)

print(response)

Results in a Pydantic ValidationError:

Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/luxor/src/lamesh/test_tele.py", line 39, in <module>
    response = client.chat.completions.create(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/miniconda3/envs/luxor/lib/python3.12/site-packages/instructor/client.py", line 172, in create
    return self.create_fn(
           ^^^^^^^^^^^^^^^
  File "/miniconda3/envs/luxor/lib/python3.12/site-packages/openinference/instrumentation/instructor/_wrappers.py", line 113, in patched_new_func
    with self._tracer.start_as_current_span(
  File "/miniconda3/envs/luxor/lib/python3.12/contextlib.py", line 137, in __enter__
    return next(self.gen)
           ^^^^^^^^^^^^^^
  File "/miniconda3/envs/luxor/lib/python3.12/site-packages/openinference/instrumentation/config.py", line 396, in start_as_current_span
    span = self.start_span(
           ^^^^^^^^^^^^^^^^
  File "/miniconda3/envs/luxor/lib/python3.12/site-packages/openinference/instrumentation/config.py", line 439, in start_span
    span.set_attributes(attributes)
  File "/miniconda3/envs/luxor/lib/python3.12/site-packages/openinference/instrumentation/config.py", line 334, in set_attributes
    self.set_attribute(k, v)
  File "/miniconda3/envs/luxor/lib/python3.12/site-packages/openinference/instrumentation/config.py", line 341, in set_attribute
    value = self._self_config.mask(key, value)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/miniconda3/envs/luxor/lib/python3.12/site-packages/openinference/instrumentation/config.py", line 273, in mask
    return value() if callable(value) else value
           ^^^^^^^
  File "/miniconda3/envs/luxor/lib/python3.12/site-packages/pydantic/main.py", line 212, in __init__
    validated_self = self.__pydantic_validator__.validate_python(data, self_instance=self)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
pydantic_core._pydantic_core.ValidationError: 2 validation errors for WeatherReport
temperature
  Field required [type=missing, input_value={}, input_type=dict]
    For further information visit https://errors.pydantic.dev/2.9/v/missing
precipitation
  Field required [type=missing, input_value={}, input_type=dict]
    For further information visit https://errors.pydantic.dev/2.9/v/missing

For some reason, moving the client = instructor.from_litellm(completion) line above the telemetry setup gives a different warning:

import instructor
from litellm import completion, drop_params
import litellm
litellm.drop_params = True

client = instructor.from_litellm(completion)

from typing import Iterable, List, Optional
from pydantic import BaseModel, ConfigDict, Field

from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import SimpleSpanProcessor

endpoint = "http://127.0.0.1:6006/v1/traces"
tracer_provider = TracerProvider()
tracer_provider.add_span_processor(SimpleSpanProcessor(OTLPSpanExporter(endpoint)))
from openinference.instrumentation.instructor import InstructorInstrumentor
from openinference.instrumentation.litellm import LiteLLMInstrumentor

InstructorInstrumentor().instrument(tracer_provider=tracer_provider)
LiteLLMInstrumentor().instrument(tracer_provider=tracer_provider)


class WeatherReport(BaseModel):
	"""The weather for today."""
	temperature: float
	precipitation: str

messages = [{"role": "user", "content": "Generate a weather report."}]

response = client.chat.completions.create(
		api_base=API_BASE,
        api_key=API_KEY,
        model=API_MODEL,
        messages=messages,
        response_model=WeatherReport
		)

print(response)

Invalid type ModelMetaclass in attribute 'output.value' value sequence. Expected one of ['bool', 'str', 'bytes', 'int', 'float'] or None
temperature=23.0 precipitation='light rain'

Expected behavior
The expected response is that the validation would work after the generation is finished.

temperature=23.0 precipitation='light rain'

Environment (please complete the following information):

• OS: Windows (Ubuntu 22 via WSL)

arize-otel                               0.5.3
arize-phoenix                            5.7.0
arize-phoenix-evals                      0.16.1
arize-phoenix-otel                       0.6.1

instructor                               1.6.3

litellm                                  1.51.0

openinference-instrumentation            0.1.18
openinference-instrumentation-dspy       0.1.13
openinference-instrumentation-instructor 0.1.2
openinference-instrumentation-litellm    0.1.5
openinference-semantic-conventions       0.1.12
opentelemetry-api                        1.27.0
opentelemetry-exporter-otlp              1.27.0
opentelemetry-exporter-otlp-proto-common 1.27.0
opentelemetry-exporter-otlp-proto-grpc   1.27.0
opentelemetry-exporter-otlp-proto-http   1.27.0
opentelemetry-instrumentation            0.48b0
opentelemetry-proto                      1.27.0
opentelemetry-sdk                        1.27.0
opentelemetry-semantic-conventions       0.48b0

vllm                                     0.6.3.post1

[mikeldking]

Thanks for the note @ikarth ! We've tagged the maintainer of the instrumentation. Let us know if you need anything!

[glebmezh]

Ran into the same issue using Instructor + Anthropic Bedrock as the provider.