Skip to main content

How to return structured data from a model

It is often useful to have a model return output that matches some specific schema. One common use-case is extracting data from arbitrary text to insert into a traditional database or use with some other downstrem system. This guide will show you a few different strategies you can use to do this.

Prerequisites

This guide assumes familiarity with the following concepts:

The .withStructuredOutput() method​

There are several strategies that models can use under the hood. For some of the most popular model providers, including Anthropic, Google VertexAI, Mistral, and OpenAI LangChain implements a common interface that abstracts away these strategies called .withStructuredOutput.

By invoking this method (and passing in JSON schema or a Zod schema) the model will add whatever model parameters + output parsers are necessary to get back structured output matching the requested schema. If the model supports more than one way to do this (e.g., function calling vs JSON mode) - you can configure which method to use by passing into that method.

Let’s look at some examples of this in action! We’ll use Zod to create a simple response schema.

Pick your chat model:

Install dependencies

tip

See this section for general instructions on installing integration packages.

yarn add @langchain/openai

Add environment variables

OPENAI_API_KEY=your-api-key

Instantiate the model

import { ChatOpenAI } from "@langchain/openai";

const model = new ChatOpenAI({
  model: "gpt-3.5-turbo",
  temperature: 0
});
import { z } from "zod";

const joke = z.object({
  setup: z.string().describe("The setup of the joke"),
  punchline: z.string().describe("The punchline to the joke"),
  rating: z.number().optional().describe("How funny the joke is, from 1 to 10"),
});

const structuredLlm = model.withStructuredOutput(joke);

await structuredLlm.invoke("Tell me a joke about cats");
{
  setup: "Why don't cats play poker in the wild?",
  punchline: "Too many cheetahs.",
  rating: 7
}

One key point is that though we set our Zod schema as a variable named joke, Zod is not able to access that variable name, and therefore cannot pass it to the model. Though it is not required, we can pass a name for our schema in order to give the model additional context as to what our schema represents, improving performance:

const structuredLlm = model.withStructuredOutput(joke, { name: "joke" });

await structuredLlm.invoke("Tell me a joke about cats");
{
  setup: "Why don't cats play poker in the wild?",
  punchline: "Too many cheetahs!",
  rating: 7
}

The result is a JSON object.

We can also pass in an OpenAI-style JSON schema dict if you prefer not to use Zod. This object should contain three properties:

  • name: The name of the schema to output.
  • description: A high level description of the schema to output.
  • parameters: The nested details of the schema you want to extract, formatted as a JSON schema dict.

In this case, the response is also a dict:

const structuredLlm = model.withStructuredOutput({
  name: "joke",
  description: "Joke to tell user.",
  parameters: {
    title: "Joke",
    type: "object",
    properties: {
      setup: { type: "string", description: "The setup for the joke" },
      punchline: { type: "string", description: "The joke's punchline" },
    },
    required: ["setup", "punchline"],
  },
});

await structuredLlm.invoke("Tell me a joke about cats", { name: "joke" });
{
  setup: "Why was the cat sitting on the computer?",
  punchline: "Because it wanted to keep an eye on the mouse!"
}

If you are using JSON Schema, you can take advantage of other more complex schema descriptions to create a similar effect.

You can also use tool calling directly to allow the model to choose between options, if your chosen model supports it. This involves a bit more parsing and setup. See this how-to guide for more details.

Specifying the output method (Advanced)​

For models that support more than one means of outputting data, you can specify the preferred one like this:

const structuredLlm = model.withStructuredOutput(joke, {
  method: "json_mode",
  name: "joke",
});

await structuredLlm.invoke(
  "Tell me a joke about cats, respond in JSON with `setup` and `punchline` keys"
);
{
  setup: "Why don't cats play poker in the jungle?",
  punchline: "Too many cheetahs!"
}

In the above example, we use OpenAI’s alternate JSON mode capability along with a more specific prompt.

For specifics about the model you choose, peruse its entry in the API reference pages.

(Advanced) Raw outputs​

LLMs aren’t perfect at generating structured output, especially as schemas become complex. You can avoid raising exceptions and handle the raw output yourself by passing includeRaw: true. This changes the output format to contain the raw message output and the parsed value (if successful):

const joke = z.object({
  setup: z.string().describe("The setup of the joke"),
  punchline: z.string().describe("The punchline to the joke"),
  rating: z.number().optional().describe("How funny the joke is, from 1 to 10"),
});

const structuredLlm = model.withStructuredOutput(joke, {
  includeRaw: true,
  name: "joke",
});

await structuredLlm.invoke("Tell me a joke about cats");
{
  raw: AIMessage {
    lc_serializable: true,
    lc_kwargs: {
      content: "",
      tool_calls: [
        {
          name: "joke",
          args: [Object],
          id: "call_0pEdltlfSXjq20RaBFKSQOeF"
        }
      ],
      invalid_tool_calls: [],
      additional_kwargs: { function_call: undefined, tool_calls: [ [Object] ] },
      response_metadata: {}
    },
    lc_namespace: [ "langchain_core", "messages" ],
    content: "",
    name: undefined,
    additional_kwargs: {
      function_call: undefined,
      tool_calls: [
        {
          id: "call_0pEdltlfSXjq20RaBFKSQOeF",
          type: "function",
          function: [Object]
        }
      ]
    },
    response_metadata: {
      tokenUsage: { completionTokens: 33, promptTokens: 88, totalTokens: 121 },
      finish_reason: "stop"
    },
    tool_calls: [
      {
        name: "joke",
        args: {
          setup: "Why was the cat sitting on the computer?",
          punchline: "Because it wanted to keep an eye on the mouse!",
          rating: 7
        },
        id: "call_0pEdltlfSXjq20RaBFKSQOeF"
      }
    ],
    invalid_tool_calls: [],
    usage_metadata: { input_tokens: 88, output_tokens: 33, total_tokens: 121 }
  },
  parsed: {
    setup: "Why was the cat sitting on the computer?",
    punchline: "Because it wanted to keep an eye on the mouse!",
    rating: 7
  }
}

Prompting techniques​

You can also prompt models to outputting information in a given format. This approach relies on designing good prompts and then parsing the output of the models. This is the only option for models that don’t support .with_structured_output() or other built-in approaches.

Using JsonOutputParser​

The following example uses the built-in JsonOutputParser to parse the output of a chat model prompted to match a the given JSON schema. Note that we are adding format_instructions directly to the prompt from a method on the parser:

import { JsonOutputParser } from "@langchain/core/output_parsers";
import { ChatPromptTemplate } from "@langchain/core/prompts";

type Person = {
  name: string;
  height_in_meters: number;
};

type People = {
  people: Person[];
};

const formatInstructions = `Respond only in valid JSON. The JSON object you return should match the following schema:
{{ people: [{{ name: "string", height_in_meters: "number" }}] }}

Where people is an array of objects, each with a name and height_in_meters field.
`;

// Set up a parser
const parser = new JsonOutputParser<People>();

// Prompt
const prompt = await ChatPromptTemplate.fromMessages([
  [
    "system",
    "Answer the user query. Wrap the output in `json` tags\n{format_instructions}",
  ],
  ["human", "{query}"],
]).partial({
  format_instructions: formatInstructions,
});

Let’s take a look at what information is sent to the model:

const query = "Anna is 23 years old and she is 6 feet tall";

console.log((await prompt.format({ query })).toString());
System: Answer the user query. Wrap the output in `json` tags
Respond only in valid JSON. The JSON object you return should match the following schema:
{{ people: [{{ name: "string", height_in_meters: "number" }}] }}

Where people is an array of objects, each with a name and height_in_meters field.

Human: Anna is 23 years old and she is 6 feet tall

And now let’s invoke it:

const chain = prompt.pipe(model).pipe(parser);

await chain.invoke({ query });
{ people: [ { name: "Anna", height_in_meters: 1.83 } ] }

For a deeper dive into using output parsers with prompting techniques for structured output, see this guide.

Custom Parsing​

You can also create a custom prompt and parser with LangChain Expression Language (LCEL), using a plain function to parse the output from the model:

import { AIMessage } from "@langchain/core/messages";
import { ChatPromptTemplate } from "@langchain/core/prompts";

type Person = {
  name: string;
  height_in_meters: number;
};

type People = {
  people: Person[];
};

const schema = `{{ people: [{{ name: "string", height_in_meters: "number" }}] }}`;

// Prompt
const prompt = await ChatPromptTemplate.fromMessages([
  [
    "system",
    `Answer the user query. Output your answer as JSON that
matches the given schema: \`\`\`json\n{schema}\n\`\`\`.
Make sure to wrap the answer in \`\`\`json and \`\`\` tags`,
  ],
  ["human", "{query}"],
]).partial({
  schema,
});

/**
 * Custom extractor
 *
 * Extracts JSON content from a string where
 * JSON is embedded between ```json and ``` tags.
 */
const extractJson = (output: AIMessage): Array<People> => {
  const text = output.content as string;
  // Define the regular expression pattern to match JSON blocks
  const pattern = /```json(.*?)```/gs;

  // Find all non-overlapping matches of the pattern in the string
  const matches = text.match(pattern);

  // Process each match, attempting to parse it as JSON
  try {
    return (
      matches?.map((match) => {
        // Remove the markdown code block syntax to isolate the JSON string
        const jsonStr = match.replace(/```json|```/g, "").trim();
        return JSON.parse(jsonStr);
      }) ?? []
    );
  } catch (error) {
    throw new Error(`Failed to parse: ${output}`);
  }
};

Here is the prompt sent to the model:

const query = "Anna is 23 years old and she is 6 feet tall";

console.log((await prompt.format({ query })).toString());
System: Answer the user query. Output your answer as JSON that
matches the given schema: ```json
{{ people: [{{ name: "string", height_in_meters: "number" }}] }}
```.
Make sure to wrap the answer in ```json and ``` tags
Human: Anna is 23 years old and she is 6 feet tall

And here’s what it looks like when we invoke it:

import { RunnableLambda } from "@langchain/core/runnables";

const chain = prompt
  .pipe(model)
  .pipe(new RunnableLambda({ func: extractJson }));

await chain.invoke({ query });
[
  { people: [ { name: "Anna", height_in_meters: 1.83 } ] }
]

Next steps​

Now you’ve learned a few methods to make a model output structured data.

To learn more, check out the other how-to guides in this section, or the conceptual guide on tool calling.

Was this page helpful?


You can also leave detailed feedback on GitHub.