LLMs are trained to produce natural language. Getting them to reliably produce machine-parseable output — JSON, specific schemas, typed fields — requires deliberate design. Without it you get valid JSON most of the time, malformed JSON occasionally, prose sometimes, numbers returned as strings, and markdown code fences wrapping the JSON unpredictably. At low volume you can handle failures manually. At scale, unreliable structured output breaks pipelines. If Cartara flagged this in your diff, you're likely using tool_choice, response_format, Pydantic models, or a JSON extraction layer in an LLM call.
The Reliability Ladder
From least to most reliable — use the highest rung available:
```
- Native structured output (provider feature) ← most reliable
- Tool / function calling
- Prompt-only with schema + examples
- Post-processing / repair ← least reliable
```
Native Structured Output
The most reliable approach. The provider constrains the model's output at the decoding level — it's physically impossible for the model to produce output that doesn't match your schema.
Anthropic (Claude) — use tool calling with tool_choice forcing a specific tool:
```python
import anthropic
client = anthropic.Anthropic()
response = client.messages.create(
model="claude-sonnet-4-5",
max_tokens=1024,
tools=[{
"name": "extract_person",
"description": "Extract person details from text",
"input_schema": {
"type": "object",
"properties": {
"name": {"type": "string"},
"age": {"type": "integer"},
"email": {"type": "string", "format": "email"}
},
"required": ["name", "age"]
}
}],
tool_choice={"type": "tool", "name": "extract_person"},
messages=[{"role": "user", "content": "John Smith is 34. Email: j@smith.com"}]
)
tool_use = next(b for b in response.content if b.type == "tool_use")
data = tool_use.input # {"name": "John Smith", "age": 34, "email": "j@smith.com"}
```
OpenAI — use strict structured outputs with a Pydantic model:
```python
from pydantic import BaseModel
from openai import OpenAI
client = OpenAI()
class Person(BaseModel):
name: str
age: int
email: str | None = None
response = client.beta.chat.completions.parse(
model="gpt-4o",
response_format=Person,
messages=[{"role": "user", "content": "John Smith is 34. Email: j@smith.com"}]
)
person = response.choices[0].message.parsed # Typed Person object
```
Tool / Function Calling
When native structured output isn't available, tool calling is the next most reliable approach. Define a "tool" whose parameters match your desired schema. The model "calls" the tool and you extract the arguments. Models are trained extensively on tool use and produce well-formed arguments more reliably than free-form JSON.
Name the tool something neutral like save_result or return_answer rather than a domain-specific action name.
Prompt-Only Approach
When you can't use native structured output or tool calling, use a detailed prompt with schema definition and examples.
```
Extract the following fields and return a JSON object:
{
"company_name": string, // Full legal company name
"founded_year": integer | null, // null if unknown
"employee_count": integer | null,
"is_public": boolean
}
Return ONLY the JSON object. No explanation, no markdown fences, no other text.
The first character of your response must be { and the last must be }.
Example:
Input: "Stripe was founded in 2010. ~8,000 employees, remains private."
Output: {"company_name": "Stripe", "founded_year": 2010, "employee_count": 8000, "is_public": false}
Input: {{actual_text}}
Output:
```
The explicit "ONLY JSON" instruction sounds tedious but significantly reduces format errors.
Output Parsing and Repair
Even with the best prompts, output occasionally needs cleanup. Build a parsing layer:
```python
import re, json
def extract_json(text: str) -> dict:
# Try direct parse first
try:
return json.loads(text.strip())
except json.JSONDecodeError:
pass
# Try extracting from code fences
match = re.search(r'```(?:json)?\s(\{.?\})\s*```', text, re.DOTALL)
if match:
try:
return json.loads(match.group(1))
except json.JSONDecodeError:
pass
# Try finding the outermost JSON object
match = re.search(r'\{.*\}', text, re.DOTALL)
if match:
try:
return json.loads(match.group())
except json.JSONDecodeError:
pass
raise ValueError(f"Could not extract JSON from: {text[:200]}")
```
Validation
Always validate parsed output against your schema before using it downstream.
Python with Pydantic:
```python
from pydantic import BaseModel, ValidationError
from typing import Optional
class ExtractedPerson(BaseModel):
name: str
age: int
email: Optional[str] = None
try:
person = ExtractedPerson(**raw_dict)
except ValidationError as e:
print(e.errors())
```
TypeScript with Zod:
```typescript
import { z } from 'zod'
const PersonSchema = z.object({
name: z.string(),
age: z.number().int().positive(),
email: z.string().email().optional()
})
const result = PersonSchema.safeParse(rawData)
if (!result.success) {
console.error(result.error.issues)
}
```
Schema Design Tips
Use simple, flat structures. Deeply nested schemas have higher failure rates.
Use enums for categorical fields. Without enums, you get "Pending", "PENDING", "in progress" — all meaning the same thing:
```json
"status": {"type": "string", "enum": ["pending", "active", "cancelled"]}
```
Mark optional fields as nullable. Models hallucinate values for required fields when information is absent. Use {"type": ["string", "null"]} and add "Use null for any field where the information is not present."
Be specific with field names. date is ambiguous — use event_start_date or publication_date.
Retry Strategy
For production pipelines, implement automatic retry on parse failure. On retry, include the previous failure in the prompt so the model can self-correct:
```python
def extract_with_retry(text: str, schema: dict, max_retries: int = 3) -> dict:
error_context = ""
for attempt in range(max_retries):
try:
raw = llm.complete(build_prompt(text, schema, error_context))
data = extract_json(raw)
return validate(data, schema)
except (ValueError, ValidationError) as e:
if attempt == max_retries - 1:
raise
error_context = f"Previous attempt failed: {e}. Fix the issue."
```
Related Concepts
- Prompting Best Practices
- RAG - Retrieval-Augmented Generation
- Bounding Boxes in AI
- The MCP Ecosystem