LLMs

CrewAI integrates with multiple LLM providers through LiteLLM, giving you the flexibility to choose the right model for your specific use case. This guide will help you understand how to configure and use different LLM providers in your CrewAI projects.

What are LLMs?

Large Language Models (LLMs) are the core intelligence behind CrewAI agents. They enable agents to understand context, make decisions, and generate human-like responses. Here’s what you need to know:

LLM Basics

Large Language Models are AI systems trained on vast amounts of text data. They power the intelligence of your CrewAI agents, enabling them to understand and generate human-like text.

Context Window

The context window determines how much text an LLM can process at once. Larger windows (e.g., 128K tokens) allow for more context but may be more expensive and slower.

Temperature

Temperature (0.0 to 1.0) controls response randomness. Lower values (e.g., 0.2) produce more focused, deterministic outputs, while higher values (e.g., 0.8) increase creativity and variability.

Provider Selection

Each LLM provider (e.g., OpenAI, Anthropic, Google) offers different models with varying capabilities, pricing, and features. Choose based on your needs for accuracy, speed, and cost.

Available Models and Their Capabilities

Here’s a detailed breakdown of supported models and their capabilities, you can compare performance at lmarena.ai and artificialanalysis.ai:

Model	Context Window	Best For
GPT-4	8,192 tokens	High-accuracy tasks, complex reasoning
GPT-4 Turbo	128,000 tokens	Long-form content, document analysis
GPT-4o & GPT-4o-mini	128,000 tokens	Cost-effective large context processing
o3-mini	200,000 tokens	Fast reasoning, complex reasoning

1 token ≈ 4 characters in English. For example, 8,192 tokens ≈ 32,768 characters or about 6,000 words.

Setting Up Your LLM

There are three ways to configure LLMs in CrewAI. Choose the method that best fits your workflow:

The simplest way to get started. Set these variables in your environment:

# Required: Your API key for authentication
OPENAI_API_KEY=<your-api-key>

# Optional: Default model selection
OPENAI_MODEL_NAME=gpt-4o-mini  # Default if not set

# Optional: Organization ID (if applicable)
OPENAI_ORGANIZATION_ID=<your-org-id>

Never commit API keys to version control. Use environment files (.env) or your system’s secret management.

Advanced Features and Optimization

Learn how to get the most out of your LLM configuration:

CrewAI includes smart context management features:

from crewai import LLM

# CrewAI automatically handles:
# 1. Token counting and tracking
# 2. Content summarization when needed
# 3. Task splitting for large contexts

llm = LLM(
    model="gpt-4",
    max_tokens=4000,  # Limit response length
)

Best practices for context management:

Choose models with appropriate context windows
Pre-process long inputs when possible
Use chunking for large documents
Monitor token usage to optimize costs

Token Usage Optimization

Choose the right context window for your task:

Small tasks (up to 4K tokens): Standard models
Medium tasks (between 4K-32K): Enhanced models
Large tasks (over 32K): Large context models

# Configure model with appropriate settings
llm = LLM(
    model="openai/gpt-4-turbo-preview",
    temperature=0.7,    # Adjust based on task
    max_tokens=4096,    # Set based on output needs
    timeout=300        # Longer timeout for complex tasks
)

Lower temperature (0.1 to 0.3) for factual responses
Higher temperature (0.7 to 0.9) for creative tasks

Best Practices

Monitor token usage
Implement rate limiting
Use caching when possible
Set appropriate max_tokens limits

Remember to regularly monitor your token usage and adjust your configuration as needed to optimize costs and performance.

Provider Configuration Examples

Code
# Required
OPENAI_API_KEY=sk-...

# Optional
OPENAI_API_BASE=<custom-base-url>
OPENAI_ORGANIZATION=<your-org-id>

Example usage:

Code
from crewai import LLM

llm = LLM(
    model="gpt-4",
    temperature=0.8,
    max_tokens=150,
    top_p=0.9,
    frequency_penalty=0.1,
    presence_penalty=0.1,
    stop=["END"],
    seed=42
)

Code
# Option 1: Gemini accessed with an API key.
# https://ai.google.dev/gemini-api/docs/api-key
GEMINI_API_KEY=<your-api-key>

# Option 2: Vertex AI IAM credentials for Gemini, Anthropic, and Model Garden.
# https://cloud.google.com/vertex-ai/generative-ai/docs/overview

Get credentials:

Code
import json

file_path = 'path/to/vertex_ai_service_account.json'

# Load the JSON file
with open(file_path, 'r') as file:
    vertex_credentials = json.load(file)

# Convert the credentials to a JSON string
vertex_credentials_json = json.dumps(vertex_credentials)

Example usage:

Code
from crewai import LLM

llm = LLM(
    model="gemini/gemini-1.5-pro-latest",
    temperature=0.7,
    vertex_credentials=vertex_credentials_json
)

Code
# Required
AZURE_API_KEY=<your-api-key>
AZURE_API_BASE=<your-resource-url>
AZURE_API_VERSION=<api-version>

# Optional
AZURE_AD_TOKEN=<your-azure-ad-token>
AZURE_API_TYPE=<your-azure-api-type>

Example usage:

Code
llm = LLM(
    model="azure/gpt-4",
    api_version="2023-05-15"
)

Code
AWS_ACCESS_KEY_ID=<your-access-key>
AWS_SECRET_ACCESS_KEY=<your-secret-key>
AWS_DEFAULT_REGION=<your-region>

Example usage:

Code
llm = LLM(
    model="bedrock/anthropic.claude-3-sonnet-20240229-v1:0"
)

Code
# Required
WATSONX_URL=<your-url>
WATSONX_APIKEY=<your-apikey>
WATSONX_PROJECT_ID=<your-project-id>

# Optional
WATSONX_TOKEN=<your-token>
WATSONX_DEPLOYMENT_SPACE_ID=<your-space-id>

Example usage:

Code
llm = LLM(
    model="watsonx/meta-llama/llama-3-1-70b-instruct",
    base_url="https://api.watsonx.ai/v1"
)

Code
# Required
CEREBRAS_API_KEY=<your-api-key>

Example usage:

Code
llm = LLM(
    model="cerebras/llama3.1-70b",
    temperature=0.7,
    max_tokens=8192
)

Cerebras features:

Fast inference speeds
Competitive pricing
Good balance of speed and quality
Support for long context windows

Code
OPENROUTER_API_KEY=<your-api-key>

Example usage:

Code
llm = LLM(
    model="openrouter/deepseek/deepseek-r1",
    base_url="https://openrouter.ai/api/v1",
    api_key=OPENROUTER_API_KEY
)

Open Router models:

openrouter/deepseek/deepseek-r1
openrouter/deepseek/deepseek-chat

Structured LLM Calls

CrewAI supports structured responses from LLM calls by allowing you to define a response_format using a Pydantic model. This enables the framework to automatically parse and validate the output, making it easier to integrate the response into your application without manual post-processing.

For example, you can define a Pydantic model to represent the expected response structure and pass it as the response_format when instantiating the LLM. The model will then be used to convert the LLM output into a structured Python object.

Code
from crewai import LLM

class Dog(BaseModel):
    name: str
    age: int
    breed: str


llm = LLM(model="gpt-4o", response_format=Dog)

response = llm.call(
    "Analyze the following messages and return the name, age, and breed. "
    "Meet Kona! She is 3 years old and is a black german shepherd."
)
print(response)

Common Issues and Solutions

Most authentication issues can be resolved by checking API key format and environment variable names.

# OpenAI
OPENAI_API_KEY=sk-...

# Anthropic
ANTHROPIC_API_KEY=sk-ant-...

Getting Help

If you need assistance, these resources are available:

LiteLLM Documentation

Comprehensive documentation for LiteLLM integration and troubleshooting common issues.

GitHub Issues

Report bugs, request features, or browse existing issues for solutions.

Community Forum

Connect with other CrewAI users, share experiences, and get help from the community.

Best Practices for API Key Security:

Use environment variables or secure vaults
Never commit keys to version control
Rotate keys regularly
Use separate keys for development and production
Monitor key usage for unusual patterns

Get Started

Core Concepts

How to Guides

Tools

Telemetry

What are LLMs?

LLM Basics

Context Window

Temperature

Provider Selection

Available Models and Their Capabilities

Setting Up Your LLM

Advanced Features and Optimization

Provider Configuration Examples

Structured LLM Calls

Common Issues and Solutions

Getting Help

LiteLLM Documentation

GitHub Issues

Community Forum

Get Started

Core Concepts

How to Guides

Tools

Telemetry

​What are LLMs?

LLM Basics

Context Window

Temperature

Provider Selection

​Available Models and Their Capabilities

​Setting Up Your LLM

​Advanced Features and Optimization

​Provider Configuration Examples

​Structured LLM Calls

​Common Issues and Solutions

​Getting Help

LiteLLM Documentation

GitHub Issues

Community Forum

What are LLMs?

Available Models and Their Capabilities

Setting Up Your LLM

Advanced Features and Optimization

Provider Configuration Examples

Structured LLM Calls

Common Issues and Solutions

Getting Help