LLMs

CrewAI integrates with multiple LLM providers through LiteLLM, giving you the flexibility to choose the right model for your specific use case. This guide will help you understand how to configure and use different LLM providers in your CrewAI projects.

What are LLMs?

Large Language Models (LLMs) are the core intelligence behind CrewAI agents. They enable agents to understand context, make decisions, and generate human-like responses. Here’s what you need to know:

LLM Basics

Large Language Models are AI systems trained on vast amounts of text data. They power the intelligence of your CrewAI agents, enabling them to understand and generate human-like text.

Context Window

The context window determines how much text an LLM can process at once. Larger windows (e.g., 128K tokens) allow for more context but may be more expensive and slower.

Temperature

Temperature (0.0 to 1.0) controls response randomness. Lower values (e.g., 0.2) produce more focused, deterministic outputs, while higher values (e.g., 0.8) increase creativity and variability.

Provider Selection

Each LLM provider (e.g., OpenAI, Anthropic, Google) offers different models with varying capabilities, pricing, and features. Choose based on your needs for accuracy, speed, and cost.

Available Models and Their Capabilities

Here’s a detailed breakdown of supported models and their capabilities, you can compare performance at lmarena.ai and artificialanalysis.ai:

Model	Context Window	Best For
GPT-4	8,192 tokens	High-accuracy tasks, complex reasoning
GPT-4 Turbo	128,000 tokens	Long-form content, document analysis
GPT-4o & GPT-4o-mini	128,000 tokens	Cost-effective large context processing

1 token ≈ 4 characters in English. For example, 8,192 tokens ≈ 32,768 characters or about 6,000 words.

Setting Up Your LLM

There are three ways to configure LLMs in CrewAI. Choose the method that best fits your workflow:

The simplest way to get started. Set these variables in your environment:

# Required: Your API key for authentication
OPENAI_API_KEY=<your-api-key>

# Optional: Default model selection
OPENAI_MODEL_NAME=gpt-4o-mini  # Default if not set

# Optional: Organization ID (if applicable)
OPENAI_ORGANIZATION_ID=<your-org-id>

Never commit API keys to version control. Use environment files (.env) or your system’s secret management.

Advanced Features and Optimization

Learn how to get the most out of your LLM configuration:

CrewAI includes smart context management features:

from crewai import LLM

# CrewAI automatically handles:
# 1. Token counting and tracking
# 2. Content summarization when needed
# 3. Task splitting for large contexts

llm = LLM(
    model="gpt-4",
    max_tokens=4000,  # Limit response length
)

Best practices for context management:

Choose models with appropriate context windows
Pre-process long inputs when possible
Use chunking for large documents
Monitor token usage to optimize costs

Token Usage Optimization

Choose the right context window for your task:

Small tasks (up to 4K tokens): Standard models
Medium tasks (between 4K-32K): Enhanced models
Large tasks (over 32K): Large context models

# Configure model with appropriate settings
llm = LLM(
    model="openai/gpt-4-turbo-preview",
    temperature=0.7,    # Adjust based on task
    max_tokens=4096,    # Set based on output needs
    timeout=300        # Longer timeout for complex tasks
)

Lower temperature (0.1 to 0.3) for factual responses
Higher temperature (0.7 to 0.9) for creative tasks

Best Practices

Monitor token usage
Implement rate limiting
Use caching when possible
Set appropriate max_tokens limits

Remember to regularly monitor your token usage and adjust your configuration as needed to optimize costs and performance.

Provider Configuration Examples

Code
# Required
OPENAI_API_KEY=sk-...

# Optional
OPENAI_API_BASE=<custom-base-url>
OPENAI_ORGANIZATION=<your-org-id>

Example usage:

Code
from crewai import LLM

llm = LLM(
    model="gpt-4",
    temperature=0.8,
    max_tokens=150,
    top_p=0.9,
    frequency_penalty=0.1,
    presence_penalty=0.1,
    stop=["END"],
    seed=42
)

Code
# Option 1. Gemini accessed with an API key.
# https://ai.google.dev/gemini-api/docs/api-key
GEMINI_API_KEY=<your-api-key>

# Option 2. Vertex AI IAM credentials for Gemini, Anthropic, and anything in the Model Garden.
# https://cloud.google.com/vertex-ai/generative-ai/docs/overview

Example usage:

Code
llm = LLM(
    model="gemini/gemini-1.5-pro-latest",
    temperature=0.7
)

Code
# Required
AZURE_API_KEY=<your-api-key>
AZURE_API_BASE=<your-resource-url>
AZURE_API_VERSION=<api-version>

# Optional
AZURE_AD_TOKEN=<your-azure-ad-token>
AZURE_API_TYPE=<your-azure-api-type>

Example usage:

Code
llm = LLM(
    model="azure/gpt-4",
    api_version="2023-05-15"
)

Code
AWS_ACCESS_KEY_ID=<your-access-key>
AWS_SECRET_ACCESS_KEY=<your-secret-key>
AWS_DEFAULT_REGION=<your-region>

Example usage:

Code
llm = LLM(
    model="bedrock/anthropic.claude-3-sonnet-20240229-v1:0"
)

Code
# Required
WATSONX_URL=<your-url>
WATSONX_APIKEY=<your-apikey>
WATSONX_PROJECT_ID=<your-project-id>

# Optional
WATSONX_TOKEN=<your-token>
WATSONX_DEPLOYMENT_SPACE_ID=<your-space-id>

Example usage:

Code
llm = LLM(
    model="watsonx/meta-llama/llama-3-1-70b-instruct",
    base_url="https://api.watsonx.ai/v1"
)

Code
# Required
CEREBRAS_API_KEY=<your-api-key>

Example usage:

Code
llm = LLM(
    model="cerebras/llama3.1-70b",
    temperature=0.7,
    max_tokens=8192
)

Cerebras features:

Fast inference speeds
Competitive pricing
Good balance of speed and quality
Support for long context windows

Common Issues and Solutions

Most authentication issues can be resolved by checking API key format and environment variable names.

# OpenAI
OPENAI_API_KEY=sk-...

# Anthropic
ANTHROPIC_API_KEY=sk-ant-...

Getting Help

If you need assistance, these resources are available:

LiteLLM Documentation

Comprehensive documentation for LiteLLM integration and troubleshooting common issues.

GitHub Issues

Report bugs, request features, or browse existing issues for solutions.

Community Forum

Connect with other CrewAI users, share experiences, and get help from the community.

Best Practices for API Key Security:

Use environment variables or secure vaults
Never commit keys to version control
Rotate keys regularly
Use separate keys for development and production
Monitor key usage for unusual patterns

Get Started

Core Concepts

How to Guides

Tools

Telemetry

What are LLMs?

LLM Basics

Context Window

Temperature

Provider Selection

Available Models and Their Capabilities

Setting Up Your LLM

Advanced Features and Optimization

Provider Configuration Examples

Common Issues and Solutions

Getting Help

LiteLLM Documentation

GitHub Issues

Community Forum

Get Started

Core Concepts

How to Guides

Tools

Telemetry

​What are LLMs?

LLM Basics

Context Window

Temperature

Provider Selection

​Available Models and Their Capabilities

​Setting Up Your LLM

​Advanced Features and Optimization

​Provider Configuration Examples

​Common Issues and Solutions

​Getting Help

LiteLLM Documentation

GitHub Issues

Community Forum

What are LLMs?

Available Models and Their Capabilities

Setting Up Your LLM

Advanced Features and Optimization

Provider Configuration Examples

Common Issues and Solutions

Getting Help