LLMs
A comprehensive guide to configuring and using Large Language Models (LLMs) in your CrewAI projects
CrewAI integrates with multiple LLM providers through LiteLLM, giving you the flexibility to choose the right model for your specific use case. This guide will help you understand how to configure and use different LLM providers in your CrewAI projects.
What are LLMs?
Large Language Models (LLMs) are the core intelligence behind CrewAI agents. They enable agents to understand context, make decisions, and generate human-like responses. Here’s what you need to know:
LLM Basics
Large Language Models are AI systems trained on vast amounts of text data. They power the intelligence of your CrewAI agents, enabling them to understand and generate human-like text.
Context Window
The context window determines how much text an LLM can process at once. Larger windows (e.g., 128K tokens) allow for more context but may be more expensive and slower.
Temperature
Temperature (0.0 to 1.0) controls response randomness. Lower values (e.g., 0.2) produce more focused, deterministic outputs, while higher values (e.g., 0.8) increase creativity and variability.
Provider Selection
Each LLM provider (e.g., OpenAI, Anthropic, Google) offers different models with varying capabilities, pricing, and features. Choose based on your needs for accuracy, speed, and cost.
Setting Up Your LLM
There are three ways to configure LLMs in CrewAI. Choose the method that best fits your workflow:
The simplest way to get started. Set these variables in your environment:
Never commit API keys to version control. Use environment files (.env) or your system’s secret management.
The simplest way to get started. Set these variables in your environment:
Never commit API keys to version control. Use environment files (.env) or your system’s secret management.
Create a YAML file to define your agent configurations. This method is great for version control and team collaboration:
The YAML configuration allows you to:
- Version control your agent settings
- Easily switch between different models
- Share configurations across team members
- Document model choices and their purposes
For maximum flexibility, configure LLMs directly in your Python code:
Parameter explanations:
temperature
: Controls randomness (0.0-1.0)timeout
: Maximum wait time for responsemax_tokens
: Limits response lengthtop_p
: Alternative to temperature for samplingfrequency_penalty
: Reduces word repetitionpresence_penalty
: Encourages new topicsresponse_format
: Specifies output structureseed
: Ensures consistent outputs
Provider Configuration Examples
CrewAI supports a multitude of LLM providers, each offering unique features, authentication methods, and model capabilities. In this section, you’ll find detailed examples that help you select, configure, and optimize the LLM that best fits your project’s needs.
Streaming Responses
CrewAI supports streaming responses from LLMs, allowing your application to receive and process outputs in real-time as they’re generated.
Enable streaming by setting the stream
parameter to True
when initializing your LLM:
When streaming is enabled, responses are delivered in chunks as they’re generated, creating a more responsive user experience.
Enable streaming by setting the stream
parameter to True
when initializing your LLM:
When streaming is enabled, responses are delivered in chunks as they’re generated, creating a more responsive user experience.
CrewAI emits events for each chunk received during streaming:
Structured LLM Calls
CrewAI supports structured responses from LLM calls by allowing you to define a response_format
using a Pydantic model. This enables the framework to automatically parse and validate the output, making it easier to integrate the response into your application without manual post-processing.
For example, you can define a Pydantic model to represent the expected response structure and pass it as the response_format
when instantiating the LLM. The model will then be used to convert the LLM output into a structured Python object.
Advanced Features and Optimization
Learn how to get the most out of your LLM configuration:
Common Issues and Solutions
Most authentication issues can be resolved by checking API key format and environment variable names.
Most authentication issues can be resolved by checking API key format and environment variable names.
Always include the provider prefix in model names
Use larger context models for extensive tasks