Skip to main content

​
Overview

The URLReadTool reads the content at a URL and returns it as text. PDF, DOCX and XLSX bodies have their text extracted, HTML is reduced to visible text, and text-based types (plain text, Markdown, JSON, XML, YAML, CSV) are decoded as-is. It works with presigned cloud-storage links (S3, R2, Google Drive, OneDrive, SharePoint) even when the server labels the file application/octet-stream. The tool does not authenticate, so the link must already grant access.

​
Installation

​
Usage Example

Code

​
Arguments

The agent supplies these at runtime:
  • url: The URL to read.
  • start_line: (Optional) The line to start from (1-indexed). Defaults to 1.
  • line_count: (Optional) The number of lines to return. If omitted, returns everything from start_line.
You set these when constructing the tool:
  • max_bytes: (Optional) Largest response body to accept. Defaults to 5 MiB.
  • timeout: (Optional) Per-request timeout in seconds. Defaults to 30.
  • headers: (Optional) Extra request headers.
  • encoding: (Optional) Force a text encoding instead of the one the server declares.
If the fetch fails, the tool returns a failure that includes the reason, such as a blocked address or a connection error.

​
Network safety

Because the URL is usually chosen by an LLM at runtime, requests are checked against SSRF:
  • Each hostname is resolved, and the request is rejected if any resolved address is private, loopback, link-local or otherwise reserved. This covers cloud metadata endpoints and internal services.
  • Redirects are not followed automatically. Every hop is checked again, and credentials are dropped on cross-origin hops.

​
Allowing an internal host

Some deployments serve files from a host that resolves to a private address, such as a self-hosted platform behind an internal load balancer. To permit it, list its exact hostname in CREWAI_TOOLS_ALLOWED_PRIVATE_HOSTS:
Examples of what each value allows, for a request to https://files.internal.example.com/report.pdf:
  • Entries are comma-separated. Each can be a bare hostname, host:port or a full URL. Only the hostname is used, and it is matched exactly and case-insensitively. Wildcards such as *.example.com are not supported.
  • IP addresses are never allowed through this list.
  • A listed host may resolve to any private address, so list only hostnames whose DNS you control and trust.
  • Redirects are still checked, so a redirect to another internal host must be listed as well.
  • The setting applies to every crewai-tools tool that makes safe requests, not only URLReadTool.
  • TLS certificates are still verified. A host that uses a private certificate authority must chain to a CA the tool trusts.
This is narrower than CREWAI_TOOLS_ALLOW_UNSAFE_PATHS=true, which disables path and SSRF protection process-wide and should be avoided. The allowlist is ignored while CREWAI_TOOLS_FORCE_SAFE_PATHS=true is set, because that lock exists so a deployment cannot relax these checks. To use the allowlist, leave CREWAI_TOOLS_FORCE_SAFE_PATHS unset, set CREWAI_TOOLS_ALLOWED_PRIVATE_HOSTS, and remove CREWAI_TOOLS_ALLOW_UNSAFE_PATHS, since that flag takes effect again once the lock is off. Do this only on workers you operate, not on shared or multi-tenant workers.