> ## Documentation Index
> Fetch the complete documentation index at: https://docs.bland.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Scrape Websites

> Creates a new knowledge base item by scraping content from web URLs.

Create a knowledge base item by scraping content from one or more web URLs. This is perfect for creating knowledge base items from documentation sites, blog posts, or other web-based content.

### Headers

**authorization** `string` *required*

Your API key for authentication.

**content-type** `string` *required*

Must be `application/json` for web scraping knowledge base items.

### Body Parameters

**type** `string` *required* (default: web)

Must be `"web"` for web scraping knowledge base items.

**name** `string` *required*

Name for the knowledge base item.

**description** `string`

Optional description of the knowledge base item content.

**urls** `string[]` *required*

Array of URLs to scrape (maximum 100 URLs).

### Response

**data** `object`

The created knowledge base item object.

**data.id** `string`

Unique identifier for the knowledge base item.

**data.name** `string`

Name of the knowledge base item.

**data.description** `string`

Description of the knowledge base item (if provided).

**data.status** `string`

Current status: `"PROCESSING"`, `"COMPLETED"`, `"FAILED"`, or `"DELETED"`.

**data.type** `string`

Will be `"WEB_SCRAPE"` for web scraping knowledge base items.

**data.source\_urls** `string`

Source URLs that were scraped (comma-separated).

**data.base\_url** `string`

Base URL derived from the provided URLs.

**data.created\_at** `string`

ISO timestamp of creation.

**data.updated\_at** `string`

ISO timestamp of last update.

**data.error\_message** `string`

Error message if status is `"FAILED"`.

**errors** `null`

Will be `null` on successful creation.

```bash cURL theme={null}
curl -X POST https://api.bland.ai/v1/knowledge/learn \
  -H "authorization: Bearer YOUR_API_KEY" \
  -H "content-type: application/json" \
  -d '{
    "type": "web",
    "name": "Documentation",
    "description": "Complete product documentation",
    "urls": [
      "https://docs.example.com/overview",
      "https://docs.example.com/api-reference",
      "https://docs.example.com/tutorials"
    ]
  }'
```

```json Single URL theme={null}
{
  "type": "web",
  "name": "Blog Posts",
  "description": "Latest company blog posts",
  "urls": [
    "https://example.com/blog/latest-features"
  ]
}
```

```json Multiple URLs theme={null}
{
  "type": "web",
  "name": "Support Documentation",
  "description": "Customer support and FAQ pages",
  "urls": [
    "https://support.example.com/faq",
    "https://support.example.com/troubleshooting",
    "https://support.example.com/getting-started",
    "https://support.example.com/advanced-features"
  ]
}
```

```json Success Response theme={null}
{
  "data": {
    "id": "kb_01H8X9QK5R2N7P3M6Z8W4Y1V7V",
    "name": "Documentation",
    "description": "Complete product documentation",
    "status": "PROCESSING",
    "type": "WEB_SCRAPE",
    "source_urls": "https://docs.example.com/overview,https://docs.example.com/api-reference,https://docs.example.com/tutorials",
    "base_url": "https://docs.example.com",
    "created_at": "2025-01-15T12:00:00Z",
    "updated_at": "2025-01-15T12:00:00Z"
  },
  "errors": null
}
```

```json Error Response theme={null}
{
  "data": null,
  "errors": [
    {
      "error": "INVALID_INPUT",
      "message": "URLs array cannot exceed 100 items"
    }
  ]
}
```

***

Docs for agents: [llms.txt](https://docs.bland.ai/llms.txt)


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.