Scrape multiple webpages in a single batch request. This endpoint is ideal for bulk extraction jobs where you need to process multiple URLs with shared options.

🧰 Using with SDKs

Prefer code over curl? Crawlio offers official SDKs for seamless integration with your stack:

Node.js SDK (npm) – Perfect for backend automation, agents, and JS projects.
Python SDK (PyPI) – Ideal for data science, AI/ML workflows, and scripting.

📖 View full usage docs: 👉 Node.js SDK Docs 👉 Python SDK Docs

We are working on an extensive documentation on our SDKs. Thanks for your cooperation!

Cost

Name	Cost	Type
Scrape	Number of scraped pages	Scrape

📦 `POST /scrape/batch`

Scrape multiple webpages in a single batch request. This endpoint is ideal for bulk extraction jobs where you need to process multiple URLs with shared options.

📥 Request

Endpoint:

POST https://crawlio.xyz/api/scrape/batch

Headers:

Authorization: Bearer YOUR_API_KEY  
Content-Type: application/json

Request Body Parameters:

Field	Type	Required	Description
`url`	`array of strings`	✅ Yes	A list of URLs to scrape in one batch.
`options`	`object`	❌ No	Additional config such as headers, timeouts, etc. (future use)
`exclude`	`array of strings`	❌ No	CSS selectors to remove elements from all pages in the batch.
`markdown`	`boolean`	❌ No	Return scraped content in Markdown format for each page.

🧾 Example Request

POST /scrape/batch
Content-Type: application/json
Authorization: Bearer YOUR_API_KEY

{
  "url": [
    "https://example.com/page1",
    "https://example.com/page2"
  ],
  "exclude": [".ads", "footer"],
  "markdown": true
}

📤 Response

On success, Crawlio will respond with a unique batch identifier which you can use to retrieve results when the job is complete.

Field	Type	Description
`batchId`	`string`	Unique ID for the batch scraping job.

📦 Example Response

{
  "batchId": "batch_789ghi"
}

You can use the batch ID with a job result or status endpoint (if available) to monitor progress and retrieve full output once processing completes.

See Postman Collection for more detailed information on getting the batch data and job status :

What and Why?

The Batch Scrape feature is designed to let you extract content from multiple URLs in one go, making it ideal for bulk scraping tasks where efficiency and consistency are key.

Instead of sending individual requests for each page, you can group your targets and handle them together — saving time, reducing overhead, and simplifying result tracking.

Use Cases:

🛒 Scraping product detail pages from a category listing
📝 Collecting multiple blog posts or news articles
📈 Monitoring a set of competitor pages
🔄 Feeding a batch of URLs from a search or database into a scraper

Key Capabilities:

✅ Submit multiple URLs at once for scraping
🧹 Optionally exclude elements like footers, ads, or navigation bars
📝 Return content as Markdown for easier storage or post-processing
🪝 Perfect for pairing with /search or internal lists of known URLs

The /scrape/batch endpoint is a fast, scalable solution when you need to extract structured content from many pages with shared configuration.

📦 Batch Scrape