🪝 Webhooks

Crawlio supports webhooks for notifying your server when important events occur — such as the completion of crawl or scrape jobs. With webhooks, you can automate your workflows, trigger pipelines, or log job outcomes in real time.

🔧 Setup Webhooks

To start receiving webhook events:

Go to the Dashboard → Settings → Webhooks.
Enter your server endpoint (must support HTTPS).
Copy your Signing Secret — used to verify authenticity of incoming payloads.

✅ Tip: Each user has a unique signing secret. Keep this secret safe and do not expose it publicly.

📬 Events

Crawlio sends webhook events for the following job types:

Type	Description	Payload Contents
`crawl`	When a crawling job completes	`crawlId`, `status`, `type`
`scrape`	When a scraping job completes	Full job result data
`batch-scrape`	When a batch scrape finishes processing	`batchId`, `status`, `type`

🧾 Payload Structure

All webhook payloads follow the structure below:

interface WebhookEventPayload {
  userId: string;
  type: "crawl" | "scrape" | "batch-scrape";
  data: object; // varies by type
  status: "success" | "failed" | "errored";
}

Example Payload: `scrape`

{
  "type": "scrape",
  "status": "success",
  "data": {...},
  "jobId": "asdhasdhj",
  "request" : {...} // options used for initiating the scrape request
}

Example Payload: `crawl`

{
  "type": "crawl",
  "status": "success",
  "crawlId": "asjdkw",
  "request" : {...} // options used for initiating the crawl request
}

Example Payload: `batch-scrape`

{
  "type": "batch-scrape",
  "status": "success",
  "batchId": "asjdkw",
  "request" : {...} // options used for initiating the batch scrape request
}

🔐 Authenticating Webhooks

Each webhook request is signed using your unique secret key and the HMAC SHA-256 algorithm.

Signature Header

Each request includes an X-Signature header:

X-Signature: <sha256-hmac-hex>

Signature Generation

const crypto = require("crypto");

const signature = crypto
  .createHmac("sha256", SIGNING_SECRET)
  .update(JSON.stringify(payload))
  .digest("hex");

Verify this signature on your server to ensure the request came from Crawlio.

📥 Handling Webhooks

Here’s a simple Node.js/Express example of how to handle and verify a webhook:

import express from "express";
import crypto from "crypto";

const app = express();
app.use(express.json());

const SIGNING_SECRET = process.env.CRAWLIO_SECRET;

app.post("/webhook", (req, res) => {
  const payload = JSON.stringify(req.body);
  const signature = req.headers["x-signature"];

  const expectedSignature = crypto
    .createHmac("sha256", SIGNING_SECRET)
    .update(payload)
    .digest("hex");

  if (signature !== expectedSignature) {
    return res.status(403).send("Invalid signature");
  }

  // Process the event
  console.log("Received webhook:", req.body);
  res.status(200).send("OK");
});

app.listen(3000, () => console.log("Listening for webhooks"));

🧪 Testing Webhooks

To test your webhook integration:

Use a tool like Webhook.site or ngrok to expose a local server.
Trigger test jobs from the Crawlio Dashboard and verify receipt and handling.

📘 Summary

Feature	Supported
Event Types	`crawl`, `scrape`, `batch-scrape`
Signature Headers	✅
Custom Payloads	✅ (by event type)

🪝 Webhooks

On this page