Skip to main content

GET /v1/crawl/

Retrieve the status and results of a crawl job using its ID.

Authentication

This endpoint requires authentication using a Bearer token. Include your API key in the Authorization header:

Path Parameters

string
required
The ID of the crawl job (UUID format)

Response

string
The current status of the crawl. Can be scraping, completed, or failed.
integer
The total number of pages that were attempted to be crawled.
integer
The number of pages that have been successfully crawled.
integer
The number of credits used for the crawl.
string
The date and time when the crawl will expire (ISO 8601 format).
string
The URL to retrieve the next 10MB of data. Returned if the crawl is not completed or if the response is larger than 10MB.
array
The crawled page data. Each item contains:

Example Request

Example Response

Error Responses

Pagination

If the response data exceeds 10MB, the next field will contain a URL to retrieve the next batch of results. Continue fetching using the next URL until it becomes null.

Status Values

  • scraping: The crawl is currently in progress
  • completed: The crawl has finished successfully
  • failed: The crawl encountered an error and stopped