> ## Documentation Index > Fetch the complete documentation index at: https://www.cometchat.com/docs/llms.txt > Use this file to discover all available pages before exploring further. > Retrieves all URLs discovered during a website crawl. Use this endpoint to review which pages were found before processing them. # Get Discovered URLs For the complete error reference, see [Error Guide](/rest-api/ai-agents-apis/error-codes). ## OpenAPI ````yaml get /ai-agents/agent-builder/knowledge-base/website/{parentUniqueId}/discovered-urls openapi: 3.0.0 info: title: AI Agents APIs description: API reference for CometChat AI Agents service version: '1.0' servers: - url: https://{appId}.api-{region}.cometchat.io/v3 variables: appId: default: appId description: (Required) App ID region: enum: - us - eu - in default: us description: Select Region security: [] tags: - name: ai-agent description: '' paths: /ai-agents/agent-builder/knowledge-base/website/{parentUniqueId}/discovered-urls: get: tags: - crawl-web-pages summary: Get Discovered URLs description: >- Retrieves all URLs discovered during a website crawl. Use this endpoint to review which pages were found before processing them. operationId: CrawlWebPagesController_getDiscoveredUrls parameters: - name: parentUniqueId required: true in: path schema: type: string - name: cursor required: false in: query description: To be used with affix for pagination. schema: example: '1701789000000' type: string - name: limit required: false in: query description: Maximum number of results to return per page. schema: example: 20 type: number - name: affix required: false in: query description: >- To paginate over results, use the value of affix as prepend or append. schema: enum: - append - prepend type: string - name: status required: false in: query description: Filter by URL processing status schema: enum: - discovered - processing - processed - failed - skipped type: string - name: startTime required: false in: query description: >- Start timestamp (Unix timestamp in milliseconds) to filter records created after this time schema: example: 1701780000000 type: number - name: endTime required: false in: query description: >- End timestamp (Unix timestamp in milliseconds) to filter records created before this time schema: example: 1701790000000 type: number - name: parentId required: true in: path description: Unique ID of the parent crawl (from WebPages collection) schema: example: uuid-1234-5678-9abc responses: '200': description: Discovered URLs retrieved successfully with pagination metadata content: application/json: schema: type: object properties: data: type: array description: Array of discovered URL records items: type: object properties: urlId: type: string description: Unique identifier for the discovered URL example: uuid-abcd-1234 url: type: string description: The discovered URL example: https://docs.example.com/guide title: type: string description: Page title extracted from the URL example: Getting Started Guide description: type: string description: Page description or meta description example: Learn how to get started... content: type: string description: Page content preview (truncated to 500 characters) example: Page content preview... status: type: string description: Processing status of the URL example: processed lastCrawledAt: type: number description: Timestamp when URL was last crawled example: 1701789123456 retryCount: type: number description: Number of retry attempts example: 0 createdAt: type: number description: Creation timestamp example: 1701789000000 updatedAt: type: number description: Last update timestamp example: 1701789123456 vectorizationStatus: type: string description: Status of content vectorization example: completed chunksCreated: type: number description: Number of text chunks created example: 5 embeddingsGenerated: type: number description: Number of embeddings generated example: 5 meta: type: object description: Pagination metadata for navigation properties: current: type: object properties: limit: type: number description: Records per page limit used for this request count: type: number description: Number of records returned in this page previous: type: object description: >- Navigation info for previous page (newer records). Present only if there are newer records available. properties: affix: type: string enum: - prepend description: Use "prepend" to get previous/newer records cursor: type: string description: >- Cursor value to use in next request for previous page next: type: object description: >- Navigation info for next page (older records). Present only if there are older records available. properties: affix: type: string enum: - append description: Use "append" to get next/older records cursor: type: string description: Cursor value to use in next request for next page example: data: - urlId: uuid-abcd-1234 url: https://docs.example.com/guide title: Getting Started Guide description: Learn how to get started... content: Page content preview... status: processed lastCrawledAt: 1701789123456 retryCount: 0 createdAt: 1701789000000 updatedAt: 1701789123456 vectorizationStatus: completed chunksCreated: 5 embeddingsGenerated: 5 meta: current: limit: 10 count: 1 previous: affix: prepend cursor: '1701789000000' next: affix: append cursor: '1701789000000' security: - apiKey: [] components: securitySchemes: apiKey: type: apiKey description: API Key (i.e. Rest API Key from the Dashboard). name: apikey in: header ````