> ## Documentation Index
> Fetch the complete documentation index at: https://www.cometchat.com/docs/llms.txt
> Use this file to discover all available pages before exploring further.

> Marks discovered URLs for exclusion and initiates batch crawling of the remaining URLs. Use this endpoint to selectively process pages discovered during a website crawl.

# Process Discovered URLs

For the complete error reference, see [Error Guide](/rest-api/ai-agents-apis/error-codes).


## OpenAPI

````yaml post /ai-agents/agent-builder/knowledge-base/website/{parentUniqueId}/discovered-urls/process
openapi: 3.0.0
info:
  title: AI Agents APIs
  description: API reference for CometChat AI Agents service
  version: '1.0'
servers:
  - url: https://{appId}.api-{region}.cometchat.io/v3
    variables:
      appId:
        default: appId
        description: (Required) App ID
      region:
        enum:
          - us
          - eu
          - in
        default: us
        description: Select Region
security: []
tags:
  - name: ai-agent
    description: ''
paths:
  /ai-agents/agent-builder/knowledge-base/website/{parentUniqueId}/discovered-urls/process:
    post:
      tags:
        - crawl-web-pages
      summary: Process Discovered URLs
      description: >-
        Marks discovered URLs for exclusion and initiates batch crawling of the
        remaining URLs. Use this endpoint to selectively process pages
        discovered during a website crawl.
      operationId: CrawlWebPagesController_processDiscoveredUrls
      parameters:
        - name: parentUniqueId
          required: true
          in: path
          schema:
            type: string
      requestBody:
        required: true
        description: >-
          Discovered URLs processing configuration. App ID is taken from header,
          not body.
        content:
          application/json:
            schema:
              $ref: '#/components/schemas/ProcessDiscoveredUrlsDto'
      responses:
        '200':
          description: Batch processing initiated successfully
          content:
            application/json:
              schema:
                type: object
                properties:
                  success:
                    type: boolean
                    example: true
                  message:
                    type: string
                    example: Batch processing initiated successfully
                  data:
                    type: object
                    properties:
                      batchId:
                        type: string
                        example: batch-1701789123456
                      urlsProcessed:
                        type: number
                        example: 2
                      totalUrls:
                        type: number
                        example: 2
                      webhookUrl:
                        type: string
                        example: >-
                          https://webhook.site/b4994000-62d9-4e17-928f-790282cb5815
                      status:
                        type: string
                        example: processing
              example:
                success: true
                message: Batch processing initiated successfully
                data:
                  batchId: batch-1701789123456
                  urlsProcessed: 2
                  totalUrls: 2
                  webhookUrl: https://webhook.site/b4994000-62d9-4e17-928f-790282cb5815
                  status: processing
      security:
        - apiKey: []
components:
  schemas:
    ProcessDiscoveredUrlsDto:
      type: object
      properties:
        exclude:
          description: Array of unique IDs to exclude from crawling
          default: []
          type: array
          items:
            type: string
      required:
        - exclude
  securitySchemes:
    apiKey:
      type: apiKey
      description: API Key (i.e. Rest API Key from the Dashboard).
      name: apikey
      in: header

````