Connecting Amazon Q Business to Web Crawler using the console - Amazon Q Business

Connecting Amazon Q Business to Web Crawler using the console

On the Web Crawler page, enter the following information:

  1. Name – Name your data source for easy tracking.

    Note: You can include hyphens (-) but not spaces. Maximum of 1,000 alphanumeric characters.

  2. In Source choose from the following options:

    • Source URLs – Add up to 10 seed/starting point URLs of the websites you want to crawl. You can also include website subdomains.

    • Source sitemaps – Add up to 3 sitemap URLs of the websites you want to crawl.

    • Source URLs file – Add up to 100 seed/starting point URLs listed in a text file in Amazon S3. Each URL should be on a separate line in the text file.

    • Source sitemaps file – Add up to 3 sitemap XML files stored in Amazon S3. You can also zip the XML files.

    Note

    If you choose to use a text file that includes a list of up to 100 seed URLs or to use a sitemap XML file, you specify the path to the Amazon S3 bucket where your file is stored.

    You can also combine multiple sitemap XML files into a .zip file. Otherwise, you can manually enter up to 10 seed or starting point URLs, and up to three sitemap URLs.

    Note

    If you want to crawl a sitemap, check that the base or root URL is the same as the URLs listed on your sitemap page. For example, if your sitemap URL is https://example.com/sitemap-page.html, the URLs listed on this sitemap page should also use the base URL "https://example.com/".

    Note

    If you want to later edit your data source to change your seed URLs with authentication to sitemaps, you must create a new data source.

    Amazon Q configures the data source using the seed URLs endpoint information in the Secrets Manager secret for authentication. Therefore, Amazon Q can't reconfigure the data source when changing to sitemaps.

  3. In Authentication, choose the type of authentication you want to use and enter the following information in your AWS Secrets Manager secret:

    • No authentication – Choose to crawl a public website without any authentication.

    • Basic authentication – Enter a name for the secret, plus the username and password

    • NTLM/Kerberos authentication – Enter a name for the secret, plus the username and password. NTLM authentication protocol includes password hashing, and Kerberos authentication protocol includes password encryption

    • Form authentication – Enter a name for the secret, and the username and password. Use XPath for the username field. Use XPaths for the password field and button, and login page URL. You can find the XPaths (XML Path Language) of elements using your web browser's developer tools. XPaths usually follow this format: //tagname[@Attribute='Value']

    • SAML authentication – Enter a name for the secret, plus the username and password. Use XPath for the username field and for the username button. Use XPaths for the password field and button, and login page URL. You can find the XPaths (XML Path Language) of elements using your web browser's developer tools. XPaths usually follow this format: //tagname[@Attribute='Value']

  4. Web proxy – optional – Enter the host name and the port number of the proxy server that you want to use to connect to internal websites. For example, the host name of https://a.example.com/page1.html is "a.example.com" and the port number is 443, the standard port for HTTPS. If web proxy credentials are required to connect to a website host, you can create an AWS Secrets Manager secret that stores the credentials.

  5. Configure VPC and security group – optional – Choose whether you want to use a VPC. If you do, enter the following information:

    1. Subnets – Select up to 6 repository subnets that define the subnets and IP ranges the repository instance uses in the selected VPC.

    2. VPC security groups – Choose up to 10 security groups that allow access to your data source. Ensure that the security group allows incoming traffic from Amazon EC2 instances and devices outside your VPC. For databases, security group instances are required.

    For more information, see VPC.

  6. IAM role – Choose an existing IAM role or create an IAM role to access your repository credentials and index content.

    For more information, see IAM role.

  7. In Sync scope, enter the following information:

    1. Sync domain range – Choose whether to sync website domains with subdomains only, or also crawl other domains that the webpages link to (Sync everything). By default, Amazon Q only syncs the domains of the websites that you want to crawl.

    2. For Maximum single file size – Specify the file size limit in MBs that Amazon Q will crawl. Amazon Q will crawl only the files within the size limit you define. The default file size is 50MB. The maximum file size should be greater than 0MB and less than or equal to 50MB.

    3. In Additional configuration – optional – Configure the following settings:

      • Scope settings, choose from the following:

        • Crawl depth – The depth, or number, of levels from the seed level to crawl. For example, the seed URL page is depth 1 and any hyperlinks on this page that are also crawled are depth 2.

        • Maximum single file size – The maximum size in MB of a webpage or attachment to crawl.

        • Maximum links per page – The maximum number of URLs on a single webpage to crawl.

        • Maximum throttling – The maximum number of URLs crawled per website host per minute.

      • Include files that web pages link to – Choose to crawl files that the webpages link to.

      • Crawl URL patterns – Add regular expression patterns to include or exclude crawling specific URLs, and indexing any hyperlinks on these URL webpages.

      • URL pattern to index – Add regular expression patterns to include or exclude crawling specific URLs, and indexing any hyperlinks on these URL webpages.

  8. In Sync mode, choose how you want to update your index when your data source content changes. When you sync your data source with Amazon Q for the first time, all content is synced by default.

    • Full sync – Sync all content regardless of the previous sync status.

    • New, modified, or deleted content sync – Sync only new, modified, and deleted documents.

  9. In Sync run schedule, for Frequency – Choose how often Amazon Q will sync with your data source. For more details, see Sync run schedule.

  10. Tags - optional – Add tags to search and filter your resources or track your AWS costs. See Tags for more details.

  11. Field mappings – A list of data source document attributes to map to your index fields. Add the fields from the Data source details page after you finish adding your data source. You can choose from two types of fields:

    1. Default – Automatically created by Amazon Q on your behalf based on common fields in your data source. You can't edit these.

    2. Custom – Automatically created by Amazon Q on your behalf based on common fields in your data source. You can edit these. You can also create and add new custom fields.

      Note

      Support for adding custom fields varies by connector. You won't see the Add field option if your connector doesn't support adding custom fields.

    For more information, see Field mappings.

  12. To finish connecting your data source to Amazon Q, select Add data source.

    You are taken to the Data source details, where you can view your data source configuration details.

  13. In Data source details, choose Sync now to allow Amazon Q to begin syncing (crawling and ingesting) data from your data source. When the sync job finishes, your data source is ready to use.

    Note

    You can also choose to view CloudWatch logs for your data source sync job by selecting View CloudWatch logs. If you get a Resource not found exception when you try to view your CloudWatch logs for a data source sync job in progress, it can be because the CloudWatch logs are not available yet. Wait for some time and check again.