Prepare web content for Amazon Bedrock Knowledge Bases with Crawl4AI

When you build Retrieval Augmented Generation (RAG) applications, answer quality depends on the data that you ingest. Amazon Bedrock Knowledge Bases is the fully managed RAG capability of Amazon Bedrock. With Amazon Bedrock Managed Knowledge Base , you connect your data sources, and Amazon Bedrock manages ingestion, storage, indexing, and retrieval for you. You don’t provision or scale a vector store, and embedding and reranking use service-managed models by default. For web content, start with the Amazon Bedrock Web Crawler data source connector for managed knowledge bases. You give it the seed URLs or sitemaps that you’re authorized to crawl, and it crawls within the scope and rate limits that you set. Scheduled syncs keep your knowledge base current as pages change, and the connector can wait for content that loads with JavaScript. There’s no crawler for you to build, host, or operate, so this connector is the starting point for most public website content. Some sources need more control over what reaches the knowledge base. You might want to index only the main content region of each page, remove navigation and other boilerplate, or attach your own metadata attributes to every document. For that class of source material, you can extend the reach of your knowledge base with a preparation step built with Crawl4AI , an open source Python crawler. The step writes Markdown and a metadata sidecar for each page to Amazon Simple Storage Service (Amazon S3), and the knowledge base ingests both through its Amazon S3 data source. In this post, we show you how to choose the appropriate web ingestion path and implement custom preparation for content that needs additional processing. You then start ingestion into a managed knowledge base and validate retrieval quality. The companion aws-samples repository provides the complete workflow as a sample implementation. This is batch ingestion of known web properties, not real-time web access at inference time such as Web Grounding with Amazon Nova. Solution overview Start with the Amazon Bedrock Web Crawler when it can reach and index the pages you need. It respects robots.txt , and after the first full crawl, each sync captures only added, updated, and deleted content. Use the Crawl4AI preparation path when public source material needs one or more of the following before ingestion: Structured Markdown preparation : Use DefaultMarkdownGenerator with a pruning content filter to reduce navigation and other boilerplate while preserving headings, lists, and tables. Because pruning is heuristic, review representative output from your sites. Bounded, targeted extraction : Extract only the page region that an optional CSS selector matches. Constrain breadth-first traversal by seed origin, page/path/host scope, depth and page caps, and URL patterns. Custom metadata : Attach source URLs, crawl timestamps, content type, and your own attributes to each document, so that you can filter retrieval results on them. Because managed and customer-managed knowledge bases both support Amazon S3 data sources, the Crawl4AI path works with either type. This post uses a managed knowledge base and notes where a customer-managed knowledge base differs. The rest of this post follows the companion sample. Treat its architecture and defaults as a starting point, not as requirements of Amazon Bedrock Knowledge Bases. By default, the sample runs the AWS Fargate task in a public subnet, where it reaches your configured websites through an internet gateway. Crawl4AI’s role ends once it has prepared the source documents and metadata sidecars. From ingestion onward, Amazon Bedrock Knowledge Bases handles parsing, chunking, embedding, indexing, and retrieval. The following diagram shows the Crawl4AI path from the schedule through to retrieval. Figure 1: The Crawl4AI-to-Amazon S3 ingestion path for Amazon Bedrock Knowledge Bases The numbered steps in the diagram map to the following workflow: Amazon EventBridge starts the Amazon Elastic Container Service (Amazon ECS) task on AWS Fargate on the configured schedule. The task crawls the public websites you configured and prepares Markdown with a metadata sidecar for each page. The task uploads the prepared files to the Amazon S3 bucket that backs the knowledge base data source. The task starts an Amazon Bedrock Knowledge Bases ingestion job. Amazon Bedrock Knowledge Bases reads the new and changed objects, parses them, and applies the chunking strategy you selected. It creates embeddings and indexes them in the knowledge base’s vector store, which Amazon Bedrock manages for a managed knowledge base. Your generative AI application queries the knowledge base for relevant chunks or for a generated response with citations. The task reads its non-secret pipeline configuration from AWS Systems Manager Parameter Store at startup. The workflow processes public pages only and doesn’t implement authenticated sessions. For sites that require sign-in, evaluate the Amazon Bedrock Web Crawler authentication options first, which include basic, form-based, and SAML authentication. Extending the sample with authentication requires session handling, credentials in AWS Secrets Manager, and a security review. How the workflow works The AWS Cloud Development Kit (AWS CDK) app packages the implementation in a container image during deployment. When the schedule fires, the container runs scripts/container_entrypoint.py and executes crawl, upload, and ingestion in sequence. The following three steps explain the core logic. Step 1: Crawl and extract content The crawl step performs bounded crawling and writes a Markdown file with a metadata sidecar for each successfully prepared page. The following excerpt is simplified. The module src/crawl4ai_bedrock_kb/crawl.py contains the complete seed pacing, URL filtering and scoping, page limits, and redirect checks: async def crawl_and_prepare(config: PipelineConfig, output_dir: str) -> list[str]: written, seen = [], set() pages_crawled = 0 browser_config = BrowserConfig(headless=True, java_script_enabled=True) async with AsyncWebCrawler(config=browser_config) as crawler: for seed in config.seed_urls: remaining = config.crawl.max_pages - pages_crawled if remaining = config.crawl.max_pages: break pages_crawled += 1 requested_url = result.url source_url = getattr(result, "redirected_url", None) or requested_url if not SeedScopeFilter(seed, config.crawl.scope).apply(source_url): continue # Recheck the effective destination after redirects. canonical = _canonical_url(source_url) if canonical in seen: continue seen.add(canonical) if not result.success or not result.markdown: continue md_result = result.markdown markdown = ( getattr(md_result, "fit_markdown", None) or getattr(md_result, "raw_markdown", None) or (md_result if isinstance(md_result, str) else "") ) if not markdown: continue # The complete implementation adds title, crawl timestamps, content # type, and configured custom attributes to this metadata. written.extend( _write_document( output_dir, _base_name(source_url), markdown, {"source_url": source_url}, ) ) return written Amazon Bedrock Knowledge Bases supports several document formats for Amazon S3 data sources, including plain text, HTML, Markdown, PDF, and DOCX. The workflow writes Markdown because it preserves headings, lists, and tables without carrying all raw HTML into ingestion. Step 2: Upload to Amazon S3 The upload step sends each prepared Markdown document and its matching metadata sidecar to the configured S3 bucket, under the workflow’s s3Prefix . For page.md , the sidecar must be named page.md.metadata.json , the complete source filename with .metadata.json appended. The workflow stores each sidecar in the same S3 location as its document. Amazon Bedrock limits each metadata sidecar to 10 KB. If a sidecar would exceed that limit, the run stops with an error before it writes either file. Each attribute in the sidecar uses the typed format that both managed and custo