A multithreaded 🕸️ web crawler that recursively crawls a website and creates a 🔽 markdown file for each page, designed for LLM RAG
markdown html-to-markdown markdown-parser web-scraper rag html2md llm llmops html-to-markdown-converter markdown-scraper markdown-crawler md-crawler
-
Updated
Aug 13, 2024 - Python