Library / Code, Git & Developer Tools
Code, Git & Developer Tools
Discover Hidden Website API Endpoints with Regex and AI
When you run it, it fetches a website's HTML and JavaScript files, extracts API endpoints with regex, and uses AI models to clean and organize them.
What it does
# 1. API Endpoints Extraction With a Predefined Regex * Paste the URL of the target website into the URL field of the Configuration node. * Fetch the HTML page of the site and extract the URLs of the JS files. * Keep relevant JS files with potential API endpoints (e.g. local, bundled, CDN sources). You can customise the logic to your needs: add or remove URL conditions. * Load JS content from kept URLs: convert references to JS files into properly formatted, complete URLs. * Apply predefined regex to extract API endpoints. * Keep JS files with enough URLs (>4 endpoints). Assume that JS files with fewer URLs do not contain relevant API endpoints. Discard them to send only the necessary data to the LLM. * Clean up the extracted URLs by removing duplicated items.
The steps
- Auto-fixing Output Parser
- Start API Discovery
- Configuration
- Fetch Website HTML
- Extract URLs of JS files
- Split URLs of JS files
- Keep Relevant JS Files
- Fetch JS Content
- Extract API Endpoints
- Check Endpoints Count
- Split Endpoints
- Remove Duplicates
- AI Endpoints Analysis
- Add Source Metadata
42 steps in total.
Similar automations
Email AutomationDraft beginner-friendly AI Instagram post drafts with OpenRouter and Google Sheets17 stepsBuilding Blocks & UtilitiesSet a Close Date Three Weeks Out Using Regex6 stepsCode, Git & Developer ToolsTest LLM Outputs with a GPT-4 Judge in Sheets12 stepsRAG & Knowledge BasesRoute Website Questions to Pinecone Knowledge with DeepSeek28 steps