Skip to content
NEWMarketplace and the trig8n Agent: sell what you build, and get plans and changes that are tested before you see them. See what is new

Library / Code, Git & Developer Tools

Code, Git & Developer Tools

Discover Hidden Website API Endpoints with Regex and AI

When you run it, it fetches a website's HTML and JavaScript files, extracts API endpoints with regex, and uses AI models to clean and organize them.

What it does

# 1. API Endpoints Extraction With a Predefined Regex * Paste the URL of the target website into the URL field of the Configuration node. * Fetch the HTML page of the site and extract the URLs of the JS files. * Keep relevant JS files with potential API endpoints (e.g. local, bundled, CDN sources). You can customise the logic to your needs: add or remove URL conditions. * Load JS content from kept URLs: convert references to JS files into properly formatted, complete URLs. * Apply predefined regex to extract API endpoints. * Keep JS files with enough URLs (>4 endpoints). Assume that JS files with fewer URLs do not contain relevant API endpoints. Discard them to send only the necessary data to the LLM. * Clean up the extracted URLs by removing duplicated items.

The steps

  1. Auto-fixing Output Parser
  2. Start API Discovery
  3. Configuration
  4. Fetch Website HTML
  5. Extract URLs of JS files
  6. Split URLs of JS files
  7. Keep Relevant JS Files
  8. Fetch JS Content
  9. Extract API Endpoints
  10. Check Endpoints Count
  11. Split Endpoints
  12. Remove Duplicates
  13. AI Endpoints Analysis
  14. Add Source Metadata

42 steps in total.

Similar automations