Skip to content
NEWMarketplace and the trig8n Agent: sell what you build, and get plans and changes that are tested before you see them. See what is new

Library / Web Scraping & Research

Web Scraping & Research

Clean Scraped HTML and Convert to Markdown for OpenAI Processing

Sub-workflow called by other workflows that strips scripts, media, and comments from fetched pages, optionally replaces URLs and images, and converts the rest to Markdown for OpenAI.

What it does

## Post-processing of the HTML page: 1. Keep only content 2. Remove inline tag entirely, as well as: NOSCRIPT, IFRAME, OBJECT, EMBED, VIDEO, AUDIO, SVG, and HTML comments. 3. In case query parameter method=simplified, replace all page URLs (a href) and IMG (src) with NOURL / NOIMG - this may save up to 20% of the page length 4. Convert the remaining HTML to Markdown. This step further reduces the length of the page: long HTML tags and styles are eliminated, but the markdown syntax keeps some page structure. This gives much better results compared to just a blank text. 5. Finally, check the page length. If it's too long, send an "ERROR: PAGE CONTENT TOO LONG" instead of the actual page. Of course, you could split the page content in chunks, but sometimes long pages just don't have a needed content, so it makes little sense to burn tokens on them.

The steps

  1. When chat message received
  2. AI Agent
  3. OpenAI Chat Model1
  4. When Executed by Another Workflow
  5. QUERY_PARAMS
  6. CONFIG
  7. HTTP Request
  8. Is error?
  9. Stringify error message
  10. Exctract HTML Body
  11. Remove extra tags
  12. Simplify?
  13. Simplify output
  14. Convert to Markdown

16 steps in total.

Similar automations