Skip to content
NEWMarketplace and your team: sell what you build, and get plans and changes that are tested before you see them. See what is new

Library / Web Scraping & Research

Web Scraping & Research

Fetch Web Pages over HTTP and Convert HTML to Markdown for OpenAI Agent

Agent that fetches a web page, strips scripts, media and comments, converts the remaining HTML to Markdown and passes it to OpenAI. Runs when another workflow calls it.

What it does

## Post-processing of the HTML page: 1. Keep only content 2. Remove inline tag entirely, as well as: NOSCRIPT, IFRAME, OBJECT, EMBED, VIDEO, AUDIO, SVG, and HTML comments. 3. In case query parameter method=simplified, replace all page URLs (a href) and IMG (src) with NOURL / NOIMG - this may save up to 20% of the page length 4. Convert the remaining HTML to Markdown. This step further reduces the length of the page: long HTML tags and styles are eliminated, but the markdown syntax keeps some page structure. This gives much better results compared to just a blank text. 5. Finally, check the page length. If it's too long, send an "ERROR: PAGE CONTENT TOO LONG" instead of the actual page. Of course, you could split the page content in chunks, but sometimes long pages just don't have a needed content, so it makes little sense to burn tokens on them.

The steps

  1. On new manual Chat Message
  2. OpenAI Chat Model
  3. HTTP Request
  4. Exctract HTML Body
  5. Is error?
  6. Stringify error message
  7. Execute Workflow Trigger
  8. Remove extra tags
  9. Simplify output
  10. Simplify?
  11. QUERY_PARAMS
  12. CONFIG
  13. ReAct AI Agent
  14. Convert to Markdown

16 steps in total.

Similar automations