Library / Web Scraping & Research
Web Scraping & Research
Fetch Web Pages over HTTP and Convert HTML to Markdown for OpenAI Agent
Agent that fetches a web page, strips scripts, media and comments, converts the remaining HTML to Markdown and passes it to OpenAI. Runs when another workflow calls it.
What it does
## Post-processing of the HTML page: 1. Keep only content 2. Remove inline tag entirely, as well as: NOSCRIPT, IFRAME, OBJECT, EMBED, VIDEO, AUDIO, SVG, and HTML comments. 3. In case query parameter method=simplified, replace all page URLs (a href) and IMG (src) with NOURL / NOIMG - this may save up to 20% of the page length 4. Convert the remaining HTML to Markdown. This step further reduces the length of the page: long HTML tags and styles are eliminated, but the markdown syntax keeps some page structure. This gives much better results compared to just a blank text. 5. Finally, check the page length. If it's too long, send an "ERROR: PAGE CONTENT TOO LONG" instead of the actual page. Of course, you could split the page content in chunks, but sometimes long pages just don't have a needed content, so it makes little sense to burn tokens on them.
The steps
- On new manual Chat Message
- OpenAI Chat Model
- HTTP Request
- Exctract HTML Body
- Is error?
- Stringify error message
- Execute Workflow Trigger
- Remove extra tags
- Simplify output
- Simplify?
- QUERY_PARAMS
- CONFIG
- ReAct AI Agent
- Convert to Markdown
16 steps in total.
Similar automations
Web Scraping & ResearchConvert web pages to clean Markdown with TopAI Web-to-Markdown API2 stepsWeb Scraping & ResearchConvert Web Pages to Markdown with Firecrawl10 stepsWeb Scraping & ResearchFetch Web Pages as Markdown with an OpenAI Agent16 stepsContent Writing & GenerationConvert Airtable Markdown to HTML via Webhook18 steps