Library / Web Scraping & Research
Web Scraping & Research
Clean Scraped HTML and Convert to Markdown for OpenAI Processing
Sub-workflow called by other workflows that strips scripts, media, and comments from fetched pages, optionally replaces URLs and images, and converts the rest to Markdown for OpenAI.
What it does
## Post-processing of the HTML page: 1. Keep only content 2. Remove inline tag entirely, as well as: NOSCRIPT, IFRAME, OBJECT, EMBED, VIDEO, AUDIO, SVG, and HTML comments. 3. In case query parameter method=simplified, replace all page URLs (a href) and IMG (src) with NOURL / NOIMG - this may save up to 20% of the page length 4. Convert the remaining HTML to Markdown. This step further reduces the length of the page: long HTML tags and styles are eliminated, but the markdown syntax keeps some page structure. This gives much better results compared to just a blank text. 5. Finally, check the page length. If it's too long, send an "ERROR: PAGE CONTENT TOO LONG" instead of the actual page. Of course, you could split the page content in chunks, but sometimes long pages just don't have a needed content, so it makes little sense to burn tokens on them.
The steps
- When chat message received
- AI Agent
- OpenAI Chat Model1
- When Executed by Another Workflow
- QUERY_PARAMS
- CONFIG
- HTTP Request
- Is error?
- Stringify error message
- Exctract HTML Body
- Remove extra tags
- Simplify?
- Simplify output
- Convert to Markdown
16 steps in total.
Similar automations