Skip to content
NEWMarketplace and your team: sell what you build, and get plans and changes that are tested before you see them. See what is new

Library / Web Scraping & Research

Web Scraping & Research

Scrape Any Web Page with Selenium Behind a Webhook, Including Login Pages

Webhook-triggered workflow that uses a self-hosted Selenium container to collect data from any web page, with or without login, and return it.

What it does

## N8N Ultimate Scraper - Workflow This workflow's objective is to collect data from any website page, whether it requires login or not. For example, you can collect the number of stars of the n8n-ultimate-scraper project on GitHub. ## Requirements Selenium Container: Selenium is an open-source automation framework for web applications, enabling browser control and interaction through scripts in various programming languages. You can deploy the Docker Compose file from the associated GitHub project to set up your Selenium container and configuration: {{ $env.WEBHOOK_URL }} Residential Proxy Server: To scrape data at scale without being blocked, I personally recommend GeoNode. They offer affordable, high-quality residential proxies: {{ $env.WEBHOOK_URL }} OpenAI API Key: For using GPT-4. ## Optional Session Cookies Collection: To use login functionality with the n8n Ultimate Scraper, you need to collect session cookies from the target website. You can do this using the extension created for this application in the GitHub project: {{ $env.WEBHOOK_URL }} Follow the installation procedure to use it. ## How to use Deploy the project with all the requiremnts and request your webhook. Example of request: curl -X POST {{ $env.WEBHOOK_URL }} \ -H "Content-Type: application/json" \ -d '{ "subject": "Hugging Face", "Url": "github.com", "Target data": [ { "DataName": "Followers", "description": "The number of followers of the GitHub page" }, { "DataName": "Total Stars", "description": "The total numbers of stars on the different repos" } ], "cookies": [] }' You can also scrape link like this : curl -X POST {{ $env.WEBHOOK_URL }} \ -H "Content-Type: application/json" \ -d '{ "Target Url": "{{ $env.WEBHOOK_URL }}", "Target data": [ { "DataName": "Followers", "description": "The number of followers of the GitHub page" }, { "DataName": "Total Stars", "description": "The total numbers of stars on the different repo" } ] }' Note The maximum nimber of Target data is 5.

The steps

  1. Extract First Url Match
  2. OpenAI Chat Model
  3. Clean Webdriver
  4. Delete Session
  5. Delete Session2
  6. If Block1
  7. Delete Session3
  8. Limit
  9. Delete Session1
  10. Delete Session4
  11. Success with cookie
  12. Respond to Webhook2
  13. Code
  14. Delete Session5

56 steps in total.

Similar automations