Kütüphane / Web Kazıma ve Araştırma
Web Kazıma ve Araştırma
Webhook ile Tetiklenen Selenium ile Giriş Sayfaları Dahil Her Web Sayfasını Kazı
Kendi sunucunuzda çalışan bir Selenium konteyneri kullanarak, giriş gerektiren veya gerektirmeyen herhangi bir web sayfasından veri toplayan ve döndüren, webhook ile tetiklenen bir iş akışı.
Ne yapar
## N8N Ultimate Scraper - Workflow This workflow's objective is to collect data from any website page, whether it requires login or not. For example, you can collect the number of stars of the n8n-ultimate-scraper project on GitHub. ## Requirements Selenium Container: Selenium is an open-source automation framework for web applications, enabling browser control and interaction through scripts in various programming languages. You can deploy the Docker Compose file from the associated GitHub project to set up your Selenium container and configuration: {{ $env.WEBHOOK_URL }} Residential Proxy Server: To scrape data at scale without being blocked, I personally recommend GeoNode. They offer affordable, high-quality residential proxies: {{ $env.WEBHOOK_URL }} OpenAI API Key: For using GPT-4. ## Optional Session Cookies Collection: To use login functionality with the n8n Ultimate Scraper, you need to collect session cookies from the target website. You can do this using the extension created for this application in the GitHub project: {{ $env.WEBHOOK_URL }} Follow the installation procedure to use it. ## How to use Deploy the project with all the requiremnts and request your webhook. Example of request: curl -X POST {{ $env.WEBHOOK_URL }} \ -H "Content-Type: application/json" \ -d '{ "subject": "Hugging Face", "Url": "github.com", "Target data": [ { "DataName": "Followers", "description": "The number of followers of the GitHub page" }, { "DataName": "Total Stars", "description": "The total numbers of stars on the different repos" } ], "cookies": [] }' You can also scrape link like this : curl -X POST {{ $env.WEBHOOK_URL }} \ -H "Content-Type: application/json" \ -d '{ "Target Url": "{{ $env.WEBHOOK_URL }}", "Target data": [ { "DataName": "Followers", "description": "The number of followers of the GitHub page" }, { "DataName": "Total Stars", "description": "The total numbers of stars on the different repo" } ] }' Note The maximum nimber of Target data is 5.
Adımlar
- Extract First Url Match
- OpenAI Chat Model
- Clean Webdriver
- Delete Session
- Delete Session2
- If Block1
- Delete Session3
- Limit
- Delete Session1
- Delete Session4
- Success with cookie
- Respond to Webhook2
- Code
- Delete Session5
Toplam 56 adım.
Benzer otomasyonlar
Web Kazıma ve AraştırmaHerhangi Bir Web Sitesi için OpenAI'li Selenium Web Kazıyıcı56 adımWeb Kazıma ve AraştırmaSelenium ve OpenAI ile herhangi bir web sitesini kazıma362 adımWeb Kazıma ve AraştırmaSelenium ve OpenAI ile Herhangi Bir Web Sitesini Kazıma56 adımWeb Kazıma ve AraştırmaSelenium ve OpenAI ile herhangi bir web sitesini kazıma362 adım