İçeriğe geç
YENİPazar yeri ve ekibin: geliştirdiğini sat; planlar ve değişiklikler sen görmeden önce test edilsin. Yeniliklere bak

Kütüphane / Web Kazıma ve Araştırma

Web Kazıma ve Araştırma

Webhook ile Tetiklenen Selenium ile Giriş Sayfaları Dahil Her Web Sayfasını Kazı

Kendi sunucunuzda çalışan bir Selenium konteyneri kullanarak, giriş gerektiren veya gerektirmeyen herhangi bir web sayfasından veri toplayan ve döndüren, webhook ile tetiklenen bir iş akışı.

Ne yapar

## N8N Ultimate Scraper - Workflow This workflow's objective is to collect data from any website page, whether it requires login or not. For example, you can collect the number of stars of the n8n-ultimate-scraper project on GitHub. ## Requirements Selenium Container: Selenium is an open-source automation framework for web applications, enabling browser control and interaction through scripts in various programming languages. You can deploy the Docker Compose file from the associated GitHub project to set up your Selenium container and configuration: {{ $env.WEBHOOK_URL }} Residential Proxy Server: To scrape data at scale without being blocked, I personally recommend GeoNode. They offer affordable, high-quality residential proxies: {{ $env.WEBHOOK_URL }} OpenAI API Key: For using GPT-4. ## Optional Session Cookies Collection: To use login functionality with the n8n Ultimate Scraper, you need to collect session cookies from the target website. You can do this using the extension created for this application in the GitHub project: {{ $env.WEBHOOK_URL }} Follow the installation procedure to use it. ## How to use Deploy the project with all the requiremnts and request your webhook. Example of request: curl -X POST {{ $env.WEBHOOK_URL }} \ -H "Content-Type: application/json" \ -d '{ "subject": "Hugging Face", "Url": "github.com", "Target data": [ { "DataName": "Followers", "description": "The number of followers of the GitHub page" }, { "DataName": "Total Stars", "description": "The total numbers of stars on the different repos" } ], "cookies": [] }' You can also scrape link like this : curl -X POST {{ $env.WEBHOOK_URL }} \ -H "Content-Type: application/json" \ -d '{ "Target Url": "{{ $env.WEBHOOK_URL }}", "Target data": [ { "DataName": "Followers", "description": "The number of followers of the GitHub page" }, { "DataName": "Total Stars", "description": "The total numbers of stars on the different repo" } ] }' Note The maximum nimber of Target data is 5.

Adımlar

  1. Extract First Url Match
  2. OpenAI Chat Model
  3. Clean Webdriver
  4. Delete Session
  5. Delete Session2
  6. If Block1
  7. Delete Session3
  8. Limit
  9. Delete Session1
  10. Delete Session4
  11. Success with cookie
  12. Respond to Webhook2
  13. Code
  14. Delete Session5

Toplam 56 adım.

Benzer otomasyonlar