Project Overview We currently use Make.com + Apify to scrape product data. While functional, the process is too slow for large datasets. We need to move to a faster, scalable architecture built in N8N, capable of efficiently scraping: -5,000 – 20,000 SKUs per batch -Large datasets with parallelized execution -Modular architecture for future scraper expansion We believe the solution may require: -Distributed workflows -Multiple concurrent scraping actors/workers -Queue-based batching system -Rate limiting + proxy handling -Smart retry logic -We are open to the best architectural recommendation from you. Scope of Work Phase 1: Infrastructure -Deploy and configure N8N on our AWS environment -Recommend AWS architecture (EC2, ECS, Docker, scaling strategy) -Implement proper logging and error tracking -Ensure system stability and scalability Phase 2: High-Performance Scraper (Initial Use Case) -Build first large-scale scraping workflow -Accept large SKU list input (5k–20k SKUs) -Parallelize requests efficiently Implement: -Proxy support (if required) -Rotation logic -Retry/backoff handling -Captcha/anti-bot mitigation strategy -Structured output (JSON/CSV/database ready) Performance benchmarking -Success Criteria -Scraping 5,000+ SKUs reasonably fast -Scalable architecture for future scrapers -Clean, modular N8N workflows -Clear documentation for internal team Ideal Candidate Strong experience with: -N8N (required) -AWS infrastructure (EC2, ECS, Docker) -High-scale scraping architectures -Proxy rotation systems -Anti-bot mitigation strategies -Experience building distributed or parallelized workflows -Understands rate limiting, queues, and fault tolerance -Can recommend performance-optimized solutions Important Notes -This is the first of multiple scraping projects -Long-term work available for the right candidate -Must be comfortable working with large datasets -Must be able to think architecturally, not just build a simple scraper To Apply Please include: -Examples of scraping systems you've built -Scale handled (number of records per run) -N8N experience (live examples preferred) -Suggested high-level architecture for this project If you're someone who understands performance, scale, and clean automation design — we’d love to work with you.