About the job
About the Role
At iCapital Network, we are seeking a skilled Python Automation Engineer with a proven background in web scraping, automated portal interactions, and cloud-native deployment in AWS. The successful candidate will possess extensive hands-on experience with Playwright for browser automation, adeptly managing multi-factor authentication (MFA) flows, and deploying scalable scraping tasks using AWS Lambda and associated services. Your role will involve architecting and developing robust, secure, and scalable scraping solutions that effectively interact with complex web applications and secure portals.
Responsibilities
- Collaborate in the design and implementation of sophisticated scraping solutions utilizing Python, Playwright, and AWS services.
- Automate interactions with JavaScript-dense and authentication-secured websites, effectively handling MFA, CAPTCHAs, and session or token-based login flows.
- Architect scraping pipelines employing serverless AWS components such as Lambda, Step Functions, S3, CloudWatch, and Secrets Manager.
- Develop systems capable of scaling to support high volumes of data extraction with fault tolerance, retries, and advanced logging mechanisms.
- Integrate and manage intricate workflows across multiple portals, APIs, and data sources.
- Contribute to architectural decisions, tooling, and best practices.

