This role will sit in Charlotte, NC on-site interview required in NC or Malvern, PA
Manager prefers someone that will come in for onsite interviews
Would like to see GPA as well.
Sr. Big Data Engineer
8+ years of experience.
AWS
Python
Very Analytical / problem solver
Attention to detail,
Ability to clearly communicate thoughts and ideas
The person on this role will play a crucial role in building scalable and cost-effective data pipelines, data lakes, and analytics systems.
- Data Ingestion: Implement data ingestion processes to collect data from various sources, including databases, streaming data, and external APIs.
- Data Transformation: Develop ETL (Extract, Transform, Load) processes to transform and cleanse raw data into a structured and usable format for analysis.
- Data Storage: Manage and optimize data storage solutions, including Amazon S3, Redshift, and other AWS storage services.
- Data Processing: Utilize AWS services like AWS Glue, Amazon EMR, and AWS Lambda to process and analyze large datasets.
- Data Monitoring and Optimization: Continuously monitor and optimize data pipelines and infrastructure for performance, cost-efficiency, and scalability.
- Integration: Collaborate with data scientists, analysts, and other stakeholders to integrate AWS-based solutions into data analytics and reporting platforms.
- Documentation: Maintain thorough documentation of data engineering processes, data flows, and system configurations.
- Scalability: Design AWS-based solutions that can scale to accommodate growing data volumes and changing business requirements.
- Cost Management: Implement cost-effective solutions by optimizing resource usage and recommending cost-saving measures.
- Troubleshooting: Diagnose and resolve AWS-related issues to minimize downtime and disruptions.
- Qualifications
Must Have:
- Strong experience with AWS services on big data platforms.
- Strong Python experience in building Dash application in AWS
- Spark
- Shell scripting and Linux knowledge
Nice to have:
- Investment Knowledge preferred
- Looking for candidates who have worked in Investment/Finance space
- Machine Learning Skills
Qualifications:
- Educational Background: A bachelor's degree in computer science, information technology, or a related field is typically required.
- A master's degree is preferred with GPA of 3.8+
Problem-Solving:
- Strong analytical and problem-solving skills to address complex data engineering challenges.
- Communication Skills:
- Effective communication and collaboration skills to work with cross-functional teams.