Job Description:

We are seeking a highly skilled Senior Data Engineer with deep expertise in AWS data services, data wrangling using Python & PySpark, and a solid understanding of data governance, lineage, and quality frameworks. The ideal candidate will have a proven track record of delivering end-to-end data pipelines for logistics, supply chain, enterprise finance, or B2B analytics use cases.

Key Responsibilities:

Design and develop scalable ETL pipelines using AWS Glue 3.0+ and PySpark.
Build secure data lakes in Amazon S3 following bronze/silver/gold zoning best practices.
Write high-performance SQL queries in AWS Athena (Presto), using CTEs, window functions, and aggregations.
Develop orchestration workflows using AWS Step Functions, Event Bridge, and CloudWatch.
Manage schemas and metadata with AWS Glue Data Catalog.
Implement data quality checks with Great Expectations (null %, ranges, referential integrity).
Ensure data lineage using Open Meta data or Amundsen, tagging sensitive data and KPIs.
Collaborate with data scientists on feature engineering and ML pipeline integration (Parquet, JSON I/O).
Prepare BI-ready datasets for Sigma, with optional exposure to Power BI or Tableau.
Translate business metrics like revenue forecasts, margin trends, and utilization into technical implementations.
Work cross-functionally with QA, consultants, and business teams to finalize KPIs and logic.

Required Technical Skills:

Strong expertise in AWS services: Glue, S3, Athena, Step Functions, Event Bridge, CloudWatch, Glue Data Catalog.
Advanced programming in Python 3.x, PySpark, and Athena SQL.
Hands-on with Pandas, NumPy for data manipulation and time-series analysis.
Experience with data quality and governance tools like Great Expectations, Open Meta data, or Amundsen.
Familiarity with metadata tagging (e.g., PII, KPIs), audit logs, and data compliance standards.
Take full ownership from ingestion → transformation → validation → metadata →documentation → dashboard-ready output.
Feature engineering expertise: rolling averages, deltas, window-based tagging.
BI pipeline readiness and integration with Sigma, Power BI, or Tableau (nice to have).

Preferred Qualifications:

8+years of experience in software development.
Minimum 6–8 years of hands-on data engineering with at least 3 years in AWS-native data pipelines and governance.
Bachelor's or Master’s degree in Computer Science, Data Engineering, or a related field.
Domain experience in logistics, supply chain, enterprise finance, or B2B analytics.
Knowledge of ML pipelines, Parquet/JSON formats, and data modeling.
Agile working experience and comfort with fast-paced delivery cycles.