Data Engineer

The DataOps / Cloud Data Engineer is responsible for designing, developing, testing, implementing, and supporting enterprise data solutions across cloud and hybrid environments. The role focuses on building scalable data pipelines, data lakehouse and data warehouse solutions, and automated data integration processes that enable analytics, reporting, and business intelligence initiatives.

Responsibilities

Data Engineering & Pipeline Development

• Design, develop, test, implement, and troubleshoot enterprise data pipelines.
• Build and maintain data ingestion, transformation, and orchestration processes using cloud-native and ETL tools.
• Develop complex data transformation procedures to support business and operational reporting requirements.
• Process and integrate structured and unstructured data from multiple sources.
• Implement data quality, validation, monitoring, and error-handling mechanisms.
• Optimize data workflows for performance, scalability, and reliability.

Cloud Data Platform Development

• Develop and support cloud-based data solutions using AWS and/or Azure services.
• Build and maintain data lakehouse environments using Databricks Delta Lake.
• Develop cloud-based data warehouse solutions using platforms such as AWS Redshift.
• Implement automated data processing and orchestration workflows using services such as AWS Glue, Step Functions, Lambda, S3, and Azure Data Factory.
• Support cloud migration and data modernization initiatives.

Data Warehousing & Modeling

• Design and maintain relational and dimensional data models.
• Develop and support enterprise data warehouses, data marts, and analytical datasets.
• Implement ETL/ELT processes using tools such as Informatica IDMC, Azure Data Factory, and AWS Glue.
• Support reporting and visualization requirements by providing optimized and governed data assets.

Operations & Support

• Participate in production support, troubleshooting, incident resolution, and root-cause analysis.
• Support change management and deployment activities across development, testing, and production environments.
• Ensure adherence to security, governance, and data management standards.

Project Delivery & Stakeholder Collaboration

• Participate in all phases of the Software Development Life Cycle (SDLC), including requirements gathering, design, development, testing, deployment, and support.
• Collaborate with business stakeholders, architects, analysts, and project teams to deliver data solutions.
• Work within Agile/Scrum and Waterfall project methodologies.
• Provide technical guidance and recommendations related to data architecture and engineering best practices.

Required Skills and Experience

Data Engineering (40%)

• Strong programming and scripting experience with Python, SQL, Linux Shell, and PowerShell.
• Experience using Pandas and PySpark for data processing and analysis.
• Experience working with XLSX, CSV, JSON files, relational databases, cloud storage, and structured/unstructured data.
• Experience developing and maintaining enterprise data pipelines.

Cloud Technologies (25%)

• Experience with AWS and/or Azure cloud platforms.
• Experience with AWS services including Glue, Step Functions, Lambda, S3, and Redshift.
• Experience with Databricks Delta Lake and lakehouse architectures.
• Experience implementing automated cloud data processing solutions.

Data Warehousing & ETL (25%)

• Experience with ETL/ELT tools such as Informatica IDMC, Azure Data Factory, and AWS Glue.
• Strong understanding of relational and dimensional data modeling.
• Experience supporting enterprise reporting and analytics environments.
• Knowledge of data architecture and warehouse design principles.

Professional Skills (5%)

• Full SDLC experience.
• Agile/Scrum and Waterfall project delivery experience.
• Change and Incident Management experience.
• Strong communication, presentation, negotiation, problem-solving, and decision-making skills.

Public Sector Experience (5%)

• Experience working within the Ontario Public Sector (OPS) and/or broader public sector environments is considered an asset.

Mandatory Requirements

• Data engineering experience using Pandas and PySpark.
• Experience with Databricks Delta Lake and lakehouse solutions.
• Experience working with XLSX, CSV, JSON files, relational databases, cloud storage, and structured/unstructured data.
• Experience with AWS services including Glue, Step Functions, Lambda, and S3.
• Experience extracting, transforming, and loading data using Informatica IDMC.

Work Arrangement

• Onsite position (5 days per week).
• Standard working hours: Monday to Friday, 8:00 AM-5:00 PM.
• Expected workday: 7.25 hours excluding lunch breaks.
• Resource must work from the designated OPS office location.