Job Purpose

Data Engineers – Data Ingestion Specialists are aimed to support the ingestion of large-scale data workloads into the Azure Data Platform. The role will focus on validating source pre-requisites, configuring and duplicating Azure Data Factory pipelines, executing data ingestion into ADLS Gen2 bronze layer/Silver Layer and coordinating handover to the Quality team for validation.

Job Description

Bespineers Conduct

At Bespin, we are committed to excellence in all that we do, continually seeking opportunities for improvement and innovation. We prioritize fostering a positive and collaborative atmosphere where teamwork thrives, and we expect all our employees to embrace this commitment as well.

Key Expectations

Strive for operational excellence and process automation

Promote collaboration, transparency, and accountability in delivery

Uphold security, compliance, and quality as non-negotiable principles

Be a role model in driving modern engineering practices

Live the Bespin values and create a vibrant workplace culture

Additional Responsibilities

Data Engineering

Validate pre-requisites for each data source including schema, authentication, and connectivity.

Duplicate and configure ingestion pipelines in Azure Data Factory (ADF) for new data sources.

Execute ingestion pipelines to land raw/JSON files into Azure Data Lake Storage Gen2 bronze layer.

Monitor ingestion jobs, troubleshoot errors, and re-run failed pipelines as needed.

Document ingestion run metrics (duration, volume, error logs) for each source.

Provision access permissions and share datasets with the Quality team for validation.

Collaborate with the Quality team to address ingestion gaps; repeat ingestion if validation fails.

Maintain ingestion activity logs, operational dashboards, and progress reports.

Work closely with source system owners to resolve access or data format issues.

Ensure adherence to established standards for pipeline configuration, monitoring, and logging.

Job Specification

Education

Bachelor’s degree in IT, Computer Science or any related field

Professional Certifications

DP-900 Azure Data Fundamentals

DP-203 Azure Data Engineer Associate

DP-600 Fabric Analytics Engineer Associate

DP-700 Fabric Data Engineer Associate

Knowledge

Regular ITSM tools

Experience

4-5 years as Data Engineer experience is a MUST