Azure DevOps/ETL

 

Create, maintain, monitor, and perform activities on Azure Data Factory components like ETL pipelines, triggers, datalinks etc.

• ADF activities of copy activity, lookup, validation, switch activities etc.

• Creating CI/CD pipelines for both ADF and Databricks on version control GIT, Azure DevOps environment.

• Azure storage services, Agile application development support & Deployment

• Azure Databricks asset handing by running and implementing jobs of PySpark, SQL, Python notebooks.

Azure key, secrete & scope implementation, and security performance on business rules requirements by connected with client’s portfolios.

 

1. Azure Fundamentals: A solid understanding of the core concepts, services, and infrastructure components of the Azure cloud platform. 2. Cloud Architecture: Proficiency in designing and implementing cloud architectures, including virtual networks, storage, compute, and security in Azure. 3. Repository: In-depth knowledge of creating repos in repository tools like Bitbucket and integrating with Jenkins, Databricks and Synapse 4. CI/CD: Design, implement, and maintain CI/CD pipelines for Azure Databricks ETL jobs using Bitbucket and Jenkins. 5.Collaboration and communication: Collaborate with cross-functional teams including data engineers, data scientists, and DevOps to ensure smooth integration of pipelines. Written and spoken English communication skills are required. 6. ETL: Automate deployment processes to facilitate efficient and reliable ETL job execution. 7.Monitor: Monitor pipeline performance and troubleshoot issues to ensure optimal performance and reliability. 8.Stay updated with the latest industry trends and best practices in CI/CD methodologies and tools 9.DevOps and Automation: Familiarity with DevOps practices and tools for continuous integration, continuous deployment (CI/CD), and automation, such as Azure DevOps, GitHub Actions, or Jenkins. 10. Security and Compliance: Understanding of Azure security principles, identity and access management (IAM), data protection, encryption, network security groups (NSGs), Azure Security Center, and compliance frameworks. 11. Azure Governance: Familiarity with Azure governance frameworks, policies, and management groups to enforce and maintain compliance, resource organization, and cost control. 12. Scripting and Programming: Proficiency in scripting languages like PowerShell, Python, or Azure CLI, and ability to write code using Azure SDKs and APIs for automation and custom solutions. 13. Problem-solving and Troubleshooting: Strong analytical and problem-solving skills to diagnose and resolve issues related to Azure deployments, performance, security, and scalability. 14. Databricks: Knowledge of Clusters, Notebooks, Workspaces, Job and Datawarehouse in Azure Databricks. In-depth knowledge of Azure Databricks and its integration with CICD pipelines, including: • Databricks Delta for managing data Lakes • Databricks CLI for automation • Databricks REST API for integration 15. Certification in Azure, Databricks and DevOps will be preferred 16. Hands on experience in Jenkins and Bitbucket will be Strongly preferred 17. Mentor and train resources on DevOps setup and configuration