Specialized IT Consultant

You will drive end-to-end architecture and engineering of robust, metadata-driven data pipelines across Bronze, Silver, and Gold layers, ensuring governance, security, and performance at scale.


Advantages
🏠 Hybrid flexibility: 2 days onsite, 3 days remote
⚙️ Lead architecture and build for an enterprise-scale Databricks platform
🧱 Design layered Medallion Architecture (Bronze/Silver/Gold) with Delta Lake
🔐 Implement secure access controls with Unity Catalog and RBAC
💡 Influence data strategy, standards, and reusability frameworks
📈 Work with cutting-edge Azure, PySpark, and metadata-driven ingestion

Responsibilities
• Architect and engineer end-to-end data pipelines in Databricks using PySpark, Delta Lake, and Delta Live Tables
• Implement Medallion Architecture, defining standards across Bronze, Silver, and Gold layers
• Design secure, scalable data solutions on Azure Data Lake leveraging Unity Catalog, RBAC, and workspace ACLs
• Conduct data discovery and profiling to support data quality, governance, and ingestion logic
• Lead integration efforts with REST APIs (e.g., MDM, FHIR) and downstream systems
• Build metadata-driven ingestion/transformation pipelines using Python, SQL, and PySpark
• Optimize performance across clusters, SQL queries, ingestion logic, and data models
• Participate in agile delivery (scrum, user stories, sprint planning), and serve as a technical advisor during backlog grooming
• Define and manage business and technical metadata models and logical/physical data models
• Document pipeline logic, architecture, design patterns, and reusable frameworks
• Guide DevOps practices around notebook versioning, CI/CD, and environment management
• Collaborate with analysts, developers, and stakeholders to translate data needs into technical solutions


Qualifications
✅ Must Have:
• Hands-on expertise with Databricks, including Unity Catalog, Delta Lake, Auto Loader, PySpark
• 5+ years designing and building ETL/ELT pipelines and implementing CDC (batch + streaming)
• In-depth knowledge of Azure Cloud data architecture and Azure Data Lake
• Proven experience designing Medallion Architecture and metadata-driven workflows
• Strong SQL development and debugging skills
• Experience with RBAC and secure workspace configuration in Unity Catalog
• Ability to lead architecture, design, and deployment from discovery through delivery
• Skilled in profiling large datasets for structure, quality, and completeness
• Strong collaboration skills in Agile environments and experience translating user stories into pipelines



⭐ Nice to Have:
🔄 Experience integrating with FHIR and MDM services
📊 Familiarity with Delta Lake optimizations and DevOps practices for Databricks
🛠 Proficiency with Postman, Azure DevOps, and Git for API and code management
🎯 Experience designing data governance frameworks and reusable patterns
📚 Mentorship and training experience in data engineering best practices


Summary
Join as a Senior Data Architect / Platform Data Engineer and lead the design and implementation of a modern, secure, and scalable data lakehouse solution using Databricks and Azure technologies. You will shape the future of enterprise data platforms by applying Medallion Architecture, ensuring robust data quality, governance, and performance. This is an exciting opportunity for a data leader ready to own delivery in a fast-paced, hybrid work environment.
If you are interested in this role, please apply online at www.randstad.ca. Qualified candidates will be contacted.

Randstad Canada is committed to fostering a workforce reflective of all peoples of Canada. As a result, we are committed to developing and implementing strategies to increase the equity, diversity and inclusion within the workplace by examining our internal policies, practices, and systems throughout the entire lifecycle of our workforce, including its recruitment, retention and advancement for all employees. In addition to our deep commitment to respecting human rights, we are dedicated to positive actions to affect change to ensure everyone has full participation in the workforce free from any barriers, systemic or otherwise, especially equity-seeking groups who are usually underrepresented in Canada's workforce, including those who identify as women or non-binary/gender non-conforming; Indigenous or Aboriginal Peoples; persons with disabilities (visible or invisible) and; members of visible minorities, racialized groups and the LGBTQ2+ community.

Randstad Canada is committed to creating and maintaining an inclusive and accessible workplace for all its candidates and employees by supporting their accessibility and accommodation needs throughout the employment lifecycle. We ask that all job applications please identify any accommodation requirements by sending an email to accessibility@randstad.ca to ensure their ability to fully participate in the interview process.