We have an immediate hiring need for a Senior Data Engineer who is seasoned in PySpark and Spark SQL api. As a Senior Data Engineer, you will be responsible for developing best practices and making architectural decisions to rapidly improve critical data processing & analytics pipelines. You will tackle hard problems to improve the platform’s reliability, resiliency, and scalability. We are looking for someone who thrives on autonomy and has experience driving long-term projects to completion. You are detail and quality oriented, and excited about the prospects of having a big impact with data at our firm. Our tech stack includes Airflow, Hive, EMR, PySpark, Presto, Jenkins, Snowflake, Datadog and various AWS services. AdvantagesWork with a large well known enterprise shop with ability to grow with a new dynamic team !Responsibilities· You will be tasked with modernizing and expanding a critical set of existing systems · You will need a product-focused mindset. It is essential for you to understand business requirements and architect systems that will scale and extend to accommodate those needs · Break down complex problems, document technical solutions and sequence work to make fast, iterative improvements · Build and scale data infrastructure that powers batch and real-time data processing of billions of records · Develop CI/CD ETL pipelines using Jenkins for deployment and automation testing. · Take ownership of ETL pipelines, own data quality for your areas, and ensure SLAs are met. · Interface with data engineers, data scientists, product managers and all data stakeholders to understand their needs and promote best practices Qualifications· 3-5+ years of relevant industry experience in big data systems, data processing and SQL data warehouses · At least 3+ years of experience on working with large Hadoop projects and building scalable applications using PySpark by leveraging Spark DataFrame and Spark SQL api. · At least 2+ years of experience in optimizing Spark applications, spark memory management, and e2e spark process optimization. · Strong overall programming skills, able to write modular, maintainable code, preferably Python & SQL · Fundamental knowledge of distributed data processing systems, storage mechanisms, and compression techniques. · Experience building code-driven infrastructure on public cloud platforms, preferably AWS · Understanding of SQL, dimensional modeling, and analytical data warehouses, like Snowflake, Redshift · Knowledge of Data Modeling techniques and high volume ETL/ELT design. · Familiar with workflow management tools, like Airflow and Oozie · Experience with CI and code management tools like Git, Jenkins · Familiar with BI tools, like Power BI, Looker, Qlikview · Problem solver with excellent written and interpersonal skills; ability to make sound, complex decisions in a fast-paced, technical environment · Bachelor’s degree in Computer Science, Engineering or related field, or equivalent training, fellowship, or work experience Good to have: · Any Certification in Spark · Good Understanding of AWS EMR ecosystem and deploying spark applications on EMR. SummaryIf you feel your skills match - then don't delay and apply immediately !Contact Sohil Jivani email@example.comRandstad Canada is committed to fostering a workforce reflective of all peoples of Canada. As a result, we are committed to developing and implementing strategies to increase the equity, diversity and inclusion within the workplace by examining our internal policies, practices, and systems throughout the entire lifecycle of our workforce, including its recruitment, retention and advancement for all employees. In addition to our deep commitment to respecting human rights, we are dedicated to positive actions to affect change to ensure everyone has full participation in the workforce free from any barriers, systemic or otherwise, especially equity-seeking groups who are usually underrepresented in Canada's workforce, including those who identify as women or non-binary/gender non-conforming; Indigenous or Aboriginal Peoples; persons with disabilities (visible or invisible) and; members of visible minorities, racialized groups and the LGBTQ2+ community.Randstad Canada is committed to creating and maintaining an inclusive and accessible workplace for all its candidates and employees by supporting their accessibility and accommodation needs throughout the employment lifecycle. We ask that all job applications please identify any accommodation requirements by sending an email to firstname.lastname@example.org to ensure their ability to fully participate in the interview process.