Senior Data Engineer - Databricks (Remote)

Company: Premier Staffing Partners
Location: Fort Lauderdale
Posted on: June 24, 2022

Job Description:

Senior Data Engineer - Databricks (Remote)**** Candidate must reside in Eastern or Central Time ZonesThe purpose of the Data Platform Engineer (Remote) is to ensure all Databricks workloads conform to the organization's architecture, engineering, and security standards, are optimized for the best price performance and reliability possible.
Senior Data Engineer - Databricks (Remote) Responsibilities (Remote)
1. Contribute to Databricks architecture designs and discussions.2. Provision Databricks workspaces, clusters and cluster policies, instance profiles, and other resources as needed.3. Define and implement role-based access control for the Databricks platform and workspaces.4. Ensure security best practices are followed and built into any automation for workspace provisioning or code deployments.5. Define and document best practices for Delta table creation and management. Ensure best practices are followed by all development teams.6. Define and document best practices for Databricks managed MLFlow. Ensure best practices are being followed by development teams.7. Define and implement high-availability and recovery strategies for Databricks workspaces and jobs congruent with RTOs and RPOs.8. Contribute to the design and implementation of CICD pipelines for Databricks.9. Participate in product design discussions related to Databricks workloads.10. Participate in defining coding and engineering standards and best practices. Evangelize standards and best practices across all teams that are deploying workloads on Databricks.11. Perform frequent code reviews for any workloads being deployed to Databricks to prevent inefficient or insecure code from being deployed into the production environment.12. Define strategies for leveraging Databricks SQL congruent with Pilot's defined Data and Analytics architecture strategies.13. Perform cost analysis on Databricks workloads looking for opportunities for optimization, and planning for future usage estimates.14. Identify other opportunities within the organization to leverage Databricks to drive value or optimize expense.15. Keep up to date with new features and releases that could be leveraged to drive better price performance, security, and/or functionality for current, proposed, or future workloads.16. Assist with troubleshooting any issues related to the Databricks platform.
Senior Data Engineer - Databricks (Remote)

• Expert level knowledge of SQL
• Expert level knowledge of Apache Spark
• Strong understanding of AWS networking, infrastructure, and database services such as VPC, S3, EC2, IAM, RDS, and DynamoDB
• Strong understanding of and experience with Infrastructure as Code tools such as Terraform, AWS CloudFormation, or equivalent
• Strong understanding of and experience with CICD pipeline and orchestration tools such as Jenkins, Docker, CLI (AWS and Databricks), DBX, or equivalent
• Experience with data engineering processes, tools, and methodologies
• Experience with data science processes, tools, and methodologies
• Strong understanding of MLFlow, model lifecycle and sharing
  Keywords: Data Platform Engineer, Databricks, SQL, Apache Spark, AWS, Data Engineering, Data Platform Engineer, Databricks, SQL, Apache Spark, AWS, Data Engineering

Keywords: Premier Staffing Partners, Fort Lauderdale , Senior Data Engineer - Databricks (Remote), Engineering , Fort Lauderdale, Florida