Data Engineer Team Lead (Malaysia)
Cyberjaya, Selangor
|Onsite
|Direct hire
|Job ID 7572|Posted Dec 10, 2024JOB DESCRIPTION
On a given day, you will:
- Team Leadership: Lead and mentor a team of data engineers to design, build, and optimize scalable data pipelines for analytics, reporting, and operational needs.
- Unified Data Pipeline Management: Create and manage a unified data pipeline that integrates multiple data sources, facilitating efficient access for both internal and external systems, including third-party tools and client-facing platforms.
- Utilization of Low-Code/No-Code Tools: Leverage low-code/no-code integration tools such as Fivetran or RudderStack to accelerate data workflows and improve integration efficiency.
- Cross-Functional Collaboration: Work closely with various teams to identify data needs and ensure smooth data flows between systems, enhancing high availability and integration.
- Data Governance and Quality Assurance: Implement and manage data governance and quality standards, ensuring data consistency and reliability throughout the organization.
- Real-Time and Batch Data Architecture: Architect and oversee the implementation of real-time and batch data pipelines using data streaming tools like Kafka or Pub/Sub, ensuring low-latency, high-performance data processing.
- Technical Guidance and Support: Provide technical guidance to your team, aiding in troubleshooting complex data issues and optimizing performance and cost-effectiveness while adhering to best practices.
- Continuous Improvement: Stay informed on the latest in data engineering technologies and best practices, especially concerning real-time processing and low-code/no-code integrations, to enhance our data infrastructure continually.
- Stakeholder Coordination: Communicate with stakeholders to ensure timely delivery of data solutions that effectively meet business requirements.
- Experience in fintech, trading, or other data-heavy industries.
- Familiarity with API-based integrations and third-party system integrations.
- Knowledge of additional data streaming technologies and best practices.
- Experience with automating data flows using low-code platforms.
- Bachelor's degree in computer science, data engineering, or a related field.
- 10 years of experience in data engineering, with at least 4 years in a leadership role.
- Expertise in designing, building, and maintaining data pipelines and data warehouse solutions.
- Strong knowledge of cloud platforms such as Google Cloud or AWS, and tools like BigQuery, PostgreSQL, and Airflow.
- Proficiency in programming languages such as Python or similar, with experience in ETL/ELT processes.
- Hands-on experience with low-code/no-code data integration tools like Fivetran, RudderStack, or similar platforms.
- Deep understanding of real-time data streaming and data processing tools such as Kafka, Pub/Sub, or similar.
- Experience in building unified data pipelines that serve both internal and external systems, ensuring consistent, secure data flows.
- Experience working with both real-time and batch data processing systems.
- Strong understanding of data governance, security, and compliance practices.
- Excellent communication skills, with the ability to convey technical concepts to non-technical stakeholders and mentor junior team members.
Horizontal is proud to be an Equal Opportunity and Affirmative Action Employer.
We seek to provide employment opportunities to talented, qualified candidates regardless of race, color, sex/gender including gender identity and/or expression, national origin, religion, sexual orientation, disability, marital status, citizen status, veteran status, or any other protected classification under federal, state or local law.
In addition, Horizontal will provide reasonable accommodations for qualified individuals with disabilities. If you need to request a reasonable accommodation in order to complete the application or interview process, please contact us.
All applicants applying must be legally authorized to work in the country of employment.