Data Engineer

Description:

Application closing date: Wednesday, 10 April 2024 • 11:59pm, Canberra time

Estimated start date: Wednesday, 01 May 2024

Location of work: NSW

Working arrangements: Flexible work arrangements available

Length of contract: 12 months

Contract extensions: 2 x 12 months

Security clearance: Must be able to obtain Baseline

Rates: $100 - $130 per hour (inc. super)

The Biosecurity Analytics Centre, Digital Business Division, requires skilled data engineers to work in a high performing data warehousing team delivering analytics products to support business outcomes.

The team member will provide services under direction including:

1. Follow the principles defined¬ in the team’s Data Engineering quality blueprint and training materials.
2. Collaborate with data analysts, data scientists, and business stakeholders to understand data requirements and deliver solutions.
3. Design, develop, and maintain data pipelines and code-first ELT processes using team standards and best practices.
4. Optimise pipeline performance through performance-tuning techniques.
5. Ensure data quality, reliability, and availability across different sources and destinations.
6. Design and maintain reference data tables according to the team’s reference data standards using manual and automated mechanisms.
7. Implement defined data governance, security, and privacy policies and standards.
8. Monitor, troubleshoot, and optimize data pipeline performance and issues.
9. Write clear, business meaningful, code documentation that explains the intent and mechanics of scripts.

The staff will be part of a team that is:

1. Business-focused – highest priority is given to considering the business problem and ensuring the technical work is solving real-world problems.
2. Code-first – highest technical rigor in code is demanded, expectations are clearly defined and to be met.
3. Multi-disciplinary – team members are to be considerate of how staff from other professions contribute to the overall outcome, and collaborate accordingly.
4. Open – code, communication, documentations, peer-review are open-by-default, including early drafts of work.

Essential Criteria

1. Data engineering - ability to create high business value, robust and efficient SQL pipelines to curate data from complex and high-volume sources. The goal of data engineering is to create useful and trustworthy data representations of real-world business processes. The engineer aspires to be the pillar of the multidisciplinary team by providing a constant access to the data needed to solve the next problem. They appropriately question requirements and take responsibility for checking outputs against independent testing. Ability to tackle complex SQL problems is paramount. (Response Weighting 34%)
2. Business analysis – business analysis of complex digital systems through stakeholder engagement, SQL data analysis and researching available resources. The goal of the analysis is to capture the intent of business processes and the flow of information through these processes. The analyst aspires to be the subject matter expert for the multi-disciplinary team. They communicate findings effectively to technical and non-technical audiences. Ability to verify hypotheses using SQL data analysis is essential. (Response Weighting 33%)
3. Team player – ability to work independently with limited guidance to create self-direction in uncertain and rapidly changing environments, displaying the characteristics of creativity and resilience Ability to work openly and collegiately in a multi-disciplinary team, displaying a genuine consideration of how members from other disciplines contribute to the overall outcome. They have a strong focus on people and business outcomes over formal methodologies. (Response Weighting 33%)

Desirable Criteria

1. Reference data - ability to integrate disparate data assets through judicious use of reference data. By using a common standard as a translation of data from disparate systems, decision makers or analysts obtain a seamless view of multiple systems. The data engineer achieves this by designing golden reference tables, maintaining mapping tables between individual systems to the common reference, and create automated monitoring for outdated references.
2. Performance tuning - ability to maintain the optimal execution performance of a complex data pipeline. The engineer takes pride in the background work to ensure the rest of the warehouse are not encumbered by the ever-growing SQL and Power BI pipeline. The engineer does this through daily monitoring of the pipeline execution, apply performance tuning techniques such as SQL optimisation, designing incremental extractions in SQL and partitioned refreshes in Power BI.
3. Familiarity with the Australia’s biosecurity systems. Familiarity with the imported cargo pathway will be highly regarded.
4. Familiarity with the Australia’s exports systems.