Data Engineer

Data Engineer

Job Title: Data Engineer
Contract Type: Permanent
Location: Sydney
Reference: 918
Contact Name: Matt Mathauda
Job Published: July 27, 2018 10:00

Job Description

We are looking for an experienced big data engineer who is passionate about a wide range of big data technologies including MapReduce, Hadoop, Spark, Hive, Spark SQL, BigQuery, Redshift, HortonWorks, Cloudera, MapR

You will be leading a small development team, taking ownership of the requirement, proposing, evaluating, and driving solutions through to completion, delivering software that is ready to production.

You will be working across the full software development lifecycle in an Agile environment. You must take a quality approach to software delivery.

You must be able to continue to learn new technologies, and evolve the platform to keep pace with technology changes, always working towards making our platform a robust and secure platform that is simple for customers to use.

To approach problem-solving logically, research options, avoid biases and focus on meaningful data to draw the right conclusions—and be able to make good decisions even under pressure.

Anticipate consequences, both intended and unintended

Mentoring and supporting junior and intermediate developers.

Experience and Skills

  • Big Data Technologies: Hadoop, Spark, ETL, MapReduce, Data Lake, Data Warehouse

  • Languages: Java, Python, Scala

  • Server Technologies: event-pipeline, pub/sub, kafka/kinesis - data streaming, distributed caching, big data computing and storage

  • Databases: NoSQL - MongoDB, DynamoDb, Cassandra, SQL - Hive, MS-SQL, Postgres

  • Revision Control: Git (Git Flow)

  • Infrastructure: Cloud platform (AWS, Google, Azure, …)

  • Methodologies: Scrum and Agile Development


  • Team collaboration: Jira, Confluence

  • Continuous Integration: Bamboo, Stash/Bitbucket, Docker

Key Traits

  • Knowledge in public cloud Big Data services - storage, compute, AWS EMR, Athena, Glue, data pipeline, Google DataFlow, Redshift, BigQuery etc.

  • Passion for big data technology, design, innovation and latest technologies ?

  • Experience (and a preference for) working in an Agile environment ?

  • Enjoys working with diverse stakeholders: architects, senior managers, designers, customers, ?investors, data analysts ?

  • Proven delivery of challenging big data technology platform ?and services

  • Passion for data and its potential to change businesses, industries and communities for the better ?

  • Enthusiasm for developing strategy through to executing small details ?

  • Grasp on soft skills such as: facilitation, communication and active listening ?

  • Team leadership and mentoring capabilities.

  • You love simplicity and code that just works.

  • You know how to structure, develop and maintain relational databases (MySQL/PostGres)

  • Experience with key/value store databases, for example, DynamoDB, Cassandra, MongoDB, Redis. You understand code testability, build pipelines, docker and CI/CD systems

  • Experience hiring and managing software engineers