Available for new work Data Engineer · Sydney, Australia

Data pipelines that hold up in production.

I'm Richard Friedman — a data engineer with 7+ years designing scalable ETL/ELT pipelines and cloud-native platforms across enterprise environments. Python, PySpark and SQL at the core; certified across Databricks, AWS, Snowflake and Airflow.

Get in touch See my experience
7+
Years in data engineering
9
Industry certifications
5
Cloud & data platforms

About

I turn business requirements into robust, maintainable data architectures — and reliable pipelines the team can run without me.

Across enterprise and consulting work I've built ETL/ELT pipelines, cloud-native platforms and ML systems that cut processing times, automate manual workflows and raise data quality for stakeholders. My foundation is CI/CD, Docker and infrastructure-as-code — the unglamorous parts that keep a system dependable in production.

Programming

  • Python
  • SQL
  • JavaScript
  • C#
  • C

Data engineering

  • PySpark
  • PostgreSQL
  • ETL / ELT
  • Databricks
  • Snowflake
  • Airflow
  • Kestra

Machine learning

  • scikit-learn
  • TensorFlow
  • Keras
  • PyTorch
  • Time-series

Cloud & infra

  • AWS
  • Azure
  • GCP
  • Docker
  • Terraform
  • CI/CD
  • Git

Analytics & BI

  • Pandas
  • NumPy
  • Streamlit
  • Flask
  • Power BI
  • Tableau

Practices

  • Agile
  • Data quality
  • Data governance
  • Documentation

Certifications

  • AWS Cloud Practitioner
  • AWS AI Practitioner
  • Azure Fundamentals
  • Databricks Data Engineering
  • Databricks Fundamentals
  • Databricks GenAI
  • Databricks AI Agent
  • Airflow 3 Fundamentals
  • Snowflake — Zero to Snowflake

Experience

Where I've shipped, and what came of it.

  • Imagineered Consulting

    Principal Data Consultant & Sports Trader

    2015 – 2025

    Built and ran end-to-end analytics pipelines for sports-betting markets on Databricks — API ingestion, PySpark ETL, predictive ML models and automated alerting delivered to subscription clients. Engineered deep-learning architectures to process high-frequency odds data for near real-time feature engineering and backtesting across millions of events.

    → Sustained a profitable edge against efficient global markets over a 10-year period, measured by positive closing line value (CLV).

    • Databricks
    • PySpark
    • Python
    • ML
    • Time-series

  • Ventia Australia

    Data Engineer

    Nov 2021 – 2023

    Designed ETL pipelines for NBN data validation and GIS extraction, and built cloud-based Python applications that turned manual workflows into self-service tools and automated reporting dashboards.

    → Cut design errors 30% and processing time from 8h to 1h. Eliminated 40% of repetitive tasks and reduced downstream rework 25%.

    • Python
    • PostgreSQL
    • ETL
    • Azure
    • Power BI

  • Insight Analytics

    Data Analyst

    2020 – 2021

    Queried and validated data from central databases with SQL to support client deliverables under tight deadlines, operating fully remote with secure data transfers and compliance for sensitive datasets.

    → Streamlined high-volume data workflows on Databricks for faster stakeholder decision-making.

    • SQL
    • Databricks
    • Cloud

  • SIRCA Australia

    Data Scientist

    2019

    Orchestrated Hadoop workloads to ingest and transform terabytes of structured and unstructured financial data, with secure server-side operations over SSH and full audit trails.

    → Delivered client data requests from centralised systems to SLA, with accuracy and completeness verified end to end.

    • Hadoop
    • Python
    • SQL
    • SSH

  • UTS Summer Studio

    Data Scientist & Project Leader

    2012 – 2013

    Led a 4-person agile team to build an LSTM Forex-forecasting prototype with TensorFlow/Keras, deployed in a Flask web application for live stakeholder demonstrations.

    → Presented results at an industry showcase event.

    • TensorFlow
    • Keras
    • Flask
    • LSTM

Projects

Public code, end to end.

A few open projects that show how I work — reliable data systems, analytics built for real decisions, and product code with tests.

Data Pipeline Orchestrator

↗

A modular ETL framework for Databricks and Spark — configurable extractors, transformers and loaders with CDC, incremental loads and data-quality checks built for production.

  • PySpark
  • Databricks
  • CDC
  • Python

Retail Sales Analytics

↗

A structured analytics workspace for retail performance: trend and seasonality analysis, customer and product segmentation, and reporting designed for non-technical decision-makers.

  • Python
  • EDA
  • Plotly
  • Statistics

Sentiment Classifier (PyTorch)

↗

A production-minded PyTorch sentiment classifier — transformer, CNN and LSTM experimentation with proper training loops, evaluation and inference structure.

  • PyTorch
  • NLP
  • Deep learning

Song Year Guesser

↗

A multiplayer music game built with Express and browser JavaScript — real-time gameplay, audio previews and a leaderboard, with unit, API and Playwright coverage.

  • JavaScript
  • Express
  • Playwright
More on GitHub →

Contact

Let's talk.

Tell me about the data problem you're working on and what good looks like. I read every message and reply within a working day.

By email

admin@richdata.au

Elsewhere

LinkedIn GitHub
Based
Sydney, Australia · AEST
Status
Australian citizen
Open to
Data engineering roles & contracts