Summary

The Data Engineering team is part of the broader DevOps department, and sits at the core of the business,  bridging the fast-paced and never-ending data feeds from the BitMEX trading platform to the entirety of the firm. Whether it is about business development, financial products, risk management, compliance & market surveillance, marketing consistent and reliable data is key, making Data Engineering the cornerstone to the firms success.

Responsibilities

• Design and maintain enhancements to our data warehouse, data lake, and data pipelines to increase their reliability and consistency
• Improve queriability of large historical datasets through industry-standard tools, careful data representation and aggregation for both technical and business units
• Ensure data governance and security/retention policies can be implemented and enforced
• Ensure that operational system integrations driven from the data stack are running, monitored and available
• Continually review the data platform to ensure that it is fit for purpose and meeting the needs of the business
• Support and maintain downstream integrations from our data lake, for example business intelligence and visualization tools and third party systems

Qualifications

• Experience in the data engineering field with demonstrated design and technical implementation of data warehouses
• Experience with OLAP databases, how they differ from OLTP databases, and data structuring/modeling with understanding of key data points in a business sense for trade-offs between storage/performance and usability
• Experience with building, deploying, and troubleshooting reliable, idempotent and consistent data pipelines working with disparate in-house and external data-sources, e.g. using Airflow DAGs
• Experience with AWS Redshift, Glue Data Catalog, S3, PostgreSQL, Parquet, Iceberg, Trino, and how they are managed using Terraform & Kubernetes
• Experience with data loading, extraction, manipulation and preparing data for ingestion and integration with visualization platforms such as Tableau.

Skills

• AWS
• Data Structures
• Database Management
• SQL