Data Engineering
- www.astronomer.io 3 Key Takeaways from Airflow Summit 2023
Last week, the first-ever in-person Airflow Summit occurred in Toronto, Canada. Over 500 attendees from 20+ countries came together for all things Airflow, orchestration, and open source.
-
Unified Star Schema
Hi all,
I was recently reading about the Unified Star Schema and the Puppini Bridge. I’m curious whether anyone here has experience with it and what their thoughts are.
TIA
-
Visualising Kafka Internals
I plan to run a few tests to determine if Kafka is suitable for a certain usecase I have in mind.
My idea is to run a local cluster of Kafka servers (either VMs or containers), produce/consume a series of messages, observe a bunch of metrics (Prometheus & Grafana) and custom business logic outcomes.
What are some good tools to record and visualise the internals of Kafka cluster?
I'm looking for things like consumer lag, topic replication, possibly tracing messages, ...
Originally posted on https://mastodon.social/@bahmanm/110662538718523380
-
One big table vs a dimensional model
Hi fellow data engineers,
Currently I’m restructuring a pipeline written with pyspark on Databricks. Since it’s a lot of transformations, results in an extensive DAG, but it’s cool to spend some extra processing resources to make a standard dimensional model (apart from the necessary transformations).
Was wondering what real benefits you have seen a star schema design has from the “one big table” approach, I could preach to my team? (My goal mainly would be to have a resulting smaller PowerBI model.)
And as a side question, what tools do you use to create a dimensional model such a star schema with code?
Thanks a lot!
-
Free resource books
books.goalkicker.com Free Programming Books; HTML5, CSS3, JavaScript, PHP, Python...Free Programming Books on Android development, C, C#, CSS, HTML5, iOS development, Java, JavaScript, PowerShell, PHP, Python, SQL Sever and more
Thought I’d share this link. I’m not affiliated in any way.
-
Data Vault 2.0 Advanced Material
Hey there community!
Does anyone have any resources they could share relating to Data Vault 2.0, specifically the joining of SAL and PIT tables? The two main books on the architecture are very sparse on this area, which I would have thought would be a fairly key component for any mid-to-large organisation.
-
Welcome to the community, want to join the mod team?
Fellow data engineers, looking forward to your contribution/participation to the communiy. If you want to help in managing the community, get in touch to join the team