[ad_1]
One morning at 8 am, I woke as much as the Cupboard Minister of India calling me. He stated, “Prukalpa, the quantity on this dashboard doesn’t appear proper.”
Frantic, I opened up my laptop computer and loaded the dashboard to appreciate the quantity was clearly off. And but, at that second, there was nothing I might do to clarify it. I might really feel myself dropping the credibility and hard-earned belief that had taken months to construct.
I known as my Challenge Supervisor, who was incredible at stakeholder administration however couldn’t perceive the nitty-gritties of knowledge. She known as our Information Analyst, who regarded on the dashboard and stated, “Looks as if one thing broke down within the pipeline”. Our Analyst then known as our solely Information Engineer, who pulled out logs from Apache Airflow. However he couldn’t troubleshoot it as a result of he didn’t know what the variables meant and didn’t have the information context.
It took us 8 hours and 4 folks to determine what went unsuitable. We misplaced time that day.
However extra importantly, we misplaced belief. Belief with our buyer. Belief in our group.
Belief is commonly not about issues breaking. In years of working with knowledge, I’ve realized that knowledge will all the time be chaos. However when issues break and you discover out too late, or you may’t clarify why one thing broke, that’s what breaks belief.
Think about if, at that second when the cupboard minister known as me, I might shortly open a dashboard and say, “Sure, looks as if the pipeline didn’t run on time at this time. We’ve obtained an alert and it has already been escalated to knowledge engineering.” And even higher, think about if the dashboard had an alert on it, signaling to the minister that one thing was unsuitable and he shouldn’t use it.
At the moment we’re excited to announce that Atlan natively integrates with Apache Airflow. For knowledge groups in all places, this implies extra transparency and belief, and fewer time spent debugging pipelines after a damaged dashboard or mismatched metrics.
Atlan + Airflow: Constructing an ecosystem of belief and transparency
With this integration, knowledge groups can construct higher knowledge engineering experiences centered round constructing data and belief of their knowledge.
First, Atlan’s integration with Airflow brings much-needed pipeline context to knowledge belongings.
Now you may share any sort of metadata from Airflow pipelines to Atlan knowledge asset profiles, the place knowledge analysts, scientists, and enterprise customers have entry to it. This opens up pipeline context and makes it absolutely clear in order that knowledge groups and customers can all the time know the standing of the information pipeline related to every knowledge asset.
Listed below are some nice context fields that we’ve seen folks deliver from Airflow to Atlan:
- Freshness: When was my desk final up to date?
- Run schedule: Did the pipeline run as anticipated?
- Pipeline standing: Was the final pipeline run profitable?
Atlan already connects to knowledge warehouses (e.g. Snowflake, Redshift) and BI instruments (e.g. Tableau and Looker). Bringing Airflow into this ecosystem additionally signifies that knowledge groups can now map relationships throughout all of their knowledge. Whether or not you’re loading in new knowledge, revising a pipeline, or establishing a dashboard, now you can assemble and visualize knowledge lineage from finish to finish.
Much less time debugging, extra time constructing
Getting an pressing name about damaged knowledge is likely one of the worst experiences for a knowledge group. As a substitute of calling everybody who has ever touched the information, now you can diagnose the issue in seconds.
All it takes is opening a knowledge asset profile and checking the pipeline standing and metrics. No extra hours of scrambling or damaged belief, Atlan and Airflow’s integration helps you to see all your knowledge and its context in a single place.
Able to get began with this integration? Take a look at a demo of Atlan.
[ad_2]
