Producing and Viewing Lineage by means of Apache Ozone

Posted in Technical |
August 10, 2021 6 min learn

Observe your knowledge in object storage on-premises

As companies look to scale-out storage, they want a storage layer that’s performant, dependable and scalable. With Apache Ozone on the Cloudera Knowledge Platform (CDP), they’ll implement a scale-out mannequin and construct out their subsequent era storage structure with out sacrificing safety, governance and lineage. CDP integrates its present Shared Knowledge Expertise (SDX) with Ozone for a straightforward transition, so you possibly can start using object storage on-prem. On this article, we’ll deal with producing and viewing lineage that features Ozone belongings from Apache Atlas.

About Ozone integration with Atlas

With CDP 7.1.4 and later, Ozone is built-in with Atlas out of the field, and entities like Hive, Spark course of, and NiFi flows, will lead to Atlas creating Ozone path entities. For instance, writing a Spark dataset to Ozone or launching a DDL question in Hive that factors to a location in Ozone. Previous to that, entities created with an Ozone path resulted in creating HDFS path entities.

This integration mechanism doesn’t present a direct Atlas Hook or Atlas Bridge choice to take heed to the entity occasions in Ozone. As such, Atlas doesn’t have a direct hook and solely the trail data is supplied. As we’ll see, Atlas populates just a few different attributes for Ozone entities.

Earlier than we start

This text assumes that you’ve got a CDP Non-public Cloud Base cluster 7.1.5 or increased with Kerberos enabled and admin entry to each Ranger and Atlas. As well as, you will have a person that may create databases and tables in Hive, and create volumes and buckets in Ozone.

Additionally, to offer some context for these new to Ozone, it supplies three principal abstractions. If we take into consideration them from the attitude of conventional storage, we will draw the next analogies:

• Volumes are much like mount factors. Volumes are used to retailer buckets. Solely directors can create or delete volumes. As soon as a quantity is created, customers can create as many buckets as wanted.
• Buckets are much like subdirectories of that mount level. A bucket can include any variety of objects, however buckets can’t include different buckets. Ozone shops knowledge as objects which dwell inside these buckets.
• Keys are much like the totally certified path of recordsdata.

This enables for safety insurance policies on the quantity or bucket degree so you possibly can isolate customers, because it is sensible to your necessities. For instance, my knowledge quantity might include a number of buckets for each stage of the info, and I can management who accesses every stage. One other state of affairs may very well be every line of enterprise will get their very own quantity, and so they can create buckets and insurance policies because it is sensible for his or her customers.

Loading knowledge into Ozone

Let’s begin with loading knowledge into Ozone. With the intention to do this, we first have to create a quantity and a bucket utilizing the Ozone shell. The instructions to take action are as follows:

ozone sh quantity create /knowledge
ozone sh bucket create /knowledge/tpc

I’ve chosen these names as a result of I’ll be utilizing a straightforward technique for producing and writing TPC-DS datasets, together with creating their corresponding Hive tables. You’ll find the device I’m utilizing on the repo right here. Nevertheless, be at liberty to select your labels for the quantity and bucket and produce your knowledge. Earlier than that, let’s confirm the bucket exists in two methods:

1. Utilizing the Ozone shell

   ozone sh bucket record /knowledge

   You must see one thing much like

   {
    "metadata" : { },
     "volumeName" : "knowledge",
     "identify" : "tpc",
     "storageType" : "DISK",
     "versioning" : false,
     "usedBytes" : 0,
     "usedNamespace" : 0,
     "creationTime" : "2021-07-21T17:21:18.158Z",
     "modificationTime" : "2021-07-21T17:21:18.158Z",
     "encryptionKeyName" : null,
     "sourceVolume" : null,
     "sourceBucket" : null,
     "quotaInBytes" : -1,
     "quotaInNamespace" : -1
   }

2. Utilizing the Hadoop CLI
   

   hdfs dfs -ls ofs://ozone1/knowledge/tpc

   The Ozone URI takes the type of ofs://<ozone_service_id>/<quantity>/<bucket>. If you happen to don’t know the primary half, you possibly can simply discover it in Cloudera Supervisor by going to Clusters > Ozone and selecting the configuration tab. From there you possibly can seek for service.id and see one thing like

With our bucket created, let’s finalize safety for Hive. I discussed in the beginning that you just’d require a person with pretty open entry in Hive and Ozone. Nevertheless, regardless of that entry, there’s nonetheless another coverage required for us to create tables that time to a location in Ozone. Now that we’ve created that location, we will create a coverage like beneath:

As you possibly can see, this makes use of the URI in Ozone the place we loaded knowledge. By creating this coverage, we’re permitting SQL actions entry to the quantity and the bucket inside it. Thus, we’re in a position to create tables in Hive.

Now we will start loading knowledge.

• If you happen to’re bringing your personal, it’s so simple as creating the bucket in Ozone utilizing the Hadoop CLI and placing the info you need there:
  

  hdfs dfs -mkdir ofs://ozone1/knowledge/tpc/take a look at
  hdfs dfs -put <filename>.csv ofs://ozone1/knowledge/tpc/take a look at

• If you wish to use the info generator, clone the repo domestically and alter into the listing.
  

  git clone https://github.com/dstreev/hive-testbench.git

• From there, you possibly can run the next:
  

  hdfs dfs -mkdir ofs://ozone1/knowledge/tpc/tpcds
  ./tpcds-build.sh
  ./tpcds-gen.sh --scale 10 --dir ofs://ozone1/knowledge/tpc/tpcds

• Upon success, you’ll see these messages:
  

  TPC-DS textual content knowledge era full.
  Loading textual content knowledge into exterior tables.

If you should create a desk to your knowledge, it’s so simple as launching a DDL question with the situation pointing to the Ozone tackle the place the info was loaded. 

Viewing lineage in Atlas

As soon as tables are in place, we will start to see lineage in Atlas. To view the tables you’ve created, pull down the Search By Sort bar and enter hive_table. As soon as that’s been chosen, click on the search button to view all tables. 

Click on on the desk you wish to view and spot the lineage tab on the proper (see beneath). You’ll discover that Hive tables and processes are current however so are Ozone keys.

Now that we’ve got knowledge in Hive, let’s propagate the lineage with Spark.

Writing to Ozone in Spark

Because the goal of this weblog is to point out lineage for knowledge that exists in Ozone, I’m going to do a easy transformation within the Spark shell and write the info out to Ozone. Be at liberty to convey your code or run queries as you’d like towards the info you’ve got there. If you wish to comply with alongside, listed below are the steps:

1. Launch spark-shell in shopper mode
   

   spark-shell --master yarn --deploy-mode shopper --conf spark.yarn.entry.hadoopFileSystems=ofs://ozone1/knowledge/tpc/

2. Create a dataset from the client desk
   

   val customerDs = spark.sql("SELECT * FROM tpcds_text_10.buyer")

3. Fill in any nulls for the columns c_birth_year, c_birth_month, and c_birth_day. On this instance, we’re filling null years with 1970 and null months and days with 1.
   

   val noDobNulls = customerDs.na.fill(1970, Seq("c_birth_year")).na.fill(1, Seq("c_birth_month", "c_birth_day"))

4. Left pad the month and day columns in order that they’re all the identical size and create the birthDate column to concatenate 12 months, month, and day separated by a splash.
   

   val paddedMonth = lpad(col("c_birth_month"), 2, "0")
   val paddedDay = lpad(col("c_birth_day"), 2, "0")
   val birthDate = concat_ws("-", col("c_birth_year"), paddedMonth, paddedDay)

5. Add the c_birth_date column to the dataset the place we stuffed in null values
   

   val customerWithDob = noDobNulls.withColumn("c_birth_date", birthDate)

6. Write the remodeled knowledge to Ozone in ORC format
   

   customerWithDob.write.mode(SaveMode.Overwrite).orc("ofs://ozone1/knowledge/tpc/tpcds/10/customer_orc")

Going again to Atlas, you possibly can see the lineage has propagated from our Spark course of.

Abstract

Now you’ve got seen how CDP supplies a better transition to an on-prem, personal cloud structure with out sacrificing essential elements of safety, governance, and lineage. With Ozone in place, you possibly can start to scale-out compute and storage individually and overcome the constraints of HDFS. If you wish to proceed experimenting with Ozone and Atlas, you possibly can attempt writing to Ozone through Kafka utilizing our documented configuration examples. Then you possibly can import Kafka lineage utilizing the Atlas Kafka import device supplied with CDP. After that, checkout our Data Hub to see what all is accessible to you with CDP and the best way to get there!