Saying the GA of Cloudera DataFlow for the Public Cloud on Microsoft Azure

Posted in Enterprise |
February 10, 2022 6 min learn

After the launch of Cloudera DataFlow for the Public Cloud (CDF-PC) on AWS a number of months in the past, we’re thrilled to announce that CDF-PC is now typically accessible on Microsoft Azure, permitting NiFi customers on Azure to run their information flows in a cloud-native runtime. 

With CDF-PC, NiFi customers can import their present information flows right into a central catalog from the place they are often deployed to a Kubernetes based mostly runtime by means of a easy circulate deployment wizard or with a single CLI command. CDF-PC supplies a central monitoring dashboard for circulate deployments and provides customized KPI monitoring and alerting permitting prospects to remain on prime of what issues to them.

Determine 1: CDF-PC permits organizations to deploy their NiFi information flows to a cloud-native run time whereas offering central monitoring and cataloging capabilities

The necessity for a cloud-native Apache NiFi service on Microsoft Azure

And not using a cloud-native service to run NiFi flows on Microsoft Azure, organizations resorted to constructing and working NiFi clusters on both digital machines or their very own container based mostly infrastructure. Whereas Azure providers like Digital Machines, Managed Disks, Digital Networks and Azure Kubernetes Providers (AKS) make infrastructure provisioning and administration simpler, organizations had been nonetheless answerable for configuring, securing and working NiFi. This finally compelled NiFi groups to spend so much of time on managing the cluster infrastructure, stopping them from constructing new information flows and onboarding new use instances.

As we noticed a rising variety of organizations eager to run NiFi information flows on Azure however battling the operational challenges, it grew to become clear that there was a necessity for a cloud service that takes care of infrastructure administration and NiFi configuration to permit NiFi customers to concentrate on what issues most to them: Constructing new information flows and guaranteeing that these information flows meet the enterprise SLAs. 

Fixing Frequent Information Integration Use Circumstances with CDF-PC on Azure

CDF-PC helps Azure prospects implement key information integration use instances that require information motion, filtering and transformation at scale. Apache NiFi’s wealthy processor library supplies Azure centered processors like ADLS Gen2, Occasion Hub, Blob Storage or Cosmos DB out of the field. Extra Azure providers might be simply built-in by means of their APIs utilizing customizable NiFi processors like InvokeHTTP. 

SIEM Optimization

Determine 2: Shifting utility log information from Azure Occasion Hub to ADLS Gen2 and SIEM techniques

A standard use case on Azure is SIEM Optimization (SIEM=safety data and occasion administration) for analyzing utility log information. Cloud purposes might be configured to ship their logs to a central Azure Occasion Hub from the place CDF-PC circulate deployments choose up the log recordsdata to curate the occasions for the SIEM system. On the similar time the occasions might be saved in ADLS Gen2 storage for customized evaluation outdoors of the SIEM utility. Utilizing NiFi for this use case helps cut back the prices of the SIEM system and establishes a normal device which may take any utility log recordsdata and put together them for the SIEM system.

Processing Streaming Information

Determine 3: Shifting information from Azure Occasion Hub to ADLS Gen2

Fashionable purposes usually present streaming interfaces to ship transaction information in real-time to exterior techniques for evaluation. Apache Kafka deployments are generally used to buffer these messages for downstream consumption. Prospects can use Streams Messaging clusters in CDP Public Cloud to create enterprise grade Kafka deployments on Microsoft Azure. Since not each downstream utility is ready to immediately learn from Kafka matters, CDF-PC circulate deployments are sometimes used to learn and curate the occasions for evaluation by downstream techniques. A standard integration level for Azure providers is ADLS Gen2 for which NiFi supplies out of the field connectivity choices. On this use case NiFi deployments on CDF-PC are the bridge between streaming information and providers counting on information being accessible in ADLS Gen2.

Information Ingest for Microsoft Sentinel 

Determine 4: Shifting information from community infrastructure gadgets to Microsoft Sentinel

Microsoft Sentinel is an Azure native SIEM resolution that organizations use for assault detection, risk visibility, proactive searching, and risk response. Whereas Microsoft Sentinel supplies level integration for a lot of supply techniques, not each vendor or product is supported and might be immediately related. CDF-PC circulate deployments might help bridge the hole between unsupported gadgets and purposes by turning the uncooked machine log recordsdata right into a format that Microsoft Sentinel understands and ingesting it by means of its HTTP API. 

Getting a head begin with ReadyFlows

To assist organizations who are usually not as skilled with NiFi, CDF-PC comes with an built-in ReadyFlow Gallery which makes circulate deployments for fashionable use instances straightforward. As soon as they’ve recognized their ReadyFlow of selection, all they should do is begin the Deployment Wizard to offer connection parameters for supply and vacation spot techniques and the primary circulate deployment will probably be up and operating inside minutes. Immediately, CDF-PC helps Azure optimized ReadyFlows to maneuver information from Kafka to ADLS and between two totally different ADLS areas. Sooner or later we are going to present extra Azure optimized ReadyFlows to cowl the use instances talked about above.

Determine 5: Uncover Azure centered information flows within the built-in ReadyFlow Gallery

Leveraging key Microsoft Azure applied sciences to offer elastic, auto-scaling information flows

CDF-PC is powered by Microsoft Azure providers to offer a scalable infrastructure for NiFi information flows. CDF-PC manages the lifecycle of those infrastructure providers, liberating up NiFi directors from infrastructure upkeep duties resembling performing upgrades or making use of hotfixes for safety points.

Determine 6: CDF-PC excessive degree structure on Microsoft Azure

As Determine 6 reveals, CDF-PC creates and manages an AKS cluster in a digital community that consists of two node swimming pools – one for operating Cloudera infrastructure providers and one for operating CDF-PC and the precise NiFi circulate deployments. Every NiFi circulate deployment is created in its personal Kubernetes namespace for useful resource isolation functions. The NiFi circulate deployments can scale up and down based mostly on CPU utilization whereas AKS auto-scales the node swimming pools based mostly on useful resource utilization of scheduled pods throughout the cluster. CDF-PC additionally depends on ADLS Gen2 for storing utility and circulate deployment log recordsdata and an Azure Postgres database to retailer utility information. 

When CDF-PC is first enabled, customers can configure the minimal and most numbers of Nodes within the CDF-PC Node Pool which is able to scale up and down inside the boundaries as required. 

CDF-PC helps totally different networking setups and permits customers to configure which of the accessible subnets in a digital community must be used for the AKS cluster, whether or not customers ought to have the ability to entry CDF-PC by means of a public endpoint, in addition to limiting entry to CDF-PC to a listing of CIDR ranges.

Determine 8. Networking settings when enabling CDF-PC for an Azure surroundings

CDF-PC’s structure and configurable choices throughout service enablement make it versatile to work in any Azure setup whereas abstracting the complexity of the underlying infrastructure by means of easy wizards.

Abstract & Getting Began

With the Normal Availability of Cloudera DataFlow for the Public Cloud on Azure, we’re getting into a brand new period of operating Apache NiFi information flows in multi-cloud environments. For the primary time ever, Apache NiFi customers can handle and monitor information flows operating on Microsoft Azure or AWS from a single administration console. CDF-PC takes care of infrastructure administration, abstracts the variations between cloud suppliers and permits NiFi customers to actually concentrate on growing and operating their information flows.

Take our interactive product tour to get an impression of CDF-PC in motion or join a free trial.

Hyperlinks: