Asserting the GA of Cloudera DataFlow for the Public Cloud

Posted in Enterprise |
August 16, 2021 3 min learn

Are you able to turbo-charge your knowledge flows on the cloud for max velocity and effectivity? We’re excited to announce the overall availability of Cloudera DataFlow for the Public Cloud (CDF-PC) – a model new expertise on the Cloudera Knowledge Platform (CDP) to deal with among the key operational and monitoring challenges of normal Apache NiFi clusters which can be overloaded with high-performant flows. Deploy, handle and monitor your commonplace NiFi flows operating on-premises or on CDP Knowledge Hub into cloud-native flows operating on Kubernetes clusters on AWS. 

Key operational and monitoring challenges of a NiFi administrator

• Useful resource rivalry – When a number of high-performant flows are loaded onto a single cluster, there could also be intervals when these flows could compete in opposition to one another for shared sources like CPU, reminiscence, I/O operations and so on. This will result in an total efficiency impression for all flows in that cluster.
• Proper-sizing the cluster – So as to deal with the crowded cluster state of affairs described above, directors typically err on the facet of warning and oversize the cluster. This will lead to some idle nodes throughout non-peak intervals, which may add to your infrastructure prices unnecessarily.
• Manually scaling the cluster on-demand – It isn’t a sustainable course of for an administrator to maintain watching the utilization of sources, after which including nodes when wanted and eradicating nodes when not wanted. There’s a robust want for automation right here.
• Centralized monitoring – As soon as knowledge flows transfer to the cloud, there could be a number of environments the place such flows could also be operating. It is going to be very exhausting for an administrator to look throughout the panorama of deployments and perceive the well being of all flows. Troubleshooting bottlenecks or figuring out chokepoints can be difficult with out a centralized dashboard.

NEW Cloudera DataFlow for the Public Cloud

We acknowledged these challenges from our personal buyer base and determined to make the Circulate Ops expertise rather a lot easier. The brand new providing allows NiFi/cloud directors and builders with key capabilities to make the method of circulation deployment / administration / monitoring very simple.

• Environments – You may allow DataFlow for any AWS surroundings you might have registered with CDP. The enablement course of creates the Kubernetes infrastructure required by CDF and every surroundings maps to 1 Kubernetes cluster.
• Circulate Catalog – A model new circulation catalog is now obtainable to import commonplace NiFi circulation definitions out of your current clusters into the brand new CDF for the Public Cloud. The catalog would be the start line for any circulation deployment. The catalog additionally helps versioning of flows.
• Deployment Wizard – Simplify the method of deploying flows into the brand new cloud-native surroundings with the model new deployment wizard. Utilizing the straightforward steps within the wizard, you’ll be able to provide configuration parameters, auto-scaling settings and KPI definitions to your circulation deployment.
• Dashboard – Monitor all of your deployed flows throughout all of your registered cloud environments in a single single view with the brand new centralized dashboard. For every circulation deployment, you’ll be able to open the deployment particulars pane which reveals you KPIs you might have outlined, system metrics in addition to system occasions and alerts.
• ReadyFlows – These are pre-built NiFi flows prepared to be used for among the most typical use circumstances like shifting knowledge from Kafka to an S3 bucket. Simply choose a Readyflow, present some configuration parameters after which deploy.

Key Advantages

• Increase your operational efficiencies by deploying flows in a streamlined method and by defining key metrics to measure their efficiency.
• Optimize your infrastructure setup by permitting CDF for Public Cloud to auto-scale your flows based mostly on wants. This ensures that you’re not over-sizing your infrastructure unnecessarily.
• Stop useful resource rivalry on crowded clusters by isolating your flows to their very own particular person cloud-native clusters.
• Monitor all of your circulation deployments throughout a number of cloud clusters from a single dashboard. This allows seamless troubleshooting from an administrator’s perspective. In-built alerting capabilities additionally make administration rather a lot simpler.
• Increase productiveness by getting a headstart on widespread circulation use circumstances by leveraging pre-built flows from a gallery of ReadyFlows

After collaborating within the technical preview program for Cloudera DataFlow for the Public Cloud, our prospects have been raving about it. We’re certain you’ll rave too. Take a look at cloudera.com/cdfx for extra data and property like datasheets, a product tour and extra. For a deep dive into the product and its options, try the technical deep dive weblog put up written by Michael Kohs, Sr. Product Supervisor, Cloudera.

Additionally, if you’re eager about seeing the product reside, we’re doing a reside launch webinar on August 18th, 2021. Register for it right now.