What’s new in Cloudera Streaming Analytics 1.5?

Posted in Technical |
October 12, 2021 4 min learn

On the finish of Could, we launched the second model of Cloudera SQL Stream Builder (SSB) as a part of Cloudera Streaming Analytics (CSA). Amongst different options, the 1.4 model of CSA surfaced the expressivity of Flink SQL in SQL Stream Builder by way of including DDL and Catalog help, and it tremendously improved the combination with different Cloudera Information Platform elements, for instance by way of enabling stream enrichment from Hive and Kudu. 

Since then, we now have added a RESTful API as a first-class citizen to SSB, doubled down on Flink SQL for outlining all facets of SQL jobs, and upgraded to Apache Flink 1.13. Now we’re releasing a brand new model of our product that takes the consumer expertise, technical capabilities, and manufacturing readiness to the subsequent stage.

Function Highlights

• Flink SQL scripts
• Templates for producing sinks for queries
• RESTful API for programmatic job submission
• Change Information Seize help
• Java UDF help

Flink SQL scripts

We’ve got enabled writing totally fledged SQL scripts in the principle editor window on the Compose tab of Streaming SQL Console, together with SET, DDL and DML statements with even a number of INSERT INTO statements in a single script. For instance, the next snippet is executable:

SET execution.goal=yarn-per-job;

CREATE TABLE IF NOT EXISTS datagen_sample (

  `col_int` INT,

  `col_ts` TIMESTAMP(3),

  WATERMARK FOR `col_ts` AS `col_ts` - INTERVAL '5' SECOND

) WITH (

  'connector' = 'datagen'

);

CREATE TABLE IF NOT EXISTS blackhole_sample (

  `col_int` INT,

  `col_ts` TIMESTAMP(3)

) WITH (

  'connector' = 'blackhole'

);

INSERT INTO blackhole_sample SELECT * FROM datagen_sample;

Because of this variation, we now have eliminated the choice so as to add Flink DDL tables utilizing the wizard on the Tables tab, and we encourage customers to outline them just like the above instance as an alternative.

When executing a number of INSERT INTO statements in a single job, SSB attaches the sampling to the final assertion. Moreover, SET statements can be utilized to configure any Flink configuration parameters. The presently set values are displayed on the Session tab. 

Sink templates

We’ve got added the Templates performance to generate a sink desk matching the schema inferred from the consumer’s question. 

When the templates are referred to as with an empty editor, they supply a default schema. In any other case, they infer it from the script accessible or chosen within the editor. Now that this performance is out there, we now have eliminated the power to create schemaless or “Dynamic schema” tables as they didn’t conform to our desk mannequin.

RESTful API for SQL Stream Builder

On this launch, we’re introducing a RESTful API for all SQL Stream Builder operations. This allows programmatic entry and automation of SQL Stream Builder jobs. The accompanying Swagger web page is out there as a part of our documentation. For instance the next name creates a self-contained new job:

curl --location --request POST '<streaming_sql_engine_host>:<streaming_sql_engine_port>/api/v1/ssb/sql/execute' 

--header 'Content material-Sort: utility/json' 

--data-raw '{

    "sql": "CREATE TABLE IF NOT EXISTS datagen_sample (col_int INT, col_ts TIMESTAMP(3), WATERMARK FOR col_ts AS col_ts - INTERVAL '''5''' SECOND) WITH ('''connector''' = '''datagen''');nSELECT * FROM datagen_sample;",

    "job_parameters": {

        "job_name": "production_job"

    }

}'

The GUI internally makes use of the identical endpoints, so the outcomes could be additionally noticed from the SQL Jobs tab. The default port for the Streaming SQL Engine is 18121.

Change Information Seize

We’re including help for Change Information Seize streams from relational databases based mostly on a group mission that wraps Flink as a runtime round logic imported from Debezium. This method doesn’t require modifications to the replicated database tables, as an alternative it hooks into the replication stream of the database.

For instance the next desk could be outlined to connect with an Oracle RDBMS CDC stream:

Supported CDC connector implementations can be found from the Templates characteristic.

Java Person Outlined Capabilities

SQL Stream Builder already had help for Javascript UDFs outlined on the GUI. Now we now have added the choice to make use of Flink SQL Java UDFs too by way of including them to the classpath.

For instance, the next easy increment operate applied as a Flink Java Operate:

package deal com.cloudera;

import org.apache.flink.desk.capabilities.ScalarFunction;

public class FlinkTestJavaUDF extends ScalarFunction {

    public Integer eval(Integer i) {

        return i + 1;

    }

}

Will be added after which used the next method on the aforementioned datagen_sample desk:

CREATE FUNCTION incrementer AS 'com.cloudera.FlinkTestJavaUDF' LANGUAGE java;

SELECT col_int, incrementer(col_int) as inc FROM datagen_sample;

Abstract

In Cloudera Streaming Analytics 1.5, we now have considerably improved the SQL Stream Builder performance and consumer expertise. We’ve got doubled down on Flink SQL by way of exposing SQL scripts and Java UDFs, added new performance with the Change Information Seize connectors, and enabled programmatic entry with a first-class citizen REST API.

Take the subsequent step and be taught extra about Cloudera Streaming Analytics.