[ad_1]
Amazon Redshift is a quick, absolutely managed, extensively in style cloud information warehouse that powers the fashionable information structure that empowers you with quick and deep insights and machine studying (ML) predictions utilizing SQL throughout your information warehouse, information lake, and operational databases. A key differentiating issue of Amazon Redshift is its native integration with different AWS companies, which makes it simple to construct full, complete, and enterprise-level analytics purposes. The AWS Analytics Automation Toolkit allows automated provisioning and integration of not solely Amazon Redshift, however database migration companies like AWS Database Migration Service (AWS DMS) and the AWS Schema Conversion Software (AWS SCT).
This submit discusses new additions to the AWS Analytics Automation Toolkit, which allow you to carry out superior load testing on Amazon Redshift. That is completed by provisioning Apache JMeter as a part of the analytics stack.
Resolution overview
Apache JMeter is an open-source load testing utility written in Java that you should use to load take a look at internet purposes, backend server purposes, databases, and extra. Within the database context, it’s a particularly useful instrument for repeating benchmark checks in a constant method, simulating concurrency workloads, and scalability testing on totally different database configurations.
For instance, you should use JMeter to simulate a single enterprise intelligence (BI) person or a whole lot of BI customers concurrently operating varied SQL queries on an Amazon Redshift cluster for efficiency benchmarking, scalability, and throughput testing. Moreover, you possibly can rerun the identical actual simulation on a unique Amazon Redshift cluster that maybe has twice as many nodes as the unique cluster, to match the value/efficiency ratio of every cluster.
Equally, you should use JMeter to load take a look at and assess the efficiency and throughput achieved for combined extract, rework, and cargo (ETL) and BI workloads operating on totally different Amazon Redshift cluster configurations.
For a deeper dialogue of JMeter and its use for benchmarking Amazon Redshift, confer with Constructing high-quality benchmark checks for Amazon Redshift utilizing Apache JMeter.
Though JMeter set up is a comparatively easy course of, consisting primarily of downloading and putting in a Java digital machine and JMeter, the considered having to obtain, set up, and arrange any instrument for benchmarking functions can typically function a detractor for a lot of. Ranging from scratch for a take a look at setup is also intimidating.
The AWS Analytics Automation Toolkit now consists of the choice to robotically deploy JMeter on Amazon Elastic Compute Cloud (Amazon EC2) in the identical digital non-public cloud (Amazon VPC) as Amazon Redshift. This features a devoted Home windows occasion, with all required JMeter dependencies, corresponding to JVM and a pattern take a look at plan, thereby simply enabling highly effective load testing capabilities on Amazon Redshift. On this submit, we reveal using the AWS Analytics Automation Toolkit for JMeter load checks on cloud benchmark information, utilizing Amazon Redshift as a goal atmosphere.
This resolution has the next options:
- It deploys assets robotically, together with JMeter
- You possibly can level JMeter to an present Amazon Redshift cluster, or robotically create a brand new cluster
- You possibly can deliver your personal information and queries, or use a pattern TPC dataset
- You possibly can simply customise the take a look at plan into separate threads, every with totally different workloads and concurrency as wanted
To make use of the AWS Analytics Automation Toolkit to run a JMeter load take a look at, deploy the toolkit with the JMeter possibility, load information into your Amazon Redshift cluster, and customise the default take a look at plan as you see match.
The next diagram illustrates the answer structure:
Stipulations
Previous to deploying the AWS Analytics Automation Toolkit, full the conditions and put together the config file, user-config.json. For directions, confer with Automate constructing an built-in analytics resolution with AWS Analytics Automation Toolkit.
The config file has a brand new parameter referred to as JMETER, on the high part. To provision a JMeter occasion, enter the worth CREATE for this parameter.
To provision a brand new Amazon Redshift cluster, enter the worth CREATE for the REDSHIFT_ENDPOINT parameter. Then fill in values for the fields within the redshift part of the config file. You should utilize the sizing calculator on the Amazon Redshift console to suggest the right cluster configuration based mostly in your information measurement.
In order for you load an industry-standard pattern TPC-DS information (3TB) into your cluster, enter the worth “Y” for the “loadTPCdata” parameter, beneath the redshift part.
To make use of an present cluster, enter the endpoint of your cluster for the REDSHIFT_ENDPOINT parameter within the high part of the user-config.json file.
Deploy assets utilizing the AWS Analytics Automation Toolkit
To deploy your assets, full the next steps:
- Launch the toolkit as described in Automate constructing an built-in analytics resolution with AWS Analytics Automation Toolkit.
- Create tables and ingest your take a look at information into your Amazon Redshift cluster.
- Should you selected to load the pattern TPC-DS 3TB information, this can take a while to load, so please enable for this. If you’re loading your personal information then chances are you’ll do this at this level.
Launch JMeter
To launch JMeter, full the next steps:
- Utilizing RDP, log in to the JMeter EC2 Home windows occasion created by the AWS Analytics Automation Toolkit.
- Launch the JMeter GUI by selecting (double-clicking) the shortcut JMETER on the Home windows Desktop.
In our expertise, altering the JMeter Look and Really feel choice to Home windows (as an alternative of darkish mode) leads to elevated JMeter stability, so we extremely suggest making that change and selecting Sure to restart the GUI.
Customise the JMeter take a look at plan
To customise the JMeter take a look at plan, we modify the JDBC connection, and optionally modify the thread ramp-up schedule and optimize the SQL.
- Utilizing the JMeter GUI, open the AWS Analytics Automation Toolkit’s default take a look at plan file
c:JMETERapache-jmeter-5.4.1Redshift Load Take a look at.jmx. - Select the take a look at plan title and edit the
JdbcUserworth to the right person title on your Amazon Redshift cluster.
Should you used the CREATE cluster possibility, this worth is identical because the master_user_name worth in your user-config.json file.
- In JDBC connection, edit the
DatabaseURLworth and password with the right values on your cluster.
Should you used the CREATE cluster possibility, the password is stored in a secret named <stackname>-RedshiftPassword. You could find the endpoint by selecting the brand new Amazon Redshift cluster and copying the endpoint worth on the higher proper.
This take a look at plan already has a initialization command set to show off the end result cache function of Amazon Redshift: set enable_result_cache_for_session to off. No motion is required to configure this.
- Optionally, modify the thread ramp-up schedule.
This take a look at plan makes use of the Final Thread Group, which is robotically put in whenever you open the take a look at plan. Every thread group accommodates a ramp-up schedule, in addition to a question or set of queries. Modify these to in accordance with your testing preferences and dataset. Should you loaded the TPCDS dataset, the queries included by default within the three thread teams will work.
Within the following instance, Smallthread Group has 4 rows, every of which launches the precise variety of classes at staggered timings. The ramp-up time to attain the utmost session rely is 45 seconds, as a result of the final thread doesn’t begin till 15 seconds into the take a look at, and has a 30-second begin time. You possibly can regulate the ramp-up schedule, in addition to the maintain period and shutdown time, by modifying, including, or deleting rows within the Thread Schedule part. The graph is then robotically adjusted.
- Optionally, you possibly can customise the SQL.
Within the following instance, we select the JDBC request merchandise beneath the identical SmallThread Group, referred to as SmallSQL, and assessment the question run by this thread group. Should you added your personal information, insert the question or queries you need to run for this thread group. Do the identical for the medium and huge thread teams, or delete or add thread teams as wanted.
Run the take a look at
Run the take a look at plan by selecting the inexperienced arrow within the GUI.
Alternatively, enter the next command in a Home windows command immediate to run JMeter in command line mode:
Use case instance: Evaluating concurrency scaling advantages
For our instance use case, we need to decide how Amazon Redshift concurrency scaling advantages our workload efficiency. We will use the JMeter setup outlined on this submit to rapidly reply this query.
Within the following instance, we ran the pattern take a look at plan as is towards the pattern TPC-DS information, with 80 concurrent customers throughout three thread teams. We ran the take a look at first with concurrency scaling enabled, then reran it after disabling concurrency scaling on the Amazon Redshift cluster.
To watch the outcomes, open the Amazon Redshift console, select Clusters within the navigation pane, and select the cluster you’re utilizing for the efficiency take a look at. On the Cluster efficiency tab, you possibly can monitor the CPU utilization of all of the nodes in your cluster, as proven within the following screenshot.
On the Question monitoring tab, you possibly can monitor the queue exercise, in addition to concurrency scaling exercise of your cluster.
The previous graphs cowl two totally different checks with a 4-node ra3.16xlarge cluster. On the left aspect, concurrency scaling was enabled for the cluster, and on the fitting aspect it was disabled. Within the take a look at with concurrency scaling enabled, the queueing is much less, and take a look at completion period is shorter.
Be aware that the take a look at period is the time to run the workload on Amazon Redshift, to not obtain question outcomes.
Lastly, to assessment the precise take a look at end result question efficiency, you possibly can obtain the file C:JMETERapache-jmeter-5.4.1SummaryReportIndividualRecords.csv from the EC2 occasion, and assessment the question efficiency for every thread. The next screenshot is a abstract plot of the take a look at outcomes for this instance, with and with out concurrency scaling.
Because the chart illustrates, concurrency scaling can considerably cut back the latency on your workloads, and is especially helpful for brief bursts of exercise in your utility.
Conclusion
JMeter lets you create a versatile and highly effective load take a look at on your Amazon Redshift clusters to evaluate the efficiency of the cluster. With the brand new functionality within the AWS Analytics Automation Toolkit, you possibly can provision and configure JMeter, together with a take a look at plan, in a fraction of the time it will usually take.
Concerning the Authors
Samir Kakli is an Analytics POC Specialist Options Architect based mostly out of Florida. He’s targeted on serving to clients rapidly and successfully align Amazon Redshift’s capabilities to their enterprise wants.
Asser Moustafa is an Analytics Specialist Options Architect at AWS based mostly out of Dallas, Texas. He advises clients within the Americas on their Amazon Redshift and information lake architectures and migrations, ranging from the POC stage to precise manufacturing deployment and upkeep.
[ad_2]
