Construct a knowledge sharing workflow with AWS Lake Formation in your knowledge mesh

A key good thing about a knowledge mesh structure is permitting totally different traces of enterprise (LOBs) and organizational models to function independently and provide their knowledge as a product. This mannequin not solely permits organizations to scale, but in addition offers the end-to-end possession of sustaining the product to knowledge producers which can be the area specialists of the info. This possession entails sustaining the info pipelines, debugging ETL scripts, fixing knowledge high quality points, and conserving the catalog entries updated because the dataset evolves over time.

On the patron facet, groups can search the central catalog for related knowledge merchandise and request entry. Entry to the info is completed by way of the knowledge sharing function in AWS Lake Formation. As the quantity of knowledge merchandise develop and doubtlessly extra delicate data is saved in a corporation’s knowledge lake, it’s necessary that the method and mechanism to request and grant entry to particular knowledge merchandise are performed in a scalable and safe method.

This publish describes how you can construct a workflow engine that automates the info sharing course of whereas together with a separate approval mechanism for knowledge merchandise which can be tagged as delicate (for instance, containing PII knowledge). Each the workflow and approval mechanism are customizable and must be tailored to stick to your organization’s inner processes. As well as, we embrace an optionally available workflow UI to display how you can combine with the workflow engine. The UI is only one instance of how the interplay works. In a typical massive enterprise, you too can use ticketing methods to mechanically set off each the workflow and the approval course of.

Resolution overview

A typical knowledge mesh structure for analytics in AWS accommodates one central account that collates all of the totally different knowledge merchandise from a number of producer accounts. Customers can search the obtainable knowledge merchandise in a single location. Sharing knowledge merchandise to shoppers doesn’t really make a separate copy, however as a substitute simply creates a pointer to the catalog merchandise. This implies any updates that producers make to their merchandise are mechanically mirrored within the central account in addition to in all the patron accounts.

Constructing on prime of this basis, the answer accommodates a number of elements, as depicted within the following diagram:

The central account contains the next elements:

• AWS Glue – Used for Information Catalog functions.
• AWS Lake Formation – Used to safe entry to the info in addition to present the info sharing capabilities that allow the info mesh structure.
• AWS Step Features – The precise workflow is outlined as a state machine. You possibly can customise this to stick to your group’s approval necessities.
• AWS Amplify – The workflow UI makes use of the Amplify framework to safe entry. It additionally makes use of Amplify to host the React-based utility. On the backend, the Amplify framework creates two Amazon Cognito elements to assist the safety necessities:
  • Consumer swimming pools – Present a person listing performance.
  • Identification swimming pools – Present federated sign-in capabilities utilizing Amazon Cognito person swimming pools as the placement of the person particulars. The identification swimming pools vend short-term credentials so the workflow UI can entry AWS Glue and Step Features APIs.
• AWS Lambda – Accommodates the appliance logic orchestrated by the Step Features state machine. It additionally supplies the mandatory utility logic when a producer approves or denies a request for entry.
• Amazon API Gateway – Supplies the API for producers to simply accept and deny requests.

The producer account accommodates the next elements:

The buyer account accommodates the next elements:

• AWS Glue – Used for Information Catalog functions.
• AWS Lake Formation – After the info has been made obtainable, shoppers can grant entry to its personal customers by way of Lake Formation.
• AWS Useful resource Entry Supervisor (AWS RAM) – If the grantee account is in the identical group because the grantor account, the shared useful resource is offered instantly to the grantee. If the grantee account just isn’t in the identical group, AWS RAM sends an invite to the grantee account to simply accept or reject the useful resource grant. For extra particulars about Lake Formation cross-account entry, see Cross-Account Entry: How It Works.

The answer is break up into a number of steps:

1. Deploy the central account backend, together with the workflow engine and its related elements.
2. Deploy the backend for the producer accounts. You possibly can repeat this step a number of instances relying on the variety of producer accounts that you just’re onboarding into the workflow engine.
3. Deploy the optionally available workflow UI within the central account to work together with the central account backend.

Workflow overview

The next diagram illustrates the workflow. On this specific instance, the state machine checks if the desk or database (relying on what’s being shared) has the pii_flag parameter and if it’s set to TRUE. If each circumstances are legitimate, it sends an approval request to the producer’s SNS subject. In any other case, it mechanically shares the product to the requesting shopper.

This workflow is the core of the answer, and could be custom-made to suit your group’s approval course of. As well as, you’ll be able to add customized parameters to databases, tables, and even columns to connect further metadata to assist the workflow logic.

Conditions

The next are the deployment necessities:

You possibly can clone the workflow UI and AWS CDK scripts from the GitHub repository.

Deploy the central account backend

To deploy the backend for the central account, go to the foundation of the challenge after cloning the GitHub repository and enter the next code:

yarn deploy-central --profile <PROFILE_OF_CENTRAL_ACCOUNT>

This deploys the next:

• IAM roles utilized by the Lambda features and Step Features state machine
• Lambda features
• The Step Features state machine (the workflow itself)
• An API Gateway

When the deployment is full, it generates a JSON file within the src/cfn-output.json location. This file is utilized by the UI deployment script to generate a scoped-down IAM coverage and workflow UI utility to find the state machine that was created by the AWS CDK script.

The precise AWS CDK scripts for the central account deployment are in infra/central/. This additionally contains the Lambda features (within the infra/central/features/ folder) which can be utilized by each the state machine and the API Gateway.

Lake Formation permissions

The next desk accommodates the minimal required permissions that the central account knowledge lake administrator must grant to the respective IAM roles for the backend to have entry to the AWS Glue Information Catalog.

Workflow catalog parameters

The workflow makes use of the next catalog parameters to offer its performance.

You should use UpdateDatabase and UpdateTable so as to add parameters to database and column-level granularity, respectively. Alternatively, you should utilize the CLI for AWS Glue so as to add the related parameters.

Use the AWS CLI to run the next command to test the present parameters in your database:

aws glue get-database --name <DATABASE_NAME> --profile <PROFILE_OF_CENTRAL_ACCOUNT>

You get the next response:

{
  "Database": {
    "Identify": "<DATABASE_NAME>",
    "CreateTime": "<CREATION_TIME>",
    "CreateTableDefaultPermissions": [],
    "CatalogId": "<CATALOG_ID>"
  }
}

To replace the database with the parameters indicated within the previous desk, we first create the enter JSON file, which accommodates the parameters that we wish to replace the database with. For instance, see the next code:

{
  "Identify": "<DATABASE_NAME>",
  "Parameters": {
    "data_owner": "<AWS_ACCOUNT_ID_OF_OWNER>",
    "data_owner_name": "<AWS_ACCOUNT_NAME_OF_OWNER>",
    "pii_flag": "true"
  }
}

Run the next command to replace the Information Catalog:

aws glue update-database --name <DATABASE_NAME> --database-input file://<FILE_NAME>.json --profile <PROFILE_OF_CENTRAL_ACCOUNT>

Deploy the producer account backend

To deploy the backend in your producer accounts, go to the foundation of the challenge and run the next command:

yarn deploy-producer --profile <PROFILE_OF_PRODUCER_ACCOUNT> --parameters centralMeshAccountId=<central_account_account_id>

This deploys the next:

• An SNS subject the place approval requests get printed.
• The ProducerWorkflowRole IAM position with a belief relationship to the central account. This position permits Amazon SNS publish to the beforehand created SNS subject.

You possibly can run this deployment script a number of instances, every time pointing to a special producer account that you just wish to take part within the workflow.

To obtain notification emails, subscribe your e-mail within the SNS subject that the deployment script created. For instance, our subject is named DataLakeSharingApproval. To get the total ARN, you’ll be able to both go to the Amazon Easy Notification Service console or run the next command to record all of the matters and get the ARN for DataLakeSharingApproval:

aws sns list-topics --profile <PROFILE_OF_PRODUCER_ACCOUNT>

After you’ve the ARN, you’ll be able to subscribe your e-mail by working the next command:

aws sns subscribe --topic-arn <TOPIC_ARN> --protocol e-mail --notification-endpoint <EMAIL_ADDRESS> --profile <PROFILE_OF_PRODUCER_ACCOUNT>

You then obtain a affirmation e-mail by way of the e-mail handle that you just subscribed. Select Verify subscription to obtain notifications from this SNS subject.

Deploy the workflow UI

The workflow UI is designed to be deployed within the central account the place the central knowledge catalog is positioned.

To begin the deployment, enter the next command:

This deploys the next:

• Amazon Cognito person pool and identification pool
• React-based utility to work together with the catalog and request knowledge entry

The deployment command prompts you for the next data:

• Challenge data – Use the default values.
• AWS authentication – Use your profile for the central account. Amplify makes use of this profile to deploy the backend assets.

UI authentication – Use the default configuration and your username. Select No, I’m performed when requested to configure superior settings.

• UI internet hosting – Use internet hosting with the Amplify console and select handbook deployment.

The script offers a abstract of what’s deployed. Coming into Y triggers the assets to be deployed within the backend. The immediate seems to be just like the next screenshot:

When the deployment is full, the remaining immediate is for the preliminary person data akin to person title and e-mail. A brief password is mechanically generated and despatched to the e-mail supplied. The person is required to vary the password after the primary login.

The deployment script grants IAM permissions to the person by way of an inline coverage hooked up to the Amazon Cognito authenticated IAM position:

{
   "Model":"2012-10-17",
   "Assertion":[
      {
         "Effect":"Allow",
         "Action":[
            "glue:GetDatabase",
            "glue:GetTables",
            "glue:GetDatabases",
            "glue:GetTable"
         ],
         "Useful resource":"*"
      },
      {
         "Impact":"Permit",
         "Motion":[
            "states:ListExecutions",
            "states:StartExecution"
         ],
         "Useful resource":[
"arn:aws:states:<REGION>:<AWS_ACCOUNT_ID>:stateMachine:<STATE_MACHINE_NAME>"
]
      },
      {
         "Impact":"Permit",
         "Motion":[
             "states:DescribeExecution"
         ],
         "Useful resource":[
"arn:aws:states:<REGION>:<AWS_ACCOUNT_ID>:execution:<STATE_MACHINE_NAME>:*"
]
      }


   ]
}

The final remaining step is to grant Lake Formation permissions (DESCRIBE for each databases and tables) to the authenticated IAM position related to the Amazon Cognito identification pool. You could find the IAM position by working the next command:

cat amplify/team-provider-info.json

The IAM position title is within the AuthRoleName property below the awscloudformation key. After you grant the required permissions, you should utilize the URL supplied in your browser to open the workflow UI.

Your short-term password is emailed to you so you’ll be able to full the preliminary login, after which you’re requested to vary your password.

The primary web page after logging in is the record of databases that customers can entry.

Select Request Entry to see the database particulars and the record of tables.

Select Request Per Desk Entry and see extra particulars on the desk stage.

Going again within the earlier web page, we request database-level entry by coming into the patron account ID that receives the share request.

As a result of this database has been tagged with a pii_flag, the workflow must ship an approval request to the product proprietor. To obtain this approval request e-mail, the product proprietor’s e-mail must be subscribed to the DataLakeSharingApproval SNS subject within the product account. The small print ought to look just like the next screenshot:

The e-mail seems to be just like the next screenshot:

The product proprietor chooses the Approve hyperlink to set off the Step Features state machine to proceed working and share the catalog merchandise to the patron account.

For this instance, the patron account just isn’t a part of a corporation, so the admin of the patron account has to go to AWS RAM and settle for the invitation.

After the useful resource share is accepted, the shared database seems within the shopper account’s catalog.

Clear up

In case you now not want to make use of this resolution, use the supplied cleanup scripts to take away the deployed assets.

Producer account

To take away the deployed assets in producer accounts, run the next command for every producer account that you just deployed in:

yarn clean-producer --profile <PROFILE_OF_PRODUCER_ACCOUNT>

Central account

Run the next command to take away the workflow backend within the central account:

yarn clean-central --profile <PROFILE_OF_CENTRAL_ACCOUNT>

Workflow UI

The cleanup script for the workflow UI depends on an Amplify CLI command to provoke the teardown of the deployed assets. Moreover, you should utilize a customized script to take away the inline coverage within the authenticated IAM position utilized by Amazon Cognito in order that Amplify can absolutely clear up all of the deployed assets. Run the next command to set off the cleanup:

This command doesn’t require the profile parameter as a result of it makes use of the prevailing Amplify configuration to deduce the place the assets are deployed and which profile was used.

Conclusion

This publish demonstrated how you can construct a workflow engine to automate a corporation’s approval course of to realize entry to knowledge merchandise with various levels of sensitivity. Utilizing a workflow engine allows knowledge sharing in a self-service method whereas codifying your group’s inner processes to have the ability to safely scale as extra knowledge merchandise and groups get onboarded.

The supplied workflow UI demonstrated one attainable integration state of affairs. Different attainable integration situations embrace integration together with your group’s ticketing system to set off the workflow in addition to obtain and reply to approval requests, or integration with enterprise chat purposes to additional shorten the approval cycle.

Lastly, a excessive diploma of customization is feasible with the demonstrated method. Organizations have full management over the workflow, how knowledge product sensitivity ranges are outlined, what will get auto-approved and what wants additional approvals, the hierarchy of approvals (akin to a single approver or a number of approvers), and the way the approvals get delivered and acted upon. You possibly can reap the benefits of this flexibility to automate your organization’s processes to assist them safely speed up in the direction of being a data-driven group.

Concerning the Creator

Jan Michael Go Tan is a Principal Options Architect for Amazon Net Companies. He helps clients design scalable and revolutionary options with the AWS Cloud.