Created:        2020-05-25 Mon
Last modified:  2022-05-14 Sat

Big Data Workflows on AWS

This post is an overview of few articles from AWS blog regarding Big Data Workflows

• Orchestrate big data workflows with Apache Airflow, Genie, and Amazon EMR (Oct 2019)

• Build a Concurrent Data Orchestration Pipeline Using Amazon EMR and Apache Livy (Jul 2018)

• Orchestrate an ETL process using AWS Step Functions for Amazon Redshift (Jul 2019)

Orchestrate big data workflows with Apache Airflow, Genie, and Amazon EMR (Oct 2019)

• AWS Blog (Part 1)

• AWS Blog (Part 2)

CloudFormation Stack files:

• 1-MasterStack.yaml

• 2-SetupIAMRolesForAirflowGenieZookeeperAmazonEMRStack.yaml

• 2-SetupVPC.yaml

• 3-SetupSecurityGroupsForAirlfowGenieZookeeperRDS.yaml

• 4-SetupBastion.yaml

• 4-SetupEFSRDSRedis.yaml

• 5-SetupAirflowZKGenie.yaml

• 6-SetupAirflowInstanceRole.yaml

• 6-SetupAmazonEMRIAMRoles.yaml

• 6-SetupBastionInstanceRole.yaml

• 6-SetupGenieRoles.yaml

• 6-SetupZookeeperRoles.yaml

• 8-SetupEFS.yaml

• 8-SetupRDS.yaml

• 8-SetupREDIS.yaml

• 9-SetupAmazonEMRCluster.yaml

• 9-SetupApacheAirflow.yaml

• 9-SetupGenie.yaml

• 9-SetupZookeeperInstance.yaml

Option 1

Apache Airflow + Genie + Amazon EMR and Amazon S3

Cluster

• always-on clusters

• transient clusters

Custom Airflow Operators:

• GenieOperator

• EMR Airflow Operator can be used to spin up Amazon EMR clusters that register with Genie, run a job, and tear them down

Option 2

Apache Airflow + Apache Livy + Amazon EMR

Build a Concurrent Data Orchestration Pipeline Using Amazon EMR and Apache Livy (Jul 2018)

• AWS Blog

• AWS Sample

• CloudFormation Stack file

Apache Airflow + Apache Livy + Amazon EMR

• Apache Livy allow sending code (Scala or Python) over REST API calls

• Spark allows parralel execution, EMR Step API runs jobs serially

• Sample code uses non-reliable cluster termination, probably better to try EMR auto-termination

  • Auto-termination example

Orchestrate an ETL process using AWS Step Functions for Amazon Redshift (Jul 2019)

• AWS Blog

• CloudFormation Stack file

AWS Step Functions + AWS Lambda + AWS Batch

My notes

• Complicated workflow management

Other Useful/Useless Stuff

• Memory Management Spark on EMR

• How Verizon Media Group migrated from on-premises Apache Hadoop and Spark to Amazon EMR (Jan 2020)

• From Zero to EKS with Terraform and Helm

AWS Data Pipeline

• ETL Processing Using AWS Data Pipeline and Amazon Elastic MapReduce (Nov 2014)

• How Coursera Manages Large-Scale ETL using AWS Data Pipeline and Dataduct (Sep 2015)

AWS Glue

• Simplify data pipelines with AWS Glue automatic code generation and Workflows (Apr 2020)

• AWS Glue Workflows

Well-known workflow schedule tools

• Apache Oozie

• Apache Airflow

• Azkaban

• Cron

Other

• Apache Ni-Fi