It cannot handle load spikes at all.Īmazon Managed Workflows for Apache Airflow (MWAA) is a managed orchestration service in its nascent state as of now in 2021. It works well in a scenario where workloads are fairly fixed and growth is minimal or constant. This deployment model offers little dynamism in terms of scale-up or scale-down and the availability of the solution depends on the number of upfront provisioned instances. A set of dedicated EC2 instances takes care of the webserver components and another set of EC2 instances host the scheduler and executors. Depending on the variety of tasks and their resource requirements, EKS may need to launch a greater number of instances without maximum utilization of each node, leading to increased costs.ĭeployment of Airflow on EC2 is almost the same as you would deploy on an on-premises VM: sweet and simple, old-style deployment, pre-configured capacity, fixed nodes in the cluster, and pre-determined load balancing. The biggest drawback in the setup is that the cost for the smaller workloads may turn out to be higher. EKS keeps spawning new nodes with the Airflow executor or scheduler for handling new and heavy workloads. There is a huge community supporting Kubernetes initiatives and hence several ready-to-use configuration files are available for deploying Airflow using EKS. Kubernetes is the proven solution for auto-scaling, elasticity, and automatic resource management. What options are available to deploy Airflow on AWS?ĪWS provides a variety of options for deploying Airflow that can be categorized under IaaS, PaaS, as well as SaaS. The focus is always on the scheduler and executor components because they carry out the main workload ofĪpache Airflow and need clustering and autoscaling. The choice of deployment model is driven by the concerns such as performance, availability, and scalability. Some of the components of the Airflow, such as the Task creation and monitoring UI, need a webserver and other components, such as the scheduler and executor, need a It includes deployment of all components on a single VM or deployment of different components on separate single or load-balanced VMs. There are various deployment approaches available for Airflow.
0 Comments
Leave a Reply. |