If there is a brief moment where 1) the current tasks exceed current environment capacity, followed by 2) a few minutes of no tasks executing or being queued, then 3) new tasks being queued.Īmazon MWAA autoscaling reacts to the first scenario by adding additional workers. This can occur for the following reasons: There may be tasks being deleted mid-execution that appear as task logs which stop with no further indication in Apache Airflow. You can use the update-environment command in the AWS Command Line Interface (AWS CLI) to change the minimum or maximum number of Workers that run on your environment.Īws mwaa update-environment -name MyEnvironmentName -min-workers 2 -max-workers 10 If there are a large number of tasks that were queued before autoscaling has had time to detect and deploy additional workers, we recommend staggering task deployment and/or increasing the minimum Apache Airflow Workers. If there are more tasks to run than an environment has the capacity to run, we recommend reducing the number of tasks that your DAGs run concurrently, and/or increasing the minimum Apache Airflow Workers. If there are more tasks to run than the environment has the capacity to run, and/or a large number of tasks that were queued before autoscaling has time to detect the tasks and deploy additional Workers. This often appears as a large-and growing-number of tasks in the "None" state, or as a large number in Queued Tasks and/or Tasks Pending in CloudWatch. There may be a large number of tasks in the queue. To learn more about the best practices we recommend to tune the performance of your environment, see Performance tuning for Apache Airflow on Amazon MWAA. There are other ways to optimize Apache Airflow configurations which are outside the scope of this guide. This leads to large Total Parse Time in CloudWatch Metrics or long DAG processing times in CloudWatch Logs. If you're using greater than 50% of your environment's capacity you may start overwhelming the Apache Airflow Scheduler.
#Airflow dag not updating update
Reduce the number of DAGs and perform an update of the environment (such as changing a log level) to force a reset.Īirflow parses DAGs whether they are enabled or not. There may be a large number of DAGs defined. I received a 'Broken DAG' error when using Amazon DynamoDB operators The following topic describes the errors you may receive when running DAGs. apache-airflow-backport-providers- xyz is compatible with Apache Airflow 1.10.12. Adding apache-airflow-providers-amazon causes my environment to failĪpache-airflow-providers- xyz is only compatible with Apache Airflow v2. The following topic describes the errors you may receive when updating your requirements.txt. I see a '503' error when triggering a DAG in the CLI.I see a 'The scheduler does not appear to be running' error.
I see a 5xx error accessing the web server.I'm using the BigQueryOperator and it's causing my web server to crash.I can't connect to my MySQL server on '.'.I received an error using the BigQuery operator.I received 'Broken DAG: No module named Cython' error.I received various errors installing Google/GCP/BigQuery.I received a 'Broken DAG' error when using the Slack operators.I received 'Broken DAG: No module named psycopg2' error.I received a 'Broken DAG' error when using Amazon DynamoDB operators.Adding apache-airflow-providers-amazon causes my environment to fail.
0 kommentar(er)
