Airflow 配置celery+rabbitmq
1、安装celery和rabbitmq组件
pip3 install apache-airflow[celery]
pip3 install apache-airflow[rabbitmq]
2、系统安装rabbit
1）安装 erlang
yum install -y erlang
如果报错如下：
[ ]# yum install erlang Loaded plugins: fastestmirror, langpacks Loading mirror speeds from cached hostfile * base: mirrors.aliyun.com * extras: mirrors.aliyun.com * updates: mirrors.aliyun.com No package erlang available. Error: Nothing to do
则需要安装第三方软件库 epel：
# 下载 wget https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm # 安装完成即可 rpm -ivh epel-release-latest-7.noarch.rpm
再次执行则可以成功安装erlang了！
2）安装 rabbitmq
wget http://www.rabbitmq.com/releases/rabbitmq-server/v3.6.15/rabbitmq-server-3.6.15-1.el6.noarch.rpm rpm --import https://www.rabbitmq.com/rabbitmq-release-signing-key.asc yum install -y rabbitmq-server-3.6.15-1.el6.noarch.rpm
3）配置rabbitmq（设置用户名root，密码rabbitmq，创建虚拟主机airflow-rabbitmq）
[root@centos-master airflow]# /sbin/service rabbitmq-server stop Stopping rabbitmq-server (via systemctl): [ OK ] [root@centos-master airflow]# /sbin/service rabbitmq-server start Starting rabbitmq-server (via systemctl): [ OK ] [root@centos-master airflow]# rabbitmqctl add_user root rabbitmq Creating user "root" [root@centos-master airflow]# rabbitmqctl add_vhost airflow-rabbitmq Creating vhost "airflow-rabbitmq" [root@centos-master airflow]# rabbitmqctl set_user_tags root airflow-rabbitmq Setting tags for user "root" to ['airflow-rabbitmq'] [root@centos-master airflow]# rabbitmqctl set_permissions -p airflow-rabbitmq root ".*" ".*" ".*" Setting permissions for user "root" in vhost "airflow-rabbitmq" [root@centos-master airflow]# rabbitmq-plugins enable rabbitmq_management The following plugins have been enabled: amqp_client cowlib cowboy rabbitmq_web_dispatch rabbitmq_management_agent rabbitmq_management
4）修改airflow配置
[root@centos-master airflow]# vi airflow.cfg executor = CeleryExecutor broker_url = amqp://root:rabbitmq@localhost:5672/airflow-rabbitmq result_backend = amqp://root:rabbitmq@localhost:5672/airflow-rabbitmq
[root@centos-master airflow]# echo "export C_FORCE_ROOT="True"" >> /etc/profile
[root@centos-master airflow]# source /etc/profile
Airflow 配置celery+redis
1.安装redis
yum -y install redis
2.修改配置文件
sudo vim /etc/redis.conf #bind 127.0.0.1 // 注释掉，使redis允许远程访问 requirepass redis // 修改这行，redis登录密码（自定义）
3.开启redis服务
systemctl start redis
4.python下安装redis库
pip3 install redis
5.安装celery
pip3 install celery
6.配置airflow.cfg
executor = CeleryExecutor broker_url = redis://127.0.0.1:6379/0 result_backend = redis://127.0.0.1:6379/0