Hadoop生态圈-HUE环境部署
作者：尹正杰
版权声明：原创作品，谢绝转载！否则将追究法律责任。
一.HUE简介
1>.HUE的由来
HUE全称是HadoopUser Experience，看这名字就知道怎么回事了吧。没错，直白来说就是Hadoop用户体验，是一个开源的Apache Hadoop UI系统，由Cloudera Desktop演化而来，最后Cloudera公司将其贡献给Apache基金会的Hadoop社区，它是基于Python Web框架Django实现的。通过使用HUE我们可以在浏览器端的Web控制台上与Hadoop集群进行交互来分析处理数据。
2>.官网地址
HUE官网地址可参考：http://gethue.com/overview/。
二.HUE安装
1>.部署环境介绍（我使用的是CentOS 7.2的操作系统）
2>.安装开发JDK版本
现在你生成环境用的Java开发版本，关于Java的下载地址可参考官网：http://www.oracle.com/technetwork/java/javase/downloads，下载后通过上传工具（如winscp等工具，或者你也可以使用linux自带的rz命令进行上传）
[yinzhengjie@s100 ~]$ sudo yum -y install lrzsz [sudo] password for yinzhengjie: Loaded plugins: fastestmirror Loading mirror speeds from cached hostfile * base: mirrors.huaweicloud.com * extras: mirrors.huaweicloud.com * updates: mirrors.aliyun.com Resolving Dependencies --> Running transaction check ---> Package lrzsz.x86_64 0:0.12.20-36.el7 will be installed --> Finished Dependency Resolution Dependencies Resolved ============================================================================================================ Package Arch Version Repository Size ============================================================================================================ Installing: lrzsz x86_64 0.12.20-36.el7 base 78 k Transaction Summary ============================================================================================================ Install 1 Package Total download size: 78 k Installed size: 181 k Downloading packages: lrzsz-0.12.20-36.el7.x86_64.rpm | 78 kB 00:00:00 Running transaction check Running transaction test Transaction test succeeded Running transaction Installing : lrzsz-0.12.20-36.el7.x86_64 1/1 Verifying : lrzsz-0.12.20-36.el7.x86_64 1/1 Installed: lrzsz.x86_64 0:0.12.20-36.el7 Complete! [yinzhengjie@s100 ~]$
[yinzhengjie@s100 ~]$ ll total 0 [yinzhengjie@s100 ~]$ rz #会弹出对话框，选择你要上传的文件即可 [yinzhengjie@s100 ~]$ ll total 181296 -rw-r--r--. 1 yinzhengjie yinzhengjie 185646832 Aug 11 18:26 jdk-8u181-linux-x64.tar.gz [yinzhengjie@s100 ~]$ [yinzhengjie@s100 ~]$ sudo mkdir /soft [yinzhengjie@s100 ~]$ sudo chown yinzhengjie:yinzhengjie /soft/ [yinzhengjie@s100 ~]$ tar zxf jdk-8u181-linux-x64.tar.gz -C /soft/ [yinzhengjie@s100 ~]$ ln -s /soft/jdk1.8.0_181/ /soft/jdk [yinzhengjie@s100 ~]$ ll /soft/ total 4 lrwxrwxrwx. 1 yinzhengjie yinzhengjie 19 Aug 12 03:33 jdk -> /soft/jdk1.8.0_181/ drwxr-xr-x. 7 yinzhengjie yinzhengjie 4096 Jul 7 04:09 jdk1.8.0_181 [yinzhengjie@s100 ~]$ [yinzhengjie@s100 ~]$ sudo vi /etc/profile [yinzhengjie@s100 ~]$ more /etc/profile | tail -3 #添加java的环境变量 #ADD JAVA_PATH JAVA_HOME=/soft/jdk PATH=$PATH:$JAVA_HOME/bin [yinzhengjie@s100 ~]$ [yinzhengjie@s100 ~]$ source /etc/profile #重新加载系统配置文件，使得配置文件在当前shell下生效 [yinzhengjie@s100 ~]$ [yinzhengjie@s100 ~]$ java -version #查看java版本 openjdk version "1.8.0_181" OpenJDK Runtime Environment (build 1.8.0_181-b13) OpenJDK 64-Bit Server VM (build 25.181-b13, mixed mode) [yinzhengjie@s100 ~]$
3>.准备环境依赖
查看官网安装文档（http://archive.cloudera.com/cdh5/cdh/5/hue-3.7.0-cdh5.3.0/manual.html），我们就可以看到相关的依赖环境如下：（CentOS具体依赖包参考：http://cloudera.github.io/hue/latest/admin-manual/manual.html#installation）
Centos安装操作系统环境如下:
[yinzhengjie@s100 ~]$sudo yum -y install apache-maven ant asciidoc cyrus-sasl-devel cyrus-sasl-gssapi gcc gcc-c++ krb5-devel libxml2-devel libxslt-devel make mysql mysql-devel openldap-devel python-devel sqlite-devel gmp-devel [sudo] password for yinzhengjie: Loaded plugins: fastestmirror Loading mirror speeds from cached hostfile * base: mirrors.huaweicloud.com * extras: mirrors.huaweicloud.com * updates: mirrors.aliyun.com .......... 接下来的信息我就省略了，安装过程以及成功后会有以下输出：（见下图）
安装过程如下：
安装成功如下：
4>.下载HUE安装包
[yinzhengjie@s100 ~]$ sudo yum -y install wget [sudo] password for yinzhengjie: Loaded plugins: fastestmirror Loading mirror speeds from cached hostfile * base: mirrors.huaweicloud.com * extras: mirrors.huaweicloud.com * updates: mirrors.aliyun.com Resolving Dependencies --> Running transaction check ---> Package wget.x86_64 0:1.14-15.el7_4.1 will be installed --> Finished Dependency Resolution Dependencies Resolved ============================================================================================================ Package Arch Version Repository Size ============================================================================================================ Installing: wget x86_64 1.14-15.el7_4.1 base 547 k Transaction Summary ============================================================================================================ Install 1 Package Total download size: 547 k Installed size: 2.0 M Downloading packages: wget-1.14-15.el7_4.1.x86_64.rpm | 547 kB 00:00:00 Running transaction check Running transaction test Transaction test succeeded Running transaction Installing : wget-1.14-15.el7_4.1.x86_64 1/1 Verifying : wget-1.14-15.el7_4.1.x86_64 1/1 Installed: wget.x86_64 0:1.14-15.el7_4.1 Complete! [yinzhengjie@s100 ~]$
5>.删除安装后的openJDK
温馨提示：使用yum安装这些包的同时，也会自动安装openJDK的依赖，所以，请自行删除安装后的openJDK
[yinzhengjie@s100 ~]$ rpm -qa | grep java javapackages-tools-3.4.1-11.el7.noarch java-1.8.0-openjdk-headless-1.8.0.181-3.b13.el7_5.x86_64 python-javapackages-3.4.1-11.el7.noarch tzdata-java-2018e-3.el7.noarch java-1.8.0-openjdk-1.8.0.181-3.b13.el7_5.x86_64 java-1.8.0-openjdk-devel-1.8.0.181-3.b13.el7_5.x86_64 [yinzhengjie@s100 ~]$ [yinzhengjie@s100 ~]$ sudo rpm -e --nodeps python-javapackages-3.4.1-11.el7.noarch [yinzhengjie@s100 ~]$ sudo rpm -e --nodeps javapackages-tools-3.4.1-11.el7.noarch [yinzhengjie@s100 ~]$ sudo rpm -e --nodeps java-1.8.0-openjdk-1.8.0.181-3.b13.el7_5.x86_64 [yinzhengjie@s100 ~]$ sudo rpm -e --nodeps java-1.8.0-openjdk-devel-1.8.0.181-3.b13.el7_5.x86_64 [yinzhengjie@s100 ~]$ sudo rpm -e --nodeps java-1.8.0-openjdk-headless-1.8.0.181-3.b13.el7_5.x86_64 [yinzhengjie@s100 ~]$ rpm -qa | grep java tzdata-java-2018e-3.el7.noarch [yinzhengjie@s100 ~]$
6>.解压HUE安装包
[yinzhengjie@s100 ~]$ ll total 216276 -rw-r--r--. 1 yinzhengjie yinzhengjie 35818977 Sep 19 2016 hue-3.7.0-cdh5.3.6.tar.gz -rw-r--r--. 1 yinzhengjie yinzhengjie 185646832 Aug 11 18:26 jdk-8u181-linux-x64.tar.gz [yinzhengjie@s100 ~]$ [yinzhengjie@s100 ~]$ tar -zxf hue-3.7.0-cdh5.3.6.tar.gz -C /soft/ [yinzhengjie@s100 ~]$ ll /soft/ total 8 drwxr-xr-x. 9 yinzhengjie yinzhengjie 4096 Jul 28 2015 hue-3.7.0-cdh5.3.6 lrwxrwxrwx. 1 yinzhengjie yinzhengjie 19 Aug 12 03:33 jdk -> /soft/jdk1.8.0_181/ drwxr-xr-x. 7 yinzhengjie yinzhengjie 4096 Jul 7 04:09 jdk1.8.0_181 [yinzhengjie@s100 ~]$
7>.编译HUE
[yinzhengjie@s100 ~]$ ll /soft/ total 8 drwxr-xr-x. 9 yinzhengjie yinzhengjie 4096 Jul 28 2015 hue-3.7.0-cdh5.3.6 lrwxrwxrwx. 1 yinzhengjie yinzhengjie 19 Aug 12 03:33 jdk -> /soft/jdk1.8.0_181/ drwxr-xr-x. 7 yinzhengjie yinzhengjie 4096 Jul 7 04:09 jdk1.8.0_181 [yinzhengjie@s100 ~]$ cd /soft/hue-3.7.0-cdh5.3.6/ [yinzhengjie@s100 hue-3.7.0-cdh5.3.6]$ ll total 64 drwxr-xr-x. 21 yinzhengjie yinzhengjie 4096 Jul 28 2015 apps drwxr-xr-x. 3 yinzhengjie yinzhengjie 4096 Jul 28 2015 cloudera drwxr-xr-x. 5 yinzhengjie yinzhengjie 54 Jul 28 2015 desktop drwxr-xr-x. 6 yinzhengjie yinzhengjie 4096 Jul 28 2015 docs drwxr-xr-x. 3 yinzhengjie yinzhengjie 23 Jul 28 2015 ext -rw-r--r--. 1 yinzhengjie yinzhengjie 11358 Jul 28 2015 LICENSE.txt -rw-r--r--. 1 yinzhengjie yinzhengjie 4715 Jul 28 2015 Makefile -rw-r--r--. 1 yinzhengjie yinzhengjie 8505 Jul 28 2015 Makefile.sdk -rw-r--r--. 1 yinzhengjie yinzhengjie 3498 Jul 28 2015 Makefile.vars -rw-r--r--. 1 yinzhengjie yinzhengjie 2192 Jul 28 2015 Makefile.vars.priv drwxr-xr-x. 2 yinzhengjie yinzhengjie 20 Jul 28 2015 maven -rw-r--r--. 1 yinzhengjie yinzhengjie 801 Jul 28 2015 NOTICE.txt -rw-r--r--. 1 yinzhengjie yinzhengjie 1517 Jul 28 2015 README drwxr-xr-x. 4 yinzhengjie yinzhengjie 65 Jul 28 2015 tools -rw-r--r--. 1 yinzhengjie yinzhengjie 932 Jul 28 2015 VERSION [yinzhengjie@s100 hue-3.7.0-cdh5.3.6]$ make apps #开始编译HUE --- Creating virtual environment at /soft/hue-3.7.0-cdh5.3.6/build/env python2.7 /soft/hue-3.7.0-cdh5.3.6/tools/virtual-bootstrap/virtual-bootstrap.py -qq --no-site-packages /soft/hue-3.7.0-cdh5.3.6/build/env --- /soft/hue-3.7.0-cdh5.3.6/build/env ready make[1]: Entering directory `/soft/hue-3.7.0-cdh5.3.6/desktop' make -C core env-install make[2]: Entering directory `/soft/hue-3.7.0-cdh5.3.6/desktop/core' --- Building egg for markdown running bdist_egg running egg_info ......... #编译时会出现下图的报错信息
解决方案：
解决方案： [yinzhengjie@s100 hue-3.7.0-cdh5.3.6]$ sudo vi /usr/include/openssl/x509.h [sudo] password for yinzhengjie: #删除以下两行： 751 X509_REVOKED *X509_REVOKED_dup(X509_REVOKED *rev); 752 X509_REQ *X509_REQ_dup(X509_REQ *req); [yinzhengjie@s100 hue-3.7.0-cdh5.3.6]$
使用root权限删除图中的751和752行代码，然后在重新编译即可。编译成功截图如下：
8>.配置HUE
[yinzhengjie@s100 hue-3.7.0-cdh5.3.6]$ vi desktop/conf/hue.ini [yinzhengjie@s100 hue-3.7.0-cdh5.3.6]$ grep secret_key= desktop/conf/hue.ini secret_key=jFE93j;2[290-eiw.KEiwN2s3['d;/.q[eIW^y#e=+Iei*@Mn<qW5o [yinzhengjie@s100 hue-3.7.0-cdh5.3.6]$ [yinzhengjie@s100 hue-3.7.0-cdh5.3.6]$ grep http_host= desktop/conf/hue.ini http_host=172.30.1.100 [yinzhengjie@s100 hue-3.7.0-cdh5.3.6]$ grep http_port= desktop/conf/hue.ini http_port=8888 [yinzhengjie@s100 hue-3.7.0-cdh5.3.6]$ grep time_zone= desktop/conf/hue.ini time_zone=Asia/Shanghai [yinzhengjie@s100 hue-3.7.0-cdh5.3.6]$
具体配置如下：
9>.启动HUE
10>.关闭服务器防火墙以及selinux服务。
[yinzhengjie@s100 ~]$ systemctl stop firewalld #临时关闭防火墙 [yinzhengjie@s100 ~]$ systemctl disable firewalld #禁用防火墙的服务开机自启 [yinzhengjie@s100 ~]$ [yinzhengjie@s100 ~]$ su root Password: [root@s100 yinzhengjie]# cat /etc/selinux/config # This file controls the state of SELinux on the system. # SELINUX= can take one of these three values: # enforcing - SELinux security policy is enforced. # permissive - SELinux prints warnings instead of enforcing. # disabled - No SELinux policy is loaded. SELINUX=enforcing # SELINUXTYPE= can take one of three two values: # targeted - Targeted processes are protected, # minimum - Modification of targeted policy. Only selected processes are protected. # mls - Multi Level Security protection. SELINUXTYPE=targeted [root@s100 yinzhengjie]# [root@s100 yinzhengjie]# getenforce Enforcing [root@s100 yinzhengjie]# [root@s100 yinzhengjie]# sed -i 's#SELINUX=enforcing#SELINUX=disabled#' /etc/selinux/config [root@s100 yinzhengjie]# [root@s100 yinzhengjie]# cat /etc/selinux/config | grep SELINUX= | grep -v ^# SELINUX=disabled [root@s100 yinzhengjie]#
11>.访问HUE的webUI并设置管理员账号
Duang~如果你出现以下界面说明你HUE启动成功啦！
三.通过git方式安装HUE
经查阅资料，发现github上也有教程教你如何安装HUE，需要你本地服务器安装git命令，详情请参考：https://github.com/cloudera/hue#development-prerequisites
[yinzhengjie@s100 ~]$ sudo yum -y install git [sudo] password for yinzhengjie: Loaded plugins: fastestmirror Loading mirror speeds from cached hostfile * base: mirrors.huaweicloud.com * extras: mirrors.huaweicloud.com * updates: mirrors.aliyun.com Resolving Dependencies --> Running transaction check ---> Package git.x86_64 0:1.8.3.1-14.el7_5 will be installed --> Processing Dependency: perl-Git = 1.8.3.1-14.el7_5 for package: git-1.8.3.1-14.el7_5.x86_64 --> Processing Dependency: rsync for package: git-1.8.3.1-14.el7_5.x86_64 --> Processing Dependency: perl(Term::ReadKey) for package: git-1.8.3.1-14.el7_5.x86_64 --> Processing Dependency: perl(Git) for package: git-1.8.3.1-14.el7_5.x86_64 --> Processing Dependency: perl(Error) for package: git-1.8.3.1-14.el7_5.x86_64 --> Processing Dependency: libgnome-keyring.so.0()(64bit) for package: git-1.8.3.1-14.el7_5.x86_64 --> Running transaction check ---> Package libgnome-keyring.x86_64 0:3.12.0-1.el7 will be installed ---> Package perl-Error.noarch 1:0.17020-2.el7 will be installed ---> Package perl-Git.noarch 0:1.8.3.1-14.el7_5 will be installed ---> Package perl-TermReadKey.x86_64 0:2.30-20.el7 will be installed ---> Package rsync.x86_64 0:3.1.2-4.el7 will be installed --> Finished Dependency Resolution Dependencies Resolved ========================================================================================================================================================== Package Arch Version Repository Size ========================================================================================================================================================== Installing: git x86_64 1.8.3.1-14.el7_5 updates 4.4 M Installing for dependencies: libgnome-keyring x86_64 3.12.0-1.el7 base 109 k perl-Error noarch 1:0.17020-2.el7 base 32 k perl-Git noarch 1.8.3.1-14.el7_5 updates 54 k perl-TermReadKey x86_64 2.30-20.el7 base 31 k rsync x86_64 3.1.2-4.el7 base 403 k Transaction Summary ========================================================================================================================================================== Install 1 Package (+5 Dependent packages) Total download size: 5.0 M Installed size: 23 M Downloading packages: (1/6): perl-TermReadKey-2.30-20.el7.x86_64.rpm | 31 kB 00:00:00 (2/6): libgnome-keyring-3.12.0-1.el7.x86_64.rpm | 109 kB 00:00:00 (3/6): perl-Error-0.17020-2.el7.noarch.rpm | 32 kB 00:00:00 (4/6): perl-Git-1.8.3.1-14.el7_5.noarch.rpm | 54 kB 00:00:00 (5/6): rsync-3.1.2-4.el7.x86_64.rpm | 403 kB 00:00:00 (6/6): git-1.8.3.1-14.el7_5.x86_64.rpm | 4.4 MB 00:00:02 ---------------------------------------------------------------------------------------------------------------------------------------------------------- Total 1.7 MB/s | 5.0 MB 00:00:02 Running transaction check Running transaction test Transaction test succeeded Running transaction Warning: RPMDB altered outside of yum. ** Found 12 pre-existing rpmdb problem(s), 'yum check' output follows: ant-1.9.2-9.el7.noarch has missing requires of java-devel >= ('0', '1.5.0', None) ant-1.9.2-9.el7.noarch has missing requires of jpackage-utils ant-1.9.2-9.el7.noarch has missing requires of jpackage-utils >= ('0', '1.7.5', None) xalan-j2-2.7.1-23.el7.noarch has missing requires of jpackage-utils xerces-j2-2.11.0-17.el7_0.noarch has missing requires of java xerces-j2-2.11.0-17.el7_0.noarch has missing requires of jpackage-utils xerces-j2-2.11.0-17.el7_0.noarch has missing requires of jpackage-utils xml-commons-apis-1.4.01-16.el7.noarch has missing requires of java xml-commons-apis-1.4.01-16.el7.noarch has missing requires of jpackage-utils xml-commons-apis-1.4.01-16.el7.noarch has missing requires of jpackage-utils xml-commons-resolver-1.2-15.el7.noarch has missing requires of jpackage-utils xml-commons-resolver-1.2-15.el7.noarch has missing requires of jpackage-utils Installing : 1:perl-Error-0.17020-2.el7.noarch 1/6 Installing : rsync-3.1.2-4.el7.x86_64 2/6 Installing : perl-TermReadKey-2.30-20.el7.x86_64 3/6 Installing : libgnome-keyring-3.12.0-1.el7.x86_64 4/6 Installing : perl-Git-1.8.3.1-14.el7_5.noarch 5/6 Installing : git-1.8.3.1-14.el7_5.x86_64 6/6 Verifying : git-1.8.3.1-14.el7_5.x86_64 1/6 Verifying : libgnome-keyring-3.12.0-1.el7.x86_64 2/6 Verifying : perl-TermReadKey-2.30-20.el7.x86_64 3/6 Verifying : 1:perl-Error-0.17020-2.el7.noarch 4/6 Verifying : rsync-3.1.2-4.el7.x86_64 5/6 Verifying : perl-Git-1.8.3.1-14.el7_5.noarch 6/6 Installed: git.x86_64 0:1.8.3.1-14.el7_5 Dependency Installed: libgnome-keyring.x86_64 0:3.12.0-1.el7 perl-Error.noarch 1:0.17020-2.el7 perl-Git.noarch 0:1.8.3.1-14.el7_5 perl-TermReadKey.x86_64 0:2.30-20.el7 rsync.x86_64 0:3.1.2-4.el7 Complete! [yinzhengjie@s100 ~]$
安装完git后按照GitHub的介绍，安装相应的软件。