大数据平台安装文档

§ 环境安装

操作系统:Linux iZ42yo1ihl8Z 2.6.32-431.23.3.el6.x86_64 #1 SMP Thu Jul 31 17:20:51 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux。

§ 操作步骤

§ 安装node.js并配置

§ 在服务器新建install目录。

cd ~ //切换到当前用户目录
mkdir install     
cd install//切换到install目录
 

§ 下载最新版本的node.js

node.js中文网下载最新版本的node-v6.2.0-linux-x64.tar.gz,并利用ftp工具将其上传到install目录下。

§ 解压node.js压缩包

tar xvf node-v6.2.0-linux-x64.tar.gz
 

§ 配置node.js的环境变量

切换到解压后的node.js文件夹所在工作目录

cd node-v6.2.0-linux-x64 
 

查看此时工作目录的绝对路径

pwd //home/scapp/install/node-v6.2.0-linux-x64
 

打开环境变量设置目录

sudo vi /etc/profile
 

在末行输入如下配置:wq 保存并退出:

export NODE_HOME=/home/scapp/install/node-v6.2.0-linux-x64 //上面获取的绝对路径
export PATH=$PATH:$NODE_HOME/bin
export NODE_PATH=$PATH:$NODE_HOME/lib/node_modules
 

让配置生效:

$ sudo -s

$ source /etc/profile
 

§ 全局安装 pm2

§ 1.在线全局安装pm2

npm install -g pm2
 

§ 2.离线安装pm2(有pm2依赖包直接跳到第2步)

1 找一台可以在线安装pm2的机器,执行 npm install pm2 -g 安装pm2,执行 npm config get prefix, 看一下这台机器的npm默认全局安装目录,比如默认安装在/home/scapp/install/node-v6.2.0-linux-x64/lib/node_modules/,然后去这个目录下就能看到你刚刚安装的pm2程序;找到pm2 执行tar czvf pm2.tar.gz pm2 命令, 把pm2打包,在这里打包的好处,是因为该目录下面已经包含了pm2依赖的所有node模块

2 打完包,同样执行 npm config get prefix 看一下这台服务器的npm默认安装目录为/home/scapp/install/node-v6.2.0-linux-x64/lib/node_modules

3 执行cd /home/scapp/install/node-v6.2.0-linux-x64/lib/node_modules/命令切换到该目录下,利用ftp工具拷贝你的 pm2.tar.gz 包到该目录下,执行tar xvf pm2.tar.gz解压。

4 下面就是把这个命令加到系统环境中

cd /usr/bin

ln -s /home/scapp/install/node-v6.2.0-linux-x64/lib/node_modules/pm2/bin/pm2 pm2
 

§ 离线安装mongodb并配置

切换到install目录

cd ~
cd install
 

利用ftp工具将压缩包拷贝到install目录,并解压

tar xvf mongodb-linux-x86_64-3.2.8.tgz
 

将 test数据库文件夹拷贝到MongoDB的安装目录下

在安装目录下重启MongoDB服务

bin/mongod --dbpath ./test/
 

后台运行MongoDB,在MongoDB安装目录下继续执行:

bin/mongod -dbpath ./test/ -logpath ./test/mongo.log -logappend -fork -port 27017
 

§ Node.js项目初始化

§ 新建node_root目录,并切换到该目录

cd ~ //切换到当前用户目录
mkdir node_root
cd node_root//切换到node_root目录
 

§ 拷贝相关web服务项目到该目录下

§ 1.直接拷贝express服务压缩包解压到当前目录即可
unzip pcitc-bigdata-express.zip  //解压express服务压缩包
 
§ 2.利用git克隆远端express服务项目
git clone git@10.248.10.170:bigdata/pcitc-bigdata-express-index.git //大数据分析平台首页

git clone git@10.248.10.170:bigdata/pcitc-bigdata-express-help.git //大数据分析平台帮助中心

git clone git@10.248.10.170:bigdata/pcitc-bigdata-express-react-platform.git //大数据分析平台工作台
                    
git clone git@10.248.10.170:bigdata/pcitc-bigdata-express-visualization-service.git //可视化工作台
 

§ 利用pm2命令启动相关服务

cd /home/scapp/node_root/pcitc-bigdata-express-index

pm2 start ./index-server-3001   //启动大数据分析平台首页服务

cd /home/scapp/node_root/pcitc-bigdata-express-help

pm2 start ./help-server-3009   //启动大数据分析平台帮助中心服务

cd /home/scapp/node_root/pcitc-bigdata-express-react-platform

pm2 start ./bin/react-visualization-server-3083   //启动大数据分析平台工作台服务

cd /home/scapp/node_root/pcitc-bigdata-express-visualization-service

pm2 start ./bin/visualization-service-3081   //启动大数据可视化工作台服务
 

此时通过http://hostName:3001 http://hostName:3009 http://hostName:3083 进行访问。

§ ngnix反向代理

§ 1.在线安装nginx

yum install nginx
 

§ 修改配置文件


cd /etc/nginx   

chmod 777 nginx.conf   

vi nginx.conf
 

将以下配置内容替换原有配置内容:



#user  nobody;
worker_processes  1;

#error_log  logs/error.log;
#error_log  logs/error.log  notice;
#error_log  logs/error.log  info;

#pid        logs/nginx.pid;


events {
    worker_connections  1024;
}


http {
    include       mime.types;
    default_type  application/octet-stream;

    #log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
    #                  '$status $body_bytes_sent "$http_referer" '
    #                  '"$http_user_agent" "$http_x_forwarded_for"';

    #access_log  logs/access.log  main;

    sendfile        on;
    #tcp_nopush     on;

    #keepalive_timeout  0;
    keepalive_timeout  65;

    #gzip  on;

    server {
        listen       80;
        server_name  localhost;
	#root /ui;

        #charset koi8-r;

        #access_log  logs/host.access.log  main;
	location / {
           # root   html;
           # index  index.html index.htm;
	   # rewrite /(.*) /$1 break;
	   proxy_pass   http://localhost:3001;
	   # try_files $uri /index.html;
        }

        location /visual/ {
           # root   html;
           # index  index.html index.htm;
           rewrite /visual/(.*) /$1 break;
           proxy_pass   http://localhost:3082;
           # try_files $uri /index.html;
           #  try_files $uri  /visual/index.html;
           #  root html;
	   #  proxy_pass   http://localhost:3082;
        }
        location /platform/ {
              rewrite /platform/(.*) /$1 break;
          # root   html;
          # index  index.html index.htm;
           proxy_pass   http://localhost:3083;
	}
        location ^~/fonts/{
	    rewrite "^(.*)/fonts(.*)$" $1/platform/fonts$2;
	 }

       location /conter/{
           rewrite /conter/(.*) /$1 break;
          # root   html;
          # index  index.html index.htm;
           proxy_pass   http://localhost:3083;
         }

       location /view/{
           rewrite /view/(.*) /$1 break;
          # root   html;
          # index  index.html index.htm;
           proxy_pass   http://localhost:3083;
         }



        location /help/ {
           # root   html;
           #index  index.html index.htm;
           rewrite /help/(.*) /$1 break;
           proxy_pass   http://localhost:3009;
           # try_files $uri /index.html;
        }

        location /ui/ {
          rewrite /ui/(.*) /$1 break;
          # root   html;
          # index  index.html index.htm;
	   proxy_pass   http://localhost:3097;
        }

        location /react/ {
           # root   html;
           # index  index.html index.htm;
           rewrite /react/(.*) /$1 break;
           proxy_pass   http://localhost:3098;
           # try_files $uri /index.html;
        }
     #   location /bootstrap/ {
     #
     #       proxy_pass   http://localhost:3008;
     #   }


        #error_page  404              /404.html;

        # redirect server error pages to the static page /50x.html
        #
        error_page   500 502 503 504  /50x.html;
        location = /50x.html {
            root   html;
        }

        # proxy the PHP scripts to Apache listening on 127.0.0.1:80
        #
        #location ~ .php$ {
        #    proxy_pass   http://127.0.0.1;
        #}

        # pass the PHP scripts to FastCGI server listening on 127.0.0.1:9000
        #
        #location ~ .php$ {
        #    root           html;
        #    fastcgi_pass   127.0.0.1:9000;
        #    fastcgi_index  index.php;
        #    fastcgi_param  SCRIPT_FILENAME  /scripts$fastcgi_script_name;
        #    include        fastcgi_params;
        #}

        # deny access to .htaccess files, if Apache's document root
        # concurs with nginx's one
        #
        #location ~ /.ht {
        #    deny  all;
        #}
    }


    # another virtual host using mix of IP-, name-, and port-based configuration
    #
    #server {
    #    listen       8000;
    #    listen       somename:8080;
    #    server_name  somename  alias  another.alias;

    #    location / {
    #        root   html;
    #        index  index.html index.htm;
    #    }
    #}


    # HTTPS server
    #
    #server {
    #    listen       443 ssl;
    #    server_name  localhost;

    #    ssl_certificate      cert.pem;
    #    ssl_certificate_key  cert.key;

    #    ssl_session_cache    shared:SSL:1m;
    #    ssl_session_timeout  5m;

    #    ssl_ciphers  HIGH:!aNULL:!MD5;
    #    ssl_prefer_server_ciphers  on;

    #    location / {
    #        root   html;
    #        index  index.html index.htm;
    #    }
    #}

}
 

§ 启动ngnix服务

sudo service nginx start
 

此时可能出现报错信息: nginx: [emerg] socket() [::]:80 failed (97: Address family not supported by protocol) ,只需修改/etc/nginx/conf.d/default.conf即可

解决办法:


vim /etc/nginx/conf.d/default.conf

listen       80 default_server;
listen       [::]:80 default_server;


改为:
listen       80;
#listen       [::]:80 default_server;
 

即可,此时通过http://hostName http://hostName/help/ http://hostName/platform/ 进行相关访问。

§ 2.离线安装nginx

由于nginx依赖于pcre和zlib,所以在安装nginx时,需要先安装pcre和zlib(为了方便起见,我把pcre、zlib、nginx的包放到了同一个目录temp下)

cd ~ //切换到当前用户目录
mkdir temp
cd temp//切换到temp目录
 

利用ftp工具将 pcre-8.39.tar.gzzlib-1.2.8.tar.gznginx-1.10.1.tar.gz上传到temp目录下

§ 安装pcre

找到 pcre-8.39.tar.gz 压缩包

tar -zxvf pcre-8.39.tar.gz 
cd pcre-8.39
./configure
make
make install
 

§ 安装zlib

找到 zlib-1.2.8.tar.gz 压缩包

tar -xvf zlib-1.2.8.tar.gz
cd zlib-1.2.8
./configure
make
make install
 

§ 安装nginx

找到 nginx-1.10.1.tar.gz 压缩包

tar -zxvf nginx-1.10.1.tar.gz
cd nginx-1.10.1.tar.gz
./configure --with-pcre=../pcre-8.39 --with-zlib=../zlib-1.2.8
make
make install
 

至此nginx安装完毕

§ 编辑配置文件

cd /usr/local/nginx/sbin 
sudo vi ../conf/nginx.conf
 

将以下配置内容替换原有配置内容:



#user  nobody;
worker_processes  1;

#error_log  logs/error.log;
#error_log  logs/error.log  notice;
#error_log  logs/error.log  info;

#pid        logs/nginx.pid;


events {
    worker_connections  1024;
}


http {
    include       mime.types;
    default_type  application/octet-stream;

    #log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
    #                  '$status $body_bytes_sent "$http_referer" '
    #                  '"$http_user_agent" "$http_x_forwarded_for"';

    #access_log  logs/access.log  main;

    sendfile        on;
    #tcp_nopush     on;

    #keepalive_timeout  0;
    keepalive_timeout  65;

    #gzip  on;

    server {
        listen       80;
        server_name  localhost;
	#root /ui;

        #charset koi8-r;

        #access_log  logs/host.access.log  main;
	location / {
           # root   html;
           # index  index.html index.htm;
	   # rewrite /(.*) /$1 break;
	   proxy_pass   http://localhost:3001;
	   # try_files $uri /index.html;
        }

        location /visual/ {
           # root   html;
           # index  index.html index.htm;
           rewrite /visual/(.*) /$1 break;
           proxy_pass   http://localhost:3082;
           # try_files $uri /index.html;
           #  try_files $uri  /visual/index.html;
           #  root html;
	   #  proxy_pass   http://localhost:3082;
        }
        location /platform/ {
              rewrite /platform/(.*) /$1 break;
          # root   html;
          # index  index.html index.htm;
           proxy_pass   http://localhost:3083;
	}
        location ^~/fonts/{
	    rewrite "^(.*)/fonts(.*)$" $1/platform/fonts$2;
	 }

       location /conter/{
           rewrite /conter/(.*) /$1 break;
          # root   html;
          # index  index.html index.htm;
           proxy_pass   http://localhost:3083;
         }

       location /view/{
           rewrite /view/(.*) /$1 break;
          # root   html;
          # index  index.html index.htm;
           proxy_pass   http://localhost:3083;
         }



        location /help/ {
           # root   html;
           #index  index.html index.htm;
           rewrite /help/(.*) /$1 break;
           proxy_pass   http://localhost:3009;
           # try_files $uri /index.html;
        }

        location /ui/ {
          rewrite /ui/(.*) /$1 break;
          # root   html;
          # index  index.html index.htm;
	   proxy_pass   http://localhost:3097;
        }

        location /react/ {
           # root   html;
           # index  index.html index.htm;
           rewrite /react/(.*) /$1 break;
           proxy_pass   http://localhost:3098;
           # try_files $uri /index.html;
        }
     #   location /bootstrap/ {
     #
     #       proxy_pass   http://localhost:3008;
     #   }


        #error_page  404              /404.html;

        # redirect server error pages to the static page /50x.html
        #
        error_page   500 502 503 504  /50x.html;
        location = /50x.html {
            root   html;
        }

        # proxy the PHP scripts to Apache listening on 127.0.0.1:80
        #
        #location ~ .php$ {
        #    proxy_pass   http://127.0.0.1;
        #}

        # pass the PHP scripts to FastCGI server listening on 127.0.0.1:9000
        #
        #location ~ .php$ {
        #    root           html;
        #    fastcgi_pass   127.0.0.1:9000;
        #    fastcgi_index  index.php;
        #    fastcgi_param  SCRIPT_FILENAME  /scripts$fastcgi_script_name;
        #    include        fastcgi_params;
        #}

        # deny access to .htaccess files, if Apache's document root
        # concurs with nginx's one
        #
        #location ~ /.ht {
        #    deny  all;
        #}
    }


    # another virtual host using mix of IP-, name-, and port-based configuration
    #
    #server {
    #    listen       8000;
    #    listen       somename:8080;
    #    server_name  somename  alias  another.alias;

    #    location / {
    #        root   html;
    #        index  index.html index.htm;
    #    }
    #}


    # HTTPS server
    #
    #server {
    #    listen       443 ssl;
    #    server_name  localhost;

    #    ssl_certificate      cert.pem;
    #    ssl_certificate_key  cert.key;

    #    ssl_session_cache    shared:SSL:1m;
    #    ssl_session_timeout  5m;

    #    ssl_ciphers  HIGH:!aNULL:!MD5;
    #    ssl_prefer_server_ciphers  on;

    #    location / {
    #        root   html;
    #        index  index.html index.htm;
    #    }
    #}

}
 

保存配置并退出

§ 非平滑重启

sudo ./nginx -s stop && sudo ./nginx
 

nginx离线配置完成,此时通过http://hostName http://hostName/help/ http://hostName/platform/ 进行相关访问。

§ 开源产品界面布局修改

§ Spark分析工具

zeppelin 0.6.2版本,部署在210节点

zeppelin.png

修改了

/opt/zeppelin-0.6.2-bin-netinst/webapps/webapp/components/navbar navbar.html
/opt/zeppelin-0.6.2-bin-netinst/webapps/webapp/assets/styles/looknfeel default.css

§ Python分析工具

jupyter

部署在207节点

jupyter.png

修改了 /home/scapp/anaconda3/pkgs/notebook-4.2.3-py35_0/lib/python3.5/site-packages/notebook/static/style style.min.css

§ R分析工具

rstudio server

部署在206节点

说明:没有进行修改

§ Zabbix监控

部署在节点211

Zabbix.png

修改了

路径:/var/www/html/zabbix/styles default.css

原文地址:https://www.cnblogs.com/cxf520/p/6484888.html