spark(一) build

(1)编译前的准备工作,安装jdk,解压maven,解压spark,解压scala并配置相关的环境变量
export JAVA_HOME=/opt/module/jdk1.6.0_45
export PATH=$PATH:$JAVA_HOME/bin

export MAVEN_HOME=/opt/module/apache-maven-3.0.5
export PATH=$PATH:$MAVEN_HOME/bin

export SCALA_HOME=/opt/module/scala-2.10.4
export PATH=$PATH:$SCALA_HOME/bin
(2)如果使用mvn方式,还需增加 export MAVEN_OPTS="-Xmx2g -XX:MaxPermSize=512M -XX:ReservedCodeCacheSize=512m",不然会出现错误错误
一切工作准备就绪,则开始编译 mvn -Pyarn -Phadoop-2.4 -Dhadoop.version=2.4.0 -DskipTests clean package
这些数值设置的是不通的参数的值,可以查看官方文档,根据提示设置其他的值。
(3)如果使用的是make-distribution.sh,则需要做以下修改
配置域名服务器
#vim /resolv.conf
内容:
nameserver 8.8.8.8
nameserver 8.8.4.4
最后:
./make-distribution.sh --tgz -Pyarn -Phadoop-2.4 -Dhadoop.version=2.6.0-cdh5.4.0 -Phive-0.13.1 -Phive-thriftserver
./make-distribution.sh --tgz -Pyarn -Phadoop-2.4 -Dhadoop.version=2.6.0-cdh5.4.0 -Phive-0.13.1 -Phive-thriftserver
为了减少时间,我们对make-distribution.sh文件做如下修改
VERSION=1.3.0 #设置spark版本
SPARK_HADOOP_VERSION=2.6.0-cdh5.4.0
SPARK_HIVE=1
等待1个半小时 。。。。。。。。。。。。。。。。
出现了下边的内容,则成功了
Saving to outputFile=/opt/module/spark-1.3.0/external/kafka-assembly/scalastyle-output.xml
Processed 0 file(s)
Found 0 errors
Found 0 warnings
Found 0 infos
Finished in 0 ms
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO]
[INFO] Spark Project Parent POM .......................... SUCCESS [20.658s]
[INFO] Spark Project Networking .......................... SUCCESS [59.540s]
[INFO] Spark Project Shuffle Streaming Service ........... SUCCESS [19.474s]
[INFO] Spark Project Core ................................ SUCCESS [17:10.312s]
[INFO] Spark Project Bagel ............................... SUCCESS [3:25.347s]
[INFO] Spark Project GraphX .............................. SUCCESS [6:48.707s]
[INFO] Spark Project Streaming ........................... SUCCESS [4:51.339s]
[INFO] Spark Project Catalyst ............................ SUCCESS [6:09.566s]
[INFO] Spark Project SQL ................................. SUCCESS [5:40.546s]
[INFO] Spark Project ML Library .......................... SUCCESS [6:50.022s]
[INFO] Spark Project Tools ............................... SUCCESS [1:02.293s]
[INFO] Spark Project Hive ................................ SUCCESS [7:17.840s]
[INFO] Spark Project REPL ................................ SUCCESS [2:37.665s]
[INFO] Spark Project YARN ................................ SUCCESS [1:23.802s]
[INFO] Spark Project Assembly ............................ SUCCESS [3:10.951s]
[INFO] Spark Project External Twitter .................... SUCCESS [2:56.559s]
[INFO] Spark Project External Flume Sink ................. SUCCESS [1:07.428s]
[INFO] Spark Project External Flume ...................... SUCCESS [34.766s]
[INFO] Spark Project External MQTT ....................... SUCCESS [1:47.120s]
[INFO] Spark Project External ZeroMQ ..................... SUCCESS [38.234s]
[INFO] Spark Project External Kafka ...................... SUCCESS [1:05.513s]
[INFO] Spark Project Examples ............................ SUCCESS [7:23.760s]
[INFO] Spark Project YARN Shuffle Service ................ SUCCESS [40.643s]
[INFO] Spark Project External Kafka Assembly ............. SUCCESS [48.520s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 1:25:14.796s
[INFO] Finished at: Mon Nov 16 23:55:37 PST 2015
[INFO] Final Memory: 65M/200M
[INFO] ------------------------------------------------------------------------

原文地址:https://www.cnblogs.com/nolonely/p/4972270.html