Spark On Yarn报警告信息 WARN yarn.Client: Neither spark.yarn.jars nor spark.yarn.archive is set, falling back to uploading libraries under SPARK

Spark On Yarn报警告信息 WARN yarn.Client: Neither spark.yarn.jars nor spark.yarn.archive is set, falling back to uploading libraries under SPARK_HOME.

1 贴出完整日志信息

/01/28 03:57:58 INFO client.RMProxy: Connecting to ResourceManager at hdp1/192.168.56.201:8032
19/01/28 03:57:58 INFO yarn.Client: Requesting a new application from cluster with 3 NodeManagers
19/01/28 03:57:58 INFO yarn.Client: Verifying our application has not requested more than the maximum memory capability of the cluster (8192 MB per container)
19/01/28 03:57:58 INFO yarn.Client: Will allocate AM container, with 4505 MB memory including 409 MB overhead
19/01/28 03:57:58 INFO yarn.Client: Setting up container launch context for our AM
19/01/28 03:57:58 INFO yarn.Client: Setting up the launch environment for our AM container
19/01/28 03:57:58 INFO yarn.Client: Preparing resources for our AM container
19/01/28 03:58:00 WARN yarn.Client: Neither spark.yarn.jars nor spark.yarn.archive is set, falling back to uploading libraries under SPARK_HOME.
19/01/28 03:58:00 INFO yarn.Client: Uploading resource file:/mnt/software/spark-2.2.0-bin-hadoop2.6/spark-3045ce36-b8d7-4655-a9aa-e0509ef0ec89/__spark_libs__8280328345325459092.zip -> hdfs://hdp1:9000/user/root/.sparkStaging/application_1548663204656_0001/__spark_libs__8280328345325459092.zip
19/01/28 03:58:06 INFO yarn.Client: Uploading resource file:/mnt/software/examples/jars/spark-examples*.jar -> hdfs://hdp1:9000/user/root/.sparkStaging/application_1548663204656_0001/spark-examples*.jar
19/01/28 03:58:06 INFO yarn.Client: Deleted staging directory hdfs://hdp1:9000/user/root/.sparkStaging/application_1548663204656_0001

2 原因，原因是因为Spark提交任务到yarn集群，需要上传Hadoop相关yarn的jar包

3 解决办法，提前上传到HDFS集群，并且在Spark配置文件指定文件路径，就可以避免每次提交任务到Yarn都需要重复上传文件

4 配置步骤 Spark版本 spark-2.2.0-bin-hadoop2.6

## 打包jars
jar cv0f spark-libs.jar -C $SPARK_HOME/jars/ .
## 新建hdfs路径
 hdfs dfs -mkdir -p /spark/jar
## 上传jars到HDFS
hdfs dfs -put spark-libs.jar /spark/jar
## 增加配置
vim spark-defaults.conf 
spark.yarn.archive=hdfs:///spark/jar/spark-libs.jar

4 重新提交任务，验证结果,没有问题，warn信息消失

root@hdp1 /mnt]#spark-submit --class org.apache.spark.examples.SparkPi --master yarn --deploy-mode cluster --driver-memory 2g --executor-memory 1g --executor-cores 1 --queue thequeue /mnt/software/spark-2.2.0-bin-hadoop2.6/examples/jars/spark-examples*.jar 10