1.Apache Tez
(Tez is the next generation Hadoop Query Processing framework written on top of YARN. 它是一款基于YARN的新一代数据处理和分析框架，亮点：支持DAG调度作业，它能把多个有依赖的作业转换成一个作业，所以能大幅提升DAG性能，从而提升M-R查询性能)
tez.am.resource.memory.mb=1024
tez.task.resource.memory.mb=1024
tez.runtime.io.sort.mb=400
1.1 现象一
ERROR : Failed to execute tez graph. org.apache.tez.dag.api.TezException: org.apache.hadoop.yarn.exceptions.InvalidResourceRequestException: Invalid resource request, requested resource type=[memory-mb] < 0 or greater than maximum allowed allocation. Requested resource=<memory:8192, vCores:1>, maximum allowed allocation=<memory:4096, vCores:32>, please note that maximum allowed allocation is calculated by scheduler based on maximum resource of registered NodeManagers, which might be less than configured maximum allocation=<memory:4096, vCores:32> 原因是: ambari中设置Tez过大,下面两个参数被设置成了8192,启动时yarn单个container内存申请资源为8192,导致所需资源过大 tez.am.resource.memory.mb=1024
tez.task.resource.memory.mb=1024
1.2 现象二
设置的tez.runtime.io.sort.mb=512,大于设计的410MB
tez.runtime.io.sort.mb 512 should be larger than 0 and should be less than 410MB
1.3 现象三
当container以512MB启动时,执行过程内存超出
Container [pid=27426,containerID=container_e07_1596508490512_0184_02_000001] is running 17154048B beyond the 'PHYSICAL' memory limit. Current usage: 528.4 MB of 512 MB physical memory used; 2.4 GB of 1.0 GB virtual memory used. Killing container.
1.4 现象四
yarn资源不足,可以调整 yarn.scheduler.capacity.maximum-am-resource-percent=0.5
That your drivers have exceeded max memory configured in Max Application Master Resources. You can either increase max memory for AM or decrease driver memory in your jobs.
Application is added to the scheduler and is not yet activated. Queue's AM resource limit exceeded. Details : AM Partition = 3ssystem; AM Resource Request = <memory:4096, vCores:1>; Queue Resource Limit for AM = <memory:6144, vCores:3>; User AM Resource Limit of the queue = <memory:6144, vCores:3>; Queue AM Resource Usage = <memory:4096, vCores:1>;
1.5 现象五 hivesever2-site的权限验证
去掉验证
1.6 现象六 hdfs的权限问题
dfs.permissions.enabled=false