hadoop cgroup+container配置

配置container-executor.cfg

vim etc/hadoop/container-executor.cfg
yarn.nodemanager.linux-container-executor.group=yarn
banned.users=root
min.user.id=1000
allowed.system.users=yarn

注意:这个配置文件是通过C语言解析的,编码格式等很敏感。曾踩在坑里很久才出来。提示“yarn.nodemanager.linux-container-executor.group”配置找不到。

Caused by: ExitCodeException exitCode=24: Can't get configured value for yarn.nodemanager.linux-container-executor.group.

  

配置文件是否正确、和权限是否正确,可通过 ./bin/container-executor --checksetup  检查。 

配置文件权限:(权限很重要,必须的步骤)

chown root:yarn bin/container-executor
chmod 6050 bin/container-executor 

  

配置yarn-site.xml

<!-- cgroup start -->
        <property>
            <name>yarn.nodemanager.container-executor.class</name>
            <value>org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor</value>
        </property>
        <property>
            <name>yarn.nodemanager.linux-container-executor.resources-handler.class</name>
            <value>org.apache.hadoop.yarn.server.nodemanager.util.CgroupsLCEResourcesHandler</value>
        </property>
        <property>
            <name>yarn.nodemanager.linux-container-executor.cgroups.hierarchy</name>
            <value>/hadoop-yarn</value>
        </property>
        <property>
            <name>yarn.nodemanager.linux-container-executor.cgroups.mount</name>
            <value>true</value>
        </property>
        <property>
            <name>yarn.nodemanager.linux-container-executor.cgroups.mount-path</name>
            <value>/cgroup</value>
        </property>
        <property>
            <name>yarn.nodemanager.linux-container-executor.group</name>
            <value>yarn</value>
        </property>
        <property>
            <name>yarn.nodemanager.resource.percentage-physical-cpu-limit</name>
            <value>90</value>
        </property>
        <property>
            <name>yarn.nodemanager.linux-container-executor.cgroups.strict-resource-usage</name>
            <value>true</value>
        </property>
        <!-- cgroup end -->

  

启动cgroup服务:

service cgconfig start

启动后在/cgroup/下多出一堆文件:

root@hadoop3 hadoop-2.7.3]# ll /cgroup/
total 0
drwxr-xr-x 2 root root 0 Sep 22 12:27 blkio
drwxr-xr-x 3 root root 0 Sep 22 12:27 cpu
drwxr-xr-x 2 root root 0 Sep 22 12:27 cpuacct
drwxr-xr-x 2 root root 0 Sep 22 12:27 cpuset
drwxr-xr-x 2 root root 0 Sep 22 12:27 devices
drwxr-xr-x 2 root root 0 Sep 22 12:27 freezer
drwxr-xr-x 2 root root 0 Sep 22 12:27 memory
drwxr-xr-x 2 root root 0 Sep 22 12:27 net_cls

还需要建立hadoop-yarn文件夹:(官方文档是说能自动建立,实际没有,可能是centos 6 和centos7区别的原因)

mkdir -p /cgroup/cpu/hadoop-yarn

如不建立可能会报如下错误:

org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor: Exit code from container executor initialization is : 22
ExitCodeException exitCode=22: Invalid permissions on container-executor binary.

Caused by: java.io.IOException: Linux container executor not configured properly (error=22)

  

./sbin/yarn-daemon.sh start nodemanager

done!  

  

  

原文地址:https://www.cnblogs.com/kisf/p/7575050.html