warden创建容器的过程

基础知识（本节转自http://www.54chen.com/architecture/cloud-foundry-warden-part1.html）

rv = unshare(CLONE_NEWNS);

unshare这个调用，可以把挂载的文件系统设置成只在新的挂载命名空间（mount namespace）中可见。

execvp(argv[0], argv);

execvp()会从PATH 环境变量所指的目录中查找符合参数file 的文件名，找到后便执行该文件，然后将第二个参数argv传给该欲执行的文件。

shopt -s nullglob

设置shell环境变量nullglob的值为on，nullglob为on时对于通配符匹配时，若匹配不到时为空（相对应的为通配符本身）。

int stat(const char *restrict pathname, struct stat *restrict buf);

提供文件名字，获取文件对应属性。

build-essential软件包

作用是提供编译程序必须软件包的列表信息，也就是说编译程序有了这个软件包，它才知道头文件在哪，才知道库函数在哪，还会下载依赖的软件包，最后才组成一个开发环境。

工具debootstrap

可以用于在系统的某个目录中安装一套基本系统，这个基本系统除了一些配置项外，与ubuntu安装程序在安装的第一阶段安装的内容基本相同。这项功能有许多有趣的功能，例如，你可以从某个定制版本的ubuntu Live光盘上通过这个命令快速的在硬盘上安装ubuntu而不需要ubuntu的安装程序，也可以把创建在硬盘上的基本系统目录作为某些涉及系统安全性服务的chroot运行环境，通过chroot进入该目录并调试和运行一些可能修改系统配置的应用程序，作为定制小型系统模板等等。

aufs

一种文件格式，可以mount到目录，同时控制只读和读写。

overlayfs

另一种文件格式，在ubuntu 11.04后开始替代aufs作为官方livecd的文件格式。

cgroup的初始化

初始化的脚本为：

cgroup_path=/sys/fs/cgroup

mount -t tmpfs none $cgroup_path

for subsystem in cpu cpuacct devices memory

do

mkdir -p $cgroup_path/$subsystem

if ! grep -q "${cgroup_path}/$subsystem " /proc/mounts

then

mount -t cgroup -o $subsystem none $cgroup_path/$subsystem

fi

done

上面的命令执行完后，可以看下挂载的结果：

# grep '^cgroup' /proc/mounts

cgroups /sys/fs/cgroup tmpfs rw,relatime,mode=755 0 0

cgroup /sys/fs/cgroup/cpu cgroup rw,relatime,cpu 0 0

cgroup /sys/fs/cgroup/cpuacct cgroup rw,relatime,cpuacct 0 0

cgroup /sys/fs/cgroup/devices cgroup rw,relatime,devices 0 0

cgroup /sys/fs/cgroup/memory cgroup rw,relatime,memory 0 0

cgroup /sys/fs/cgroup/freezer cgroup rw,relatime,freezer 0 0

网络的控制

Every container is assigned a network interface which is one side of a
virtual ethernet pair created on the host. The other side of the virtual
ethernet pair is only visible on the host (from the root namespace).
The pair is configured to use IPs in a small and static subnet. Traffic
from and to the container can be forwarded using NAT. Additionally, all
traffic can be filtered and shaped as needed, using readily available
tools such as `iptables`.网络初始化

echo 1 > /proc/sys/net/ipv4/ip_forward

# iptables-save | grep -E 'warden|\*'
*nat
:warden-instance-16al6hojp15 - [0:0]
:warden-prerouting - [0:0]
-A PREROUTING -i eth0 -j warden-prerouting
-A OUTPUT -o lo -j warden-prerouting
-A warden-prerouting -j warden-instance-16al6hojp15
*mangle
*filter
:warden-default - [0:0]
:warden-dispatch - [0:0]
:warden-instance-16al6hojp15 - [0:0]
-A INPUT -i w-+ -j warden-dispatch
-A FORWARD -i w-+ -j warden-dispatch
-A warden-dispatch -p tcp -m tcp ! --tcp-flags FIN,SYN,RST,ACK SYN -j ACCEPT
-A warden-dispatch -i w-16al6hojp15-0 -g warden-instance-16al6hojp15
-A warden-dispatch -j DROP
-A warden-instance-16al6hojp15 -g warden-default

停止apparmor：

/etc/init.d/apparmor teardown

# quotaon(8) exits with non-zero status when quotas are ENABLED

if quotaon -p $CONTAINER_DEPOT_MOUNT_POINT_PATH > /dev/null

then

mount -o remount,usrjquota=aquota.user,grpjquota=aquota.group,jqfmt=vfsv0 $CONTAINER_DEPOT_MOUNT_POINT_PATH

quotacheck -ugmb -F vfsv0 $CONTAINER_DEPOT_MOUNT_POINT_PATH

quotaon $CONTAINER_DEPOT_MOUNT_POINT_PATH

fi

文件系统

Every container gets a private root filesystem. This filesystem is

created by stacking a read-only filesytem and a read-write filesystem.
This is implemented by using `aufs` on Ubuntu versions from 10.04 up to
11.10, and `overlayfs` on Ubuntu 12.04.

The read-only filesystem contains the minimal set of Ubuntu packages and
Warden-specific modifications common to all containers. The read-write
filesystem stores files overriding container-specific settings when
necessary. Because all writes are applied to the read-write filesystem,
containers can share the same read-only base filesystem.

The read-write filesystem is created by formatting a large sparse file.
Because the size of this file is fixed, the filesystem that it contains
cannot grow beyond this initial size.

创建一个容器

set -o nounset
set -o errexit
shopt -s nullglob

cp -r skeleton "${target}"
unshare -m "${target}"/setup.sh

"${target}"/setup.sh脚本

1. 首先生成一个配置文件

id=16al6hojp15
network_netmask=255.255.255.252
network_host_ip=10.254.0.17
network_host_iface=w-16al6hojp15-0
network_container_ip=10.254.0.18
network_container_iface=w-16al6hojp15-1
user_uid=10004

2. 调用setup_fs()

mkdir -p rootfs ${target}

mount -n -t overlayfs -o rw,upperdir=rootfs,lowerdir=${1} none ${target}

3. 调用prepare.sh 删除不需要的文件，并创建必要的设备

4. 配置系统参数

# cat etc/hosts
127.0.0.1 16al6hojp15 localhost
10.254.0.17 host
10.254.0.18 container

# cat etc/hostname
16al6hojp15

# cat etc/network/interfaces
auto lo
iface lo inet loopback
auto w-16al6hojp15-1
iface w-16al6hojp15-1 inet static
gateway 10.254.0.17
address 10.254.0.18
netmask 255.255.255.252

cp /etc/resolv.conf ${target}/etc/

chroot并添加用户：useradd -mU -u ${user_uid} -s /bin/bash vcap

# Copy override directory 其实就是几个etc文件和一个sbin/warden-stop.sh文件
cp -r override/* ${target}/
chmod 700 ${target}/sbin/warden-*

5. 配置ssh

6. 配置mesg

# The `mesg` tool modifies permissions on stdin. Warden regularly passes a
# custom stdin, which makes `mesg` complain that stdin is not a tty. Instead of
# removing all occurances of `mesg`, we simply bind it to /bin/true.
chroot <<EOS
rm /usr/bin/mesg
ln -s /bin/true /usr/bin/mesg
EOS

7. 物理机上添加网卡作为容器的网关：

ifconfig ${network_host_iface} ${network_host_ip} netmask ${network_netmask}

并添加tc规则控制带宽：

qdisc tbf 8002: dev w-16al6hojp15-0 root refcnt 2 rate 8192bit burst 9b lat 24.4ms

qdisc ingress ffff: dev w-16al6hojp15-0 parent ffff:fff1 ----------------

8. 配置cgroup

# Add new group for every subsystem

for system_path in /sys/fs/cgroup/*

do

instance_path=$system_path/instance-$id

mkdir -p $instance_path

if [ $(basename $system_path) == "cpuset" ]

then

cat $system_path/cpuset.cpus > $instance_path/cpuset.cpus

cat $system_path/cpuset.mems > $instance_path/cpuset.mems

fi

echo 1 > $instance_path/cgroup.clone_children

echo $PID > $instance_path/tasks

done

echo ${PPID} >> ppid

ip link add name ${network_host_iface} type veth peer name ${network_container_iface}

ip link set ${network_host_iface} netns 1

ip link set ${network_container_iface} netns ${PID}

创建好后，可以通过ssh -i access_key root@10.254.0.18 来登陆，密码是前面安装时设置的密码