How to use the ZooKeeper driver for ServiceGroup in OpenStack Nova

ServiceGroup APIs

Nova会从ServiceGroup API 中查询节点的存活信息。

ServiceGroup API 工作流程是：

当一个compute worker (running the nova-compute daemon) 启动，它会调用join API来加入到compute group,

这样其他需要这些信息的组件如scheduler就可以查询到这些信息 (by call get_all or get_one), 或者某一node的service是否存活通过service_is_up ServiceGroup API call.

在底层，ServiceGroup client driver 会自动的更新compute worker status。也可以call leave来从ServiceGroup中移除某一个node。

ServiceGroup Drivers

目前实现的有3个：database and ZooKeeper, memcached

/opt/stack/nova/nova/servicegroup/drivers$ ls

base.py  base.pyc  db.py  db.pyc  __init__.py  __init__.pyc  mc.py  zk.py

Database ServiceGroup driver

DB是最早支持的也是默认的driver，compute worker会周期的发送update到DB，周期为service_down_time。

def join(self, member, group, service=None):
           service.tg.add_timer(report_interval, self._report_state,
                                 api.INITIAL_REPORTING_DELAY, service)

DB driver有2个缺陷：

随着部署规模的增加，DB的压力增大。
信息过时，service_down_time之前的update并不一定现在是有效的。

ZooKeeper ServiceGroup driver

How it works

和DB相反，ZooKeeper是一个分布式系统。它的负载被划分在多个server上。

在compute node建立一个ZooKeeper sesion，node上会在group directory下生成一个临时的znode。而这个临时的znode和session的生命周期是同步的，

Driver只需要”ls“group directory就可以得到group membership。

    def join(self, member, group, service=None):
        """Add a new member to a service group.

        :param member: the joined member ID/name
        :param group: the group ID/name, of the joined member
        :param service: a `nova.service.Service` object
        """
        process_id = str(os.getpid())
        LOG.debug('ZooKeeperDriver: join new member %(id)s(%(pid)s) to the '
                  '%(gr)s group, service=%(sr)s',
                  {'id': member, 'pid': process_id,
                   'gr': group, 'sr': service})
        member = self._memberships.get((group, member), None)
        if member is None:
            # the first time to join. Generate a new object
            path = "%s/%s/%s" % (CONF.zookeeper.sg_prefix, group, member)
            try:
                zk_member = membership.Membership(self._session, path,
                                                  process_id)
            except RuntimeError:
                LOG.exception(_LE("Unable to join. It is possible that either"
                                  " another node exists with the same name, or"
                                  " this node just restarted. We will try "
                                  "again in a short while to make sure."))
                eventlet.sleep(CONF.zookeeper.sg_retry_interval)
                zk_member = membership.Membership(self._session, path, member)
            self._memberships[(group, member)] = zk_member

Installation and configuration

To use ZooKeeper, you'll need two client-side Python libraries on every nova node.

首先安装ZooKeeper servers

在安装ZooKeeper clients in Ubuntu:

sudo apt-get install python-zookeeper python-pip
sudo pip install evzookeeper

python-zookeeper is the official ZooKeeper Python binding. evzookeeper is the library to make the official binding work with the eventlet threading model.

After installation, make sure you have the following configuration snippet at the end of /etc/nova/nova.conf on every node:

servicegroup_driver="zk"
 
[zookeeper]
address="192.168.0.1:2181,192.168.0.2:2181,192.168.0.3:2181"

之后，你就可以nova-manage service list 来查询compute node的存活了。

https://github.com/maoy/writeups/blob/master/ZooKeeperInNova.md

memcached driver与DB类似，之不过后端DB是memcached。

另外cinder也在试图添加类似功能：

https://blueprints.launchpad.net/cinder/+spec/add-servicegroup-using-tooz