OpenStack中的ImageCacheManager管理计算节点镜像缓存文件

假想过程

1. compute节点的driver检查本地镜像存储路径，没有找到镜像文件，于是从glance api利用image.fetch()下载镜像到本地。

2. hypervisor层使用镜像：从driver层拿到镜像路径后，hypervisor以自己的方式使用镜像，比如import等操作。

3. 使用完成后，hypervisor中已有了该镜像文件，调用返回给driver继续处理。

4. driver可以根据自身需要，比如为了节省空间，该镜像文件不再存储，删除之。

ImageCacheManager执行过程

1. 在compute.manager中以周期任务的方式运行，默认频率是image_cache_manager_interval=40分钟。

该配置值按理来说不应超过24小时（但doc中并没有给出该提示），原因在3说明。

2. 检查self.driver是否支持"has_imagecache"能力，libvirt默认支持，有的driver是不支持的。

3. storage_users.py中的两个函数register_xx和get_xx用于管理/var/lib/nova/instances/compute_nodes文件，

文件内容为dict，key为host名，value为该host更新compute_nodes文件的时间戳。由本步骤依次执行register和get可知，

该文件用于维护哪些计算节点正在使用本路径作为镜像存储路径，这些计算节点会周期性更新该文件中的时间戳，

表示：我还有可能在使用该路径中的镜像，请不要删除其中的文件！get_xx函数会判断时间戳是否在24小时之内，

并且只返回24小时之内有更新的host名称列表。

4. manager->driver->imagecache.update()，真正开始检查该目录下的镜像文件，并且执行删除无用的旧image的动作。

(1) _reset_state()清空上次检查的所有结果

(2) _scan_base_images()遍历目录下的镜像文件，根据文件名格式判断是image还是resized image还是swap文件，会分别放入不同列表中

(3) _list_running_instances()遍历all_instances，检查image/bdms等信息，返回used_images和used_swap_images，

含有local/remote信息（依据第3步的host名称列表区分）

(4) _age_and_verify_cached_images()对上一步返回的used_images信息做sha1，比对scan得到的当前目录中的images文件，

确定哪些是正在使用的文件（会调用os.utime(path)更新文件的修改时间），哪些是足够旧的不再使用的文件，最终会删除它们。

为不支持image cache manage的driver编写imagecache，要注意锁的使用（TODO）

参考libvirt.imagecache的编写思路，在_remove_old_enough_file(self, base_file, maxage, remove_lock=True)时会加锁和删除锁文件，

使用中有两种情形：_remove_swap_file()-->remove_lock=False，_remove_base_file()-->remove_lock=True，即base文件删除时，

会删除对应的锁文件，swap文件删除时，不会删除该锁文件（这又是为什么？）。

执行os.remove时，会获取一个锁文件，锁路径为self.lock_path = os.path.join(CONF.instances_path, 'locks')：nova/instances/locks/{sha1 of image_id}

锁的目的：确定用户目前不正在使用该镜像文件？在获取锁后会二次检查是否old：user of the file might have come along while we were waiting for the lock.

此时还有另外一个长期存在的锁文件：nova/instances/locks/nova-{sha1 of image_id}，comment显示：

The lock file will be constructed first time the image file was accessed.

该锁文件就是由_remove_old_enough_file()的remove_lock标志控制是否删除的，即如上所述，base文件删除时，会删除对应的锁文件，

swap文件删除时，不会删除该锁文件（这又是为什么？）。

检查锁文件的存放目录发现，该锁文件从创建后就一直存在。

由于对OpenStack锁机制的不了解，目前无法确定这两种锁在image cache manage中的用法和作用。能想到的竞态条件：如果存放路径是

一个共享目录的话，那么多个imagecahcemanager可能会对同一个image进行删除操作（这种情况应该还好，毕竟该image不再使用了）；

第二种是在判断了该image未使用而且已足够旧后，但是在os.remove删除执行前，该image被使用到了，这就比较危险，该锁应该就是前述

第一个锁的作用。

log记录

storage_register管理compute_nodes时的锁：

2月 15 18:45:30 nova-compute[16212]: DEBUG oslo_concurrency.lockutils [None req-8187e134-96ca-4adb-bfa0-385eb1b7edb0 None None] Lock "storage-registry-lock" acquired by "nova.virt.storage_users.do_register_storage_use" :: waited 0.000s {{(pid=16212) inner /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:327}}
2月 15 18:45:30 nova-compute[16212]: DEBUG oslo_concurrency.lockutils [None req-8187e134-96ca-4adb-bfa0-385eb1b7edb0 None None] Lock "storage-registry-lock" released by "nova.virt.storage_users.do_register_storage_use" :: held 0.001s {{(pid=16212) inner /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:339}}
2月 15 18:45:30 nova-compute[16212]: DEBUG oslo_concurrency.lockutils [None req-8187e134-96ca-4adb-bfa0-385eb1b7edb0 None None] Lock "storage-registry-lock" acquired by "nova.virt.storage_users.do_get_storage_users" :: waited 0.000s {{(pid=16212) inner /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:327}}
2月 15 18:45:30 nova-compute[16212]: DEBUG oslo_concurrency.lockutils [None req-8187e134-96ca-4adb-bfa0-385eb1b7edb0 None None] Lock "storage-registry-lock" released by "nova.virt.storage_users.do_get_storage_users" :: held 0.001s {{(pid=16212) inner /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:339}}

计算、匹配active的base文件：

2月 15 18:04:23 nova-compute[16212]: DEBUG nova.virt.libvirt.imagecache [None req-8187e134-96ca-4adb-bfa0-385eb1b7edb0 None None] Image id e0043447-2aa1-4c39-ae79-1e239ee519b3 yields fingerprint 1f629360db7c8ff0cde2a464f3784959aaced90c {{(pid=16212) _age_and_verify_cached_images}}
2月 15 18:04:23 nova-compute[16212]: INFO nova.virt.libvirt.imagecache [None req-8187e134-96ca-4adb-bfa0-385eb1b7edb0 None None] image e0043447-2aa1-4c39-ae79-1e239ee519b3 at (nova/instances/_base/1f629360db7c8ff0cde2a464f3784959aaced90c): checking
2月 15 18:04:23 nova-compute[16212]: DEBUG nova.virt.libvirt.imagecache [None req-8187e134-96ca-4adb-bfa0-385eb1b7edb0 None None] image e0043447-2aa1-4c39-ae79-1e239ee519b3 at (/opt/stack/data/nova/instances/_base/1f629360db7c8ff0cde2a464f3784959aaced90c): image is in use {{(pid=16212) _mark_in_use}}
2月 15 18:04:23 nova-compute[16212]: DEBUG nova.virt.libvirt.imagecache [None req-8187e134-96ca-4adb-bfa0-385eb1b7edb0 None None] Instance 9dbd9c9f-7d5b-4d1d-8ee3-4fdff68d2d70 is backed by 1f629360db7c8ff0cde2a464f3784959aaced90c {{(pid=16212) _list_backing_images}}
2月 15 18:04:23 nova-compute[16212]: INFO nova.virt.libvirt.imagecache [None req-8187e134-96ca-4adb-bfa0-385eb1b7edb0 None None] Active base files: /opt/stack/data/nova/instances/_base/1f629360db7c8ff0cde2a464f3784959aaced90c

删除base镜像的锁操作：

2月 15 19:25:54 nova-compute[16212]: DEBUG oslo_concurrency.lockutils [None req-8187e134-96ca-4adb-bfa0-385eb1b7edb0 None None] Lock "aaa29360db7c8ff0cde2a464f3784959aaced90c" acquired by "nova.virt.libvirt.imagecache._inner_remove_old_enough_file" :: waited 0.000s {{(pid=16212) inner /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:327}}
2月 15 19:25:54 nova-compute[16212]: DEBUG oslo_concurrency.lockutils [None req-8187e134-96ca-4adb-bfa0-385eb1b7edb0 None None] Lock "aaa29360db7c8ff0cde2a464f3784959aaced90c" released by "nova.virt.libvirt.imagecache._inner_remove_old_enough_file" :: held 0.001s {{(pid=16212) inner /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:339}}

删除swap文件的锁操作：

2月 15 19:25:54 nova-compute[16212]: DEBUG oslo_concurrency.lockutils [None req-8187e134-96ca-4adb-bfa0-385eb1b7edb0 None None] Lock "swap_3" acquired by "nova.virt.libvirt.imagecache._inner_remove_old_enough_file" :: waited 0.000s {{(pid=16212) inner /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:327}}
2月 15 19:25:54 nova-compute[16212]: DEBUG oslo_concurrency.lockutils [None req-8187e134-96ca-4adb-bfa0-385eb1b7edb0 None None] Lock "swap_3" released by "nova.virt.libvirt.imagecache._inner_remove_old_enough_file" :: held 0.001s {{(pid=16212) inner /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:339}}

总体过程，在swap_3被删掉后，锁nova-swap_3留了下来：

2月 15 19:25:54  nova-compute[16212]: DEBUG oslo_service.periodic_task [None req-8187e134-96ca-4adb-bfa0-385eb1b7edb0 None None] Running periodic task ComputeManager._run_image_cache_manager_pass {{(pid=16212) run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:217}}
2月 15 19:25:54  nova-compute[16212]: DEBUG oslo_concurrency.lockutils [None req-8187e134-96ca-4adb-bfa0-385eb1b7edb0 None None] Lock "storage-registry-lock" acquired by "nova.virt.storage_users.do_register_storage_use" :: waited 0.001s {{(pid=16212) inner /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:327}}
2月 15 19:25:54  nova-compute[16212]: DEBUG oslo_concurrency.lockutils [None req-8187e134-96ca-4adb-bfa0-385eb1b7edb0 None None] Lock "storage-registry-lock" released by "nova.virt.storage_users.do_register_storage_use" :: held 0.002s {{(pid=16212) inner /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:339}}
2月 15 19:25:54  nova-compute[16212]: DEBUG oslo_concurrency.lockutils [None req-8187e134-96ca-4adb-bfa0-385eb1b7edb0 None None] Lock "storage-registry-lock" acquired by "nova.virt.storage_users.do_get_storage_users" :: waited 0.000s {{(pid=16212) inner /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:327}}
2月 15 19:25:54  nova-compute[16212]: DEBUG oslo_concurrency.lockutils [None req-8187e134-96ca-4adb-bfa0-385eb1b7edb0 None None] Lock "storage-registry-lock" released by "nova.virt.storage_users.do_get_storage_users" :: held 0.001s {{(pid=16212) inner /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:339}}
2月 15 19:25:54  nova-compute[16212]: DEBUG nova.virt.libvirt.imagecache [None req-8187e134-96ca-4adb-bfa0-385eb1b7edb0 None None] Adding swap_3 into backend swap images {{(pid=16212) _store_swap_image /opt/stack/nova/nova/virt/libvirt/imagecache.py:119}}
2月 15 19:25:54  nova-compute[16212]: DEBUG nova.virt.libvirt.imagecache [None req-8187e134-96ca-4adb-bfa0-385eb1b7edb0 None None] Verify base images {{(pid=16212) _age_and_verify_cached_images /opt/stack/nova/nova/virt/libvirt/imagecache.py:348}}
2月 15 19:25:54  nova-compute[16212]: DEBUG nova.virt.libvirt.imagecache [None req-8187e134-96ca-4adb-bfa0-385eb1b7edb0 None None] Image id  yields fingerprint da39a3ee5e6b4b0d3255bfef95601890afd80709 {{(pid=16212) _age_and_verify_cached_images /opt/stack/nova/nova/virt/libvirt/imagecache.py:355}}
2月 15 19:25:54  nova-compute[16212]: DEBUG nova.virt.libvirt.imagecache [None req-8187e134-96ca-4adb-bfa0-385eb1b7edb0 None None] Image id e0043447-2aa1-4c39-ae79-1e239ee519b3 yields fingerprint 1f629360db7c8ff0cde2a464f3784959aaced90c {{(pid=16212) _age_and_verify_cached_images /opt/stack/nova/nova/virt/libvirt/imagecache.py:355}}
2月 15 19:25:54  nova-compute[16212]: INFO nova.virt.libvirt.imagecache [None req-8187e134-96ca-4adb-bfa0-385eb1b7edb0 None None] image e0043447-2aa1-4c39-ae79-1e239ee519b3 at (/nova/instances/_base/1f629360db7c8ff0cde2a464f3784959aaced90c): checking
2月 15 19:25:54  nova-compute[16212]: DEBUG nova.virt.libvirt.imagecache [None req-8187e134-96ca-4adb-bfa0-385eb1b7edb0 None None] image e0043447-2aa1-4c39-ae79-1e239ee519b3 at (/nova/instances/_base/1f629360db7c8ff0cde2a464f3784959aaced90c): image is in use {{(pid=16212) _mark_in_use /opt/stack/nova/nova/virt/libvirt/imagecache.py:329}}
2月 15 19:25:54  nova-compute[16212]: DEBUG nova.virt.libvirt.imagecache [None req-8187e134-96ca-4adb-bfa0-385eb1b7edb0 None None] a5a58b25-36b3-4bbf-b33b-645559357347 is a valid instance name {{(pid=16212) _list_backing_images /opt/stack/nova/nova/virt/libvirt/imagecache.py:169}}
2月 15 19:25:54  nova-compute[16212]: DEBUG nova.virt.libvirt.imagecache [None req-8187e134-96ca-4adb-bfa0-385eb1b7edb0 None None] 9dbd9c9f-7d5b-4d1d-8ee3-4fdff68d2d70 is a valid instance name {{(pid=16212) _list_backing_images /opt/stack/nova/nova/virt/libvirt/imagecache.py:169}}
2月 15 19:25:54  nova-compute[16212]: DEBUG nova.virt.libvirt.imagecache [None req-8187e134-96ca-4adb-bfa0-385eb1b7edb0 None None] 9dbd9c9f-7d5b-4d1d-8ee3-4fdff68d2d70 has a disk file {{(pid=16212) _list_backing_images /opt/stack/nova/nova/virt/libvirt/imagecache.py:172}}
2月 15 19:25:54  nova-compute[16212]: DEBUG oslo_concurrency.processutils [None req-8187e134-96ca-4adb-bfa0-385eb1b7edb0 None None] Running cmd (subprocess): /usr/bin/python -m oslo_concurrency.prlimit --as=1073741824 --cpu=30 -- env LC_ALL=C LANG=C qemu-img info /nova/instances/9dbd9c9f-7d5b-4d1d-8ee3-4fdff68d2d70/disk --force-share {{(pid=16212) execute /usr/lib/python2.7/site-packages/oslo_concurrency/processutils.py:372}}
2月 15 19:25:54  nova-compute[16212]: DEBUG oslo_concurrency.processutils [None req-8187e134-96ca-4adb-bfa0-385eb1b7edb0 None None] CMD "/usr/bin/python -m oslo_concurrency.prlimit --as=1073741824 --cpu=30 -- env LC_ALL=C LANG=C qemu-img info /nova/instances/9dbd9c9f-7d5b-4d1d-8ee3-4fdff68d2d70/disk --force-share" returned: 0 in 0.089s {{(pid=16212) execute /usr/lib/python2.7/site-packages/oslo_concurrency/processutils.py:409}}
2月 15 19:25:54  nova-compute[16212]: DEBUG nova.virt.libvirt.imagecache [None req-8187e134-96ca-4adb-bfa0-385eb1b7edb0 None None] Instance 9dbd9c9f-7d5b-4d1d-8ee3-4fdff68d2d70 is backed by 1f629360db7c8ff0cde2a464f3784959aaced90c {{(pid=16212) _list_backing_images /opt/stack/nova/nova/virt/libvirt/imagecache.py:187}}
2月 15 19:25:54  nova-compute[16212]: WARNING nova.virt.libvirt.imagecache [None req-8187e134-96ca-4adb-bfa0-385eb1b7edb0 None None] Unknown base file: /nova/instances/_base/aaa29360db7c8ff0cde2a464f3784959aaced90c
2月 15 19:25:54  nova-compute[16212]: INFO nova.virt.libvirt.imagecache [None req-8187e134-96ca-4adb-bfa0-385eb1b7edb0 None None] Active base files: /nova/instances/_base/1f629360db7c8ff0cde2a464f3784959aaced90c
2月 15 19:25:54  nova-compute[16212]: INFO nova.virt.libvirt.imagecache [None req-8187e134-96ca-4adb-bfa0-385eb1b7edb0 None None] Removable base files: /nova/instances/_base/aaa29360db7c8ff0cde2a464f3784959aaced90c
2月 15 19:25:54  nova-compute[16212]: DEBUG oslo_concurrency.lockutils [None req-8187e134-96ca-4adb-bfa0-385eb1b7edb0 None None] Lock "aaa29360db7c8ff0cde2a464f3784959aaced90c" acquired by "nova.virt.libvirt.imagecache._inner_remove_old_enough_file" :: waited 0.000s {{(pid=16212) inner /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:327}}
2月 15 19:25:54  nova-compute[16212]: INFO nova.virt.libvirt.imagecache [None req-8187e134-96ca-4adb-bfa0-385eb1b7edb0 None None] Removing base or swap file: /nova/instances/_base/aaa29360db7c8ff0cde2a464f3784959aaced90c
2月 15 19:25:54  nova-compute[16212]: DEBUG oslo_concurrency.lockutils [None req-8187e134-96ca-4adb-bfa0-385eb1b7edb0 None None] Lock "aaa29360db7c8ff0cde2a464f3784959aaced90c" released by "nova.virt.libvirt.imagecache._inner_remove_old_enough_file" :: held 0.001s {{(pid=16212) inner /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:339}}
2月 15 19:25:54  nova-compute[16212]: DEBUG nova.virt.libvirt.imagecache [None req-8187e134-96ca-4adb-bfa0-385eb1b7edb0 None None] Verification complete {{(pid=16212) _age_and_verify_cached_images /opt/stack/nova/nova/virt/libvirt/imagecache.py:384}}
2月 15 19:25:54  nova-compute[16212]: DEBUG nova.virt.libvirt.imagecache [None req-8187e134-96ca-4adb-bfa0-385eb1b7edb0 None None] Verify swap images {{(pid=16212) _age_and_verify_swap_images /opt/stack/nova/nova/virt/libvirt/imagecache.py:333}}
2月 15 19:25:54  nova-compute[16212]: DEBUG oslo_concurrency.lockutils [None req-8187e134-96ca-4adb-bfa0-385eb1b7edb0 None None] Lock "swap_3" acquired by "nova.virt.libvirt.imagecache._inner_remove_old_enough_file" :: waited 0.000s {{(pid=16212) inner /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:327}}
2月 15 19:25:54  nova-compute[16212]: INFO nova.virt.libvirt.imagecache [None req-8187e134-96ca-4adb-bfa0-385eb1b7edb0 None None] Removing base or swap file: /nova/instances/_base/swap_3
2月 15 19:25:54  nova-compute[16212]: DEBUG oslo_concurrency.lockutils [None req-8187e134-96ca-4adb-bfa0-385eb1b7edb0 None None] Lock "swap_3" released by "nova.virt.libvirt.imagecache._inner_remove_old_enough_file" :: held 0.001s {{(pid=16212) inner /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:339}}