cinder侧挂载卷流程分析，存储类型以lvm+iscsi的方式为分析基础
cinder侧主要调用了三个接口
1)reserve_volume: 把volume的状态改为attaching，阻止其它节点执行挂载操作。
2)initialize_connection: 这个方法负责构建和返回nova调用者需要的所有信息。返回的信息中包括CHAP credential, target-iqn 和lun 信息。
3)attach_volume: 把volume状态改为in-use，挂载成功,并创建对应的attach记录。
1、nova侧调用cinder的reserve_volume方法
nova/volume/cinder.py
@translate_volume_exception
def reserve_volume(self, context, volume_id):
cinderclient(context).volumes.reserve(volume_id)
1)cinderclient端接受到nova发送的reserve操作的http请求，其入口处理函数为
cinder/api/contrib/volume_actions.py:VolumeActionsController @wsgi.action('os-reserve') def _reserve(self, req, id, body): """Mark volume as reserved.""" context = req.environ['cinder.context'] # Not found exception will be handled at the wsgi level volume = self.volume_api.get(context, id) self.volume_api.reserve_volume(context, volume) return webob.Response(status_int=http_client.ACCEPTED)
该函数的主要作用是通过volume 的uuid，获取volume实例信息，并调用volume目录下的api模块
2)进一步调用cinder volume的api模块的reserve_volume函数，进行数据库的操作，更新卷的状态为attaching
该函数的主要作用是检查指定的卷是否为available,如果卷的状态是available，更新cinder数据库，把卷的状态标记为attaching来预留这个卷，防止其他api在别的地方使用这个卷
对于支持多路挂载的卷，有效状态包括in-use
def reserve_volume(self, context, volume): expected = {'multiattach': volume.multiattach, 'status': (('available', 'in-use') if volume.multiattach else 'available')} result = volume.conditional_update({'status': 'attaching'}, expected) if not result: expected_status = utils.build_or_str(expected['status']) msg = _('Volume status must be %(expected)s to reserve, but the ' 'status is %(current)s.') % {'expected': expected_status, 'current': volume.status} LOG.error(msg) raise exception.InvalidVolume(reason=msg) LOG.info(_LI("Reserve volume completed successfully."), resource=volume)
2、nova侧向cinder发送initialize_connection请求，请求获取卷的所有连接信息
nova/virt/block_device.py:DriverVolumeBlockDevice def attach(self, context, instance, volume_api, virt_driver, do_check_attach=True, do_driver_attach=False, **kwargs): volume = volume_api.get(context, self.volume_id) if do_check_attach: volume_api.check_attach(context, volume, instance=instance) volume_id = volume['id'] context = context.elevated() connector = virt_driver.get_volume_connector(instance) connection_info = volume_api.initialize_connection(context, volume_id, connector) if 'serial' not in connection_info: connection_info['serial'] = self.volume_id self._preserve_multipath_id(connection_info) ........
1)cinderclient接受nova发送过来的os-initialize_connection请求
@wsgi.action('os-initialize_connection') def _initialize_connection(self, req, id, body): """Initialize volume attachment.""" context = req.environ['cinder.context'] # Not found exception will be handled at the wsgi level volume = self.volume_api.get(context, id) try: connector = body['os-initialize_connection']['connector'] except KeyError: raise webob.exc.HTTPBadRequest( explanation=_("Must specify 'connector'")) try: info = self.volume_api.initialize_connection(context,volume,connector) ....
2)进一步调用volume目录下的api模块的initialize_connection函数,对该请求进行处理
cinder/volume/api.py:API类 @wrap_check_policy def initialize_connection(self, context, volume, connector): if volume.status == 'maintenance': LOG.info(_LI('Unable to initialize the connection for ' 'volume, because it is in ' 'maintenance.'), resource=volume) msg = _("The volume connection cannot be initialized in " "maintenance mode.") raise exception.InvalidVolume(reason=msg) init_results = self.volume_rpcapi.initialize_connection(context, volume, connector) LOG.info(_LI("Initialize volume connection completed successfully."), resource=volume) return init_results
3)cinder api进一步发送RPC请求给volume所在的cinder-volume服务节点，最终在cinder-volume节点，
由cinder/volume/manager.py:VolumeManager的initialize_connection处理，该函数的处理，主要包括如下内容
def initialize_connection(self, context, volume, connector): .... utils.require_driver_initialized(self.driver) step 1: self.driver.validate_connector(connector) step 2: model_update = self.driver.create_export(context.elevated(),volume, connector) step 3: volume.update(model_update) setp 4: conn_info = self.driver.initialize_connection(volume, connector) return conn_info
step 1:
对于LVM + iSCSI方式，validate_connector就是检查有没有initiator字段，即nova-compute节点的initiator信息
代码跳转过程如下：drivers/lvm.py -> targets/lio.py -> targets/iscsi.py。
cinder/volume/targets/iscsi.py:ISCSITarget def validate_connector(self, connector): # NOTE(jdg): api passes in connector which is initiator info if 'initiator' not in connector: err_msg = (_LE('The volume driver requires the iSCSI initiator ' 'name in the connector.')) LOG.error(err_msg) raise exception.InvalidConnectorException(missing='initiator') return True
step 2 :调用cinder-rtstool工具创建target，并把卷volume添加到target中创建出lun,认证信息。
def create_export(self, context, volume, connector, vg=None): if vg is None: vg = self.configuration.volume_group volume_path = "/dev/%s/%s" % (vg, volume['name']) export_info = self.target_driver.create_export( context, volume, volume_path) return {'provider_location': export_info['location'], 'provider_auth': export_info['auth'], }
最终调用的是cinder/volume/targets/iscsi.py:ISCSITarget类
def create_export(self, context, volume, volume_path): """Creates an export for a logical volume.""" # 'iscsi_name': 'iqn.2010-10.org.openstack:volume-00000001' iscsi_name = "%s%s" % (self.configuration.iscsi_target_prefix,------设置iscsi name,形式为iqn.2010-10.org.openstack:volume-uuid volume['name']) iscsi_target, lun = self._get_target_and_lun(context, volume)---返回target，和lun的编号，值为(0,0) # Verify we haven't setup a CHAP creds file already # if DNE no big deal, we'll just create it chap_auth = self._get_target_chap_auth(context, volume)------从数据库volumes表中，读取该卷的provider_auth字段，获取认证信息，若没有，则创建 if not chap_auth: chap_auth = (vutils.generate_username(),-----创建auth认证信息 vutils.generate_password()) # Get portals ips and port portals_config = self._get_portals_config()-------获取portals配置，该函数返回的字典格式如下 {'portals_ips': portals_ips,'portals_port': self.configuration.iscsi_port} # NOTE(jdg): For TgtAdm case iscsi_name is the ONLY param we need # should clean this all up at some point in the future tid = self.create_iscsi_target(iscsi_name,-----------------创建target iscsi_target, lun, volume_path, chap_auth, **portals_config) data = {} data['location'] = self._iscsi_location( self.configuration.iscsi_ip_address, tid, iscsi_name, lun, self.configuration.iscsi_secondary_ip_addresses) LOG.debug('Set provider_location to: %s', data['location']) data['auth'] = self._iscsi_authentication( 'CHAP', *chap_auth) return data
创建target操作，调用的是cinder/volume/targets/lio.py中的create_iscsi_target方法
这个函数下发的参数为
name：iqn.2010-10.org.openstack:volume-uuid
tid:0
lun:0
path:卷的路径
chap_auth:{username,passowrd}
kwargs:{'portals_ips': portals_ips,存储服务器的ip
'portals_port': self.configuration.iscsi_port,一般是3260
}
def create_iscsi_target(self, name, tid, lun, path,chap_auth=None, **kwargs): # tid and lun are not used vol_id = name.split(':')[1] LOG.info(_LI('Creating iscsi_target for volume: %s'), vol_id) chap_auth_userid = "" chap_auth_password = "" if chap_auth is not None: (chap_auth_userid, chap_auth_password) = chap_auth optional_args = [] if 'portals_port' in kwargs: optional_args.append('-p%s' % kwargs['portals_port']) if 'portals_ips' in kwargs: optional_args.append('-a' + ','.join(kwargs['portals_ips'])) try: command_args = ['cinder-rtstool', 'create', path, name, chap_auth_userid, chap_auth_password, self.iscsi_protocol == 'iser'] + optional_args self._execute(*command_args, run_as_root=True) except putils.ProcessExecutionError: LOG.exception(_LE("Failed to create iscsi target for volume " "id:%s."), vol_id) raise exception.ISCSITargetCreateFailed(volume_id=vol_id) iqn = '%s%s' % (self.iscsi_target_prefix, vol_id) tid = self._get_target(iqn) if tid is None: LOG.error(_LE("Failed to create iscsi target for volume " "id:%s."), vol_id) raise exception.NotFound() # We make changes persistent self._persist_configuration(vol_id) return tid def _iscsi_location(self, ip, target, iqn, lun=None, ip_secondary=None): ip_secondary = ip_secondary or [] port = self.configuration.iscsi_port portals = map(lambda x: "%s:%s" % (x, port), [ip] + ip_secondary) return ("%(portals)s,%(target)s %(iqn)s %(lun)s" % ({'portals': ";".join(portals), 'target': target, 'iqn': iqn, 'lun': lun}))
step 3:创建完target以后，更新cinder数据库volumes表中，该volume的provider_location，provider_auth两个字段的值
step 4：调用cinder-rtstool的add-initiator子命令，把计算节点的initiator增加到刚刚创建的target acls中,并把所有的信息拼装返回给nova使用。
cinder/volume/targets/lio.py： def initialize_connection(self, volume, connector): volume_iqn = volume['provider_location'].split(' ')[1] (auth_method, auth_user, auth_pass) = volume['provider_auth'].split(' ', 3) # Add initiator iqns to target ACL try: self._execute('cinder-rtstool', 'add-initiator', volume_iqn, auth_user, auth_pass, connector['initiator'], run_as_root=True) ...... return super(LioAdm, self).initialize_connection(volume, connector)
3、nova给cinderclient发送attach_volume命令，更改cinder数据库中，volume状态
nova/virt/block_device.py:API @translate_volume_exception def attach(self, context, volume_id, instance_uuid, mountpoint, mode='rw'): cinderclient(context).volumes.attach(volume_id, instance_uuid, mountpoint, mode=mode)
1)cinder侧接受nova更新cinder数据库的入口函数
cinder/api/contrib/volume_actions.py @wsgi.action('os-attach') def _attach(self, req, id, body): ..... self.volume_api.attach(context, volume,instance_uuid, host_name, mountpoint, mode) ....
2)最后cinder-api通过RPC请求到cinder-volume节点，更新数据库，把volume状态改为in-use，并创建对应的attach记录。
cinder/volume/manager.py:VolumeManager def attach_volume(self, context, volume_id, instance_uuid, host_name, mountpoint, mode, volume=None): """Updates db to show volume is attached."" ...... attachment = volume.begin_attach(mode ......