2013-11-04
问题1：failed: cp: omitting directory '/vmfs/volumes/1/34c24021539b1bc16507e09ad4553f9b'
Log：
Mon Nov 4 15:01:39 2013 [DiM][I]: New VM state is ACTIVE.
Mon Nov 4 15:01:39 2013 [LCM][I]: New VM state is PROLOG.
Mon Nov 4 15:01:39 2013 [TM][I]: Command execution fail: /srv/cloud/one/var/remotes/tm/shared/clone nebula:/srv/cloud/one/var/datastores/1/34c24021539b1bc16507e09ad4553f9b esxi01:/vmfs/volumes/0/1/disk.0 1 1
Mon Nov 4 15:01:39 2013 [TM][I]: clone: Cloning /vmfs/volumes/1/34c24021539b1bc16507e09ad4553f9b in esxi01:/vmfs/volumes/0/1/disk.0
Mon Nov 4 15:01:39 2013 [TM][E]: clone: Command "cd /vmfs/volumes/0/1; cp /vmfs/volumes/1/34c24021539b1bc16507e09ad4553f9b /vmfs/volumes/0/1/disk.0" failed: cp: omitting directory '/vmfs/volumes/1/34c24021539b1bc16507e09ad4553f9b'
Mon Nov 4 15:01:39 2013 [TM][E]: Error copying nebula:/srv/cloud/one/var/datastores/1/34c24021539b1bc16507e09ad4553f9b to esxi01:/vmfs/volumes/0/1/disk.0
Mon Nov 4 15:01:39 2013 [TM][I]: ExitCode: 1
Mon Nov 4 15:01:39 2013 [TM][E]: Error executing image transfer script: Error copying nebula:/srv/cloud/one/var/datastores/1/34c24021539b1bc16507e09ad4553f9b to esxi01:/vmfs/volumes/0/1/disk.0
Mon Nov 4 15:01:39 2013 [DiM][I]: New VM state is FAILED
解决办法：
nebula脚本代码中CP文件夹命令错误，缺少-r参数：
[oneadmin@nebula ~]$ vi /srv/cloud/one/var/remotes/tm/shared/clone
其中一行："cd $DST_DIR; cp $SRC_PATH $DST_PATH"
修改为："cd $DST_DIR; cp -r $SRC_PATH $DST_PATH"
*************************************************************************************************
问题2：Operation not permitted (running as non-root?)
Log：
Mon Nov 4 15:47:45 2013 [DiM][I]: New VM state is ACTIVE.
Mon Nov 4 15:47:45 2013 [LCM][I]: New VM state is PROLOG.
Mon Nov 4 15:48:02 2013 [LCM][I]: New VM state is BOOT
Mon Nov 4 15:48:02 2013 [VMM][I]: Generating deployment file: /srv/cloud/one/var/vms/4/deployment.0
Mon Nov 4 15:48:02 2013 [VMM][I]: Command execution fail: /srv/cloud/one/var/remotes/vnm/vmware/pre PFZNPjxJRD40PC9JRD48VUlEPjA8L1...................
Mon Nov 4 15:48:02 2013 [VMM][E]: pre: Error executing: ((esxcfg-vswitch vSwitch0 -l|grep one-pg-0) || /sbin/esxcfg-vswitch vSwitch0 --add-pg one-pg-0); /sbin/esxcfg-vswitch vSwitch0 -p one-pg-0 --vlan=10 on host: err: Error during version check: Failed to get vmkernel version: Operation not permitted (running as non-root?)
Mon Nov 4 15:48:02 2013 [VMM][I]: Error during version check: Failed to get vmkernel version: Operation not permitted (running as non-root?)
Mon Nov 4 15:48:02 2013 [VMM][I]: Error during version check: Failed to get vmkernel version: Operation not permitted (running as non-root?)
Mon Nov 4 15:48:02 2013 [VMM][I]: ExitCode: 255
Mon Nov 4 15:48:02 2013 [VMM][I]: out:
Mon Nov 4 15:48:02 2013 [VMM][I]: ExitCode: 0
Mon Nov 4 15:48:02 2013 [VMM][I]: Failed to execute network driver operation: pre.
Mon Nov 4 15:48:02 2013 [VMM][E]: Error deploying virtual machine
Mon Nov 4 15:48:02 2013 [DiM][I]: New VM state is FAILED
解决办法：
esxi端使得普通用户拥有执行esxcfg-vswitch命令的权限：
~# chmod +s /sbin/esxcfg-vswitch
*************************************************************************************************
问题3：deploy: Error executing: virsh -c 'esx://esxi02/?no_verify=1&auto_answer=1' start one-8
Log：
Mon Nov 4 16:23:54 2013 [DiM][I]: New VM state is ACTIVE.
Mon Nov 4 16:23:54 2013 [LCM][I]: New VM state is PROLOG.
Mon Nov 4 16:24:11 2013 [LCM][I]: New VM state is BOOT
Mon Nov 4 16:24:11 2013 [VMM][I]: Generating deployment file: /srv/cloud/one/var/vms/8/deployment.0
Mon Nov 4 16:24:11 2013 [VMM][I]: Successfully execute network driver operation: pre.
Mon Nov 4 16:24:25 2013 [VMM][I]: Command execution fail: /srv/cloud/one/var/remotes/vmm/vmware/deploy '/srv/cloud/one/var/vms/8/deployment.0' 'esxi02' 8 esxi02
Mon Nov 4 16:24:25 2013 [VMM][D]: deploy: Successfully defined domain one-8.
Mon Nov 4 16:24:25 2013 [VMM][E]: deploy: Error executing: virsh -c 'esx://esxi02/?no_verify=1&auto_answer=1' start one-8 err: ExitCode: 1
Mon Nov 4 16:24:25 2013 [VMM][I]: out:
Mon Nov 4 16:24:25 2013 [VMM][I]: error: Failed to start domain one-8
Mon Nov 4 16:24:25 2013 [VMM][I]: error: internal error Could not start domain: FileNotFound - File [0] 8/disk.0/disk.vmdk was not found
Mon Nov 4 16:24:25 2013 [VMM][I]:
Mon Nov 4 16:24:25 2013 [VMM][I]: ExitCode: 1
Mon Nov 4 16:24:25 2013 [VMM][I]: Failed to execute virtualization driver operation: deploy.
Mon Nov 4 16:24:25 2013 [VMM][E]: Error deploying virtual machine
Mon Nov 4 16:24:25 2013 [DiM][I]: New VM state is FAILED
解决办法：
官方应用市场的ttylinux镜像命名存在一点问题，需要将ttylinux.vmdk名称改为disk.vmdk
[oneadmin@nebula ~]$ cd var/datastores/
[oneadmin@nebula datastores]$ ls
0 1 2
[oneadmin@nebula datastores]$ cd 1/
[oneadmin@nebula 1]$ ls
34c24021539b1bc16507e09ad4553f9b
[oneadmin@nebula 1]$ cd 34c24021539b1bc16507e09ad4553f9b/
[oneadmin@nebula 34c24021539b1bc16507e09ad4553f9b]$ ls
ttylinux-flat.vmdk ttylinux.vmdk
[oneadmin@nebula 34c24021539b1bc16507e09ad4553f9b]$ mv ttylinux.vmdk disk.vmdk
[oneadmin@nebula 34c24021539b1bc16507e09ad4553f9b]$ ls
disk.vmdk ttylinux-flat.vmdk
[oneadmin@nebula 34c24021539b1bc16507e09ad4553f9b]$
*************************************************************************************************
问题4：sh: /var/tmp/one/vmm/vmware/attach_disk: Permission denied
Log：
Wed Nov 6 14:34:51 2013 [VMM][I]: Successfully execute transfer manager driver operation: tm_attach.
Wed Nov 6 14:34:52 2013 [VMM][I]: Remote worker node files not found
Wed Nov 6 14:34:52 2013 [VMM][I]: Updating remotes
Wed Nov 6 14:35:10 2013 [VMM][I]: Command execution fail: /var/tmp/one/vmm/vmware/attach_disk 'one-18' '/vmfs/volumes/0/18/disk.2' 'VMWare' '4' 'drv_message' 18 esxi02
Wed Nov 6 14:35:10 2013 [VMM][I]: sh: /var/tmp/one/vmm/vmware/attach_disk: Permission denied
Wed Nov 6 14:35:10 2013 [VMM][I]: ExitCode: 126
Wed Nov 6 14:35:10 2013 [VMM][I]: Failed to execute virtualization driver operation: attach_disk.
Wed Nov 6 14:35:10 2013 [VMM][E]: Error attaching new VM Disk
解决办法：
在oned.conf中修改Nebula默认脚本路径：
SCRIPTS_REMOTE_DIR=/var/tmp/one
修改为目前的脚本路径，比如：
SCRIPTS_REMOTE_DIR=/srv/cloud/one/var/remotes
*************************************************************************************************
问题5：mkimage脚本文件
Log:
Thu Nov 7 15:17:40 2013 [VMM][I]: Command execution fail: /srv/cloud/one/var/remotes/tm/shared/mkimage 256 vmdk_vmfs esxi02:/vmfs/volumes/0/36/disk.2 36 0
Thu Nov 7 15:17:40 2013 [VMM][I]: mkimage: Making filesystem of 256M and type vmdk_vmfs at esxi02:/vmfs/volumes/0/36/disk.2
Thu Nov 7 15:17:40 2013 [VMM][E]: mkimage: Command "export PATH=/usr/sbin:/sbin:$PATH
Thu Nov 7 15:17:40 2013 [VMM][I]: # export PATH=/usr/sbin:/sbin:/bin:/sbin:/usr/bin:/srv/cloud/one/bin:/usr/local/bin:/var/lib/gems/1.8/bin/:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/srv/cloud/one/bin
Thu Nov 7 15:17:40 2013 [VMM][I]: dd if=/dev/zero of=/vmfs/volumes/0/36/disk.2 bs=1 count=1 seek=256M
Thu Nov 7 15:17:40 2013 [VMM][I]: vmkfstools -U /vmfs/volumes/0/36/disk.2/disk.vmdk; rm -f /vmfs/volumes/0/36/disk.2/*; vmkfstools -c 256M -d thin /vmfs/volumes/0/36/disk.2/disk.vmdk" failed: 1+0 records in
Thu Nov 7 15:17:40 2013 [VMM][I]: 1+0 records out
Thu Nov 7 15:17:40 2013 [VMM][I]: Failed to delete virtual disk: Not a directory (1310729).
Thu Nov 7 15:17:40 2013 [VMM][I]: rm: cannot stat '/vmfs/volumes/0/36/disk.2/*': Not a directory
Thu Nov 7 15:17:40 2013 [VMM][I]: File_GetFreeSpace: Couldn't statfs /vmfs/volumes/0/36/disk.2/disk.vmdk
Thu Nov 7 15:17:40 2013 [VMM][I]: Failed to create virtual disk: One of the parameters supplied is invalid (1).
Thu Nov 7 15:17:40 2013 [VMM][E]: Could not create image /vmfs/volumes/0/36/disk.2
Thu Nov 7 15:17:40 2013 [VMM][I]: ExitCode: 255
Thu Nov 7 15:17:40 2013 [VMM][I]: Failed to execute transfer manager driver operation: tm_attach.
Thu Nov 7 15:17:40 2013 [VMM][E]: Error attaching new VM Disk: Could not create image /vmfs/volumes/0/36/disk.2
解决办法：
mkimage: Command "export PATH=/usr/sbin:/sbin:$PATH
修改/srv/cloud/one/var/remotes/tm/shared/mkimage文件，将export PATH=/usr/sbin:/sbin:$PATH修改为export PATH=/usr/sbin:/sbin:$PATH
Failed to delete virtual disk: Not a directory (1310729).
修改/srv/cloud/one/var/remotes/tm/shared/mkimage文件，修改如下：
# CREATE_IMAGE="$DD if=/dev/zero of=$DST_PATH bs=1 count=1 seek=${SIZE}M" //注销该行
CREATE_IMAGE="mkdir -p $DST_PATH" //插入改行，用于创建文件夹
保存退出；
*************************************************************************************************
问题6：为esxi虚拟机挂载磁盘时提示没有权限创建路径/srv
Log:
省略
解决办法：
在esxi上以root登录执行：
~ # mkdir -p /srv/cloud/one/var/remotes
~ # chown oneadmin:cloud /srv/cloud/one/var/remotes
其他(用于备忘)：
~ # chmod +s /sbin/vmkfstools
~ # chmod +s /bin/vim-cmd
*************************************************************************************************
问题6：ESXi5.0的SNMP配置
vicfg-snmp --server esxi01 --username root --password passwd123 --show
vicfg-snmp --server esxi01 --username root --password passwd123 -c public
vicfg-snmp --server esxi01 --username root --password passwd123 -t opennebula@162/public
vicfg-snmp --server esxi01 --username root --password passwd123 --enable
vicfg-snmp --server esxi01 --username root --password passwd123 --test
*************************************************************************************************
问题7：GenericVmConfigFault - Module DevicePowerOn power on failed
Thu Nov 14 09:05:45 2013 [DiM][I]: New VM state is ACTIVE.
Thu Nov 14 09:05:45 2013 [LCM][I]: New VM state is PROLOG.
Thu Nov 14 09:10:44 2013 [LCM][I]: New VM state is BOOT
Thu Nov 14 09:10:44 2013 [VMM][I]: Generating deployment file: /srv/cloud/one/var/vms/74/deployment.0
Thu Nov 14 09:10:44 2013 [VMM][I]: Successfully execute network driver operation: pre.
Thu Nov 14 09:10:58 2013 [VMM][I]: Command execution fail: /srv/cloud/one/var/remotes/vmm/vmware/deploy '/srv/cloud/one/var/vms/74/deployment.0' 'esxi02' 74 esxi02
Thu Nov 14 09:10:58 2013 [VMM][D]: deploy: Successfully defined domain one-74.
Thu Nov 14 09:10:58 2013 [VMM][E]: deploy: Error executing: virsh -c 'esx://esxi02/?no_verify=1&auto_answer=1' start one-74 err: ExitCode: 1
Thu Nov 14 09:10:58 2013 [VMM][I]: out:
Thu Nov 14 09:10:58 2013 [VMM][I]: error: Failed to start domain one-74
Thu Nov 14 09:10:58 2013 [VMM][I]: error: internal error Could not start domain: GenericVmConfigFault - Module DevicePowerOn power on failed.
Thu Nov 14 09:10:58 2013 [VMM][I]:
Thu Nov 14 09:10:58 2013 [VMM][I]:
Thu Nov 14 09:10:58 2013 [VMM][I]: ExitCode: 1
Thu Nov 14 09:10:58 2013 [VMM][I]: Failed to execute virtualization driver operation: deploy.
Thu Nov 14 09:10:58 2013 [VMM][E]: Error deploying virtual machine
Thu Nov 14 09:10:58 2013 [DiM][I]: New VM state is FAILED
解决办法：
当前磁盘使用的是sd模式，该模式改为hd，或者直接修改oned.conf也可。
*************************************************************************************************
问题8：disk.0/disk.vmdk was not found
Thu Nov 14 13:01:03 2013 [DiM][I]: New VM state is ACTIVE.
Thu Nov 14 13:01:03 2013 [LCM][I]: New VM state is PROLOG.
Thu Nov 14 13:04:00 2013 [LCM][I]: New VM state is BOOT
Thu Nov 14 13:04:00 2013 [VMM][I]: Generating deployment file: /srv/cloud/one/var/vms/85/deployment.0
Thu Nov 14 13:04:01 2013 [VMM][I]: Successfully execute network driver operation: pre.
Thu Nov 14 13:04:15 2013 [VMM][I]: Command execution fail: /srv/cloud/one/var/remotes/vmm/vmware/deploy '/srv/cloud/one/var/vms/85/deployment.0' 'esxi02' 85 esxi02
Thu Nov 14 13:04:15 2013 [VMM][D]: deploy: Successfully defined domain one-85.
Thu Nov 14 13:04:15 2013 [VMM][E]: deploy: Error executing: virsh -c 'esx://esxi02/?no_verify=1&auto_answer=1' start one-85 err: ExitCode: 1
Thu Nov 14 13:04:15 2013 [VMM][I]: out:
Thu Nov 14 13:04:15 2013 [VMM][I]: error: Failed to start domain one-85
Thu Nov 14 13:04:15 2013 [VMM][I]: error: internal error Could not start domain: FileNotFound - File [0] 85/disk.0/disk.vmdk was not found
Thu Nov 14 13:04:15 2013 [VMM][I]:
Thu Nov 14 13:04:15 2013 [VMM][I]: ExitCode: 1
Thu Nov 14 13:04:15 2013 [VMM][I]: Failed to execute virtualization driver operation: deploy.
Thu Nov 14 13:04:15 2013 [VMM][E]: Error deploying virtual machine
Thu Nov 14 13:04:15 2013 [DiM][I]: New VM state is FAILED
*************************************************************************************************
解决办法：
出现这类问题可能有两种原因：
目录下不存在该文件，而存在vm-name-flat.vmdk和vm-name.vmdk，那么将vm-name.vmdk更改为disk.vmdk即可。
目录下存在该文件，那么请确定是否改过vm-name-flat.vmdk的名字，如果改过会出现上述问题。
如何导入esxi虚拟机：
将esxi虚拟机的vm-name.vmdk命名为disk.vmdk，并将该文件和vm-name-flat.vmdk文件压缩，比如：
[root@opennebula CentOS6.4_mini]# ls
CentOS6.4_mini-00-flat.vmdk CentOS6.4_mini-00.vmdk
[root@opennebula CentOS6.4_mini]# mv CentOS6.4_mini-00.vmdk disk.vmdk
[root@opennebula CentOS6.4_mini]# ls
CentOS6.4_mini-00-flat.vmdk disk.vmdk
[root@opennebula CentOS6.4_mini]# tar -jcvf ../CentOS6.4_mini-131114.tar.bz2 *
*************************************************************************************************
问题9：为虚拟机添加Volatile Disk
现象：在nebula4.2环境中，为esxi虚拟机创建Volatile Disk的时候remotes/tm/shared/mkimage文件和remotes/vmm/vmware/attach_disk文件不匹配导致不能真确的创建和添加磁盘，同时由于磁盘格式不对，power off虚拟机后无法resume。
remotes/vmm/vmware/attach_disk添加配置文件的磁盘路径：
DISK_PATH="/vmfs/volumes/$DATASTORE/$VM_ID/$DISK_NAME/disk.vmdk"
remotes/tm/shared/mkimage创建磁盘的名称和路径：
CREATE_IMAGE="$DD if=/dev/zero of=$DST_PATH bs=1 count=1 seek=${SIZE}M"
该命令只是在/vmfs/volumes/$DATASTORE/$VM_ID下创建一个$DISK_NAME磁盘文件，而attach_disk对应的$DISK_NAME是一个目录，该目录中的disk.vmdk才是真正的磁盘文件。其二，使用qemu-img查看磁盘类型显示是raw格式，非vmware的vmdk格式，所有要修改mkimage脚本使其在正确路径下创建磁盘并且是vmdk格式的磁盘。
修改后的mkimage脚本：
#!/bin/bash
SIZE=$1
FSTYPE=$2
DST=$3
VMID=$4
DSID=$5
if [ -z "${ONE_LOCATION}" ]; then
TMCOMMON=/var/lib/one/remotes/tm/tm_common.sh
else
TMCOMMON=$ONE_LOCATION/var/remotes/tm/tm_common.sh
fi
. $TMCOMMON
#-------------------------------------------------------------------------------
# Set dst path and dir
#-------------------------------------------------------------------------------
DST_PATH=`arg_path $DST`
DST_HOST=`arg_host $DST`
DST_DIR=`dirname $DST_PATH`
#-------------------------------------------------------------------------------
# Make the new image (file-based)
#-------------------------------------------------------------------------------
# ssh_make_path $DST_HOST $DST_DIR
MKDIR_CMD=`/bin/mkdir -p $DST_PATH` //创建目录
CREATE_IMAGE="/usr/bin/qemu-img create -f vmdk $DST_PATH/disk.vmdk ${SIZE}M" //使用qemu-img命令创建vmdk格式的disk.vmdk
$MKDIR_CMD
$CREATE_IMAGE
log "Making filesystem of ${SIZE}M and type $FSTYPE at $DST"
exit 0
替换原有mkimage脚本，并且创建/vmfs/目录指向存储目录：
[root@nebula datastores]# ls
0 1 100 101 102 2
[root@nebula datastores]# pwd
/srv/cloud/one/var/datastores
[root@nebula datastores]# mkdir /vmfs
[root@nebula datastores]# chown oneadmin:cloud /vmfs
[root@nebula datastores]# ln -s /srv/cloud/one/var/datastores /vmfs/volumes
*************************************************************************************************
问题10：
esxi新建用户需要执行一些命令，在esxi端要做的操作：
chmod +s /sbin/vmkvsitools
chmod +s /sbin/esxcfg-vswitch
chmod +s /bin/vim-cmd
chmod +s /sbin/mkdir
*************************************************************************************************