euca-create-volume fails at times with Error communicating with Storage Controller

Looking at the front-end system after Torsten's runs all completed, I see:

# lvscan
  ACTIVE '/dev/vg-d5oqkQ../lv-I5BT_w..' [1.00 GB] inherit
  ACTIVE '/dev/vg-A2mbYQ../lv-StzyLw..' [1.00 GB] inherit
  ACTIVE '/dev/vg-X0ViPw../lv-ccnoew..' [1.00 GB] inherit
  ACTIVE '/dev/vg-qvqsYg../lv-i8T4vQ..' [1.00 GB] inherit
  ACTIVE '/dev/vg-Kg0fNg../lv-erhiXw..' [1.00 GB] inherit
  ACTIVE '/dev/vg-XbIp7g../lv-tkn_Aw..' [1.00 GB] inherit
# vgscan
  Reading all physical volumes. This may take a while...
  Found volume group "vg-d5oqkQ.." using metadata type lvm2
  Found volume group "vg-A2mbYQ.." using metadata type lvm2
  Found volume group "vg-X0ViPw.." using metadata type lvm2
  Found volume group "vg-qvqsYg.." using metadata type lvm2
  Found volume group "vg-Kg0fNg.." using metadata type lvm2
  Found volume group "vg-XbIp7g.." using metadata type lvm2
  Found volume group "vg-PXBR2A.." using metadata type lvm2

So six LVs weren't deleted properly, and seven VGs.

On Lucid, I don't get the Volume: Error communicating with Storage Controller: CreateStorageVolume:Internal Error. error anymore. However, not all volume groups are deleted:

$ sudo lvscan
  ACTIVE '/dev/vg-wsdyiQ../lv-K0hXfA..' [1.00 GiB] inherit
$ sudo vgscan
  Reading all physical volumes. This may take a while...
  Found volume group "vg-wsdyiQ.." using metadata type lvm2

Note that at times a volume is stuck in the 'creating' stage. It can be deleted via euca-delete-volume, but deletion does not seem to be complete.

The situation seems to get worse when doing the create/delete volume more often:

$ for i in $(seq 1 32); do euca-create-volume -s 1 -z torsten; done
$ euca-describe-volumes | grep avail | wc -l
29
$ euca-describe-volumes | grep creat | wc -l
3
$ for i in $(euca-describe-volumes | cut -c8-20 | xargs); do euca-delete-volume $i; done

$ for i in $(seq 1 32); do euca-create-volume -s 1 -z torsten; done
$ euca-describe-volumes | grep avail | wc -l
26
$ euca-describe-volumes | grep creat | wc -l
6
$ for i in $(euca-describe-volumes | cut -c8-20 | xargs); do euca-delete-volume $i; done

$ for i in $(seq 1 32); do euca-create-volume -s 1 -z torsten; done
$ euca-describe-volumes | grep avail | wc -l
24
$ euca-describe-volumes | grep creat | wc -l
8
$ for i in $(euca-describe-volumes | cut -c8-20 | xargs); do euca-delete-volume $i; done

Is there any workaround to remove those stalled volumes completely?

We are creating big volumes and thus, the volume limit is reached easily, but we cannot remove completely the volumes that fail on creation time.

Hi,

I would like to have the eucalyptus logs at least for the SC (and, ideally, for the CLC also). Can you please attach them here?

Thanks.

@Carlos,

the cloud of the error reports is long gone. If need be I can try to
recreate the problem on a new one next week.

Hi Torsten,

Very sorry for the delay. Yes, if you could recreate this would be marvelous, thank you.

Here we go, the problem is fixed:

$ for i in $(seq 1 32); do euca-create-volume -s 1 -z TorstenCloud; done
$ euca-describe-volumes | grep avail | wc -l
30
$ euca-describe-volumes | grep creat | wc -l
0
$ euca-describe-volumes | grep fail | wc -l
2

$ for i in $(euca-describe-volumes | cut -c8-20 | xargs); do euca-delete-volume $i; done

$ for i in $(seq 1 32); do euca-create-volume -s 1 -z TorstenCloud; done
$ euca-describe-volumes | grep avail | wc -l
32

$ for i in $(euca-describe-volumes | cut -c8-20 | xargs); do euca-delete-volume $i; done

$ for i in $(seq 1 32); do euca-create-volume -s 1 -z TorstenCloud; done
$ euca-describe-volumes | grep avail | wc -l
32

$ for i in $(euca-describe-volumes | cut -c8-20 | xargs); do euca-delete-volume $i; done

$ for i in $(seq 1 32); do euca-create-volume -s 1 -z TorstenCloud; done
$ euca-describe-volumes | grep avail | wc -l
32

[Expired for eucalyptus (Ubuntu) because there has been no activity for 60 days.]