Another tale of ESXi patching adventures. My company has some ESXi hosts in remote locations that don’t get much care. As long as they’re running the VMs and business users don’t complain, the hosts don’t get much attention.
Today I was trying to patch a server with all the ESXi 6.5 patches released to date, but it was failing with this error:
02 / 27 / 2023, 2:14:57 PM The VFAT filesystem mpx.vmhba32:C0:T0:L0:8 (UUID 577b5b9f-bf67afef-822f-246e960d8318) is ...
02 / 27 / 2023, 2:56:00 PM Could not stage image profile '(Updated) ESXi-5.5.0-20150104001-standard': ('VMware_locker_tools-light_6.5.0-3.191.20448942', '[Errno 32] Broken pipe')
Some quick searching revealed that this error is likely because the locker partition is out of space. Lo and behold, it was:
[root@server02:~] ls -ltrh / | grep store
lrwxrwxrwx 1 root root 6 Feb 27 20:21 locker -> /store
lrwxrwxrwx 1 root root 49 Feb 27 20:21 store -> /vmfs/volumes/577b5b9f-bf67afef-822f-246e960d8318
[root@server02:~]
[root@server02: / vmfs/volumes/577b5b9f-bf67afef-822f-246e960d8318/packages] df -h
Filesystem Size Used Available Use% Mounted on
NFS 4.0T 2.2T 1.8T 55% /vmfs/volumes/vmds05
NFS 4.0T 2.9T 1.1T 72% /vmfs/volumes/vmds06
NFS 4.0T 2.4T 1.6T 61% /vmfs/volumes/vmds04
NFS 4.0T 2.9T 1.1T 73% /vmfs/volumes/vmds03
NFS 4.0T 2.3T 1.7T 56% /vmfs/volumes/vmds02
NFS 7.0T 6.5T 517.3G 93% /vmfs/volumes/vmds01
vfat 285.8M 285.8M 56.0K 100% /vmfs/volumes/577b5b9f-bf67afef-822f-246e960d8318
vfat 249.7M 166.1M 83.6M 67% /vmfs/volumes/930098e3-722a9454-fdea-b663aa3b7053
vfat 249.7M 166.1M 83.6M 67% /vmfs/volumes/3bc3cee8-10f1c748-e8e5-d729b7af1e64
[root@server02:~] df /locker
Filesystem Bytes Used Available Use% Mounted on
vfat 299712512 299655168 57344 100% /vmfs/volumes/577b5b9f-bf67afef-822f-246e960d8318
Using the disk and partition information in the error, I formatted the partition with vfat:
[root@server02:~] vmkfstools -C vfat /dev/disks/mpx.vmhba32:C0:T0:L0:8
create fs deviceName:'/dev/disks/mpx.vmhba32:C0:T0:L0:8', fsShortName:'vfat', fsName:'(null)'
deviceFullPath:/dev/disks/mpx.vmhba32:C0:T0:L0:8 deviceFile:mpx.vmhba32:C0:T0:L0:8
Checking if remote hosts are using this device as a valid file system. This may take a few seconds...
Creating vfat file system on "mpx.vmhba32:C0:T0:L0:8" with blockSize 1048576 and volume label "none".
Successfully created new volume: 577b5b9f-bf67afef-822f-246e960d8318
Then, I recreated the symlinks:
ln -snf /vmfs/volumes/577b5b9f-bf67afef-822f-246e960d8318 /store
ln -snf /vmfs/volumes/577b5b9f-bf67afef-822f-246e960d8318 /locker
After a reboot, I found another host server running the same ESXi version and path level, then copied the contents of the store to my newly formatted partition. After another reboot (just to be safe), I was able to patch the host server without any problems.
References
An article I referenced when figuring this out was:
Fixing the broken/corrupt Locker Partition on Esxi