Based on Red Hat solution (https://access.redhat.com/solutions/47438), the below steps should be done before flush any multipath device, otherwise the command will be failed:
- Locate any subsystem or process holding the device open.
- For any subsystem or process holding the device open, stop the process, or issue commands to release the device.
- Some examples of possible holders of a device and the commands to release it:
- One or more partition mapping(s) still exists on the device.
- Use “
kpartx -d” on the device to remove the device partition mapping(s).
- Use “
- A filesystem exists on the device and is currently mounted.
- Unmount the filesystem and if it exists in /etc/fstab, remove it.
- The device was used by LVM, and still has device mapper state in the kernel.
- Use “
lvchange -an” to deactivate any logical volume(s) associated with the device. A list of logical volumes associated with the device may be found by examining the output of “lvs -o +devices“. - If “
lvchange -an” fails, the logical volume is only partially removed, or there are blocked processes with I/O outstanding on the device, use “dmsetup remove -f” followed by “dmsetup clear” on the device. See dmsetup man page for full explanation of these commands.
- Use “
Once all holders of the device have been removed, the device should be flushed with “multipath -f“.
But may the above instruction failed at first step because the disk is hold by a process.
Problem!
Here is an example of the error:
multipath -f {Multipath Alias}
Mar 1 09:59:10 | {Multipath Alias}: map in use
dmsetup remove -f {Multipath Alias}
device-mapper: remove ioctl on {Multipath Alias} failed: Device or resource busy
Solution!
What is the solution for this problem?
As Red Hat mentioned, “Locate any subsystem or process holding the device open”. Usually a process holding the disk or disks, so as the first step follow the below:
lsof | grep {Multipath Alias}
Output will be same as the below:
oracle 10298 oracle 257u BLK 252,8 0t0 18446612177448267048 /dev/oracleasm/disks/{Multipath Alias} (deleted)
In this example output, oracle process holding the disk and the process should be identified before any action:
ps -ef | grep 10298 oracle 10298 1 0 2016 ? 00:00:00 oraclecmsrestd (LOCAL=NO)
If there is no problem about killing the process, kill the process by the below command:
kill -9 10298
Then try to flush device again and if the problem is still exists, follow the below for troubleshooting:
- Use ‘lsof’ to attempt to find anyone holding the device open.
- Check ‘dmsetup’ output for any device mapper maps that depend on the device
- Check the if there is a device for mpath7 in /dev/mapper/mpath7
- Check multipath -v4 -ll