Network Connection Problem on HPE FlexFabric 650 (FLB/M) Adapter

We have many HPE Blade System C7000 chassis for hosting our virtual infrastructures or our customers virtual infrastructures. Also many of them are filled with HPE ProLiant BL460c Gen10. Recently, we had serious problem with some of the servers about network connection, if there was FlexFabric 650 (FLB/M) adapter installed on blade server.

What’s Exact Problem?

Our servers were working fine but after a while, we lose our vMotion network connectivity, the hosts couldn’t migrate virtual machines even between servers in same C7000 chassis. The problem was happened on one chassis and other chassis has no problem.

Network adapter was connected physically but it don’t pass any network traffic, the issue some like ARP problem!

Also network adapter status was Degraded in iLO web administration.

List of All Attempts to Recovering Issue

Try this, ask IT guy about network problem on network card, one hundred percent sure that IT guy will say:

Upgrade firmware, update driver, upgrade ….!

So, I did it! 😀

Let’s review the list:

• Upgrade Firmware for FlexFabric 650M and FlexFabric 650FLB.
• Update Driver (ESXi).
• Turn off, turn on virtual connect modules.
• Unassign and reassign server profiles.
• Check all configurations and compare with other chastises.
• Re-seat one of servers for test

After the server re-seated, the problem goes much worse. There was no connectivity at all.

Solution, The Strangest Solution

As I mentioned before, network adapters status was Degraded in iLO web administration but network adapters don’t have any physical problem and virtual connect modules as well. So after simple search with adding “iLO” to my search on Google, I found the solution:

Advisory: (Revision) HPE Integrated Lights-Out 5 (iLO 5) – iLO 5 v2.10 and v2.11 Unable To Communicate With Certain HPE Converged Networking Adapters Causing Virtual Connect (VC) Networking Issues with HPE ProLiant BL460c Gen10 Server Blades

When an HPE ProLiant BL460c Gen10 server blade is using one of the listed options in the Scope section below and running HPE Integrated Lights-Out (iLO 5) v2.10 or v2.11 and server bays are re-seated, enclosure power cords plugged in again, or eFUSE server reset, the Device Control Channel (DCC) state on the Converged Networking Adapters will be unavailable and the following Virtual Connect (VC) networking issues will occur:

Adapter shows as connected but does not appear to be working

Adapter does not have an IP address

VC networking issues

Don’t update your servers by HPE Service Pack for ProLiant 219.12.0 and if did it, update iLO firmware as soon as possible.

You can use OA PowerShell cmdlet to bulk upgrade.

Don’t forget that reset server by the instruction which mentioned in the HPE document.

Further Reading

HPE Virtual Connect 4.80