InfiniBand · basic

You are on call. A compute node was recabled during maintenance and MPI jobs refuse to start on it. ibstat on the node shows the output below. What is the most likely cause?

CA 'mlx5_0'
	CA type: MT4129
	Number of ports: 1
	Firmware version: 28.39.2048
	Port 1:
		State: Initializing
		Physical state: LinkUp
		Rate: 400
		Base lid: 65535
		LMC: 0
		SM lid: 0
		Port GUID: 0x88e9a40300c85e10
		Link layer: InfiniBand

The options

• AA bad cable — the physical layer never finished training, so the port cannot progress past Initializing; swap the cable and the transceiver at both ends
• BNo Subnet Manager is reachable — the physical link is up, but no SM has assigned a LID and moved the port to Active; check that opensm (or the switch SM) is running with sminfo
• COutdated HCA firmware — the 28.39.2048 image cannot finish port bring-up, so flash the current release with mlxfwmanager and reboot the node to reach Active
• DThe port is running an Ethernet personality instead of InfiniBand, so it can never reach Active; read LINK_TYPE_P1 with mlxconfig, set it to IB, then reset

The answer

B. No Subnet Manager is reachable — the physical link is up, but no SM has assigned a LID and moved the port to Active; check that opensm (or the switch SM) is running with sminfo

Why

State: Initializing combined with Physical state: LinkUp is the classic no-SM signature: the PHY trained fine (LinkUp, Rate 400 negotiated), but a port only progresses Down -> Initializing -> Armed -> Active when a Subnet Manager assigns it a LID and activates it. Base lid 65535 (unassigned) and SM lid 0 confirm no SM has touched the port. A bad cable would leave Physical state at Polling, not LinkUp; and a port in Ethernet mode would show Link layer: Ethernet, not an InfiniBand state machine at all. Firmware is irrelevant when the link already trained at full rate.

More InfiniBand questions

• InfiniBand link trained at 1X, not 4X

← Back to ClusterDrill