Differences

This shows you the differences between two versions of the page.

--- infinibork [2018/08/29 13:57]
ss_wiki_admin
+++ infinibork [2019/09/16 16:09] (current)
@@ Line 22: / Line 22: @@
 I run ibdiagnet one more time and see an error that catches my eye:
 //Missing master SM in the discover fabric//
-Now I don't know everything about Infiniband, but I do know from my Exadata patching days that each Infiniband switch has an SM (Subnet Master) program running, which presumably manages the devices on each subnet (you can segment Infiniband switches and devices like is done with VLANs). When patching the Infiniband switch there were instructions to ensure there is always a SM running on one of the redundant switches, and to make sure there was a master with higher priority.
+Now I don't know everything about Infiniband, but I do know from my Exadata patching days that each Infiniband switch has an SM (Subnet Manager) program running, which presumably manages the devices on each Infiniband subnet (you can segment Infiniband switches and devices like is done with VLANs). I recalled from patching the Infiniband switch there were instructions to ensure there is always a SM running on one of the redundant switches, and to make sure there was a master with higher priority.
-Long story short, logging into each switch and ensuring there was one with a higher SM priority, and then disabling it and re-enabling it and instantly a lid was assigned for each failing Infiniband interface and IPoIB kicked in and started working between the servers. Clearly the SM had been hung up.
+Long story short, logging into each Infiniband switch and ensuring there was one with a higher SM priority, and then disabling it and re-enabling SM and instantly a lid was assigned for each failing Infiniband interface and IPoIB kicked in and connectivity started working between the servers. Clearly the SM had been hung up and was back in action assigning addresses. It took a few more moments but the Infiniband cards then moved into an "Active" state.
+But back to the ILOM filesystem filling up - the ILOM firmware update didn't seem to have reduced the amount of logging - but I was able to identify the source by paying more attention to the processes on the machine. I've been aware for a long time that kipmi0 runs hot on the CPU on this machine, although with very low priority, but now the repeated ILOM logins suggested that an IPMI-interfacing tool was hanging around and perhaps the IPMI tool was logging in regularly to check for any alerts from the ILOM. On a hunch I shutdown a process I thought it could be, and I was right. //kipmi0// spawns from checks made by the Oracle Hardware Management Pack (hwmgmtd). In essence, in this case, the hwmgmtd process was the one filling up the logs on the ILOM. The ILOM should handle it better, and hopefully the newer firmware will, but time will tell if I end up asking the vendor for a fix (no newer version for this OS is currently available at this time).
-Oracle hwmgmtd fills up the logs
+Urgh, what a rabbit hole. I surely could have done that better, but I'm glad I inadvertently discovered the Subnet Manager process on the Infiniband network was failing. That could have led to a whole host of other nightmares when we get round to rebooting the Exadata servers.
+<code>
+On the management controller, type:
+# setsmpriority priority
+where priority is 0 (lowest) to 13 (highest). For example:
+# setsmpriority 3
+-------------------------------------------------
+OpenSM 3.2.6_20090717
+     Reading Cached Option File: /etc/opensm/opensm.conf
+     Loading Cached Option:routing_engine = ftree
+     Loading Cached Option:sm_priority = 13
+     Loading Cached Option:sminfo_polling_timeout = 1000
+     Loading Cached Option:polling_retry_number = 3
+Command Line Arguments:
+     Priority = 3
+     Creating config file template ’/tmp/osm.conf’.
+     Log File: /var/log/opensm.log
+-------------------------------------------------
+#
+Restart the Subnet Manager:
+# disablesm
+Stopping IB Subnet Manager..                               [  OK  ]
+# enablesm
+Starting IB Subnet Manager.                                [  OK  ]
+#
+</code>

Shaun's Wiki

User Tools

Site Tools

Differences

Page Tools