Shaun's Wiki

This is an old revision of the document!

---

This morning we receive an email from the ILOM of a Sun Oracle X3-2L server: The suspect component: /SYS/SP has defect.ilom.fs.full

A full filesystem on an ILOM? Well I never… So I log into the ILOM to have a quick poke around, both on the CLI and web interface, but there's no directly accessible filesystem and shell to do any changes to the ILO 'filesystem' (without deep dropping into the ILOM's boot sector - but that's a write-up for another day) so I figure it's just some log file that is filling up and hopefully cleaning that up will do the trick. So I cleared the log files and rebooted the ILO. After it came back up all seemed ok - the alert auto-resolved, and all is well. Except I notice in the ILO audit log there's an entry for a login every minute or so. This could be why the log eventually filled up after we patched this ILOM about 4 months ago. At this stage I figured it could just be a little bug in the logging level or something so I pulled a new firmware down and installed it, hoping for a quick fix, and a free patch in the process. Of course, part of the ILO firmware patch is also a BIOS patch, so the server needs a reboot. Luckily this particular server is Well Looked After^TM so a quick boot (well, quicker than an HP DL380g9, that's for sure) later the system is back up and running. All good.

Except no - the Infiniband interfaces, which are running IP over Infiniband, are showing as up and have link, but can't send or receive any network traffic.

So I play around with the TCP stack a little - validate configurations have not changed, bring bonded interface up, down, restart network, remove one interface from the bond, try the other, all with no luck. TCP won't go. Technically firmware is a last resort, especially with something that was working beforehand, but the support notes for the ILOM firmware update have planted it in my mind that any HBAs and HCAs on a system getting a BIOS update might need to be patched at the same time. While I'm still in a patchy mood, I verify the firmware levels as per the vendor website to make sure we're on the latest for the IB HCA. Then I started with the Infiniband diagnostic tools. ibdiagnet shows we're pretty much seeing all devices across the IB network. iblinkinfo shows plenty of active links, except the two on this particular problem server. ibswitches confirms switch connectivity.

ibstat is showing LinkUp on both ib0 and ib1, but I'm worried the interfaces are still in “Initializing” state. Also, the Base lid and SM lid still have value 0, which I'm sure should actually be an integer value. Logging into the web management interface on the Infiniband ILO shows nothing wrong, but confirms the missing lid. All the working interfaces have an lid assigned. I confirm this on another server.

I run ibdiagnet one more time and see an error that catches my eye:

Oracle hwmgmtd fills up the logs