# airec-system

Part of **AIREC**

# AIRec System Management Troubleshooting Guide

## Problem Index

| Problem | Symptom | Severity | Solution Summary |
|--------|--------|---------|------------------|
| Missing IPMI Kernel Modules | Error: `No such file or directory` when running `ipmitool lan print` or `ipmitool chassis bootdev pxe` | High | Load required IPMI kernel modules using `modprobe` |
| IPMI Device File Not Found | Commands fail with `/dev/ipmi0`, `/dev/ipmi/0`, or `/dev/ipmidev/0` not present | High | Ensure all necessary IPMI kernel modules are loaded in correct order |
| Incomplete IPMI Module Initialization | Partial functionality; some IPMI commands work, others fail | Medium | Load full set of IPMI modules including `ipmi_msghandler`, `ipmi_si`, and `ipmi_devintf` |

## Problem Details

### Problem 1: Missing IPMI Kernel Modules

**Symptoms**
- Error message: `No such file or directory`
- Behavior: `ipmitool lan print` or `ipmitool chassis bootdev pxe` fails immediately
- Context: Occurs on Linux systems where IPMI support is not pre-initialized, especially after fresh OS installation or minimal system setup

**Root Cause**
The system lacks the necessary IPMI kernel modules to expose the IPMI device interface (`/dev/ipmi0`, `/dev/ipmi/0`, or `/dev/ipmidev/0`). Without these modules, `ipmitool` cannot communicate with the Baseboard Management Controller (BMC), resulting in a "file not found" error.

**Solution**
1. Load the required IPMI kernel modules in the correct dependency order:
```bash
modprobe ipmi_msghandler
modprobe ipmi_devintf
modprobe ipmi_si
modprobe ipmi_watchdog
modprobe ipmi_poweroff
```
2. For systems managed via automation or batch scripts, use a single command with logical AND chaining:
```bash
pgm –f iplist –A 'modprobe ipmi_msghandler && modprobe ipmi_devintf && modprobe ipmi_si && modprobe ipmi_watchdog && modprobe ipmi_poweroff'
```
> **Note**: The `ipmi_msghandler` module must be loaded first as it provides core messaging infrastructure for other modules.

**Verification**
- Confirm the IPMI device file exists:
```bash
ls -l /dev/ipmi*
```
Expected output includes at least one of: `/dev/ipmi0`, `/dev/ipmi/0`, or `/dev/ipmidev/0`.
- Test with a basic IPMI command:
```bash
ipmitool lan print
```
Successful execution without error confirms the fix.

### Problem 2: IPMI Device File Not Found

**Symptoms**
- Error message: `Could not open device at /dev/ipmi0 or /dev/ipmi/0 or /dev/ipmidev/0`
- Behavior: All `ipmitool` commands fail with device access errors
- Context: Common on headless servers or virtualized environments where hardware interfaces are not auto-probed

**Root Cause**
The Linux kernel did not automatically load IPMI-related modules during boot, often due to missing firmware, disabled BMC in BIOS, or minimal kernel configuration. The absence of device files prevents user-space tools like `ipmitool` from functioning.

**Solution**
1. Verify BMC hardware is enabled in system BIOS/UEFI settings.
2. Manually load the full suite of IPMI modules:
```bash
modprobe ipmi_msghandler
modprobe ipmi_devintf
modprobe ipmi_si
```
3. If the system uses a specific hardware interface (e.g., KCS, SMIC), ensure `ipmi_si` detects it:
```bash
dmesg | grep -i ipmi
```
Look for messages indicating successful detection of the BMC interface.

**Verification**
- Check kernel logs for IPMI initialization:
```bash
dmesg | grep -i "IPMI"
```
Expected output includes lines like `IPMI message handler initialized` and `IPMI SI: Adding default devices`.
- Run a test command:
```bash
ipmitool mc info
```
Returns BMC firmware version and manufacturer if working.

### Problem 3: Incomplete IPMI Module Initialization

**Symptoms**
- Some `ipmitool` commands work (e.g., `mc info`), but others fail (e.g., `chassis bootdev pxe`)
- Intermittent errors during PXE boot configuration
- Context: Occurs when only partial IPMI modules are loaded (e.g., only `ipmi_si` without `ipmi_devintf`)

**Root Cause**
IPMI functionality is split across multiple kernel modules. While basic communication may work with minimal modules, advanced features like boot device control require the full stack, including character device interface (`ipmi_devintf`) and system interface (`ipmi_si`).

**Solution**
1. Unload any partially loaded IPMI modules to avoid conflicts:
```bash
modprobe -r ipmi_poweroff ipmi_watchdog ipmi_si ipmi_devintf ipmi_msghandler
```
2. Reload all modules in dependency order:
```bash
modprobe ipmi_msghandler
modprobe ipmi_devintf
modprobe ipmi_si
modprobe ipmi_watchdog
modprobe ipmi_poweroff
```

**Verification**
- Test both basic and advanced commands:
```bash
ipmitool mc info
ipmitool chassis bootdev pxe options=efiboot
```
Both should execute without error. The second command may return success even if PXE isn’t supported, but should not report device access errors.

## FAQ

**Q: How do I check if IPMI is supported on my system?**  
A: First, verify hardware support by checking for a BMC chip (common on server-grade hardware). Then, check if IPMI modules can be loaded: run `modprobe ipmi_si` and inspect `dmesg` for detection messages. You can also look for ACPI entries like `SSDT` tables containing "IPMI".

**Q: What permissions are needed to run ipmitool commands?**  
A: By default, `ipmitool` requires root privileges because it accesses kernel device files (`/dev/ipmi*`). To allow non-root users, configure udev rules to grant appropriate group access (e.g., add user to `ipmi` group and set device permissions to 660).

**Q: Why does modprobe fail with "Module not found"?**  
A: The required kernel modules may not be installed. Install the `ipmitool` package and associated kernel modules (e.g., `linux-modules-extra` on Ubuntu/Debian or `kernel-modules-extra` on RHEL/CentOS). Ensure your kernel version matches the module package.

**Q: Can I make IPMI modules load automatically at boot?**  
A: Yes. Add the module names to `/etc/modules-load.d/ipmi.conf`:
```text
ipmi_msghandler
ipmi_devintf
ipmi_si
ipmi_watchdog
ipmi_poweroff
```
This ensures persistent loading across reboots.

**Q: How do I enable debug logging for IPMI issues?**  
A: Increase kernel logging level for IPMI by adding `ipmi_si.force_kipmid=1 ipmi_si.kipmid_max_busy_us=100` to kernel boot parameters, or dynamically adjust via:
```bash
echo 1 > /sys/module/ipmi_si/parameters/force_kipmid
```
Then monitor `dmesg` for detailed interface activity.