What makes a good KB?

I recently came across Divio’s documentation methodology, and I think it was perfect timing. I work on a helpdesk team at a small MSP, which you might know, means that there’s basically no SOPs, no KB, and very little structure in my day-to-day. Small company means wearing a few hats. Working in IT at that small company means there aren’t many hats that I don’t wear. A small list of things that have recently come up:

  • Desktop and application support
  • PC deployments
  • Server deployments
  • Network installation and troubleshooting
  • Negotiating with ISPs
  • Furniture installation
  • Excel formula troubleshooting
  • SQL queries (handed to billing)
  • Gathering billing data, sent to analytics
  • On-prem physical security
  • Inventory
  • Incident Response and Forensics

Anyways, my point is that we’ve created a large, large, amount of technical debt. Each of our team members have been creating KB articles with little direction. It’s quickly become an unmanageable and rarely useful mess. I think using some of the ideas from Divio can help me start sorting out or environment, and maybe turn all my knowledge goop into an actual knowledge base. I’ll post the best ones I write here.

Dell PowerEdge R610 Manual Fan Control

If you’re like me, you might live in close proximity to a lot of unracked servers and switches. I recently acquired a Dell R610 that was due for recycling, but found I couldn’t put it anywhere without the fans disturbing me. Some people claim theirs is nearly silent. They must have found a magic BIOS because I have had no luck using means provided by Dell.

There is a way to alter the fan speeds manually, despite what you’ll find on Dell forums. You can write fan speed percentages directly to the fan controller on these devices! As a server, these things might be outdated and inefficient, but as a homelab they can be a workhorse.

Before you start, you should know that this isn’t something you should do to a production server. I’m just some guy on the internet. Use your best judgement.
Make sure you’ve got access to the iDRAC, and have enabled IPMI over LAN during POST. I would also recommend installing lm-sensors for monitoring the temperatures OS side.

Starting with ipmitool

Search the repos of your favorite Linux distro for ipmitool. If you can’t find it, you can grab the latest release.
Every command you pass to IPMI over LAN will start with this as the prefix.

$ ipmitool -I lanplus -H [iDRAC IP] -U root -P [iDRAC password] [command]

If you’re trying to control fans from the host, and not from a separate device, it’s a little simpler. You don’t need all the options, and can just use ipmitool.

$ ipmitool [command]

Run the following to test your connection

$ ipmitool -I lanplus -H [iDRAC IP] -U root -P [iDRAC password] chassis status
or
$ ipmitool chassis status

Toggle Manual Fan Control

Manual fan control must first be enabled. Heed my warning though. If you manually set your fan speed, the server will no longer be able to manage the fan speed to keep an optimal temperature. By enabling this you have the potential to cause damage to your equipment.

Manual fan control enable:

raw 0x30 0x30 0x01 0x00

Manual fan control disable:

raw 0x30 0x30 0x01 0x01

Setting Static Speed

The last hexadecimal value is the fan’s speed measured in percent of max. 0x14 = 20%, 0x32 = 50%, etc.

20%:

raw 0x30 0x30 0x02 0xff 0x14

Monitoring Temperatures

It’s a good idea to monitor your temperatures when you make changes to your fan speed.

$ watch sensors


Bash Script

Now you’re able to control your fan speed. Congrats! Feel free to use this script to change it dynamically.

#!/bin/bash 
TIMESTAMP="[`date +%Y-%m-%d_%H:%M:%S`]"
CPUTEMP=$(sensors -u | grep input | awk '{print $2}' | sort -r | head -1)
SYSTEMP=$(ipmitool sdr type temperature |grep Ambient |grep degrees |grep -Po '\d{2}' | tail -1)
CLIMIT=90
SLIMIT=35
FANHOST="Dell R610"

# determines fan speed based on CPUTEMP
fanSpeed() {
	if [[ $CPUTEMP > $CLIMIT ]] || [[ $SYSTEMP > $SLIMIT ]]; then
		# release fan control to server
		ipmitool raw 0x30 0x30 0x01 0x01
		echo "$TIMESTAMP CPUTEMP critical. Releasing fan controls to $FANHOST."
	elif [[ $CPUTEMP > 80 ]]; then
		setFans 50
	elif [[ $CPUTEMP > 70 ]]; then
		setFans 45
	elif [[ $CPUTEMP > 60 ]]; then
		setFans 30
	elif [[ $CPUTEMP > 50 ]]; then
		setFans 20
	elif [[ $CPUTEMP > 40 ]]; then
		setFans 15
	elif [[ $CPUTEMP > 30 ]]; then
		setFans 10
	#elif [[ $CPUTEMP > 20 ]]; then
	#	setFans 10
	else setFans 7 
	fi
}

# percentage -> fan speed
# receives appropriate fan speed, sends to fan controller as hexidecimal
setFans() {
	ipmitool raw 0x30 0x30 0x01 0x00
	echo "$TIMESTAMP Setting fans to $1%" >> ./log
	ipmitool raw 0x30 0x30 0x02 0xff 0x$(echo "obase=16; $1" | bc)
}

# BEGIN
#echo "$TIMESTAMP Script Start" >> ./log
echo "$TIMESTAMP CPU $CPUTEMP°C - AMB $SYSTEMP°C" >> ./log
fanSpeed
# echo "$TIMESTAMP Script End" >> ./log