I'm at a loss here. Which value should I change when I try to create different shades of color X? The hue? Brightness? What kind of color encoding should I use?
The hue is the color, so generally you should change the lightness and saturation to get a different shade. You should use HSL (hue-saturation-lightness) because that’s what CSS3 uses, and HSL/HSV are colorwheels designed for representing RGB (what screens use).
HSL/HSV attempt to mimic paint mixing. Hue is the color. It’s a degree because imagine all colors of the rainbow arranged in a circle, that’s the color wheel and each degree corresponds to a different color. Saturation is how much color (0% saturation at 50% lightness is grey), and lightness is how light or dark the color is (0% is black, 100% is a very bright color or white depending on the saturation).