Know how to use "fewer" and "less"? Find out.
1570s, probably from a dialectal survival of Old English wiðer "against, contrary, opposite" (see with) + plural suffix. Possibly so called because the withers are the parts of the animal that oppose the load. Cf. German Widerrist "withers," from wider "against" + Rist "wrist."