Quiz: Remember the definition of mal de mer?
1530s, "a boisterous, brutal fellow, one ready to commit any crime," from Middle French rufian "a pimp" (15c.), from Italian ruffiano "a pander, pimp," of uncertain origin, perhaps from a Germanic source related to rough (adj.), but Dutch roffiaan, German Ruffian are said to be from French. English meaning might have been influenced by similarity of sound to rough. Related: Ruffianly.
The Romanic words (e.g. Medieval Latin ruffianus, Provençal rufian, Catalan rufia, Spanish rufian) preserve the sense of "protector or owner of whores." For sense evolution in English, cf. bully (n.).