black mark
An indication of censure or failure, as in If you refuse to work late, won't that be a black mark against you? This phrase alludes to a literal black mark, such as a cross, that was put next to a person's name, indicating that he or she had incurred a rebuke or penalty of some kind. [Mid-1800s]