hold against
Think badly of someone because of some fault or occurrence. For example, Even if you're late, I won't hold it against you, or She backed right into his new car, so he's bound to hold it against her. Also see bear a grudge.
