count against
Be disadvantageous to, as in His earnings this year will count against his Social Security benefits. This idiom uses count in the sense of "make a reckoning," in this case negative. [Early 1900s]
