throw out the baby with the bath water
Discard something valuable along with something not wanted. For example, I know you don't approve of that one item in the bill but we shouldn't throw out the baby with the bath water by voting the bill down. This expression, with its vivid image of a baby being tossed out with a stream of dirty water, is probably translated from a German proverb, Das Kind mit dem Bade ausschütten ("Pour the baby out with the bath"). It was first recorded in English in 1853 by Thomas Carlyle, who translated many works from German.