What word does your mother always say?
"maker of pots" (they also sometimes doubled as bell-founders), late Old English pottere "potter," reinforced by Old French potier "potter," agent noun from root of pot (n.1). As a surname from late 12c. Potter's field (1520s) is Biblical, a ground where clay suitable for pottery was dug, later purchased by high priests of Jerusalem as a burying ground for strangers, criminals, and the poor (Matt. xxvii:7). An older Old English word for "potter" was crocwyrhta "crock-wright."
"occupy oneself in a trifling way," 1740, earlier "to poke again and again" (1520s), frequentative of obsolete verb poten "to push, poke," from Old English potian "to push" (see put (v.)). Sense of "occupy oneself in a trifling way" is first recorded 1740. Related: Pottered; pottering.