back of one's mind
The remote part of one's mind or memory, as in With the idea of quitting in the back of his mind, he turned down the next assignment. [c. 1900]
