This portrait of Peter Oliver is part of a pair with his wife, Sarah Hutchinson (1955.5.4). Oliver, the nephew of Mr. and Mrs. Andrew Oliver, was a doctor who practiced in Middleborough, Massachusetts. He fell in love with Sarah, the daughter of Massachusetts governor Thomas Hutchinson and the sister of his Harvard roommate. The courtship lasted four and a half years, so that "we had Time enough to be acquainted with one another. A more agreable and better behaved young Lady there was not in the Country—None of our young Ladies were more admired than she was by all Ranks." They were married in 1770.

In 1775, the British loyalist Oliver and Hutchinson families were caught in political turmoil. The young couple was driven by a mob of American patriots from their country home into Boston. Dr. Oliver and his wife emigrated to England in 1776. He reported that "I think I never was so rejoiced in any thing as setting feet on Old England in my life; from being a mere skeleton, in one month I am fatter than I ever was."