The oath of office taken by an individual under section 3331 of this title shall be delivered by him to, and preserved by, the House of Congress, agency, or court to which the office pertains.