Can I have a private conversation that isn't remembered?

Yes. There's an Incognito / Private Chat toggle at the bottom of the in-session menu. While it's on, the conversation isn't saved to memory and past chat bubbles are hidden. Turning it off or refreshing ends private mode.