I am brand new to these forums tonight, but have been posting on macrumors for years, and I'm in search of an answer to this specific issue.
I have been having issues with Mail.app 4.0 and gmail since i installed snow leopard on launch day. It has been the same with an upgrade install and clean install on 2 different machines.
I have the same issue stated above...So while logged into Gmail's web interface, I opened Mail.app. Mail.app predictably timed out, and upon returning to Safari, noticed that I'm now getting an "Oops… the system encountered a problem (#717) - Retrying now…" error, then clicking Sign Out changes it to a (#104) error.
Closing the tab, waiting 5 minutes, then logging back in is the fix, until i launch Mail.app again, that is. grr ...
I also can not send messages using my gmail accounts from mail.app 99% of the time. I have to retry numerous times before I get anywhere.
What the heck got broken between 10.5.8 and 10.6 mail?