My experience is that X11.app from Apple is the best if you're using OSX. It used to be that I would kill aqua and log in as >console so it would be faster. For this purpose I kept XDarwin around and it worked well.
But quartz rendering just makes everything twice as fast, even with all the baggage that goes along with aqua. Speed was my major issue with XDarwin when I wanted to use it along side my aqua apps so X11.app fixed that.
If you're doing funky things like running KDE or booting into >console then your choice may be different. For most people, X11.app will be the best, and most familiar for mac users.
**append flames from hardcore *nix types here**