Are you sure the plug is all the way into the headphone port? Sometimes there's a last little "click" that requires a slight bit of force when inserting headphone plugs...
Also, check the "Sound" pane of the System Preferences, specifically under "Output."