"codec" = "format."
H.264 is the MPEG4 variant that the video is encoded in. AC3 and AAC are the audio formats.
iTunes should be able to handle H.264 video with AAC audio. What application did you use to convert the video, and what was the original video source?