I myself have the same problem but it happens more when im using safari, I notice lots of jitters when Im scrolling through images, I think it may be a software issue, so im going to wait till 10.4.5 comes out before i contact apple on the problem
I have a 20" Inel Core Duo 1GB 256MB VRAM Imac