Check if you've set iTunes to copy the music when added to the library. If it does that and has done that for all the music you have added to its library, you can remove (or copy to your backup drive instead) all the music that is in the other locations, as it's also in iTunes.
If your iTunes is set to keep all the music anywhere it is when you added it to the library, it becomes more complicated if you ever wanted to organize or merge your music collection or location etc.
You can also save space by deleting in iTunes the music you don't like (in case there's some music like that).
And perhaps getting a bigger drive or an external drive for holding the files you don't need a constant access would not be a bad idea.
