Actually i'm running Leopard but i don't think the problem about AVI's should be about the OS. it's very strange indeed.
You can also try running the older mplayer os x that's hosted on mplayer's site to see at least if you can watch xvid's. maybe that would help trying to find what's the reason behind that problem