Dr. Frank of The Blogs of War writes (09:33 yesterday, if the link doesn't work):

I really hope the that the person who arrived at this site searching for "famous quotes relating to goats" was able to find what he was looking for.

If not, I think I can help. Here is how Suetonius describes Caligula (Lives of the Twelve Caesars, Cal. 50):

He was very tall and extremely pale, with an unshapely body, but very thin neck and legs. His eyes and temples were hollow, his forehead broad and grim, his hair thin and entirely gone on the top of his head, though the rest of his body was hairy. Because of this it was considered a capital offence either to look down on him from a higher place as he passed by, or to mention a goat for any reason whatever.

It may not be a famous quotation, but I think it deserves to be. The rest of Suetonius' Life of Caligula can be found here in English, here in Latin.