Some time ago I did start to collect quotes related to the topic complexity on my web site. A friend of mine has now suggested to add "if it is not elegant is is wrong". He did hear this phrase often during his studies.
If the essential complexity can be represented in an elegant way, then this makes a problem for sure easier to understand. I don't think that there is always an elegant representation - if there is none, then the problem may still be understood correctly.
Absolutely not true is the opposite - many things are very elegant and also completely wrong. For this I can quote the movie "A beautiful mind" where Nash says "It's elegant, but wrong" to
Alicia Larde about her work. Nevertheless she does marry him later on.