What makes it worse is you often find, particularly during your first release or demo to the users, that instead of a feeling of reward, you leave with a feeling of "this is not worth the effort". This stems from the reaction of your users or clients - because instead of admiring your perfectly working functionality, they focus on how it looks. "Why is that button off-centre?" becomes a 10 minute discussion, rather than admiration for the effort on the mind bogglingly highly optimised and complex financial calculation you managed to pull off.
This comes, I believe, from a fundamental misunderstanding between users and developers. To a user, the feel of the system is top of mind. It's easy for them to see visual progress, and non technical people often attribute looks with excellence, regardless of whether something works or not. The fact that your back end has a perfect architecture means nothing to them. If it looks like crud, they'll think the whole thing is crud.
I've personally tried a few analogies to get users to understand the concept that you need the foundation correct before you can make it look pretty. The one that has worked well is an analogy with building a house, and goes:
When you build a house, do you dig the foundation, lay a brick, cement it, plaster it, and paint it before adding the next brick? No, you complete the house first, and then you make it look pretty. Otherwise your house won't be very stable, or pretty.
And software is the same. Get the architecture and functionality right first, and then focus on the interface.