No really, have you ever been caught at a bad moment by a complete stranger then realised some time later they probably think you are a bit of an arse? It's not a nice feeling knowing someone out there doesn't know the real you. It's not nice to think that although you are a decent person someone thinks you're a dick.
Your website is often the only information a stranger has about you. How many people every day are coming to your site and leaving within a few seconds cursing you for wasting their time. How many people are telling others that your company sucks simply because a page on your website is a bit iffy?
SEO will help to drive traffic to your site but it's only part of the story. Too often in forums I read "will Google like this or will Google like that". Make sure your site gives the right message and talks to your visitors. SEO by all means but not at the expense of the reason your site exists, customers and sales.