It's a good question. There are many answers, not just one. For starters I am my own worst critic. I fail in ways that others would still consider success. But I am certainly not my only critic. All of the people we develop relationships with, friends, spouses, relatives, coworkers, bosses, customers, we can fail each one of them.
Lest this be one of my depressing rants, those people can all determine success too. However, success seems to be a harder thing to own up to. I mean, it is easier to criticize than praise. I see this in myself. I don't write a review after a really good experience with a company, but after a bad experience. People don't call customer service to say how great something is.
Failure hurts. Failure is not fun. The worst part is, the more judges there are, the more people you are bound to fail. Just look at how many people hate Apple products. That's life I suppose, just get up and keep going.