Google has been the most aggressive search engine, in terms of defeating web spam. When you get into the technicalities of how search engines work, there are two big categories - on-page factors and off-page factors.
On-page factors are things that the webmaster can control (ie, manipulate) - as a for-instance, there's what is called a "keywords meta tag." It was designed for webmasters to help tell the search engines what their content is about.
As you can imagine, this was greatly abused by spammers, causing them to rank for searches that they had nothing to do with. From the onset, Google has NEVER relied on the keywords meta tag because of this reason.
Google relies heavily on off-page factors - aka, incoming links. Instead of the webmaster telling Google what their page is about, they rely on what the incoming links for a page have to say.
As an example, if you try to optimize a page to rank for "weight loss diet," but there's a bunch of incoming links pointing to your page that say "cookie diet," Google's likely going to rank you for "cookie diet" and not "weight loss diet."
This technique is called "crowdsourcing" - it's relying on the hundreds of millions of other webpages around the world to accurately represent what different pages are about, by what words they use in the link.
Now the other guys (Yahoo! and MSN) did pick up on this, but Google has further evolved. The old school spam tricks still work on MSN, and Yahoo! does look at incoming links, but Yahoo! works on a "quantity" system, whereas Google has moved into "quality."
As you can imagine, once incoming links were found to greatly increase ranking ability, spammers set across the web to create as many incoming links as possible (this is about the time that your Myspace comments started getting hammered with spam, etc).
Since then, Google now looks at trust factors of the sites that are linking to a certain page. They heavily discount (or eliminate) links from low-value sites, and focus on the links from "trustworthy" domains.
As you can imagine, even that is prone to abuse (I can buy links from some good, high PageRank sites), and so Google has moved out to off-web factors. This deals with how the consumers interact with content - Google Toolbar monitors your web behavior, and through this, they can get an idea of the quality of a website - if a bunch of users visit and quickly leave a certain site, then that site can be assumed to have low quality.
Also, they are watching it from within the SERPs (search engine result page - the thing you see when you do a search in Google). If you click on a result, and then come back (especially if you come back quickly), then again, it can be assumed that the result is either of low quality, or not relevant for your search.
That's kind of a broad sweep as to why, but gives you a good idea. Google's biggest advantage is that they have 200 PhD's working for them, as well as tens of thousands of grad students submitting free work to them, trying to get a job.