Yahoo Unveils New Content Acquisition Program

Yahoo’s New Content Acquisition Program
› › › Search Engine Marketing

BY Chris Sherman | March 17, 2004

Just two weeks after launching a brand new search engine, Yahoo! announced a content acquisition program that consolidates all its paid inclusion programs. This marks the beginning of an aggressive new campaign to significantly expand the scope and quality of content available via Yahoo search.

The new program, Content Acquisition Program (CAP), focuses on commercial and noncommercial sites. On the commercial side, the new program is called Overture Site Match. The noncommercial effort is geared toward quality sites traditionally overlooked by search engines, such as parts of the invisible Web. “It reflects our desire to invest heavily to get more content into our search engine experience,” said Tim Cadogan, Yahoo’s search VP.

A key program aspect is outreach to new and existing customers. “Inclusion in its own right is not the most exciting component going forward,” said Cadogan. “What we are excited about is having a better relationship with the people who are providing content. There’s currently a lot of uncertainty and guesswork. We think there’s a lot of room for improvement. There’s a lot of demand for knowledge in how to participate in search.”

As with previous paid inclusion programs, rank is determined by site quality. Participating doesn’t mean any boost in Yahoo’s relevance algorithms, at least explicitly.

“Site Match helps us understand more about the site,” said Cadogan. “We have more meta information about your site, and that will allow us to position you to users more appropriately. If you’re a good site, that’s going to be great for you; and if your site is bad, it’s probably going to hurt you.”

Rolling Six Programs Into One

With its acquisitions of Inktomi and Overture, Yahoo inherited six pay-for-inclusion programs (from Inktomi, AltaVista, and AlltheWeb). Over the past year, the company ran all six. For marketers, this posed a problem. All the programs had separate terms, pricing, and distribution. From a practical standpoint, it was necessary to participate in several programs to assure the broadest reach.

The new program consolidates all six programs into one. By participating, search marketers get the broad distribution offered by the previous programs, with a critical addition — inclusion in the new Yahoo index.

It’s a major benefit. Last month, Yahoo handled about 30 percent of all U.S. search traffic, according to Nielsen//NetRatings. Until the switch to the new engine, actual search results were powered by Google, which has no pay-for-inclusion program. In effect, Yahoo now offers marketers a major new source of search traffic through one unified program.

Creating Transparency and Structure

Search marketing has traditionally been a cat-and-mouse game between search engines and content providers. “Search engines do what they will, and the Web site has to guess what they’re doing,” said Cadogan. Although this often frustrated legitimate content providers, search engines justified this less-than-forthcoming approach as necessary to prevent site owners from manipulating the system in their favor.

“We’re going to continue to do that,” said Cadogan. “But on top of that we’re going to reach out to providers and engage them. Part of the way we want to do that is to create a much more transparent, structured relationship between us and content providers.”

As part of this structured relationship, Yahoo is establishing guidelines for what it considers quality content. This should help content owners have a better understanding of Yahoo’s ranking criteria, without revealing too much detail that could help spammers.

Bucking a trend, Yahoo is encouraging content providers to provide detailed metadata for each page. Most of the major search engines have recently spurned metadata as a reliable source of content information.

Participants in the CAP program are subject to a regular review process with both automated components and input from Yahoo’s editorial team. Cadogan says a key benefit of program participation is feedback from these quality reviews, though he won’t say exactly what type of feedback will be offered, nor whether favorably reviewed sites get a ranking boost.

Pages that don’t meet Yahoo’s standards will be given the opportunity to improve or risk potential rank demotion.

Embracing the Invisible Web

In addition to consolidating paid inclusion programs, Yahoo is embarking on an aggressive program to work with noncommercial sites that offer valuable content but have technical issues with index inclusion.

“We want to get much more content. That means reaching out to providers not currently on the Web — invisible Web databases,” said Cadogan. “We are reaching out to major content providers, in academia, government, libraries, and so on, and creating a structured interaction with them.”

Yahoo is working to resolve technical issues on a case-by-case basis, revealing some terrific content previously hidden to search engines.

Yahoo worked with National Public Radio to make over 17,000 hours of audio content accessible. Included is Northwestern University’s online OYEZ project, with over 2,000 hours of U.S. Supreme Court audio, including all audio recorded since 1995. Another new source is UCLA’s Cuneiform Digital Library Initiative, which documents cuneiform tablets dating back to 3200 B.C.

These content providers are responding very favorably to Yahoo’s efforts to make their “hidden” content more accessible. “What is most exciting about this is the level of engagement,” said Cadogan. “They’re just delighted to participate. It opens up a whole new set of worlds.”

In general, Yahoo doesn’t have exclusive arrangements with these providers. Cadogan says that’s not important. What matters is broadening Yahoo’s reach into many more kinds of content. It helps Yahoo’s engineers better understand how to solve some technical challenges associated with the invisible Web. “Having spent the time to see how someone’s database works is quite valuable,” he notes.

AltaVista and AlltheWeb?

AltaVista and AlltheWeb will continue to operate as standalone search properties, though they both now use the same underlying index that powers Yahoo.com. Cadogan says they nevertheless have different features that should appeal to fans of each service. AltaVista’s advanced search capabilities remain strong, for example.

Over time, Cadogan says Yahoo may attempt to differentiate the two engines. “We’re going to optimize the properties for each audience,” he said. “We may experiment a little bit more on those sites.”

Potential Disclosure Issues

Yahoo’s new paid inclusion program could potentially open up new controversy surrounding disclosure. Though the company clearly labels paid placement “sponsor results” and places them in separate areas of result pages, the new paid inclusion listings are not explicitly differentiated — even though they generate CPC revenues for Yahoo.

This may raise issues with the Federal Trade Commission (FTC), which issued a letter of recommendation to the search engines about disclosure practices in mid-2002. Though the major search engines have clearly labeled sponsored listings since then, few fully confront the issue of clear disclosure for paid inclusion results, a central FTC concern.

Yahoo does provide a limited form of disclosure through a “what’s this” link next to the “Web results” label. Click and you’ll see a page describing the structure of the search results page. On this page, the following appears:

CAP enables content providers to submit web pages and content directly to Yahoo! for review and inclusion within the Yahoo! Search index through the Site Match or Inktomi Search Submit programs. Sites from both of these programs pay to be included in our search engine index, and those participating in Site Match also pay to be reviewed for relevance.
Searchers can’t easily distinguish between paid inclusion results and content Yahoo’s crawler found on its own. There’s no indication Yahoo is paid when a searcher clicks a paid inclusion link.

Cadogan insists this isn’t a problem as there’s no difference in treatment between pages participating in the inclusion program and those found naturally. “Payment is linked with the interaction. The payment does not connect with the ranking,” he said. “We’re going to do everything we can to reward good sites and punish bad sites, period.”

That may be the case. But it’s virtually certain vocal consumer watchdog groups will have issues with Yahoo not being fully forthright about the commercial nature of these listings. We’ll keep you posted.