Today we released our quarterly statistics regarding the rate of click fraud for Q4 2009, which came in at 15.3%.  We first began publishing industry data over four years ago, in 2006, which means we can now look at the trend for the same quarter over the past four years.  The fourth calendar quarter has traditionally been the annual high, and this year is no different.  15.3% is higher than any of the three previous quarters.  Like Willie Sutton who robbed banks because “that’s where the money is,” fraudsters find the increased search traffic during the Q4 holiday season to be a prime opportunity for illicit gain.

What’s different this year is that the trend of click fraud increasing annually, which we’ve observed for the past three years, has stopped.  For the first time, the Q4 click fraud rate has declined from 2008 to 2009.  Given that Q4 2008 was the highest click fraud rate we’ve ever reported, this isn’t too surprising.  But it’s still good news for the industry.  Even as fraud schemes become increasingly sophisticated with the advent of spyware, malware, adware, and botnets, the industry’s efforts to thwart fraud and protect advertisers seem to be working.  By the way, when I say “the industry,” I’m including the major search engines themselves.  Google, Yahoo!, and Microsoft all have active traffic quality programs in place to keep one step ahead of these new sources and methods of fraud.

Unfortunately, not every ad network, publisher, and advertiser can afford to build a team of PhD’s to constantly monitor and fight the problem.  That’s why we’re here.

While it’s easy to see how the recently discovered Bahama Botnet is cheating online advertisers out of free traffic and generating fraudulent fees for complicit parked domains and ad networks, it’s important to note that ad providers are being victimized as well.

We have conducted additional research into the behavior of the Bahama botnet and found that it acts as a sort of perverted “Robin Hood” among ad networks by robbing ad revenue from the top-tier players and delivering fraudulent traffic to second and third-tier ad networks and publishers.  Chief among the ad provider victims is the one with the biggest treasure to take: Google.

As we’ve seen in this video, when an infected user performs a search on Google.com, they get some peculiar results.  This is because, unbeknownst to the user, they’re not actually on Google.com.  The page looks like Google.com and even says Google.com in the browser’s address bar.  So how can it not be google.com?  The perpetrators behind the Bahama Botnet are able to steal traffic and revenue from Google using a trick called “DNS poisoning”.

All computers on the internet identify themselves with a set of numbers that we know as an IP address.  Computers can find one another using these numbers.  However, humans find words easier to remember than long sets of numbers, so the Domain Name System (DNS) was devised to translate these numbers into names.  When “Google.com” is typed into a browser, the computer uses DNS to translate that domain name into a number.  In the case of Google.com, that number happens to be 74.125.155.99.  The DNS method for translating domain names into numbers is fundamental to making the internet work.

However, in the case of the Bahama Botnet, this DNS translation method gets corrupted.  The Bahama botnet malware causes the infected computer to mistranslate a domain name.  Instead of translating “Google.com” as 74.125.155.99, an infected computer will translate it as 64.86.17.56.  That number doesn’t represent any computer owned by Google.  Instead, it represents a computer located in Canada.  When a user with an infected machine performs a search on what they think is google.com, the query actually goes to the Canadian computer, which pulls real search results directly from Google, fiddles with them a bit, and displays them to the searcher.  Now the searcher is looking at a page that looks exactly like the Google search results page, but it’s not.  A click on the apparently “organic” results will redirect as a paid click through several ad networks or parked domains — some complicit, some not.  Regardless, cost per click (CPC) fees are generated, advertisers pay, and click fraud has occurred.

An interesting side effect of this whole scheme is that while the perpetrators of the Bahama Botnet turn organic or natural search listings into paid links, they don’t seem to alter the final destination domains of the sponsored links that show up on a search results page.  When an infected user clicks on one of these sponsored links, they always seem to end up on the correct destination domain (so clicking a sponsored link for Dell.com, for example, will always take an infected user to dell.com).  However, due to the DNS poisoning, a click on a sponsored link will never go through Google’s own click-counting redirect.  Google never sees, and therefore never charges for, that click.   The advertiser gets a free click, instead of a paid one, and Google loses the revenue.  The Bahama Botnet strikes again.

In February of 2006, Click Forensics was just getting off the ground.  We recognized the problem of click fraud was a big problem and that building a solution would be tough technical challenge.  We decided to bring in an expert in the field of data mining and anomaly detection in clickstream analysis.  That expert was Dr. Alex Tuzhilin.  Alex spent the day with us at our offices in San Antonio and provided us a roadmap for the evolution of our approach to indentifying invalid traffic. 

His contribution to us at that point was essential and provided tremendous insight.  After reviewing our approach he commented,

“Click Forensics has good data and this is a source of their advantage over the search engines. My role is to work with them to refine the scoring methodology to improve accuracy. Their approach is to incorporate as much data as possible to improve accuracy. The search providers simply don’t have enough data to have the most accurate approach.”

Shortly after Alex’s visit to Texas, I received a call from the lead attorney representing Lane’s Gifts in their lawsuit against Google.  He said, “Tom, I just hired your Ph.D!”  He told me that the judge in that case had mandated that an outside consultant review Google’s click fraud detection methods and publish a paper on the efficacy.  Alex spent many weeks at Google and wrote an insightful paper detailing their approach, ultimately describing it as “reasonable”.  The Lane’s Gift case was settled and Alex returned to his role as a professor at NYU.

Today we are thrilled to announce that Dr. Tuzhilin has joined the Click Forensics Advisory Board.  Few individuals have had more real-world and academic experience in the measurement of online traffic quality and its effect on advertisers.  His work has helped move the industry toward standards and cooperation.  After visiting us in Austin a few weeks ago and meeting with our technology team, Alex said,

“Having firsthand experience reviewing the state of the art in ad network traffic management, I was impressed with the level of technical sophistication the team exhibits and I was impressed with the directions they are going, Click Forensics has played a leadership role in helping the online advertising community to monitor quality of clicks on ads, including identification of invalid clicks. I look forward to continuing to work with the team.”

In addition to Dr. Tuzhilin, we have also added Dr. William Wright, the Chief Scientist at Paypal.  Dr.Wright, a Ph.D. in cognitive science, is an artificial intelligence expert who has built numerous analytical and predictive systems over the past twenty years, including the Falcon Credit Card Fraud Detection System at HNC, the Advanced Fraud Screen system at CyberSource, and numerous adversarial modeling systems for the U.S. military.  After spending time with our team, William concluded,

“Click Forensics has built a strong team of developers using very advanced machine learning and data mining techniques to detect fraud and measure traffic quality, they are pioneering a new area of fraud detection and I’m finding it satisfying to work closely with them on leveraging lessons from my past experience combating credit card and banking fraud.”

One out of every five employees at Click Forensics holds a Ph.D.  Adding the expertise of Alex and William dramatically enhances our ability to meet our goal of providing the state of the art approach to traffic quality management.  I appreciate their contributions and look forward to benefiting from their knowledge in the future.

Today we announced the pay-per-click (PPC) fraud figures for Q2 2009.  The data comes from the Click Fraud Index.  Traffic across more than 300 ad networks is also reflected in the data.

Key findings from data reported for Q2 2009 include:

-    The overall industry average click fraud rate was 12.7%. That’s down from 13.8% for Q1 2009 and from the 16.2% rate reported for Q2 2008.
-    Click fraud traffic from sophisticated sources and scripted programs rose again in Q2 2009. This included a rise in the incidents of publisher collusion fraud on ad networks.

The data in Q2 also showed that many of the new click fraud schemes identified last quarter continue to increase in number and sophistication. Publisher collusion fraud was one example. This scheme occurs when online publishers use rotating IP-addresses or botnets to click ads on their own sites in order to generate inflated commissions from unprotected ad networks. Ad networks have difficultly differentiating such attacks from valid clicks.

Wow!  Click fraud is real?  Click fraud costs online advertisers millions of dollars?  Click fraud can be uncovered and the perpetrators caught and punished?   Who knew?   Well, we did.

This week Microsoft filed the a complaint in U.S. District Court (Microsoft v. Lam, et. al., case number 09-cv-0815) seeking injunctive relief and damages from a group of people found to be perpetrating click fraud through the Microsoft adCenter platform.  This is only the second time (Google sued Auctions Expert International in 2004) that a search provider has ever caught and sued an individual (or a family, in this case) for click fraud.  We congratulate Microsoft for their efforts to root out this activity and encourage them in their pursuit of relief.  Online advertisers should appreciate knowing that click fraud does not always go undetected or unpunished.

For those not familiar with the case, it’s an example of what we call “competitor click fraud.”  The motivation of the perpetrators was simply to obtain higher-placed ad positions for lower bid amounts by depleting the daily budget of their competitors.  The verticals affected were auto insurance and the online role-playing game World of Warcraft.  Microsoft identified two brothers and their mother who controlled adCenter accounts that benefited from this fraud.  They believe that this scheme affected more than just adCenter advertisers, but also the advertisers on competitive search engines.

Microsoft’s complaint, now public information, is so well written it could be used as a tutorial on click fraud detection.  The most fascinating section describes the nearly year long game of cat-and-mouse played with the defendants.  Reading from the complaint: “When Microsoft took steps to mitigate these automated attacks, the perpetrators followed by implementing countermeasures to Microsoft’s actions.  A cycle of events ensued whereby the Defendants would update their attack methods to bypass the fixes implemented by Microsoft, and Microsoft would take additional steps to combat the new click fraud attacks.”

The lessons here are pretty clear:  Click fraud is still a problem and solving it requires constant vigilance.  The online advertising community needs to work together – search engines, ad networks, advertisers, and third-party auditors – to protect ourselves from this threat.

From spyware to bots to viruses and other unimaginable hazards… the web can be a scary place.  As far back as Prodigy in the early days of the online world, scams have been a part of the party.  The Internet is simply a new way for the bad guys to rip off unsuspecting consumers.  The key difference though is that the reach is enormous and the damage can spread to more people, more quickly than ever before.

Enter scareware, new way to trick unsuspecting consumers into parting with their money.  USA Today recently had an article  about the tricks and tactics used to perpetrate this latest rip off.  Unfortunately, online advertising has become an accomplice to the crime.

Scareware is worthless software that allegedly removes viruses from your computer.  Anyone who has surfed the web knows how easy it can be to become infected with a virus.  The damage to the users computer is often measured in slowed performance, unwanted clicking and potentially even more nefarious things like key logging and password swiping.  Now, the bad guys are selling “scareware” to solve a problem that may not actually exist.

The first such program was called “SpySheriff,” built by a team of cyber crooks from Russia.  The Anti-Phishing Working Group recently reported that scareware infections rose 48% in the second half of 2008.  The growth is tied to the ease of distribution and weaknesses in online advertising and the web in general.

There are several ways these fake products are being distributed.  Phony pages are created using hot search key words such as “American Idol” or “iPhone” and drive the unsuspecting consumer to the infected page.  Recently the Facebook email scam was used to send people to a page by promoting things like “best video.”  Since these emails came from your friends, millions clicked.  Twitter has become a vehicle for distribution. Phony Twitter accounts are created and enticing titles of posts encourage people to click.
 
Additionally, the bad guys are simply buying display or search ads.  They rotate in infected pages to the landing page.  It is virtually impossible for an ad provider to scan every ad impression and linking page.  This loophole creates an opportunity for the bad guys to drive significant traffic to infected pages at a very low cost.  Microsoft reported finding 4.4M installations of one such program, so the scale is enormous.  Do the math… at $49 or $79, that is big business.

Once someone lands on the page, getting off is nearly impossible.  Immediately upon landing, a “system scan” begins.  The results are, of course, showing that your computer is infected with a number of viruses.  Conveniently you can buy the product at that point and they take your money and run.  If you try to move away from the page, or cancel, an endless number of scans take over your screen.  Essentially, users must “control/alt/delete” their way out or restart.

The danger in this scam is not limited to monetary damage to the consumer.  These type of pages and methods to attract clicks are the same methods used to install spyware, malware and perpetrate click fraud.  To their credit, USA Today has done a good job over the last few years of highlighting the dangers of the web to the average consumer.

The FTC is cracking down.  They have identified products like WinFixer, DriveCleaner and XP AntiVirus as worthless and they are going after the owners.  The problem is that like the click fraud crooks, these guys are in remote locations and move their servers often. Tracking them is a full time job and extremely difficult.  The search engines are trying to help as well.  Bing has a neat feature that highlights “at risk” url’s.  Yahoo has similar product built with McAfee.

 
Trust is what keeps consumers clicking on ads.  Without stepped up industry efforts from organizations, like the Anti Phishing Working Groups and others, trust could be diminished.  Like click fraud, scareware is damaging trust.  It takes a community effort to stay after the problem and build solutions to take the scare out of the internet.

The New York Times ran a feature article this week on click fraud.  Why you ask?  Because, like spam, click fraud is still a big problem for advertisers. The article pointed out that as the economy tilts downward, advertisers cannot afford to waste dollars. This is a good news, bad news scenario for online advertising.

The good news is that online advertising is highly measurable.  Large advertisers that traditionally have been offline are now shifting dollars online.  This fact has contributed to online advertising continuing to grow as traditional media is in decline.

The bad news however, is that this window of opportunity is narrow.  The online advertising community must embrace measurability and enhance trust to gain share of spend from the big guys. 

There was a significant event this week that helped in that effort.  The Interactive Advertising Bureau (IAB) released from draft the Click Measurement Guidelines.  This document, three years in the making, is a great start for our community to come together around standards and enhance trust. Dozens of ad providers are busily working with third party audit firms to become accredited to the new guidelines.  Advertisers will have a way to gauge the level of commitment from ad provides when this list is made public.

Click Forensics was proud to represent advertisers in this process.  In fact, we were the only traffic quality management firm to participate and were quoted in the press release from the IAB.  Many thanks are in order for the 38 members of the working group for a job well done.

Now, we find ourselves at the beginning.  An opportunity exists to build on the foundation laid by the IAB member companies.  Click fraud is going to be a problem for a long time to come.  Progress is being made.  But in order to re-accelerate the growth of online advertising we need more than standards.  We need a community effort to work together to ensure advertisers have confidence that they get what they pay for.  Articles raise awareness, documents create a process and awareness builds urgency.  But ultimately it will take the effort of everyone in the community to get to the day where trust is commonplace and online advertising becomes the marvelous, measurable media it can be.  We look forward to continuing our efforts toward that goal.

Today the Click Fraud Index for Q1 2009 was released, and the startling news would appear to be that the click fraud rate in Q1 dropped to 13.8% from over 17% a quarter ago.  But there are several plausible explanations for the sharp drop in click fraud.  The real news was in the types of fraud that were identified, and the likely targets of these new attacks.

So why the drop-off?  Several factors likely contributed.  First, the Q4 fraud rate was unusually high, the highest ever in the history of publishing the click fraud index.  Due to the huge amount of online spending during the holiday season and the eroding economic climate, there was a sort of “perfect storm” for fraudulent activity.  In Q1, by contrast, there was a drop-off in online ad spending — Google reported their first-ever quarterly revenue decline — and more importantly, a decrease in the average cost-per-click (the number of paid clicks still increased by 17%, even though total revenue was down).  Lower CPCs means less reward for fraudsters.

Maybe the most important contributor to the reduced click fraud rate, though, was the heightened awareness of bots, worms, and other forms of malware created by the Conficker reporting (anyone see “60 Minutes?”).  It would appear that the tier 1 ad providers and ad networks did a much better job of mitigating fraudulent activity in Q1 than in previous quarters.  We can only hope this is a trend that continues.

But the more interesting trend uncovered in the Q1 data were the new types of fraud and the new targets of click fraud.  The data showed several examples of malicious scripts (JavaScript) designed to perpetrate click fraud.  When a visitor lands on a site these scripts execute by opening a zero iframe or zero-pixel window and clicking on paid ads.  The site visitor never sees these hidden frames and never visits the advertiser’s site.  But the advertiser pays for the click, and the site owner reaps the commission for the paid click.  Because the clicks are generated by a real browser with a valid IP address on a real web site with no suspicious repeat clicking patterns, this type of fraud is very difficult to discover for the average advertiser or unsophisticated ad network.  (No, we can’t tell you how we find it, but we do.)

The bottom line is that the click fraud rate was down in Q1, but click fraud schemes continue to get more sophisticated.  As tier 1 ad networks and ad providers like Yahoo! and Google continue to increase their efforts and effectiveness, the fraudsters will migrate elsewhere.  Tier 2 ad networks must focus on traffic quality initiatives in order to protect their advertisers, and themselves.

This past week in New York during the Search Engine Strategies conference, we sponsored a VIP dinner for the Click Quality Council.  Attendance was robust.  Although the food and drinks were excellent (thank you, McCormick & Schmick’s) I think the enthusiastic response to the CQC invite was more about the guest list than the menu.

In attendance were a wide variety of senior executives from large ad networks, online publishers, and agencies, including at least three CEOs and a dozen VPs.  Tom Cuthbert, president of Click Forensics, served as host and MC for the evening.  Some of the special guests included Dr. Alex Tuzhilin from NYU, a recognized authority on click fraud and online traffic quality, and author of the famous Lane’s Gifts v. Google report, as well as Joe Lazlo, Director of Research for the Interactive Advertising Bureau (IAB).

Joe was an especially welcome guest in light of the imminent Click Measurement Guidelines that the IAB will be publishing very soon.  Both Click Forensics and the CQC have been active participants in the discussions and debate that have formed the proposed guidelines.  Joe updated the Council on the fact that the guidelines were currently open for public comment, and provided a brief overview of what the guidelines (and the IAB itself) were designed to accomplish: ensuring advertisers get what they pay for.  Since that’s a core tenet of the CQC, the message was well received.

We thank Joe for his participation and hope to see him again at future events, perhaps even the next CQC Dinner in San Francisco on April 21 during Ad:Tech?

The remainder of the evening was filled with lively discussion about click quality and plenty of war stories having nothing at all to do with click quality.  I hope everyone found the evening as productive and enjoyable as I did.  Kudos to Laura Wolf for organizing and executing yet another successful event!

The problem of botnets is getting worse, not better.  In fact, over 30% of all click fraud comes from botnets, maybe even YOUR computer!  Ever wonder how botnets work?  The BBC has acquired control of 22,000 computers and have been demonstrating how it works.  The videos are easy to understand and  very interesting.

Cyber crime risk exposed
How cyber criminals attack websites
Is your PC doing a hacker’s dirty work?

Click fraud is costing advertisers millions of dollars a year.  So how can you protect your computer from becoming a party to the crime?  Again the BBC site has an excellent article with practical steps called, “How to keep your computer secure“.  Take time to read it and be sure you are doing your part to reduce click fraud.