In February of 2006, Click Forensics was just getting off the ground.  We recognized the problem of click fraud was a big problem and that building a solution would be tough technical challenge.  We decided to bring in an expert in the field of data mining and anomaly detection in clickstream analysis.  That expert was Dr. Alex Tuzhilin.  Alex spent the day with us at our offices in San Antonio and provided us a roadmap for the evolution of our approach to indentifying invalid traffic. 

His contribution to us at that point was essential and provided tremendous insight.  After reviewing our approach he commented,

“Click Forensics has good data and this is a source of their advantage over the search engines. My role is to work with them to refine the scoring methodology to improve accuracy. Their approach is to incorporate as much data as possible to improve accuracy. The search providers simply don’t have enough data to have the most accurate approach.”

Shortly after Alex’s visit to Texas, I received a call from the lead attorney representing Lane’s Gifts in their lawsuit against Google.  He said, “Tom, I just hired your Ph.D!”  He told me that the judge in that case had mandated that an outside consultant review Google’s click fraud detection methods and publish a paper on the efficacy.  Alex spent many weeks at Google and wrote an insightful paper detailing their approach, ultimately describing it as “reasonable”.  The Lane’s Gift case was settled and Alex returned to his role as a professor at NYU.

Today we are thrilled to announce that Dr. Tuzhilin has joined the Click Forensics Advisory Board.  Few individuals have had more real-world and academic experience in the measurement of online traffic quality and its effect on advertisers.  His work has helped move the industry toward standards and cooperation.  After visiting us in Austin a few weeks ago and meeting with our technology team, Alex said,

“Having firsthand experience reviewing the state of the art in ad network traffic management, I was impressed with the level of technical sophistication the team exhibits and I was impressed with the directions they are going, Click Forensics has played a leadership role in helping the online advertising community to monitor quality of clicks on ads, including identification of invalid clicks. I look forward to continuing to work with the team.”

In addition to Dr. Tuzhilin, we have also added Dr. William Wright, the Chief Scientist at Paypal.  Dr.Wright, a Ph.D. in cognitive science, is an artificial intelligence expert who has built numerous analytical and predictive systems over the past twenty years, including the Falcon Credit Card Fraud Detection System at HNC, the Advanced Fraud Screen system at CyberSource, and numerous adversarial modeling systems for the U.S. military.  After spending time with our team, William concluded,

“Click Forensics has built a strong team of developers using very advanced machine learning and data mining techniques to detect fraud and measure traffic quality, they are pioneering a new area of fraud detection and I’m finding it satisfying to work closely with them on leveraging lessons from my past experience combating credit card and banking fraud.”

One out of every five employees at Click Forensics holds a Ph.D.  Adding the expertise of Alex and William dramatically enhances our ability to meet our goal of providing the state of the art approach to traffic quality management.  I appreciate their contributions and look forward to benefiting from their knowledge in the future.

While attendance at SES San Jose was definitely down this year, it was still a great show.  I enjoyed meeting people at our booth and appreciated the hard work of our team pulling that all together.  We met a lot of interesting folks and enjoyed hearing feedback on the new Click Forensics dashboard.

The big hit, of course, were the “Stress Einstein” squishys! Who wouldn’t want of these guys!  They reminded me of the bobble-heads in “Night at the Museum 2″  

I also enjoyed participating in a session titled, “Ads in a Quality Score World“.   Mike Grehan moderated the panel and both Yahoo (Tomaso Pozzi) and Google (Jonathan Alferness) participated.  WebProNews covered the session and wrote a nice recap. 

The other highlight of the week for me was our Click Quality Council dinner.  We had over 25 executives from ad networks, publishers, advertisers and search engines join us for a great dinner of conversation, networking and fun!

The conversation is always interesting when you have people representing all corners of the online advertising ecosystem.  It was the third year we have hosted the CQC dinner in San Jose and our 13th since the Council was formed in 2006.

Congratulations to the newlyweds… after a long, long courtship Microsoft and Yahoo finally managed to get together (the prenuptials are still being sorted out!).  I have been in favor of this union for sometime now.  Google owns a ridiculous share of the pay per click advertising market and desperately needs a competitor.  Microsoft + Yahoo = Competition.  As I have said for the last several years, the lens we look through at Click Forensics is that of the advertiser.  Competition is always good for the advertiser.  

The growth of online advertising, in particular pay per click advertising, has been meteoric. It is a great model and one that has proven hugely successful for hundreds of thousands of advertisers large and small.  It is a model that will continue to grow as large advertisers shift more dollars from unmeasureable and less effective traditional media.  It will grow because it uses context, targeting and relevancy to the highest level.  Yahoo’s audience enhanced by Microsoft’s technology will mean innovation and efficiency.  There is no doubt; Google will continue to have success.  But the new partnership will make the online world even more attractive for advertisers.

Today there are standards in place to help hold the search providers accountable.  There are better reporting, campaign management and keyword tools to add to the efficiency.  I see a world in the near future where display advertising will begin to make significant gains from the data that exists in search. Context, targeting and relevance can improve every medium and this partnership will leverage that data to a much higher level than before.

So congrats to you both for a new start.  The entire advertising community is pulling for you and expecting big things. I do need to warn you… expectations are high and the honeymoon is short.  

We recently launched a new reporting interface for our ad network customers to provide more actionable insights into downstream publisher sources. The early feedback from new adopters has been very positive. In this blog post, I will provide a brief overview of the new enhancements.

So, why did we decide to roll out a new reporting platform? As we are growing with our customers, we learned to better understand their most pressing business problems and how our products can help address those. Downstream publisher management and new publisher screening are critical to daily operations of ad networks. We were looking for new ways to tackle them in a more effective manner.

Based on insightful customer feedback we had received, we worked on the new reporting interface with the following design goals in mind:

• Present actionable insights into large volumes of click data to better highlight traffic quality from downstream publisher sources
• Easily handle daily click volumes of 10 million clicks and deliver sub-second interface response times
• Provide detailed background information about new publishers during the approval process before accepting traffic

Users of the new interface are greeted by the publisher dashboard that summarizes current network and publisher activity. It is the starting point for further investigation. At a glance, ad network staff find their top publishers sorted by different attributes, for example by volume and traffic. The “Movers and Shakers” sections highlight sources with significant traffic changes in certain dimensions. Here is a screenshot of the dashboard:

Throughout the interface, we added small sparklines to offer traffic information at a glance without having to leave the page. For example, an average click score is accompanied by a score histogram (see this previous post for more details) and a volume volatility graph. These cues help users to absorb the information more quickly and spot areas that warrant further investigation:

Another tab in the interface offers publisher screening capabilities. It supports ad networks in making better decisions about which publisher applications to approve before accepting any traffic. After entering a URL, a report will present detailed background information about the specific domain, combining reputation information from the Click Forensics community database and public sources like Alexa, Compete.com and whois.

Next to the highlighted features, we have added numerous other useful capabilities for our customers. The new reporting interface is now available to all of our ad network customers. If you would like to learn more please don’t hesitate to contact us.

Today we announced the pay-per-click (PPC) fraud figures for Q2 2009.  The data comes from the Click Fraud Index.  Traffic across more than 300 ad networks is also reflected in the data.

Key findings from data reported for Q2 2009 include:

-    The overall industry average click fraud rate was 12.7%. That’s down from 13.8% for Q1 2009 and from the 16.2% rate reported for Q2 2008.
-    Click fraud traffic from sophisticated sources and scripted programs rose again in Q2 2009. This included a rise in the incidents of publisher collusion fraud on ad networks.

The data in Q2 also showed that many of the new click fraud schemes identified last quarter continue to increase in number and sophistication. Publisher collusion fraud was one example. This scheme occurs when online publishers use rotating IP-addresses or botnets to click ads on their own sites in order to generate inflated commissions from unprotected ad networks. Ad networks have difficultly differentiating such attacks from valid clicks.

Sonny Corleone, in The Godfather, said “I want Sollozzo. If not, it’s all-out war: we go to the mattresses.”  While the origin of the “mattress” phrase is debatable, the meaning is clear: it’s time to get serious and face this threat head-on.  (Yes, this is the line Tom Hanks referenced in “You’ve Got Mail.”)

Why am I writing about gangster movies?  Because that’s the image that “collusion fraud” conjures up, appropriately.  Collusion fraud occurs when online publishers use rotating IP addresses, or botnets, or just armies of people, to click on paid search ads on their own sites.  Collusion fraud is hard to find because the fraudulent clicks are spread across dozens (or hundreds) of sites and the clicks are generated from dozens (or hundreds) of different IP addresses.  So everything looks kosher.  But it’s not.

Standard click fraud detection mechanisms employ various anomaly detection rules.  1,000 clicks from the same visitor in the span of a minute, or an hour, or week, is a traffic anomaly that’s easy to find.  Almost all search engines and ad networks would screen this out.  But the same number of clicks spread across hundreds of sites by hundreds of “visitors” could look totally normal.  Anomaly detection rules won’t help find it.

Click Forensics employs proprietary technology utilizing high-dimensional cluster analysis to find publisher collusion.  Ad networks rely on this technology to find fraud that they would miss with their standard in-house anomaly detection rules.

The first separate “product feature” that results from this advanced cluster analysis is the Block List that we announced earlier today.  Utilizing the block list feature, ad networks can eliminate this collusion fraud from online campaigns.  They won’t pay for fraudulent clicks, and they’ll deliver higher quality traffic to advertisers.

We’ll write a lot more about this technology in future posts.  In the meantime, it’s time for ad networks to go to the mattresses.

Over the past week four major players in the online media space have announced accreditation to the Interactive Advertising Bureau’s Click Measurement Guidelines.  This list includes Yahoo!, Google, Microsoft and Business.com.  I wanted to take a moment and explore why you should care about this development and what accreditation means for advertisers.

The IAB is a publisher-focused organization that has led the process to develop click measurement guidelines.  The task force is made up of thirty or so companies representing the online advertising community.  Click Forensics has been a member since day one and participated in every step of the process.

There are three main benefits for advertisers and conversely, three concerns advertisers need to keep in mind associated with the entire process.  First, the benefits;

IAB Accreditation Represents a Commitment
The process to become accredited to the IAB guidelines is time consuming and certainly not free.  At Click Forensics, we have first hand knowledge of this and can assure you that any company that takes time and spends the money to become accredited is committed to their customers.  The level of detail the auditors go into is amazing.  Our community is fortunate to have auditors that have demonstrated a deep commitment to both the development of the process and the implementation of the guidelines.

IAB Accreditation Demonstrates Leadership
The IAB established a gating period to allow member companies and others to become accredited to the guidelines.  The companies mentioned above were the first to announce compliance.  This is important because it represents a sense of urgency among these four that enhances the urgency for others.  As an advertiser, you should reward these leaders with business.  They were first out of the gate and in my book that demonstrates leadership.

IAB Accreditation Means Better Quality Traffic
The IAB Guidelines are a lengthy narrative of “best practices” and rules in delivering quality traffic to advertisers.  While it is not intended to be a complete list, it serves as a firm foundation and includes practical steps to help ensure advertisers get what they pay for.  By working with an accredited ad provider, advertisers will be assured that the clicks they are buying have met the guidelines established by the industry.  This is a good thing and an excellent first step.

While we applaud the efforts of the IAB, Media Rating Council and member companies who participated in this process, there are things advertisers need to keep in mind.  There was a great deal of discussion and debate during the nearly three years of meetings it took to develop these guidelines.  In that process, there were a lot of valuable and important items that fell to the floor.  This is a good start, not a perfect process.  Keep in mind the following;

IAB Accreditation is a “Moment in Time” Process
The process for an ad provider to become accredited is a long one.  The auditor is invited in for a pre-assessment then the actual audit begins.  At the end of the process accreditation is awarded.  The problem is there is no mechanism for ongoing compliance.  When we buy gas at the gas station there is a meter that is routinely calibrated to ensure that when we fill our tank with 20 gallons of gasoline, we get 20 gallons.  This approach is not taken nor addressed in the guidelines.  While an annual audit is suggested in the guidelines, it is still important for advertisers to be monitoring their campaigns and holding the ad providers feet to the fire for every click.

IAB Accreditation Does Not Cover Everything
The 27 page Guideline document is quite comprehensive.  Our task force worked hard to ensure that both the guidelines are made clear and that the standard for measurement is defined.  However, when you consider that the dominant constituency in this process was multibillion-dollar ad providers, you might imagine not everything met their liking.  A few examples of chaff that hit the threshing room floor included:

Click ID – Each click should have a unique identifier so investigations can be “apples to apples”
Persistent Cookie – It’s important that ad providers can identify unique visitors to ensure they are billed for only once.
Standards for Investigation – Advertisers deserve to feel confident that they get what they pay for.  By setting an investigation format and agreeing to a timeline, ad providers can build trust with customers.

IAB Accreditation is a Roadmap
There is a Japanese proverb that says, “Beginning is easy and continuing is hard”.  There is truth in this as it relates to the guidelines.  We have begun the process.  We have released guidelines that will make the world of online advertising a better place.  Now we should look to leadership to take the next step and continue what we have begun.  The current guidelines will serve as a roadmap to the future standards.  We need to examine the items removed, listen to the community and think of better ways to ensure advertisers get what they pay for in the future.  The roadmap has been built.  Now we need to move on.

In January of 2006 as Click Forensics was just beginning as a company, I wrote the following challenge to our industry:

“Define standards for what an unwanted click looks like. We believe that there are certain characteristics or attributes that are common to a large percentage of click fraud. We are working with publishers and advertisers to agree on common ground and work together to expose it. Once this is developed it should be published so that the entire community can benefit from it.”

Today, over three years later, we have the cooperation of community leaders, the foundation of technical standards and the desire to continue to improve on what we have built.  I invite you, to join us as we build a future of ongoing growth and improving effectiveness by enhancing the process of online advertising.  I can assure you that both the Click Quality Council and Click Forensics will continue to support the work of the IAB and other industry organizations to work together to make our community a better place.  Let’s not stop with the foundation.

Wow!  Click fraud is real?  Click fraud costs online advertisers millions of dollars?  Click fraud can be uncovered and the perpetrators caught and punished?   Who knew?   Well, we did.

This week Microsoft filed the a complaint in U.S. District Court (Microsoft v. Lam, et. al., case number 09-cv-0815) seeking injunctive relief and damages from a group of people found to be perpetrating click fraud through the Microsoft adCenter platform.  This is only the second time (Google sued Auctions Expert International in 2004) that a search provider has ever caught and sued an individual (or a family, in this case) for click fraud.  We congratulate Microsoft for their efforts to root out this activity and encourage them in their pursuit of relief.  Online advertisers should appreciate knowing that click fraud does not always go undetected or unpunished.

For those not familiar with the case, it’s an example of what we call “competitor click fraud.”  The motivation of the perpetrators was simply to obtain higher-placed ad positions for lower bid amounts by depleting the daily budget of their competitors.  The verticals affected were auto insurance and the online role-playing game World of Warcraft.  Microsoft identified two brothers and their mother who controlled adCenter accounts that benefited from this fraud.  They believe that this scheme affected more than just adCenter advertisers, but also the advertisers on competitive search engines.

Microsoft’s complaint, now public information, is so well written it could be used as a tutorial on click fraud detection.  The most fascinating section describes the nearly year long game of cat-and-mouse played with the defendants.  Reading from the complaint: “When Microsoft took steps to mitigate these automated attacks, the perpetrators followed by implementing countermeasures to Microsoft’s actions.  A cycle of events ensued whereby the Defendants would update their attack methods to bypass the fixes implemented by Microsoft, and Microsoft would take additional steps to combat the new click fraud attacks.”

The lessons here are pretty clear:  Click fraud is still a problem and solving it requires constant vigilance.  The online advertising community needs to work together – search engines, ad networks, advertisers, and third-party auditors – to protect ourselves from this threat.

From spyware to bots to viruses and other unimaginable hazards… the web can be a scary place.  As far back as Prodigy in the early days of the online world, scams have been a part of the party.  The Internet is simply a new way for the bad guys to rip off unsuspecting consumers.  The key difference though is that the reach is enormous and the damage can spread to more people, more quickly than ever before.

Enter scareware, new way to trick unsuspecting consumers into parting with their money.  USA Today recently had an article  about the tricks and tactics used to perpetrate this latest rip off.  Unfortunately, online advertising has become an accomplice to the crime.

Scareware is worthless software that allegedly removes viruses from your computer.  Anyone who has surfed the web knows how easy it can be to become infected with a virus.  The damage to the users computer is often measured in slowed performance, unwanted clicking and potentially even more nefarious things like key logging and password swiping.  Now, the bad guys are selling “scareware” to solve a problem that may not actually exist.

The first such program was called “SpySheriff,” built by a team of cyber crooks from Russia.  The Anti-Phishing Working Group recently reported that scareware infections rose 48% in the second half of 2008.  The growth is tied to the ease of distribution and weaknesses in online advertising and the web in general.

There are several ways these fake products are being distributed.  Phony pages are created using hot search key words such as “American Idol” or “iPhone” and drive the unsuspecting consumer to the infected page.  Recently the Facebook email scam was used to send people to a page by promoting things like “best video.”  Since these emails came from your friends, millions clicked.  Twitter has become a vehicle for distribution. Phony Twitter accounts are created and enticing titles of posts encourage people to click.
 
Additionally, the bad guys are simply buying display or search ads.  They rotate in infected pages to the landing page.  It is virtually impossible for an ad provider to scan every ad impression and linking page.  This loophole creates an opportunity for the bad guys to drive significant traffic to infected pages at a very low cost.  Microsoft reported finding 4.4M installations of one such program, so the scale is enormous.  Do the math… at $49 or $79, that is big business.

Once someone lands on the page, getting off is nearly impossible.  Immediately upon landing, a “system scan” begins.  The results are, of course, showing that your computer is infected with a number of viruses.  Conveniently you can buy the product at that point and they take your money and run.  If you try to move away from the page, or cancel, an endless number of scans take over your screen.  Essentially, users must “control/alt/delete” their way out or restart.

The danger in this scam is not limited to monetary damage to the consumer.  These type of pages and methods to attract clicks are the same methods used to install spyware, malware and perpetrate click fraud.  To their credit, USA Today has done a good job over the last few years of highlighting the dangers of the web to the average consumer.

The FTC is cracking down.  They have identified products like WinFixer, DriveCleaner and XP AntiVirus as worthless and they are going after the owners.  The problem is that like the click fraud crooks, these guys are in remote locations and move their servers often. Tracking them is a full time job and extremely difficult.  The search engines are trying to help as well.  Bing has a neat feature that highlights “at risk” url’s.  Yahoo has similar product built with McAfee.

 
Trust is what keeps consumers clicking on ads.  Without stepped up industry efforts from organizations, like the Anti Phishing Working Groups and others, trust could be diminished.  Like click fraud, scareware is damaging trust.  It takes a community effort to stay after the problem and build solutions to take the scare out of the internet.

Q1 2009 internet media revenues were announced by the IAB today, with the industry experiencing an overall drop of 5.5% from Q1 last year.   Given all the bad economic news over the first three months of 2009, along with poor results announced by internet stalwarts Google, Yahoo, AOL and MSN, it’s not much of a surprise.  However, it is the first time since 2002 that the IAB reported number experienced a year-over-year decline.

There is some solace that online media results fell at a much lower rate than offline:  newspaper ad revenue plummeted 28% and radio revenues sunk over 24% – both some of the YOY worst declines in our lifetime.  TV revenues are also sure to fall, as soon as TNS reports Q1 results later this month.  That said, it’s never good to see our industry experience a YOY decline, no matter how small.  And, frankly, it was only a few months ago where I predicted that online media would show (much slower) YOY growth.

What’s changed was the depth of the economic crisis and the swift response by marketers to tighten up their spend.  Combine this with the free-fall in traditionally strong categories (finance, automotive to name a couple), and its surprising overall online media revenue hasn’t fallen more severely.   And, frankly, I don’t think the decline is over.  Based on conversations I’ve had over the last few months with folks whose opinions I respect, it seems like Display Media may have bottomed out, but we may see a continued decline in CPC spend.  This is being driven mainly by cuts in spending on second and third tier providers, as advertisers concentrate their spend with perceived higher quality networks and providers.

In an environment of shrinking budgets and more demanding buyers, increased transparency and tangible ROI are becoming requirements not “nice to haves.”  Those networks that actively manage and cull their traffic sources to ensure high quality traffic will continue to win out as advertisers increase demand that they “get what they pay for.”