Waterfalling, Header Bidding and New Auction Dynamics

In the early days of online advertising, ad space was purchased in the same way as print advertisements, that being that inventory was purchased directly from publishers at a set price, typically on a cost-per-mille (CPM) basis. During the late 1990s and early 2000s, a number of new advertising-technology platforms, such as ad networks, emerged to help advertisers display their ads on more sites and allow publishers to sell more of their inventory.

However, it wasn’t until 2007/2008 when the online advertising industry really hit its stride thanks to the introduction of real-time bidding (RTB). This changed the game in couple of ways.

Firstly, advertisers could purchase media on an impression-by-impression basis, which allowed them to show ads to individual users across many different websites, rather than show their ads to many different users on one website.

Secondly, it introduced a new way to buy and sell online ads—the second-price auction. The world of programmatic ad buying embraced the second-price auction model and has used it successfully for years. The implementation of second-price auctions has allowed advertisers to bid high prices to secure impressions, but ultimately pay a much lower price, typically the second-highest bid plus $0.01.

However, due to recent trends and an increased push for transparency of vendor fees, we are witnessing a steady transition to an array of new auction models, which are, in essence, more similar to the first-price auction.

Real-time bidding auctions, while originally based on second-price auctions, have recently morphed into “hybrid” second-price auctions to gain a higher price for impressions for the sellers. We discussed the possible reasons for the transition in one of our previous posts.

The turn of ad exchanges/SSPs towards first-price auctions stems from an increased pressure to reduce their take rates (that is, the fees they apply to the consumer surplus, aka reduction), which are often not known to publishers, and allow SSPs to increase their profit.

The Evolution of Bidding: From Waterfall to Header Bidding

Publishers are trying to regain some of the lost revenue caused by reductions (the difference between the bid price and the clearing price) in second-price programmatic auctions. This is the reason why SSPs and ad exchanges are increasingly implementing a combination of soft and hard price floors, essentially introducing auctions that are a hybrid of first- and second-price auctions. At the same time, we are witnessing a steady transition to newer auction types that solve yield and fill-rate problems. These include the traditional publisher waterfall and, more recently, header bidding.

Waterfalling

Before the advent of header bidding, publishers originally relied completely on waterfall auctions. Waterfalling (also known as a daisy chain or waterfall tags) is a process that aims to sell a publisher’s inventory in a sequential fashion by initiating one demand source at a time. You can read more about waterfalling here.

In waterfalling, each new impression is first processed in the publisher’s ad server with the aim of matching it to their direct campaigns. Then, if no eligible direct campaign is found, the impression is sent to the next partner in line, typically an ad network or SSP, which tries to fill the impression.

Over time, publishers came up with new methods to improve waterfall auctions by including additional ad networks and SSPs. In this way, if the first SSP failed to fill the impression, it was passed back to the ad server and sent to the second SSP, then to a third, and so forth.

The order of the waterfall is typically based on average historical yield, which is the average amount of money the demand source has made for the publisher in the past. Once an impression is filled, all the other demand sources in the waterfall are not queried, meaning all the other (possibly higher) bids are never considered. The figure below illustrates the process:

Problems With Waterfalling

Publishers used waterfalling to manage their partners and serve them in order from the highest to the lowest payer, which was found to decrease competition, as it means some bidders simply weren’t able to meet the publisher’s floor price (the minimum amount of money they are willing to sell the inventory for).

While waterfalling is an excellent method to work with multiple partners, it can be difficult to manage for a number of reasons:

• Yield risk: A publisher’s ad server was picking which partner to serve based on the average historical yield. However, this average is never a good predictor of the real-time value of any single impression. As a result, publishers were managing the inventory based on an assumption of what an impression is worth, rather than the actual value. This could result in the publisher never seeing the highest bid, since not every partner has the opportunity to participate in the auction. Consider this example:

• Fill risk: A publisher’s ad server would often decide to serve impressions to the partner with the highest average rate. Sometimes, however, the partner couldn’t fill it at all. As a result, calling the partner was not only a waste of time, but it also now requires the publisher to have a “Plan B” in place, if they want to sell the impression at all. This Plan B was usually to call another partner, or to pass it back to the ad server for the second round of decisioning, where the whole process might repeat itself. Bottom line: Calling the partner that cannot fill the ad is a major issue. Publishers needed to know if they can fill their impressions before they call the exchanges.
• Pass-back pain: Managing these complex contingency plans and constantly optimizing, analyzing, and reporting on the setup have become incredibly difficult to handle. AdOps teams have to log into many systems to get accurate reports, de-duplicate passback impressions of their own ads, and update their partners’ average-historical-yield rates to keep the whole thing running. In short, there is a lot of manual work connected with something that’s supposed to be a completely automated process—it’s called programmatic ad-buying after all.
• Publishers experience operational problems due to fill risk: Conversely, the reason they have fill risk is because they have yield risk. Today, publishers can remedy all these risks by using header bidding instead of waterfall. We discussed the difference between the two in one of our previous posts.

Header Bidding

Header bidding is a newer, smarter way to monetize a publisher’s inventory. It requires less maintenance and manual work, and increases revenue for the publisher. We explained what header bidding is and how it works in one of our previous posts.

Header bidding is the practice of accepting bids before the primary ad call is sent to the publisher’s ad server, and then including these bids in the ad-decisioning process. Header bidding is able to achieve this by sending ad requests to various demand sources via a piece of JavaScript located in the head of a web page, hence the name “header bidding.”

Header bidding was introduced to allow publishers to integrate with multiple demand sources and solve some of the most urgent problems of programmatic ad buying mentioned in the above paragraph. Here are some of the benefits of header bidding:

Higher Yields
Header bidding allows publishers to see how much advertisers are willing to pay for an impression in advance, before the final ad-serving decision takes place. This way, they know exactly what each impression is worth to many demand sources at once instead of having to rely on the average historical yield. The result is that the highest-paying demand source actually wins the impression.

Higher Fill Rates
Since higher yields inherently entail higher fill rates, the latter is also solved in the process. With header bidding, no impressions are “wasted.” The publisher always knows whether the SSP or exchange will deliver the impression or not. Lastly, by eliminating the fill risk, you can move away from the waterfall where multiple exchanges are queried in a sequential manner to a single level, where each SSP/exchange fills a 100% of what it bids on. Why does it matter? There are no complicated daisy chains to manage, there is no pass-back pain, and less manual work has to be done on the publisher’s side. Many publishers continue to use waterfalling when testing out header bidding, and the end goal is to get rid of the waterfall and just use header bidding.

Header bidding offers an immediate benefit for the publishers because it tells them how much the market will pay without sacrificing the impression. It offers fair competition between guaranteed direct-sold impressions and RTB demand, but the method is not perfect.

Problems of Header Bidding

Header bidding, although a vast improvement over the traditional waterfalling method, has issues and inefficiencies of its own.

• Privacy concerns: Because header bidding calls are sent simultaneously, every bidder gets access to data from all the users who were served impressions from the auction. Publishers can have up to a dozen header-bidding partners. There are even exchanges that allow demand-side platforms to take bid requests and basically just “listen” for data without spending money. With the increasing number of bidding partners, there is a bigger risk that the data gets to their customers and, from there, leaks out further.
• Many requests, one DSP: Header-bidding requests are sent to multiple SSPs, but they may all forward the request to the same DSPs.

• Inefficiencies and unpredictability of sequential auctions. There are sequential second-price auctions in the current header-bidding model—first at the SSP level, then at the header bidder. The model involves integration of header bidding and SSPs in the ad-buying process. This is intended to maximize fill, but may make the outcome of the auction a little unpredictable.Bottom line: the highest bidder is no longer 100% guaranteed to win the auction. Also, the publisher does not always get the maximum price the buyers would be willing to pay for the impression.

New Auction Dynamics

Header bidding was certainly an improvement over the waterfall model in terms of auction dynamics, but it still entails some problems and inefficiencies. The above problems and considerations have lead to the development of an array of new auction mechanics. Consider this example:

• A user accesses a website.
• The header-bidding wrapper makes simultaneous requests to SSPs/ad exchanges.
• Each SSP/ad exchange broadcasts the bid requests to a number of DSPs it is connected to.
• Each DSP responds with the bid responses and the max CPM price the advertiser of the matched campaign is willing to pay ($10, $5, $8, $6).
• SSPs/ad exchanges execute the second-price auctions (e.g. If the SSP received $10 and $5 bids, the winner is paying $5.01 CPM—the clearing price).
• The SSPs returns the $5.01 clearing price to the header-bidding wrapper.
• The header-bidding wrapper chooses the highest clearing price (first-price auction) and displays the impression.

How can this system be inefficient? Because there was, in fact, another buyer willing to pay more (the $10 bid from the example above). However, because there weren’t any high bidders at the SSP level, the clearing price of the second-price auction was just $5.01, less than on the other SSP level. This is a wasted opportunity for profit that publishers want to avoid.

Arguments for First-Price Auctions

Multilevel auctions produce outcomes that may heavily favor the bidders; higher bids don’t guarantee winning the impression. When a second-price auction runs in an SSP and a first-price auction runs in header bidding, lower bids can sometimes end up winning. Sellers, in a bid to maximize their revenue, have started introducing preventative measures: soft floors and hard floors. These tactics effectively make second-price auctions look more like first-price auctions, but at the same time compromise the transparency of the whole system.

The second-price auction model is no longer truthful when executed sequentially—i.e. when it is not the final auction. In sequential auctions, because there may sometimes be little competition or demand for some impressions (and the bid values of the buyers are dispersed), dynamic pricing occurs through various floor optimizations and shenanigans. Such dynamic pricing is, in essence, a kind of price discrimination, and is not advisable as a long-term business strategy. The net result of such adjustments is that advertisers would rather do business somewhere else than deal with price games.

By utilizing various floor optimizations, publishers are able to move the price floors and increase their yield until almost maximum-possible CPM is achieved. This method makes the bidders ultimately pay the price closer to the one they are willing to pay, and converts the second-price auction into a quasi-first-price auction. Such strategies have led to increasing pressure from demand-side platforms on supply-side platforms to improve the quality of their inventory and stop the shady mechanics like cascading auctions, aggregation of header and DSP bids, and the flooring tactics that have led to new auction dynamics, which are not completely clear for bidders and damage trust in the system.

Interestingly, the concerns over the shenanigans of second-price marketplaces were already felt in 2012, which was aptly described by Adam Drake in his post on Real-Time Bidding, First and Second-Price Auctions, and Transparency. The introduction of first-price auctions may be a way to restore the transparency of RTB mechanics and increase the yield of publishers.

Arguments Against First-Price Auctions

First-price auctions rarely favor the bidder. As there is no reduction in the clearing price for the auction winner, there is no consumer surplus, and the winner gets the impression for the price of his bid, meaning the bidder bears the risk associated with overbidding.

In first-price auctions, bidders submit secret, or sealed, bids and don’t know what their competition bids. Bidding in first-price auctions requires knowledge of the value of each impression. This requires bidders to employ a little more conservative bidding strategy (possibly placing little lower bids), and carefully consider how highly others bidders may be ready to bid.

Recommendations for Implementation

One thing is certain in the programmatic world: publishers appreciate the undeniable benefits of increased control and always seek new auction types to maximize yield and fill rates. While there is no one-size-fits-all formula, certain recommendations from industry leaders have emerged in an attempt to solve the pain points of RTB.

New auction implementations are standardized and usually involve the use of the OpenRTB protocol, which provides specific fields that contain the auction type. This allows SSPs to indicate which auction type it is going to run in the bid request sent to the DSP.

Dynamic Switching Between First- and Second-Price Auctions

OpenX, an ad exchange for publishers and DSPs, is one of the most vocal promoters of first-price auctions, which it has recently enabled on its platform. The decision is intended to give demand partners advantage when their bids are passed further into header bidding, and consequently increase the competitiveness of DSP bids in subsequent auctions.

OpenX recommends that a first-price model is used for all auctions that are not final (for example, when the final auction is only in the header wrapper). It is then in the demand partner’s best interest to submit a first-price bid to such an auction. This increases the likelihood of winning the final auction in the wrapper. Conversely, when OpenX runs the final auction, a second-price auction is run as usual.

Second-Price Auction is Run Only by the Publisher

Pubmatic, provider of advertising-technology solutions, is proposing an integration method. The SSP passes the first price directly to the publisher, who then determines the winner and the clearing price based upon a second-price auction. This recommendation is a reverse of the current system, where SSPs return a second-price-auction clearing price.

This model allows publishers and technology companies alike to focus on creating value and improving overall industry economics. Publishers, when given more control over auction mechanics, can also better control their revenue streams, reduce inefficiencies, and return more value to buyers. This allows for a better use of segmentation, data fidelity, addressability, and viewability. They are able to enrich impressions with data that buyers care about.

Conclusion

Programmatic ad buying has come a long way from waterfalling, through header bidding, ultimately to various first- and second-price auction optimizations. While auction mechanics have never been completely fair or transparent, we may finally be reaching a point of levelling the playing field for everyone. The ultimate goal is allowing publishers to maximize their yield and reduce inefficiencies in the system. The newly proposed auction models are believed to allow publishers and AdTech companies to stop worrying about their yield and efficiency of the system, and focus more on providing better value for the users.