Bing Ads Advertiser Science Series: The Science of Brand Bidding Webcast

Bing Ads Advertiser Science Series: The Science of Brand Bidding Webcast

Welcome, everyone, to the Bing Ads webinar on
The Science of Brand Bidding. My name is MJ DePalma, and
I will be your host today. This is the first webinar with
our Advertiser Science Team. And we are excited to share
their findings and methods so you can achieve more with
your advertising dollars and help you earn that bonus. Before we begin, Let’s review same housekeeping
items to enhance your experience with us today. Our webinar console that
you see is customizable on your side, where you can
click the content widgets that you see on your screen and
move them around or size them for your screen size. You can expand your slide area
by clicking on the maximize icon on the top right of
your slide area or by dragging the bottom right
corner of that slide area. If you have any
technical difficulties, please click on help widget. It has a question mark icon and
covers common technical issues. We recommend, if you can, to leave all the widgets as they
are so that you have the ability to see all the great resources
and Q and A at the same time. For Q and A, please feel free to input your
questions as you have them. But for the most part, we will
be getting to that list of questions at the end
of the webinar, and we will try to answer
as many as possible. Any questions we don’t
have time to answer today, we’ll post a blog. Actually, Frances here, who I’ll
be introducing in a moment, will post a blog with all the
frequently asked questions and any unanswered questions. And you can find that
at You’ll also see a resource
widget in the upper right corner, and we’ll refer to
that during the webinar. So go ahead right now, and there’s an Excel icon to
download a t-test spreadsheet, so you’ll be ready to follow
along later in the presentation. And also for your convenience, the deck is already available
for download there as well. Also, there are links to stay
in touch with us in the deck, so you’ll get that later. And lastly, there’s a Twitter
widget that’ll enable you to tweet right from the webinar
console with the hashtags bingadwebinars and bingads so
we can see those streaming live. And those will appear right
there on the right-hand side. An on-demand version of
the webcast will be available approximately four days after
the webcast, and can be accessed at the link at the bottom of
all the slides in our deck. It’s our Bing Ads YouTube
channel’s webinar playlist. And the last housekeeping item
before we get started, we would love your participation in our
survey at the end of the webinar so that we can improve
in any way possible. Let’s begin by introducing who
we have on the webcast today. As you heard,
my name is MJ DePalma, and I’ll be your host today. And I’ll be monitoring the Q and
A and making sure, as much as possible, that things
could move along smoothly. And then we also
have Lars Hirsch, our Director of Advertiser
Science here at Bing Ads. Super excited to have him
today and be working with him. And we have
Frances Donegan-Ryan, our SEM Pro Marketing Manager,
SEM guru. So really happy to have
you both here today talking about the science
of brand bidding.>>Great,
excited to be here, woo-hoo!>>Let’s get into it. The agenda, here’s what
you’re gonna learn today, super cool stuff. The Bing Ads Advertiser Science
Team’s overall brand experiment results, and we’ve done it
in a couple of verticals, so we’ll take a look at that. It’ll be retail,
finance, and travel. And also, even more exciting, I
think, is how to design your own brand term experiment to
really finally put to rest that office debate to bid
on your brand terms or not to bid on your brand terms. So we’ll teach you how to
actually look at your own data and decide that. So the most important part of
it is to make sure that your experiment is
statistically relevant. And so
in the upper right-hand corner, like I had mentioned earlier,
you’ll download that statistical relevance worksheet called
a t-test template for download. And it has the formulas already
built in, so that’s super cool. We’re trying to make it
as easy as possible for you guys to get the most out
of your advertising dollars. So without further ado,
I wanna pass it on to Lars to tell us exactly
what Advertiser Science is.>>Yeah, thank you, MJ. So my name is Lars Hirsch. I lead the group called
Advertiser Science, and we are the data science
team for the Bing Ad business. And in
the Advertiser Science Group, we have three main priorities. Number one is to analyze and
interpret advertiser behavior and impact relative to
the marketplace dynamics. And second,
we identify, test, and measure strategies that
help our advertisers to take full advantage of
the Bing Ads platform. And finally, we enable
sales channels and partners to leverage data insights to
prioritize, drive efforts against the most meaningful
activities and campaign changes. So we’ve been working on this
question about brand queries and brand ads for a long time now. And we started off
thinking about, that there wasn’t really a
consensus among our advertisers. Does it make sense to
bid on brand terms, or does it not make sense? Of course, the question is, does brand terms give
additional clicks? Or does it primarily cannibalize
clicks that you would have gotten for free if you didn’t
pay through organic search? So we really wanted to
understand the dynamics here and really discover the truth
though scientific methods. So let’s dive into
the brand analysis. So we compared two scenarios. We compared brand
queries that had brand ad from the brand
owner present. And we compared that to brand
queries where there wasn’t a brand ad from
the brand owner present. And when we talk
about brand ads, maybe we should specify
what we talk about. When we talk about
brand queries, it’s a query that
contains either the name, like the brand name of the
company, it could be a product. It could be that brand,
it could be that plus a modifier, such as Kayak
flights or Expedia flights. But we only include queries
that have above a certain threshold number. So we look at all brand queries
in a platform that has at least 1,000 searches per month.>>Great, so thanks, Lars,
that’s setting our stage. And so what we wanted to show
you is the click reality. So what happened, and what
were the results that we saw? So we looked at one
of our advertisers, who’s a social network. We looked at what happened
when someone searches for their brand or
modifiers of their brand, and they were not running
a brand ad campaign. They would capture
about 55% of the clicks resulted from that search. When they were running a brand
ad campaign, they were able to not only increase
the organic results clicks. But also ensure that they were
capturing 100% of the clicks once someone searched for
their brand. So this obviously was a really
impressive growth for them to be able to move from only capturing
about 65% of the clicks to 100%. When we did the same test and
we looked at another one of our advertisers who’s
a wireless brand. What we saw is, when they did
not run a brand ad campaign, again, they were only capturing
just over 60% of the clicks. But not only that, they were losing clicks
to their competitors. So they were losing clicks to
their competitor’s organic listing, and their competitor was running
a brand campaign of their own. And they were able to really
take a chunk of clicks from our advertiser’s
own brand name. And so what we saw when they
started including their own brand ad They were
able to retain almost, looks like 99% of
the clicks there, and only lost a sliver to
their competitor’s ad. We ran the same test for one of
our telecom advertisers, and again, saw the same
types of results. And when we started to look at
even more competitive industries like telecom and particularly retail, you can
really see the magnificent results that they saw with
the brand ad campaign. So if we look at the retail, they were only capturing
under 40% of the clicks. And their competitor, in fact,
was getting more percent of the clicks when someone
searched their brand, between their own organic
listing and their brand ad. And once our retail,
this is a women’s retail brand. Once they added their own brand
ad campaign, they were able to really capture and hold on
to what looks like there 90% of the clicks when someone
searched under their brand. And really,
only losing that 10%, rather than
the almost 50% before.>>Yeah, that’s a pretty
dramatic increase there. And I think it’s worth
maybe reiterating a bit, this is real data. This is for specific brand queries that
we’ve seen on our network. And sort of for
that retail brand, you can see, well, They almost
eliminate the competitor, the clicks the competitors, and
more than double the clicks to their own website by
advertising on the brand term. And when you think about
the different types of brand queries, it’s worthwhile to
think about the degree of competitiveness in the business. Of course, retail is very,
very competitive. Social network is probably very
much less competitive, right? So those are kinda
like extremes. It’s also worth thinking
about if the query reflects a subscription or ongoing
relationship or if it’s more transactional, if it’s somebody
who’s clearly looking for one particular location. If somebody searched for
Netflix login, for example, that person’s probably
gonna go to Netflix, even if competitors place
an ad in front of that user. It’s also worth thinking
about if the brand query is sort of like a proxy for
a category. And that’s what we see
with this retail brand, that it’s somebody that
searches for a specific brand, because that’s very top of mind. But it’s not necessarily
that somebody is dedicated to actually buying
from that particular brand. It’s more that that’s
the top of mind brand name or top of mind query.>>Yeah, and I just wanted
to say, this visual, when I look at it from, I’m
looking at it kinda far away. Looking at all of the bars
on the left compared to all the bars on the right. So it’s the dark gray and
the orange stacked on top of one another are much higher
than the ones on the right. So the big key takeaway, you’re gonna capture a lot more
if you do the brand on the left than if you don’t,
just boiling it down there.>>Yes.
>>[LAUGH]>>And so what we wanna show you now is, so like Lars mentioned
before, this is real data from individual brands,
advertisers on Bing Ads. And so
when we were doing this for these specific advertisers,
Lars and his team decided to look at
the verticals overall as well to see what we could find when
we looked at a bigger picture. So Lars is now gonna take you
through what we saw when we analyzed doing brand campaigns
across the travel industry.>>Thanks, Frances. So we included all travel
advertisers in this study, and we included all travel
queries that had at least 1,000 queries per month. So this is the result, but everything is included
within those thresholds. And this slide shows you
the click yield to travel advertisers when they don’t
bid on their brand terms. This is average across
all travel advertisers. And when we talk about
click yield, that is, out of 100 searches, how many
clicks did brand owner get, in this example? So of 100 searches on the brand
query, the brand only would get an average 61 clicks,
61 clicks per 100 searches. And you can see on
the right-hand side, you see a typical screenshot. So this 61% number is
not related to Kayak, this is an average. But you see on the right side,
in the example with Kayak, we see somebody searching for
Kayak flights. There’s a bunch of competitors
on top who are advertising, and then you see Kayak
further down on the page. And this might be something
of a category search, where people are looking for
flights in general. Kayak is top of mind, and
they might be more or less likely to click on
some of the competitors. And of course, as the brand
only get 61% of the clicks, the competitors would get 39%
of the clicks in this example.>>So again, just to point out,
it’s not only are you not capturing the full percentage
of the people who are doing this search with your
brand term in it. You’re losing
a significant chunk of those searchers
to your competitor.>>Yeah, so I wanna correct
something that I said before. So this isn’t a click yield, this is actually
the percentage of clicks. So out of 100 clicks,
the brand owner gets 61% or 61 clicks, and
the competitor gets 39 clicks. So that’s the split of
the total clicks on that query. And so those numbers were
averages across travel. And here, we’re showing some
real examples with the actual brand names redacted. But these are real queries, where you can see that
without a brand ad present, competitors captured
nearly half of the clicks. So it’s 27% for and in the 40s for some of the other what are
called brand modified queries, where you have a brand
plus another search term. But think about it, like for, there’s somebody
that’s clearly looking for a very specific destination,
right? But still, competitors are able
to capture more than a quarter of those clicks on that search. It’s pretty astonishing.>>Yeah, and particularly, and
we’ll mention this time and again during this webinar. But the amount of work and
dollars and marketing that’s gone
into having [LAUGH] a customer retain
the and the effort that they’ve
done to do that search. To lose over a quarter,
in fact, almost a third, is a huge loss after the
investment that you’ve made for them to know your brand name and be interested in going to
your actual website as well. So you really don’t
wanna lose that. And it’s not only
the opportunity to increase your leads, increase your customers, increase your conversion,
like we’ve mentioned. [LAUGH] It’s really a way to
retain your customers and your leads that you’ve
already invested in if they’re already using
your brand term. So don’t let your competitors
rob them from you and take them away from you.>>So let’s see what happens if
the brand owner is bidding on its own brand terms. So as you remember, again, this is average across
travel vertical. If the brand owner
doesn’t advertise, they get 61% of the clicks. If they do advertise,
then what we see is, on average, the brand owner will
get 88% of the clicks. So that’s actually a 45% growth
in the total number of clicks that the brand owner would see. It’s pretty significant,
I would say.>>Yeah,
that’s very significant. So if we break that 88% down,
Yeah, so the question remains, how much of the clicks do
you not have to pay for? If you don’t advertise,
of course, all your clicks are free,
it’s ultra organic. If you do advertise,
you’re gonna pay for some of those organic clicks. But, or if the clicks
that were organic now, they’re not organic now,
they’re paid clicks. What you see is that it
doesn’t look all that bad. Yes, you are going to pay. You will cannibalize some
of the organic clicks. But most of the clicks that
you got for free when you didn’t advertise, you still get
for free when you do advertise. So you retain most of
those clicks for free. And then you have a big chunk
that is just incremental.>>Yeah, so this seems to be,
well, definitely when I’ve had conversations with
advertisers and when I meet customers at events,
this tends to be the main argument against
bidding on your brand. And my background’s
actually in SEO. I’ve been doing paid search
now two and a half years. But before that, it was SEO. And so even when I was an SEO,
there was always a debate. The amount of money, energy, and time you’re putting into getting
that mainline number one, number two, number three spot
for your organic listing, why would you pay then to
perhaps cannibalize that? And really what we saw time and again was that it was
such a small percentage. And the fact that you were
able to retain such a larger percentage of the overall
clicks that were coming to that search results page, that
really, it was worth your time. And in fact, helped you
maintain your organic listing. And made it still relevant and
still popular.>>It’s like that old analogy. Sometimes you have to take
one step back to go two steps forward.>>Yeah, exactly, and I think
what you’ll see here is when you start running these tests for
yourself, this overlaps. In this example, the average
within travel was about 18% is not going to be high enough generally for you to shy away
from bidding on your own brand. Because the result of
having that total 88% of the clicks is much
stronger than the 61. And then adding on top of that
the fact that those then are not going to your competitors
makes them even more valuable.>>Yeah, let’s take a closer
look at that competitor angle that you mentioned.>>So yes, so
here’s some data on that.>>So what we saw is that if the
brand owner does not advertise, competitors are getting
39% of the clicks. But if the brand
owner do advertise, then the competitors will
get 12% of the clicks. So that is less than one-third.>>So within that 88%, where we saw on the previous slide
there was where you will see a little bit of cannibalization
into your organic clicks. But when you really start
comparing the larger numbers against what your competitor
is able to take from you, it shows that that sacrifice or that paying for those additional
clicks is so much more valuable.>>Lars, we had a question come
in that I thought maybe we could just touch on real quick. Would you clarify how you
calculate the overlap percentage in clicks?>>Hop back to that.>>If we were to go back to
that slide just real quick. And that 18% is overlap.>>Yeah, so what we saw is
that if you advertise on the brand terms,
43% of the clicks. Well, if you look at
the out of the 88% chunk, the 43% chunk is now organic,
not paid for, while the 35% is paid clicks.>>So that difference
between 35 and 43?>>It’s the difference between.>>61, I’m sorry.>>61 and 43, yeah.>>Got it, yeah,
perfect, thank you.>>So I’ll jump back. So again, let’s talk about what
this then looks like versus your competitors when you’re
running your brand ad campaign.>>Yeah, so these are some
more real examples. If you remember, we were
looking at some real examples, so what happens if the brand
owner is not advertising, where we saw for [INAUDIBLE] 27% of clicks
going to competitors. It turned out to be totally
navigational, right?>>Yeah.
>>It’s pretty astonishing. And this is a similar
travel advertiser. These are both two very large, major travel advertisers
that are comparable. So it’s not comparing apples and
oranges there. All of a sudden,
on, instead of 27% of the clicks
going to competitors, only 1% of the clicks
are going to competitors. And this is some of
the other examples. Remember back the travel
advertisers that didn’t advertise have up to almost
60% of the clicks going to competitors. Here there’s no more than
11% in this example going to competitors. There’s a pretty big difference. I’m pretty astonished, especially on the
term that we saw 27% going to competitors in
the other example. And here we have only 1%.>>Yeah,
retaining those customers, we’ve all heard in
any business course, any business article [LAUGH]
we’ve ever read that keeping a customer is far less expensive
than acquiring new customers. And this is a strategy to make
sure that you’re doing that. Keeping those customers that are
already interested in looking for you specifically if they’re
doing a Only losing 1% or
less to your competitor is a phenomenal increase over
the 27% you’re losing.>>Yeah, that’s a really
good point, Frances. And again, think about this in
the context of remarketing, too. That you definitely wanna
remarket only brand terms. So competitors don’t steal your
customers that have already visited your website, maybe
already purchased from you. All of a sudden,
if you’re not there to capture them with an ad when
they search for you, they might be going to
your competitors and become your competitors’
customers instead.>>Yeah, so
that was a quick look at travel. Like we mentioned before, we did this study across
the retail vertical industry. And we also did it across
financial services. And the same sort of study, we looked at 3 million
impressions for retail searches. And worked with retail
advertisers in December. I was gonna say of last year,
but I realize now [LAUGH]
it’s December. Not last year, the year before. So only 13 months ago.>>[LAUGH]
>>Not too long ago. But we have the results of
that study in the appendix of this presentation. So when you download it,
you’ll see it there. But we saw very similar things
than what we saw in travel in terms of how they were able
to increase their clicks.>>Yeah, and I think
the results were even a little bit stronger than what
we saw in travel. I think for travel, I think
there’s a pretty compelling case for
bidding on your brand terms. For retail,
that case was even stronger. So yeah, like Frances said,
slides are in the appendix. Is that right?>>Yeah, the slides are there. So you can take a look
through them there. They’re pretty much identical
to the travel slides, just the numbers are representative
of brand advertisers and some examples of
retail advertisers. We just wanted to make sure we
had enough time to show you how to run this experiment for
yourself. So what I’m gonna do is just
a quick recap on what we’ve learned from the tests we ran. And then Lars will take you
through Have you actually run this experiment for
yourself? So I’m a massive sports fan. Pretty much any sport out there. I love watching anything
from rugby and soccer, and American football
all the way to golf. I love watching golf. And so the major thing
that you learn in sports, particularly team sports, is your great offense are gonna
win games here in Seattle. Obviously a big Seahawks fan,
and so we know that Russel [LAUGH] and his offensive team
are great at winning games. But we would not have won
the Superbowl without our phenomenal defense team. And I think you see that
across all sorts of sports. If you are looking for
winning a championship, winning in the long term you
must have as strong a defense as you do an amazing offense.>>I have to say, I have to just chime in Francis,
this is my mantra. I was a soccer goal keeper.>>[LAUGH]
>>So I held on to the fact that the
defense won the championships, at least in my mind. Of course I loved the forward
that always score and get all the-
>>Glory [LAUGH].>>Glory, but I held on to that belief that
we really need a strong defense. [LAUGH]
>>Yeah, and that’s true. And really, you see the same thing in your
brand term bidding strategy. Bidding on your brand term,
helps you deliver more clicks. It helps you deliver more leads,
it improves your convergence. Certainly, but what if which
frankly might even be more powerful to you based on how
competitive your industry is, is that it absolutely means fewer
clicks to your competitors. And it reduces that opportunity
for your competitors to capture either existing customers or
new customers. And that mindshare that they’re
able to peel from you when you don’t have a brand ad present. And we’ve talked about this,
the risk there of not having it is not only fewer
clicks on your organic. Even if you have a strong
SEO presence you’re going to fewer clicks on our organic listings if you do not
have a brand ad campaign. And like Lars and I mentioned
earlier, you put how much money, time and energy into TV, radio,
print, in-person events, perhaps sales at your store
if you’re brick and mortar. All of this marketing
activity and lead generation only to
loose it at the last moment.>>Yeah, it’s definitely worth
reiterating that a lot of people see search at the end
of the marketing panel. So you put all of this money
into upstream marketing channels only to have your competitors
capitalize on the search teal ground. So basically you’re paying,
to drive traffic to your competitors if you’re not
able to capture that and make the funnel clicks.>>Yeah, so this also a great
strategy that allows you to sit down, and
connect to other teams. So, if you have separate teams
that are working on SEOs, separate teams that
are working on acquisitions, separate teams that
are working on paid search. As someone who’s worked on
all three of those things, [LAUGH] it’s a really awesome
opportunity to come together, work as a team, and ensure that
search, and paid search in particular is supporting
the work of your SEO work. Is supporting the work of
your acquisition team and not burning that money and not
losing out on that customer that you’ve already put so
much work into nurturing.>>We just had that question. So stay tuned. We are gonna teach you how
to do an experiment and possibly help your
company just like that.>>Yeah, so I just wanted
to do a quick overview. So what we saw when we did
these larger studies across these larger tests and
experiments across industries as a whole we looked at financial
services retail and travel. Financial services are average
brand click lift with 33%. Retail like Lars mentioned
was just astonishing at 51% increase in click lift and
travel at 44. So if you fall into any of these
categories these are the types of number you could the average
across the industry. And talking about
our awesome defense, here’s what we saw
competitor click loss. Massive in financial services. 57% retail at 43% and
travel again at that 44%. Really showing that not only are
you increasing the opportunity for your ad and your organic
listings to be clicked on. But you’re also really shutting
down the opportunity for your competitor to take
those away from you.>>We had a question
to that point, Francis coming from iCrossing. What is an average impression
share a competitor receives when brand ads are not
being displayed? So I think in a way this
slide is directional and is vertical, and so if you don’t
actually run your brand ads, this is what you can pretty much
see your competitors getting.>>Yeah, so we haven’t looked
specifically at impressions we’ve been focusing on clicks.>>Key point impressions
versus clicks.>>We should have that
data somewhere, so we might do that on our
followup study possibly. But it’s a very good
question for sure.>>Okay.>>We’ll take note
of that question and see if either in
the blog posts or in another webinar in this series
if we can take a look at that.>>Fantastic, so
I wanna jump into how do you do this for yourself? So this web series is all about
the science of advertising, so how do you set up
your experiment? Or your test to really use
your own data to look at this phenomenon for yourself?>>One of the things that I
have been working with Lars to produce this webinar that I came
to understand very clearly from my college days of setting
up actually psychological experiments because my
major was in psychology. Is that you have to
learn how to set up statistically significant
experiments. And to do that, make sure that you download the t-test in
the upper right hand corner. It’s an Excel template
that Lars built for for you guys, with the formulas
already built in. But you definitely want
a statistically significant difference in your experiment so
that it yields reliable results. And so in order to set up
your t-test just a general idea is you’re gonna want
access to two data sets. So you’re gonna want organic
clicks and the paid clicks. And you want
the amount of data for the organic clicks to be equal
to the length that you’re gonna be running your experiment
during the paid clicks. Is that right Lars?>>Yeah, so you need the organic
click data as well as the paid click data for
the length of the experiment. And possibly also for a time
period before the experiment equal to the length
of the experiment.>>Perfect.>>We’ll get back to
more details in the next couple of days.>>That means probably
partnering with your SEO team and->>Yes, you need access to that
SEO click data for sure.>>So
tell us about the first type?>>[LAUGH]
>>So we have two different ways that you can set up experiments
to find the effect of terms. One is to run a brand campaign
on alternating days of the week. So that means that your brand ads will be active
every other day. And you need to do this for
at least two weeks. Possibly you should probably do
this for maybe four weeks, but at least two weeks because
of the day of week effect. So you have one Monday
with brand ads active. While the next Monday,
you would have them inactive. So you don’t skew the results
because of different Patterns in different days a week. So the first thing that you need
to do is to organise your brand keywords into one or more separate campaigns that
only have one brand keywords and all brand keywords are contained
into those campaigns. So you can apply targeting
specifically to those Brand campaign. So once you have that organized,
you should then go to Bing Ads’ Day
of Week targeting to enable and disable brand campaigns
every other day on that. That’s a targeting setting that
let’s you target individual weekdays. So you can, for example, for the
first week you enable the brand campaigns only on Mondays,
Wednesday, Friday and Sunday. Then you will, at the end of the
week, you will have to go and change that setting because
it doesn’t let you alternate the alternating weeks. The next week, you would enable
the brand campaigns on Tuesday, Thursday, and Saturday. Did I get that right, Frances?>>Yes.
>>I think so.>>Yes.
[LAUGH]>>And then you run this for at least two, maybe four weeks, it’s gotta
be an even number of weeks. And then you compare each
day’s total grand clicks for the brand, enable day with the
brand, disable it using t-test. And Jay mentioned, there’s
a Word template for that. You can just fill the numbers
into the Excel template, and it will tell
you the results.>>So
this is looking at your brand clicks as the days of the week. And so
once you collect that data, you drop it into the Excel
template, you have your results. This is how you figure them out.>>Yeah, so you want the t-value
to be less than 0.1 to be statistically significant. And what that means is that
there’s only a 10% chance that these two data sets come from
the same distribution, and a 90% chance that there
is actually a difference, which is what’s a usual,
a normal threshold for determining statistical
significance.>>Mm-hm.>>And I think [INAUDIBLE].>>Yeah, I eluded to
the fact that if it isn’t statistically relevant, well,
then chances are that you don’t have a large enough sample set,
so you’re gonna wanna run the campaign longer
to get more data. But there’s also
the other off chance that it’s telling you not
to run Brand terms. There’s that also option.>>Because there’s no
difference between the total [INAUDIBLE] term,
you’ll basically get the same clicks through organic if you
don’t advertise any brand terms. [INAUDIBLE] the result
we have seen for the absolute most part support
the second [INAUDIBLE]. Well, maybe it didn’t run it for
long enough. But of course,
if you have an experiment, you’re set out to prove or
disprove something. So there’s also a possibility
that actually there isn’t that effect right?>>Exactly.>>Yep.
>>Exactly, you wanna be fair.>>Wanna be fair [LAUGH].
>>Yeah, and also, just to mention that
the day of the week test is really good for SMBs rather
than this one, the GL, correct?>>Well, specifically,
for local businesses, so that other way that we-
>>Local, yes.>>That you can set up a brand experiment is to target
different geographies and say, well, we’re only gonna run
brand campaigns in geographies. And of course, if you
are limited to a single or a few geographies,
you can’t do this. You have to use the day
of week approach.>>Yeah, okay.>>This is a good approach. The geo approach is a good
approach for national advertisers that advertise
across all or many geographies. And here, what you need to start off with
is to identify pairs of metro areas with similar customer
profiles to your business. You can think about how
do your customers differ. I mean, you wanna look at metro
areas that are similar across those important attributes. It could be age, gender,
interest, what have you, whatever is important for you. Cuz you want Pairs
of metro areas with similar competition
with regards to that. It is very important
to your customers.>>Mm-hm.>>That make sense, MJ?>>Yeah, makes sense. If you have any questions,
please keep sending them. We’ll get to them at the end.>>So start with as many
as you can identify. I think 20 pairs is
probably a good number, so you have 40 natural areas total. And then from each pair, you
have to randomly select one of the metro area from each pair
into what we called treatment group, and one metro area from
that area then go to control. And this has to be
picked on randomly. What’s important
is that you don’t have a pattern of how
you select it, right? That’s why we say it’s
gonna be rounded to ten. We shouldn’t select. One of my pairs in each pair
is always a little larger, or skews one particular way,
it’s important that that person always is put in
the same group into treatment. It must always be
done in the same way. There’s the link to random
generator here that you can use. There’s probably a million
of them out there. Can just search for
them on the Internet. That shouldn’t be to hard.>>Yeah.>>Then what you do is to
measure combine organic and paid clicks by metro area
before the experiment. And this has to be done in
a time period that is same as the duration of
the experiment. So if you’re running
two experiments and you measure
the combined organic and pick clicks also the two
weeks before the experiment. So what you want to do is to You
used that to set a baseline that you measure the total clicks
during the experiment against. If it’s a two week experiment,
you look at the two weeks before the experiment,
the total clicks. If you have a four week
experiment then you look at four weeks before
the experiment et cetera. I think two to four
weeks is probably a good length to run the experiment.>>The final step, you enable the geo targeting
on brand campaigns. So you put all the the
geographies that are assigned to the treatment group. Those are enabled for the brand
campaign, so you only run your brand campaigns for
the treatment geo areas.>>So this is the same whenever
you run an experiment whether it was in high school or college classes, you always
had the control group, right? You always had the group that
you aren’t modifying so you can see those results, and then the
group that you are modifying or testing so that you can compare
them back and forth, and that’s what you mean
by the treatment.>>Yeah, yeah, exactly, so
the treatment group, Geo alias, gets brand campaigns enabled and they control do not get
brand control enabled. And for all of the Geo areas
that are under the experiment does matter. You can just do what
you usually do.>>Brilliant, and just to remind
you guys, in the upper right hand corner, the resource list
this is what it looks like so you can just go
download that t-test. And there’s also other
resources there, a few of you have asked for
soft copies of the presentation. There’s already a PDF there for you to download so
go ahead and do that. And there’s also some pretty
nerdy links there to more information on statistical
relevance and t-test. So wanted to show you quickly,
and we can have large chat about this as well cuz he created
this Excel document. This is just a screenshot of
what you are gonna see when you download that template. And because we’ve talked about
two different ways to run the test, we included how to
calculate for both of those. So when you open the Excel doc,
you will see tabs along the bottom that let you run
either day of the week, if you’re running that version
of the test, or geo targeting. And I’ll just give it a little
bit close up look and Lars can show you how
to use this here.>>Yeah, so for this tab, this
is the geo targeting example, you see the yellow
columns to the left side. That’s where you put
the total number of clicks. In the pre period before
the experiment started. As an engineer I have to look
at a total number of clicks for each metro area in
that time period. To say, the identical time
period to run the experiment, but you look at a time period
before the experiment starts. And that’s going to
set the baseline, that’s what you put on
the left-hand side. You see the first upper
left yellow cells, you put in the total number
of clicks to Boston, paid. As I said, total number of
clicks of brand queries to Boston on
the upper left cell. Then the upper right cell Boston
is, let’s say paired with Philadelphia, so that’s
the first pair of metro areas. You have the total
number of clicks on brand queries
to Philadelphia, etc., down those
two yellow columns. As you punch those
numbers in there. You see both of those two
columns, if you can see the left column is the treatment,
the right column is control. Those are the metrics that don’t
get the brand campaigns enabled. You just go through the column. Everything will populate once
you fill in those yellow numbers and then you see in the black
rectangle, underneath, these are the t-test results. In this example
the t-values .04, which is a statistically
significantly value. You see the average quick
increases 6% which is, I should point out this is fake
data, this is not the real data we expect the average quick
increase to actually be much much bigger than that. This is just basically
random numbers so.>>Yeah,
just to be able to show you once you run your reports to
whatever analytical tools you’re using whether it’s Google
Analytics or something else. Once you run your report, these are the columns that
you drop your data into.>>Right.>>You’d obviously need the paid
click reports from your Bing ads campaign, and then the
calculations are already there and will spit out
the t-test results. Again, it’s showing you if this
is statistically significant data and then will also spit out
the average click increase that you are likely to see when you
do run a big ad brand campaign. So, just another sort of recap. When you’re running
an experiment, my sister as a chemical
engineer so I went to her and chatted with her when
she was in college. What are-
>>You are too.>>You are too, yeah.>>[LAUGH]
>>So I was like Margaret I
need some experiment terms when you’re running
experiments in your lab, what are the things that
you have to keep a eye on? She was like we all have
safety considerations. So, this is from my sister. So safety consideration really,
it just means hey use a reminder so like MJ said, you
do need to have patience when you’re running an experiment,
any type of experiment. So, making sure that you give
yourself the time to collect the data that’s going to be the strongest data
set that you can use. This is really basic, but I have definitely done a ton
of research on a topic to decide hey, is this something
I should be doing or not? And then never really followed
through with the implementation. And so we really just want to
remind you, after you’ve gone through all of this work,
you’ve shown your patience, you’ve collected your data,
go ahead and implement it. Make sure that you set up
the opportunity if this the fact that you might need more budget. For example, if you’ve worked
with your agency you worked with your customer to make sure
that you can get that. If you need more budget to
be able to take from other campaigns move to this one. Just make sure that you’re set
up to be able to implement particularly if your results are
showing a really great increase in your click rate. And then wash, rinse and repeat
all of us who work in search know that this is not
a static industry.>>[LAUGH]
>>It’s changing all the time, those on the SCS side and
on the paid search side. And so this is probably not
a test that you will do once and never revisit. It might be something you
want to do once a year, it might something you want
to do during a slow season versus a busy season. It’s certainly not something you
need to be doing monthly, but make sure that it’s
something scheduled into the auditing or the quality
checks that you do on your paid search campaigns overall. If you do kind of an audit
yearly this is probably one of the things that you
want to be looking at. So make sure that it’s
built into your process and that your teams and your
customers are aware that it’s something you’re really
taking advantage of. I’m gonna hand back to MJ. We wanna make sure we have
enough time for questions. Like MJ mentions, whatever
questions we don’t get to answer for the rest of this hour,
I will be writing a blog post. It’ll go live on Thursday and those will include the most
frequently asked questions as well as anything that
go back to here and you can always Tweet us at Bing
Ads or myself at Francis DR. Lars you also-
>>Lars Hirsch.>>Lars Hirsch at Twitter so
we’re always available on there to help you either with
these types of questions or really anything else
about Bing Ads.>>Thank you so much Lars and
Francis, I appreciate that. So I just wanna recap real
quick, and make sure I heard what you guys have been
saying this whole time. And that is, really you gotta
first decide what type of experiment is right for
your business. So the GO versus
the days of the week and then just have patience
because you may not have enough data in
the first time around and just you’ll probably have to do
the experiment several times to reach a statistically relevant
pool of data that you need. The other thing that I wanted
to mention is someone actually asked the question that is gonna
take a lot more time to answer, but you know we live in
a very complex world and there’s a lot of
variables going on. So we’re not saying
like this one single test is really going to be
a real clear cut, like a 100%. Not considering like
maybe there was news about your brand that
comes into play. You have to really kind of think
about a lot of different things and that question’s very
interesting and we’re gonna have to dig into that a little bit
more about all the variables. So thank you for that question
but really use the results to expand your brand keywords if
the experiment’s telling you to. It’s playing offense and
defense as we know in sports really helps you win
championships not just games. So hopefully you have now
the information to go run your own experiment and to finally
put the office debate to rest, with maybe your SEO team,
I’m not sure. To bid or not to bid, so I hope you guys have gotten
some great information. Let’s get into the Q&A, cuz we
do have a ton of questions, we’ve got a nice good
chunk at ten minutes. Let’s see, our first question
let’s see here we go. Do you see one Lars
that you liked to->>Yeah so I have a couple of questions around targeting
your competitors’ brands. There were several of those, there was also a question about
what to do with your SEM and you don’t really have
a known brand name. And I think all of
those questions suggest looking at the same data
from a different angle. It’s obvious that targeting your
competitor’s brand is something that worked to drive
traffic to your website. Especially, one if competitor
is not fitting on his or her own brand terms,
brand name, and second, if there’s a very
competitive vertical. If you’re gonna try to
be a social network, maybe it doesn’t pay to
advertise on Because people that search for
Facebook will go to Facebook, because that’s what
they’re looking for. But if you’re on a very
competitive vertical, a very competitive segment of
retail, or travel, or financial services, you will probably be
successful at driving traffic to your website by bidding
other competitive brands. Well, of course follow
the policies that are relevant in your country. Like in the US, of course, you shouldn’t put competitors
brand names in your actual ad, but you can bid on
competitor’s brand query.>>All right.>>Yeah, so
you can’t bid, and this, I saw this question come up. You can bid against competitor
brand terms, and brand names, and the different brand
modifiers that we spoke about like brand name flights,
brand name dresses etc. But you cannot use their
brand names in your ad copy. Not sure if I should share this. But currently,
you can use your competitor’s brand name in your
custom URL text. And that is a strategy I
have seen very competitive brands use. So you’re putting your
destination URL that’s hidden and then you have your
custom URL that’s visible. You can get creative there in
how you use different brand terms. You have to be careful still,
and to be honest, I’m not sure how long that
will continue, there is a case pending in Europe, actually,
Ryanair is suing Google over that currently, so I’m not
sure where that will end up. But if you really wanna
get savvy with your paid searches versus your competitor,
you can and.>>I don’t know, do unto others
as they have do unto you, that’s my motto.>>We should mention, in Europe
the policies are little bit stricter then in the US, too. Our policies did not allow
bidding on competitor’s brand terms in some
countries in Europe.>>Correct, yeah, so
if you’re a global advertiser, you may have different
strategies in different markets. If you’re a national advertiser, you can definitely get
competitive here in the US. Some of the other questions
we saw that touch results against different
verticals in industry. So I saw somebody has asked
about education, someone else had asked about, and I’m not
sure if I saw another specific example, but Lars’s team has
looked at some others if you.>>We have data on all, this is
sort of like a deep dive into three of the major verticals for
us. Not sure if we are going to
produce additional charts, and slides, and more verticals,
that could come in the future, I don’t know. But right now, we focus
on these three verticals.>>So if there is a vertical
that you would like to see them do another study on,
the squeaky wheel gets the oil?>>Yep, if we get an enough
request then we might.>>Just tweet your request,
for sure.>>Also just to mention, if you
have a Bing ads account team, they have access to this data. And so you can work directly
with your Bing ads account team to dig into some more data if
you’re a different [CROSSTALK].>>Yeah, that’s a really
good point Frances. Yes, you can get personalized
data through your Bing Ads account and they have access to
all the data on every single major brand query. Again, same threshold,
1,000 searches per month. So you can look at
the individual queries and whatever you want.>>Yeah, do you have test
results for incremental clicks? We have answered that question,
on other verticals or industries. So we just answered that. Do you have examples of impact
on not bidding on branded when the competition is
also not bidding?>>So, we do have, so that is
also part of the study, and the impact is smaller. You would still drive incremental clicks if you
bid on your brand term. All of a sudden, you will own
both the ad space as well as the top organic space, and
you will still push some of the competitors organic results
further down on the page. But the impact is definitely
smaller than when there is a competitor ad on top.>>Yeah, we did see an example,
again, specifically in retail, which tends to be
seriously competitive. That even if the other
brand isn’t bidding, they don’t have paid ads for
that result, they still will have
organic results. And like Lars said,
you still will lose clicks to those competitor organic
results with or without ads from either of you, or sorry,
without ads from either of you. But what we did see
once that brand started their own bid,
brand bidding campaign. They were able to increase the
number of clicks coming to them and decrease any risk of losing those customers
to their competitors.>>Yeah, it’s very intuitive, think about owning more of
the space above the fold, right? More of the page space
above the fold and your competitor owning less
of the above the fold space.>>It’s all about that mind
share and I think with MJ’s background in psychology
as well, you do see that. And when I was in the SEO world,
we always talked about having the magic one, two,
three, which was the ad, the top organic listing, and
then a referral listing, right? A result speaking about you,
but from a valued third party. So having those one, two, three
is always going to increase the mind share and that is gonna
help you retain customers as well as capture new ones.>>So I have another question
that’s kind of interesting and I was thinking a second
version for this question. The question is,
does the date of the year and when there is a news story
about the company brand? And my second part of that
question is, besides our data, what if you are running
an experiment and something happens and your brand
gets splashed all over the news, do you stop the experiment and
reset?>>So it depends. So their assumption is that new
stories should affect the three control groups equally if
you’re running in treatment and control geographies. This has a huge effect on
one particular geography, one particular area you might
wanna exclude the pair that that area is a part of. So say we have Boston and
Philadelphia pair that we can control, something happens
that’s only relevant in Boston, you want to just exclude that. But if it affects across
the board, then you’re good because the experimental
design will take care of it. So you have, essentially,
it will treat the control in a similar way across the board
and their results will even out.>>Yeah, great,
love that, thanks. So kind of to follow up on that
question, there was a question. Lars is there ever a time
when you should stop doing a brand campaign? Again, do you reach a threshold
where it’s no longer necessary? My assumption is to say no,
because as soon as you stop, you’re going to start losing the
advantage that you’ve gained.>>Yeah, I agree with that,
Frances, that in general, you probably don’t want to
stop doing brand campaigns. But if you wanna
test the waters, then you should run one
of these experiments. You can think about it
as a reverse setup, so instead of measuring the effect
of introducing brand ads, you can think one
that’s measuring the effect of stopping your
brand campaigns, right? So you can say the treatment is
not having the brand ads active. And it’s basically the same
set up, and you can, again, get the results of what
happens if you stop advertising on your brand.>>Great, great. Wanna respect everyone’s time, we have probably about
45 more seconds. And I wanted to remind everyone
that if you have additional questions, just tweet us
with the #AskBingAds and we’ll definitely follow up. #AskBingAds and it wouldn’t
hurt to also throw in #BingAdsWebinar, so we know
it’s connected to the webinar. And also coming up, we have
a February webinar that you can already sign up for, with Lars,
Frances, and myself. And it’s the Kevin Bacon
Approach to Keyword Attribution, totally fun and
interesting way to look at keywords that lead up to that
last click conversion keyword. So feel free, sign up, we hope
to see you on the next one, and thank you so much everyone. As soon as this webinar is over,
we would love your feedback, a survey will pop up. Thanks again and
thanks for the feedback.>>Thanks everyone,
see you on Twitter.>>Thanks everyone.


  • Michael Gorman

    February 6, 2016

    Really poor sound quality, what were you using to record this webinar?

  • LondonWebFactory.Com

    February 8, 2016

    I'm sure the content is great but I agree with +Michael Gorman, the sound is awful. I dropped out pretty quickly.

  • Tim Banaitis

    February 8, 2016

    Absolutely no sound issues during my session a few minutes ago. Good details with clear, concise approaches for everyday clients.

  • Lisa Edwards

    March 12, 2016

    Awesome presentation guys..thank you so much:-)

  • Kirti Sinha

    April 11, 2016

    The transcript of this webinar can be viewed here:


Leave a Reply