About Endai Admin

This author has not yet filled in any details.
So far Endai Admin has created 3 blog entries.

Using Predictive Analytics for Marketing [The Future of Marketing Automation]

USING PREDICTIVE ANALYTICS FOR MARKETING

[THE FUTURE OF MARKETING AUTOMATION]

What is Predictive Analytics?

Predictive analytics is the utilization of data, statistical models and machine learning capabilities in order to pinpoint the probability of future results based on historical, demographic, and other behavioral data.

Through careful examination of what has happened in the past, you can determine the greatest likelihood of what outcomes to expect down the road – and act accordingly. This maximizes the power of your data assets, and puts your business in the best position to succeed.

This guide will describe the benefits of predictive analytics (when properly utilized), and show why its role in marketing is important. We will also break down the collaborative connection between analytics and automation, and offer insight on how artificial intelligence (AI) will continue to evolve these techniques in the years to come.

What is the Predictive Process?

Predictive Analytics has been a cornerstone of data-driven marketing, and it is gaining momentum as the size and variety of data keep increasing. Not only does it consolidate and simplify data, it makes it useful by charting a course of action for obtaining and retaining customers to maximize profits.  Simply, predictive analytics provides answers to questions.

However, in order to be truly effective, the framework must be designed, and algorithms must be developed and deployed properly. If not, the consequences can be time-consuming and costly – devouring precious resources and damaging hard-earned credibility.

Fundamentally, predictive analytics should answer three critical, personalized questions:

  1. Who should you reach out to?
  2. What should you offer when you have their attention?
  3. Which channel should you communicate through?

Obtaining these answers is the chief goal of predictive analytics in marketing. But exactly how is this done?

Filling Blank Spaces in Big Data

We live in an age with an immeasurable amount of data, where every single step along the customer journey is tracked across multiple channels and stored in massive databases.

With the Big Data movement in full swing, there are plenty of holes inhibiting the most accurate answers to the key questions listed above. However, proper data strategies, predictive techniques, and technologies can color in these blank spaces in order to make the marketing landscape more vivid and readable.

While data collection is more thorough and comprehensive than ever, it is still impossible to know every single thing about every person – especially regarding their future behavior. In fact, the sheer magnitude, diversity, and speed of big data have generated a range of obstacles inhibiting its efficient consolidation and implementation.

These days, in order for predictive analytics to be the most effective, carefully designed data collection and refinement steps are key. By obtaining multiple sources of data from the widest variety of consumer channels (both online and offline) and building a 360-degree customer view around each target audience, predictive analytics opens doors to areas previously invisible to marketers.

Predictive Modeling

That is precisely why predictive models are built – to fill in these big data gaps through data mining in order to obtain a cohesive, digestible, and actionable future outlook regarding buyer behavior. These can be used to model your best customer behavior and suggest which actions to take on a personal level. Some applicable modeling techniques include customer segmentation, customer lifetime value models (CLTV), product affinity models, response models, churn prevention models, etc.

Keep in mind that – as with many aspects of life – there are no truly definitive answers in predictive analytics. Conceptually, modeling is about making the most out of what you have available – not about creating flawless datasets.

Model Scores

One critical function of a predictive model is to summarize and condense convoluted, seemingly disparate data points into simple, easy-to-read “scores.”

A basic, yet exemplary predictive modeling method involves generating a personalized likelihood score for each buyer – for example, on a scale of 1 to 10. In this case, the higher the score, the more likely the chance that person will engage with your business, gravitate to your products, respond to offers, and make a purchase via specific channel (depending on the nature of the models).

For instance, all retailers deal with loyal and valuable customers as well as one-and-done bargain seekers. The latter are undesirables. Without a doubt, attracting the most loyal and valuable buyers (Most Valuable Buyers, or MVBs) who not only come back, but continue to buy again over a long period – is the key factor of true retail success.  

So how can retailers find MVBs in such a large pool of prospects? It can seem like finding Waldo in a crowded picture.

Determine Your Target

The answer is to “target” such potential MVBs via loyalty or value models that can “mimic” their behaviors.

The following is a breakdown of the high level steps for developing MVB models for the purpose of prospecting.

  1. First, we must determine what “value” means in specific terms. Value could mean sheer dollar amount (either lifetime or the past 12 months), frequency of purchase within a set time period, longevity of relationship, time elapsed since the last purchase, and – more importantly – combinations of all of these factors. To be effective, the target should be neither too large or too small. To start, you can focus on the top 15-20% of customers in terms of lifetime spending level, more than two past purchases, and the last transaction within the past 12 months. Such factors depend heavily on your respective business model.
  2. Decide what the comparison universe (the opposite of the target) should be, since a model is an algorithmic expression of two dichotomous groups. In other words, the non-target is as important as the target universe.
  3. If finding potential MVB in the “customer base” is the goal, a trained analyst should examine all available historical transaction, promotion and response data, demographic data, and other behavioral data (such as web browsing data) to identify what factors differentiate potential MVBs from all others. For prospecting purposes, mostly non-transaction data would be used (since purchasing behavior wouldn’t be available).
  4. Once you have the the algorithm that would “score” potential MVBs, apply it to the general target universe, so that the user can easily target “high” score individuals.

These are just high level steps, but the key takeaway here is that defining both the target and the comparison universe is the most critical part of all. To accomplish this, marketers must be on the same page as the analysts who actually develop the models.  

In other words, the users must be clear about what they want out of the models, such as longevity of relationship, high revenue (e.g., high average amount, multiple transaction per year, etc.).

Applying Your Scores

Model algorithms are generally developed with samples. To be useful, final models must be applied to the pool of names used for marketing campaigns.

Once the resultant model scores are available to users, decision makers can easily identify the potentially high-value customers by gauging scores for MVB models (the higher the score, the better). They can then focus on guiding them towards becoming full-scale MVBs through personalized, specialized treatment. On the other hand, the low scorers indicate would-be priority customers who are showing relatively low potential, leading to 1-time purchase.

In another instance, a Loyalty Score would differentiate customers with long-term potential from customers who are showing signs of attrition or are already becoming dormant.

In the latter instance, a series of proactive, targeted, individualized messages could be sent autonomously (see below) in order to prevent churn and “rescue” these fading customers.

An example would be a set of properly spaced, carefully worded emails highlighting “exclusive” discounts designed to get those priority customers re-engaged with your brand and back into the fold.

Winning back potentially high-value customers at this point is crucial. If you lose a customer, the odds of him/her ever returning drop drastically. This results in a series of wasted opportunities..

Without a properly defined, targeted, and coordinated predictive modeling mechanism, you cannot know which customers to prioritize in your marketing campaign. Consequently, precious time and resources would be wasted sending the wrong messages to the wrong people.

Marketing Automation – Making Analytics Actionable

A marketing automation platform allows marketers to synchronize, streamline, and execute their omnichannel marketing campaigns from a single, centralized apparatus.

It is not enough to simply identify and understand the browsing/buying habits of your current and potential customers. In order to initiate a sale, that knowledge must be put into action.

Proper implementation of targeting and messaging will allow you to reach the right customer, at the right time, on the right channel, with the right message. In many cases, this type of advanced personalization can make all the difference between winning or losing a customer.

Aspiration, Adaptation and Application

Let’s say you’ve put in the work. You’ve asked the right questions, defined your goal, extracted and assembled your data, built your model, and got your answers. Now you want to use that information to grow and nurture your sales and customer base through a targeted marketing campaign.

Keep in mind that customers are becoming more fickle, spontaneous, and impatient than ever. With such a wide variety of consumer channels at their fingertips (web, social, email, mobile, tablets, and other addressable media), their interactions with brands can seem scattershot and haphazard.

Meanwhile, every single message you send should be individually targeted towards a specific buyer who can be at any given point along the customer journey at any time.

Sending such a timely, targeted, personalized message/offer designed to acquire or retain a buyer requires speed, proficiency, and accuracy. Thanks to advances in predictive analytics – where even modeling development can be automated via Machine Learning and AI – the whole process discussed above can lead to success in less time and (with fewer resources) than the past.

Know and Grow Your Buyers [To Their Full Potential]

We’ve reached the point where efficiently measuring and responding to consumer behavior across such a wide array of available channels is beyond the realm of simple legacy Email Service Providers (ESPs).

In this elaborate, multi-dimensional landscape, marketing automation tools can implement algorithms to hone in on your target audiences and proactively adapt to their behavior. As the software ingests more data over time, the system begins to recognize patterns and enhance its recommendations on its own. This radically simplifies and streamlines each customer interaction while simultaneously maximizing business potential.

Forbes recently referenced a series of surveys that shed light on the important role automation plays in the marketing field. Here are some of the findings:

  • 67% of marketing leaders rely on marketing automation
  • 21% plan to implement a new marketing automation platform in the year ahead
  • 82% of marketers recognized a positive return on investment (ROI) from marketing automation and said it makes them more efficient.

With the right combination of predictive analytics and marketing automation software, you will continuously keep your brand ahead of the curve – and on the minds of your customers.

The Future of Predictive Analytics and AI

The scope and efficiency of AI’s capabilities has been growing rapidly and will continue to do so in the years to come. In fact, the amount of funds spent on AI worldwide is predicted to climb to almost $40 billion by 2025.

Meanwhile, predictive marketing platforms are becoming increasingly automated, with less human involvement necessary than ever. Does this mean that humans will eventually be able to just sit back and let the machines do all of the work for them?

The answer is no, because machines still need people to define goals and lay out objectives in ways that make mathematical sense. Despite how sophisticated machines may evolve, they still won’t be able to compute and respond to irrational commands.

In addition, machines will not be able to completely understand why humans want specific tasks done, which is a critical component of any marketing campaign.

Basically, machines will continue to increase the speed, scope, and precision of both predictive analytics and marketing automation in the future, but they will still require logical, perceptive humans at the helm to chart a course of action and deliver instructions. After all, automation is really about executing what humans know how to do already.

Conclusion

With the proper predictive analytics and marketing automation software, marketers can utilize big data to sculpt insightful, actionable customer profiles, segments, and models.

Equipped with this valuable information about each individual buyer’s interests, inclinations, and position along the customer journey, predictive marketing platforms can autonomously initiate carefully targeted, personalized conversations across any available channel at any time.

This is the age of one-to-one marketing, where reaching buyers on a personal level via the proper channel is becoming more essential than ever. In a field where every competitor is perpetually striving for relevancy, a predictive marketing automation system is vital to success.

At BuyerGenomics, a well educated client is our best customer, and we’ve poured our experience and resources into this piece to that end. If you picked up even one or two new insights, you’ve succeeded today.

If you’re interested in learning more about the BuyerGenomics Predictive Marketing Automation Software here.

By | 2018-12-14T15:46:29+00:00 December 13th, 2018|Blog|0 Comments

What is a Customer Data Platform?

CUSTOMER DATA PLATFORM

[THE MARKETER’S GUIDE]

What is a Customer Data Platform (CDP)?

If you’ve heard the term “CDP” but don’t know just what it is – you are not alone.

The term “CDP” has generated considerable coverage in modern marketing over the past couple of years, but it has never been more relevant than right now. Essentially, Customer Data Platform software is a web-based interface that consists of three core components – a database, the ability to connect to multiple channels, and a marketer-friendly interface – that help drive marketing and sales initiatives.

What You Will Learn Here

There are other platforms that contain some of these capabilities, but not all (and not to the same degree). That’s why nailing the true definition can be so confusing. However, with over 20 years of innovation in marketing, analytics, and technology under our belt, and access to a vast network of experts, we not only know what a CDP really is, we’ve figured out what sets the best Customer Data Platform apart from the rest.

With so much disparate information out there, we sought to do more than simply define a CDP and its capabilities. Plenty of other marketing companies and media outlets have already attempted to do that (with varying degrees of success).

Instead, we want you to know exactly what a CDP can do to substantially enhance your company’s customer base, form (and build) legitimate, lasting consumer relationships, and increase profits.

After we break down exactly what a CDP is, we’ll move on to how you can (and why you should) utilize this highly advanced – yet incredibly practical and accessible – technology to not just attract new customers, but identify and retain the most valuable ones that drive your business.

What a CDP is [and What it Does]

Make no mistake. There is no shortage of customer data. The true problem is that much of it is disorganized and incomplete. As a result, businesses struggle with ways to efficiently sort through it all and consolidate it into something useful, actionable, targeted, and personalized..

Customers tend to be fickle, spontaneous, and impatient. Today, customers simply do not interact with brands in a linear fashion. Instead, they come into contact with numerous ones across a multitude of channels (web, social, email, mobile, tablets, and other addressable media) at any given time – generating a multitude of data points along the way.

These days, a firm grasp of the Customer Experience (CX) is viewed as a critical way to stand out from competitors. CDP software utilizes an intelligence model that sifts through these scores of fragmented data, extracts what is relevant, and uses it to form unique, up-to-date customer profiles, build relationships, via data-based personalization, and ultimately increase sales.

Gartner defines a CDP as a marketer-friendly, web-based interface that integrates four core capabilities:

  • Data Collection – Autonomously collect/sort an unlimited amount of first-party, personalized data from various sources.
  • Profile Unification – Synthesize profiles at the individual level and conjoin customer characteristics to specific identities.
  • Segmentation – Develop and regulate rule-based segments. This can include propensity models or automated segment discovery.
  • Activation – Send segments (with instructions) to actualize email campaigns, mobile messaging, advertising, or any other channel activity. Specialized components may include next-best recommendations/actions, multivariate testing, dynamic creative optimization (DCO), and testing/self-optimization functions.

In addition, a CDP can be piggybacked onto any pre-existing system mentioned above, in a specialized manner that unique to the client and its needs. Other functions include analytics, reporting, tracking, and BI (Business Intelligence). With such a versatile set of tools, the unpredictable becomes legible.

cdp-data

CDP vs Other Platforms

People often ask about the differences between a CDP and other marketing platforms. The most prominent queries are how a CDP compares to a CRM (Customer Relationship Management), a DMP (Data Management Platform), and Data Warehouses and Lakes. This can create some confusion, especially since they all seem relatively similar on the surface.

In reality, there are a number of clear-cut distinctions that differentiate a CDP and make it stand out from the rest. Other systems – like the ones listed above – were designed with comparable goals in mind, but with differing (and generally fewer) functions, scopes, and capabilities.

CDP vs CRM

While a CRM works to connect with consumers and utilize data to form customer profiles (like a CDP) – they simply are not designed to filter enormous quantities of data from so many sources. They also limit the amount of detail of ingested data, lack advanced identity matching capabilities, and restrict outside access to their internal databases.

Essentially, CRMs work well for targeted tasks within specific channels, but lack the depth and versatility to fully manage modern customer data like a CDP.

CDP vs DMP

Meanwhile, a DMP mostly utilizes 3rd party data sources (anonymous, mass audiences, no PII (Personally Identifiable Information), while a CDP largely employs 1st party data (specific, tied to actual individuals and their personal behaviors in real time). For DMPs, these pre-built audiences are used to enhance targeted display ads. On the other hand, a CDP uses predictive analytics to discern patterns, simplify data, and put it to use.

DMP data is also cookie-based, which means that a typical DMP profile only lasts for about 90 days before terminating. A CDP employs persistent, finely detailed, real-time customer profiles for unlimited amounts of time.

CDP vs Data Lake/Warehouse

Data Lakes and Data Warehouses are specially-developed IT endeavors that typically cost more time and money to install than a CDP. Oftentimes, data warehouses are updated at designated periods, unlike CDPs – in which data ingestion is an ongoing process.

Also, since data warehouses are designed and run by IT teams, marketers have to frequently depend on the IT Department, which slows down the whole process. While technical participation and know-how is still necessary with a CDP, it is much more accessible overall. Therefore, marketers can directly access and run a CDP much more smoothly and efficiently.

cdp-framework

Classifying Your Customers [With a CDP]

In Gartner’s 2017-2018 CMO Survey, marketing leaders said they invested two-thirds of their budget in supporting customer retention and growth through digital marketing.

One of the reasons why this is so important is because there are different types of customers (i.e. gender, income, interests) who are always at different stages of purchasing and skipping across multiple platforms. By identifying who these customers are (through response, engagement, and conversion data) understanding their tendencies, and tracking their buyer lifecycle, a CDP increases Customer Lifetime Value (CLTV) and transforms variables into certainties.

Strike While the Iron is Hot

It isn’t enough to simply understand the browsing and buying habits of current and potential customers. For many, in order to initiate a sale, you have to reach the right customer, at the right place, at the right time.

We’ve come to the point where efficiently measuring consumer behavior across such a wide array of available channels simultaneously is beyond the realm of human capability.

That’s where a CDP comes into play. It utilizes the most advanced forms of artificial intelligence (AI) and machine learning to sift through and unify an incredibly extensive array of data across all channels and devices not just quickly – but autonomously through predictive analytics.

Applicable statistical techniques include logistic regression, advanced data mining techniques, and neural networks. These can be used to model your best customer behavior and suggest which actions to take on a personal level. As the software ingests more data over time, the system continues to learn on its own and enhance its recommendations.

With a centralized, granular, complete, 360-degree customer view now in place, a CDP can create a feedback loop that develops a specific, singular customer profile. From there, it calculates buyer potential and takes proactive action to maximize purchase probability through automated decision-making.

The One-Time Buyer

With the incredibly high bar set by mega-retailers like Amazon, Walmart, and Target, many retailers are struggling and in dire need of an innovative, yet effective way to form personalized and lasting relationships with their customers. Additionally, subscription-based platforms – from Netflix to Dollar Shave Club – have experienced a meteoric rise by essentially guaranteeing year-round customer retention.

This leads to a critical conundrum: Can traditional retail/e-commerce companies find ways to not just attract new buyers (which is already incredibly costly), but retain them in a manner similar to modern subscription businesses?

With so many purchasing options out there that can be accomplished with a simple swipe of a finger or the click of a button, consumers are – by and large – extremely fickle.

The truth is, most retail buyers do not come back after their first purchase about 75 percent to be exact. This means that most of a retailer’s customer base contributes little to negative profit. To put it lightly, this is not a recipe for sustained success – especially since it costs so much to attract most of these one-time buyers in the first place.

The MVB (Most Valuable Buyer)

On the other end of the spectrum is the MVB. These customers, while only making up about 15-20% of a company’s total customer base, typically account for more than three-quarters of all revenue.

These are the loyal consumers who are willing to spend more money, more often than the rest. Obtaining a one-time buyer is basically equivalent to gaining a subscriber, except they are not spending a fixed amount over a structured period of time – like $9.99 per month. In fact, there is nothing holding them back from spending as much as they want, as many times as they choose.

Great businesses are built on great customers, and MVBs form the backbone of any successful business. With an all-encompassing, 360-degree, multi-channel view of consumer behavior, a CDP can help you find prospective MVBs, develop them accordingly, and continue to identify more along the way.

In a 2018 Forbes Insights/Treasure Data Survey, 44% of organizations reported that a CDP was helping to drive their customer loyalty and ROI.

Here are some other highlights of the survey:

  • 93% anticipated that employment and analysis of customer data in decisions and campaigns would create a noticeable shift in their ability to meet disruptive and competitive challenges.
  • 53% said that the transparency provided through CDP platforms enabled their teams to react more quickly to changes in markets or customer preferences.

Inactive/Fading Customers

A CDP can prevent the dreaded churn by automatically identifying shifts in their status across and strategically reaching out to fading customers before they fall into attrition. One way this can be done is by employing ESP (Email Service Provider) functions to send a special offer for your product or service at a premeditated moment. This act of “rescuing” fading or inactive customers is extremely critical – because once you lose a buyer, you’ll have to fight twice as hard to win them back.

A CDP’s RFM (Recency, Frequency, Monetary Value) scoring engine classifies customers and sends the right follow-up message at the right time. The best CDP’s go far beyond RFM, and calculate near-time or real-time model scores, for every customer, predicting their likelihood of purchase or attrition.

Real Life Customer Journeys  [The “Customer” in CDP]

Within each moment is an opportunity. When those moments are captured and managed in an intelligent, methodical fashion, a CDP enables you to transform transactions into relationships.  

We spoke with a few model consumers who fall into the MVB category and were willing to share why.

Sneakerhead

The first is a man in his mid-20’s who is an avid sneaker collector (“sneakerhead’). For him, footwear goes far beyond mere practicality. He was an avid basketball fan as a child, which branched out into a heavy interest in sneakers over time.

In high school, he always made sure he had two or three solid pairs to choose from every morning. At the time, that was all he could afford.

But once he entered the workforce and began earning a respectable income, the scope and price of his purchases increased drastically. Eventually, he developed an entire closet just for his sneakers – many of which were rarely used – if at all.

“To me, it’s a lifestyle,” he said. “It’s part of my identity, and I feel like I’m a member of a community. It’s a culture.”

Fashionista

The second is a woman in her late-20s who considers her closet to be the centerpiece of her upscale city apartment. It is a sizeable walk-in filled with a wide variety of high-end designer outfits and accessories – each for different occasions.

While she always had good fashion sense and liked to dress well, her fascination with fashion truly took root while shopping for business outfits at her first job.  She bought what she could afford at the time, but looked forward to when she could shop freely without limitations.

After a hard-earned big promotion, she started branching out and expanding her horizons. She followed all of the latest trends, and developed an impressive collection of dresses, tops, pants, jackets, handbags, and footwear.

“Fashion is more than just a hobby,” she said. “What I wear affects how I carry myself and how I’m viewed by others. I’m on display everywhere I go.

“It literally applies to any situation,” she added. “Whether I’m at work, out with my girlfriends, relaxing at home, or dressed up for a special occasion. When I shop, I’m rewarding myself and celebrating who I am.”

Key Takeaway

Equipped with a CDP, any retailer could easily identify either subject as a current or potential MVB. From there, they could monitor purchases and channel activity, gauge/influence brand loyalty, and deliver targeted, timely updates on the latest releases and promotions in order to actively increase the likeliness of another transaction.

For instance, if either consumer regularly demonstrates an affinity for one particular style or category of a product, they can actively respond by offering corresponding product suggestions for the next purchase.

An elite CDP helps to paint accurate, intricate portraits of your buyers. By knowing exactly who you are selling to, you are ahead of the game. In turn, the odds of securing and/or retaining MVBs skyrocket.

Is a CDP Right For You?

Many companies are dealing with customer data issues. In 2017, ClickZ referenced a recent Campaign survey of over 100 global CMOs and marketing executives. When asked what they felt were the biggest challenges currently facing marketers, they found some pertinent results:

  • 57% felt hindered in their ability to carry out broad digital information.
  • 22% felt their data and analytics capabilities were lacking.
  • 13% were unable to improve targeting and personalization.
  • 10% endeavored to build better marketing automation.
  • 9% wished to better understand their customers’ journey.

Clearly, properly grasping and implementing customer data is a considered a key problem in the field. However, a CDP is the most practical under a particular set of circumstances.

One factor to take into consideration is cost. CDPs employ a vast set of sophisticated, cutting-edge technologies. This requires investing a substantial amount of time and money. Generally, most CDPs fall in the realm of six figures per year.

In addition, your company should be large enough and possesses the standard customer-facing systems and staff to properly utilize/analyze the technology. While it is a powerful tool, a CDP’s true effectiveness depends upon how it is designed, implemented, and managed.

However, if employed the right way, the benefits can easily outweigh the costs. A CDP lessens the amount of time and money necessary for gathering, filtering, and activating data while actually increasing its utility. In fact – with the best software – the resulting spikes in sales can pay for the for product multiple times over.

Conclusion

Simply put, a CDP’s function is to collect, cleanse, organize consumer data, and transform it to become highly actionable. In fact, it can process any kind or class of data currently available without limit. But it is more than just a multi-channel database and interface – it has become an interactive and innovative analytics and intelligence model.

With a clear, coherent customer data strategy, the right implementation tools, and a proper cross-functional team in place, a CDP can hasten, strengthen, and broaden your organization’s marketing framework.

Businesses do not just want to collect, assemble, and assess data. They want to apply it into something useful and generate concrete results. They want to grow their customer base, develop real, personalized, lasting relationships with their buyers (MVBs), and maximize revenue.

At BuyerGenomics, a well-educated client is our best customer, and we’ve poured our experience and resources into this piece to that end. If you picked up even one or two new insights, you’ve succeeded today. Not everyone completes reading an entire work of this scope. If you did, you’ve done what most readers do not, and distinguished yourself in the process. Congratulations — knowledge is power.

You can accomplish all the goals – and more than you would expect – from a CDP effectively, intuitively, and swiftly.  

To request a demo, click here.

By | 2018-12-03T15:01:37+00:00 November 29th, 2018|Blog|0 Comments

A/B Testing – How do you declare a winner?

A/B TESTING – HOW DO YOU DECLARE A WINNER?

Key Takeaways:

  1. When declaring a winner it has to be statistically valid. In other words, there has to be a significant enough difference, that you really set a new course in whatever you do.
  2. To understand the statistical significance of your A/B test you have to remember 3 specific parameters:
    1. Sample Size
    2. Test Size
    3. Confidence
  3. Make sure you’re testing something that can actually have an impact.
  4. A smart and well thought out test is important, you want to learn something, even if you fail. 

Below is a lightly edited transcript of Episode 31 of the Inevitable Success Podcast with Damian and special guest Stephen Yu. (Listen Here)

Transcript:

Damian: So in today’s episode Stephen Yu and I are going to be talking about different ways that you can test to improve your marking program. So for example, you know, we’re big proponents of the champion/challenger methodology, basically always having an incumbent winning approach to all of your marketing that you’re constantly challenging and we always prefer to do this in a testable format. Now that said, sometimes the metrics that come back not so clear, sometimes you look at the wrong metrics. So today we want to go a little bit deeper as to how would you determine if you have a winner or not?

Stephen: Of course, and I’ve been saying this for a long time, that test results are not baseball scores. In a baseball match, with the World Series going on right now, well if you won by one run that’s fine. It’s a one-run game, maybe it was a pitchers duel. But in testing it’s not like that, it has to be statistically valid. In other words, there has to be a significant enough difference, that you really set a new course in whenever you do.

Damian: So the takeaway is, just because you have a test that would have won a baseball game doesn’t mean that you actually have a winning idea.

Stephen: I call it conclusive evidence that you have a winner.

Damian: So I’ve totally experienced this myself, you know, I’ve run probably hundreds of tests in order for types of medians at this point. Actually, the most common result that I have found, especially if the test is not aggressive enough, is inconclusive. It’s a very common result. You know, me personally when I’m working on optimizing things, I actually love to go after bold aggressive changes and here’s why. When you’re testing, the fact you’re testing, you’re already managing the risk of rolling out a bad idea.

Stephen: OK. Hopefully, it doesn’t stink too much.

Damian: Well yeah but you’re going to not necessarily roll out to everybody, you can manage that too. But, you know, I love the idea of actually avoiding having inconclusive tests. I either want something to work phenomenally or prove that I should never do that again quickly. And I think when you look for things that can have big changes, the odds of learning nothing and just spinning your wheels go down from there.

Stephen: I think you’re describing is what we call scientific approach.

Damian: Yes.

Stephen: What is the scientific approach? We all know it, we don’t practice it, but we all know it. We took some science classes in school, it’s a social science. In the beginning, there is the hypothesis – if we do this, this will happen, or if you give this drug to somebody they’ll be better, or not one to this person drop the trial, it’s the same method, right? The biggest challenge in any analytics is to come up with a hypothesis. In other words, whatever you test here, that idea should come from human beings.

Damian: Yeah, and to that end, if you design it really well, like you have a null hypothesis too, so even if you fail, you learn something as well. So you know, I think that it’s really important you know, to make sure you’re testing something that can actually have an impact.

Stephen: But having said that, when we talk about baseball for a moment, but sometimes the winning game should be like a baseball match, that if you’re testing some creative, and the difference is slight, maybe a difference in opinion, the cost of being wrong is not that great. Therefore if you had to just pick one, then fine don’t worry too much about statistical validity. Just declare a winner and go bat away. But if you’re testing some different Audience if you will, or the result of not mailing anybody at all and seeing if your mailing is doing any good, in those cases the winners should be declared carefully because you will change the way you acquire your prospect lists, the way you talk to your customers for a foreseeable time, you really want to make sure that what you learn here is something that is sustainable.

Damian: Yeah, and it should be able to be summed up in a quick conversation of what you learned, what you tested –

Stephen: When you say one thing is better than the other, it becomes quite a bit of a history-making endeavor.

Damian: Yes, just to kind of, I think to maybe, you made me think about how to clarify a little bit more – the test that I feel like we need to avoid at all costs as marketers is the imperceptible test. It’s where you if you’re using creative as an example to the same target, it’s when you show two creatives side by side and the average person actually doesn’t know what’s different between the two of them.

Stephen: Yeah the green button, red button test.

Damian: Yeah. What if it’s like, you know, blue button and slightly less blue button right? And I see these tests happen a lot.

Stephen: We have a joke here, we have a lot of developers version of streams. You’ve been in both high-definition streams, the colors are not always the same.

Damian: Exactly, but here’s another example though. You know if it’s a certain slight color blue, eight percent of the population, the male population can’t even see it because they’re colorblind. You know? We actually had a story about this. So now they’re treated like that’s the result.

Stephen: Yeah, so there’s the so what question. In fact, let’s talk about the whole scientific approach. You set up a hypothesis, set up the test rules, execute the test, declare a winner. There’s the last step which is, so what? You always have to end every test with a so what question. So what are we going to do about it? So is this something that you’re going to do forever? Is it that significant? So yeah, I’m using the word significance again.

Damian: Yes let’s dig into that one a little bit.

Stephen: I think we should dig into what statistical significance is for the people who are not stat majors. Simply for non-stat majors, you just have to remember certain parameters that you don’t jump to conclusions too hasty. One is, what is what is a sample size? It’s an easy example, so okay they do the A/B testing and whether the A or B, the difference is three clicks. Well, I don’t even have to test it, three clicks out of how many, about a few thousand. You know what, that’s not a difference.

Damian: You know what, there’s some math into it.

Stephen: Oh there’s some total math into this, but we’re starting out easy.

Damian: There are a couple of like good ways of thinking about this that I’ve approached over the past few years. So sample size, there are some general rules, of course, larger is typically always better. Right? And the other thing too is, if, and I’ll give you a really clear example of this, you can have a small sample size and still get the statistical significance.

Stephen: The difference is bigger.

Damian: Exactly, and that’s what people miss.  

Stephen: That’s exactly what I’m talking about. So you’ve got to have all three in your mind. I’ll give you three problems. One is the general sample size. Now people get scared of the large sample for valid reasons. Let’s say you have some holdouts some mailing or emailing holdouts but you’re not going to touch them. Well if you don’t touch them they’re not going to respond. That’s the belief, right? That’s why we do these things. Well, if I have a big holdout sample, I’m going to lose my money making opportunity. That is not a wrong way to see it but you’ve got to still test. So what is a good test size? Again the size matters. Now, I talk about it as a response size, not as a test size. Why? Because now you have to think about what is the typical difference that you’re trying to measure. Are you trying to measure the difference in 0.1%? Or plus or minus 1% is good enough for you.

Damian: So define response in this context.

Stephen: In other words in this context this – and by the way, if you are testing alternate click-through rates, they are normally in double-digit percentages, it’s easy. But in a mailing situation or like the alternate response for it, that is the number of actual conversions divided by the number of touches. That number generally is very small but that’s the ultimate number, isn’t it? Like who cares if you have all these opens if nobody bought it. Because that’s the ultimate barometer of success: is how much actual conversion did you see, and how much money did they bring in? So you even have measurements like revenue generated by a thousand touches and stuff like that. That’s why we have that ultimate merchant, because of the money talks. Now, what is the typical difference between, say you have a sale that you know you would touch and you have a mail sale here and one gets 1.2% response and the other gets like 0.18 difference – is that a real difference? You have to think about the size of the difference that you’re trying to measure, the smaller that you want to see the result, the bigger the sample size. That’s another thing.

Damian: Yep.

Stephen: There’s a third element. How confident do you want to be?

Damian: Confidence.

Stephen: Do you want to be 98% confident all the time, or 95% confidence or even 80% is good enough for you.  

Damian: Let’s go a little deeper on that. What is the difference, like practically, between how long you have to wait for 95% confidence versus like 98% confidence?

Stephen: That is at a confidence level most directly related to sample size, at the time that you read. Now it’s slightly related because it could read longer, of course, you have a larger sample. What does not change is that the test you universally created, all that happened in the beginning. Just because you waited longer doesn’t mean that the test universe gets bigger. So this question should be answered before then. So you have to have some idea of the time you are probably going to measure by, you have some idea of what kind of a difference you are going to measure. So they have to know the typical response –

Damian: Right, a range of outcomes.

Stephen: Exactly you’ve got to have some idea that oh, yeah so I want to measure within just a 5% difference in open rate. That’s fine. So these things determine the size of the sample and of course the confidence level is higher figures into it.  

Damian: Right. And you know what, one time I actually remember having this conversation and I said, I think I started saying that there was pushback that I either got from a client or somebody that was new here about the sample size being like a truth always, you know, more sample better. And I said, “Just think about it this way. The variance in the range of outcome has a massive impact on how many people you need.” I said, “Go through this thought experiment. Let’s say you’re AB testing two landing pages. The test goes to a fully functioning landing page that you can check out on. The control goes to a 404 dead page. You going to know very quickly you don’t need a high sample size to figure out that one is better than the other.” And that’s such a good logical test to be like, “Oh I understand the math of this.” And that’s powerful when you really understand how this stuff works because then you can start to wrap your head around what you can believe and what you don’t have to.

I mean even in medical testing, there are conditions where they’ll test that one drug is so much more powerful than the other or dangerous, that they end the test early because it’s such, you know, if people start dying then it’s very easy to tell that there’s a problem early. And that’s another thing, ending a test early when you hit a large variance in outcome.

Stephen: Tell you what it ruins baseball. Like you know what, this pitcher stinks, let’s not even continue and further agonize the team. But what you said kind of reminded me of what, a lot of marketers are too greedy about the things that they test. Please don’t do that, because I’ve seen so many tests where they’re testing everything. This source, creatives, segments, and then they go, “Well we’ll just look at all the responders and test group and divide them into all these different cells composed of like three dimensions like this segment, creative.” That’s a lot already right? But that means some cells are big enough by accident, but some cells can be so small we cannot read any result for all those dimensions. Now, when that happens I say go back to economics class again. What is the economic theory proving? We always say things like, “With all things – “

Damian: Of equal.

Stephen: Of equal what is the outcome?

Damian:  I’ve got that one. You’re testing me.

Stephen: Now say it again in Latin. I’m just kidding. But the point is, if you do that, then you know what for this report I’m going to only see from a segment point of view, so which segment? Now you may have enough sample responders in it to see the result. And then you, okay so all other things being equal in terms of creative, which one? You could do it that way too.

Damian: Yeah. Tell me if this is what you, I think you might be saying something else, but you made me think of another idea. This is what happens, you just start thinking of past experience and I’m going to share it. So, I remember doing a test early in my career and I think it was for a landing page of some sort, and I remember that just so happened the randomness because random doesn’t mean even when an AB test is routing traffic, okay? And one of the test pages in hindsight had gotten so much more brand traffic than the non-brand traffic. And for anybody that knows search, brand traffic tends to convert much higher than non-brand traffic. Right? Sometimes like 10 to 1, and the slight skew in one part of the experiment, randomly through traffic randomization, when I isolated that after the fact, completely change the results. So that taught me that being able to get a fair target is really important in constructing a test.

Stephen: Oh my god yes. That’s like saying that, I even wrote an article about this, why were all these people dead wrong about predicting Trump winning the election. You know what it was? It was a sampling error. They under sampled a survey of an area. You cannot predict the outcome of an election without fair representation. Think about it, if you just survey a whole bunch of city folks, guess what they are going to say? I mean there’s a regional bias in all of us right? So it was a sampling thing. Also when the sample size was so small, then you are talking about a town with really few people living in it and what if you missed out on a major household by just randomness. The only way to fix it is well, of course, you have to have a fair randomization routine, otherwise, it’s fraudulent.

Damian: Well this is the whole thing like the randomization, this is confidence level, right? Like that the randomization the higher the sample size you have the more confident you can be right?

Stephen: That’s right, that’s a result that the way we say it is that the higher the sample size, it is more likely to resemble the real universe.

Damian: Right.

Stephen: That’s the key. It’s not about being just like the universe or that you have to call everybody and nobody’s going to do that.

Damian: Yeah. Whenever possible, and this is not possible for everybody. But like let’s say if I could do a paid search test, I would try to like I organize, I just want people to type these keywords in, you know? There’s sometimes you can do that but sometimes it makes it so small that unless you think you’re going to get a big variation in outcome, you don’t learn. But this is where really understanding the math of how all these things tie together, can help you figure out what the best thing to test is you know? If you know that you are going to have a smaller sample size and you’re not sure if it’s going to have a big range of outcomes, you may have to take a different testing approach or maybe think about how could you bubble this up into something thematically bigger to test as a bigger idea to a bigger universe because you’re spinning your wheels with inconclusive to low confidence results.

Stephen: Yeah 100 inconclusive small tests don’t mean anything.

Damian: Yeah, well it does. It means you wasted a lot of time, a lot of money.

Stephen: Somebody kept their job by doing busy work, yeah.

Damian: For some period of time.

Stephen: I’ve found that those people are really good at keeping their jobs. I’m being sarcastic. So going back to the baseball analogy let’s just end with a baseball analogy. We started with the whole baseball thing.

Damian: This is America it’s America’s favorite pastime.

Stephen: And it’s the World Series going on. Now, just like baseball analogies, which is by the way statistically significant because there are like over 160 games so that’s why there’s enough number of pitches and hits and walks that we can predict these things right? That’s why during the postseason it’s harder to predict based on just the statistics alone.

Damian: Right, if you’re trying to do the Moneyball at little league would be harder.

Stephen: Now, why are baseball coaches so good at what they do? Because they’re much smarter us? Maybe they are, but the real reason is because they’ve seen everything. When they move certain players in the field, it’s because they’ve seen it before. That means, just like these testers, if you play this game a lot, you’ll get better at it. So, we only talked about rough guidelines today. But, having a testing mindset is the hardest part. A lot of digital marketers just don’t test.

Damian: It’s very freeing I think to embrace testing.

Stephen: I think it is, you don’t want to be wrong.

Damian: You eliminate yourself from the outcome.

Stephen: It’s the math – that’s the way.  So I think the hardest thing is having the scientific approach and actually race it, and you actually to commit to a test. And if you’re wrong, don’t give up, do it again. That’s baseball league, you don’t give up after one loss. Just keep at it and you will be better at it, you will think of more dimensions of a test as you do it.

Damian: I also think there’s this intuitiveness that comes from experience in designing test. That’s a hard one to quantify, but over time you will see, you know, everybody has a supercomputer in their head which is our brain. And this is one of the things that, you know, intuition is basically, we’re actually calculating that and figuring out it’s probably on something objective.

Stephen: And the mother of a hypothesis. Think about it. Now, even with all the automated tests scheduling, one thing the machine never determines is what test.

Damian: Right.

Stephen: Sorry that’s coming from you.

Damian: Yeah exactly. So you know you’ll gradually like figure out things that are worth testing and ways to test it where you can learn something win or lose.

Stephen: And the next time you do it you will know the expected response rate and such things so you can design a better test. That’s how it works.

Damian: All right. This was a lot of fun.

Stephen: As always.

Damian: Yeah, it really is. This is a topic that I find incredibly stimulating and a lot of other people do, and I see it done wrong so frequently so I’m glad we spend some time on it.

Stephen: That’s why I call it don’t treat the test result as a baseball game.

Damian: Not a baseball game. All right take care.

Stephen: Thank you.

Damian: If you enjoy today’s episode we ask that you please leave a rating and write a review. Or better yet share it with another marketer. Be sure to subscribe to the podcast for new episodes. Also, check out the show description for complete show notes and links to all resources covered in today’s episode. If you’d like to speak to someone about any topics covered in today’s episode please visit BuyerGenomics.com and start a chat with the BG team today.

By | 2018-11-28T17:27:14+00:00 November 13th, 2018|Blog, Inevitable Success Podcast|0 Comments