We are excited to bring Transform 2022 back to life on 19th July and virtually 20th July – 3rd August. Join AI and data leaders for discreet discussions and exciting networking opportunities. Learn more
In a previous post, I described how to ensure that marketers minimize bias when using AI. When bias comes in, it will have a significant impact on efficiency and ROAS. Therefore, it is important for marketers to develop concrete steps to ensure minimal bias in the algorithms we use, whether it’s your own AI or AI solutions from third-party vendors.
In this post, we’re going to take the next step and document specific questions to ask any AI vendor to make sure they’re minimizing bias. These questions can be part of an RFI (Request for Information) or RFP (Request for Introduction), and they can serve as a structured approach to periodic reviews of AI vendors.
Marketers’ relationships with AI vendors can take many forms, including AI building blocks in-house vs. At one end of the external spectrum, marketers often take advantage of AI that is completely off-the-shelf from the seller. For example, marketers may campaign against an audience that is pre-built within their DSP (demand-side platform), and that audience may be the result of a look-alike model based on the seed set of vendor-source audience data.
At the other end of the spectrum, marketers can choose to use their own training data sets, train and test their own, and take advantage of just one external tech platform to manage the process, or “BYOA” (“your own algorithm”). Bring “, the growing trend towards DSP). There are many flavors in between, such as providing marketers with first-party data from sellers to create custom models.
The list of questions below is for a scenario in which the marketer is taking advantage of a fully baked, off-the-shelf AI-powered product. This is largely because these scenarios are most likely to be offered to the marketer as a black box and therefore come with the greatest uncertainty and possibly the highest risk of unfamiliar bias. It is also difficult to distinguish between black boxes, which makes it very difficult to compare sellers.
But as you can see, all of these questions relate to any AI-based product, no matter where it is made. So if parts of the AI creation process are internal, these same questions are important to raise internally as part of that process.
Here are five questions to ask vendors to make sure AI is reducing bias:
1. How do you know if your training data is accurate?
When it comes to AI, put the trash inside, take out the trash. Having excellent training data does not mean having excellent AI. However, poor training data confirms poor AI.
There are many reasons why certain data is bad for training, but the most obvious if it is inaccurate. Most marketers do not realize how much inaccuracy there is in the datasets they rely on. In fact, the Advertising Research Foundation (ARF) has published only a rare look at the accuracy of demographic data across the industry, and its findings are eye-opening. Industry-wide, data for “children’s presence at home” is inaccurate 60% of the time, “single” marriage status is incorrect 76% of the time, and “small business ownership” is incorrect 83% of the time! To be clear, these are not the results of models predicting consumer positions; Instead it is the inaccuracies in the datasets that are being used to train the model!
Inaccurate training confuses the process of data algorithm development. For example, let’s say the algorithm is optimizing dynamic creative elements for a travel campaign according to geographical location. If the training data is based on inaccurate location data (a very common occurrence with location data), it may appear as an example that a customer in the southwestern U.S. responded to an announcement about a driving vacation on a Florida beach, or that Seattle consumers The Ozark responded to a fishing trip in the mountains. That would result in a very confusing model of reality, and thus a suboptimal algorithm.
Never assume that your data is accurate. Consider the source, compare it with other sources, check for consistency and whenever possible check against truth sets.
2. How do you know if your training data is complete and varied?
Good training data should also be complete, meaning you need plenty of examples outlining all the imaginative scenarios and outcomes you’re trying to run. The more complete, the more confident you can be about the pattern you find.
This is especially true for AI models designed to optimize rare results. The Freemium Mobile Game Download Campaign is an excellent example here. Games like these rely heavily on a small percentage of “whales”, with users making a lot of in-game purchases, while other users making few or no purchases. To train a whale finding algorithm, it is very important to make sure that the dataset contains tons of whale consumer travel examples so that the model can learn the pattern of who the whale is. The training dataset is bound to be biased towards non-whales as it is very common.
Another angle to add to this is diversity. If you are using AI to market a new product, for example, your training data is likely to be composed mostly of early adopters, who may differ in certain ways in terms of HHI (household income), life cycle, age and other factors. . As you strive to “cross the line” with your product to a more mainstream consumer audience, it is important to ensure that you have a diverse training data set that includes not only early adopters but also an audience that is more representative of later adopters. Is.
3. What has been tested?
Many companies focus their AI testing on overall algorithm success, such as accuracy or precision. Certainly, it is important. But especially for bias, the test cannot be stopped there. A great way to test for bias is to document specific subgroups that are key to primary use cases for the algorithm. For example, if an algorithm is set up to optimize for conversion, we have a large ticket item vs. We want to run separate tests. Small ticket items, or new customers vs. Existing customers or a variety of creative. Once we have that list of subgroups, we need to track the same set of algorithm success metrics for each individual subgroup to find out where the algorithm performs significantly weaker than the overall.
The latest IAB (Interactive Advertising Bureau) report on AI bias provides a complete infographic to guide marketers through the decision tree process for this subgroup testing method.
4. Can we run our own tests?
If a marketer is using a seller’s tool, it is highly recommended to run your own using not only to rely on the seller’s tests but also some key subgroups that are especially important for your business.
It is key to track the performance of the algorithm in subgroups. That unlikely performance would be the same between them. If not, can you live with different levels of performance? Should the algorithm be used only for certain subgroups or use cases?
5. Have you tested for bias on both sides?
When I think about the potential effects of AI bias, I see the risk for both the algorithm and the inputs in the output.
In terms of inputs, imagine using a conversion optimization algorithm for a high-consideration product and a low-consideration product.
The algorithm may be more successful in optimizing for low-consideration products because all consumer decisions are made online and is therefore a more direct route to purchase.
For a high-end product, the consumer can research offline, visit the store, chat with friends, and there is very little direct digital way to shop and thus the algorithm for such campaigns may be less accurate. .
In terms of output, imagine a mobile commerce campaign optimized for conversions. AI engines are more likely to generate more training data from short tail applications (such as ESPN or words with friends) than long tail applications. Thus, it is possible that the algorithm may drive more short-tail inventory because it has better data on these applications and is therefore more capable of detecting performance patterns. A marketer may discover over time that his campaign is over-indexing with expensive short tail inventory and potentially lose what could be a very efficient long tail inventory.
The list of questions above can help you develop or fine-tune your AI efforts to keep bias as low as possible. In a more diverse world than ever before, your AI solution must reflect that. Incomplete training data, or insufficient testing, will lead to suboptimal performance, and it is important to remember that bias testing is something that should be repeated systematically as long as the algorithm is in use.
Jack Moskowitz is Vice President and Head of Data Strategy Imodo Institute at Ericsson Imodo.
Welcome to the VentureBeat community!
DataDecisionMakers is where experts, including tech people working on data, can share data-related insights and innovations.
If you would like to read about the latest ideas and latest information, best practices and the future of data and data tech, join us at DataDecisionMakers.
You might even consider contributing to your own article!
Read more from DataDecisionMakers