📩 Subscribe for Updates

GDPR Proof Attribution. Is Online Media Mix Modeling the Answer? - EP020

By Rick Dronkers in episode 20 —
GDPR Proof Attribution. Is Online Media Mix Modeling the Answer? - EP020

In this episode I’m joined by Petri Mertanen. Petri is a digital analytics consultant from Finland who has been active in the space for a long time. Recently, he has been working a lot on “Online Media Mix Modeling” to offer an alternative to digital analytic attribution modeling as we know it from tools like Google Analytics. It’s a potentially very interesting alternative for websites that have enough traffic to make use of it. ‌‌

You can follow Petri on LinkedIn or check out his company website.‌‌

Make sure you follow the show:

If you want to help us out, please share the link to this episode page with anyone you think might be interested in learning about Digital Marketing in a Post-GDPR world.‌‌

Talk to you next week!

-Rick Dronkers

Transcription Disclaimer PLEASE NOTE LEGAL CONDITIONS: Data to Value B.V. owns the copyright in and to all content in and transcripts of the Life aFTEr GDPR Podcast, with all rights reserved, as well as the right of publicity.

WHAT YOU’RE WELCOME TO DO: You are welcome to share the below transcript (up to 500 words but not more) in media articles, on your personal website, in a non-commercial article or blog post (e.g., Medi), and/or on a personal social media account for non-commercial purposes, provided that you include attribution to “Life After GDPR” and link back to the https://lifeafterGDPR.eu URL. For the sake of clarity, media outlets with advertising models are permitted to use excerpts from the transcript per the above.

WHAT IS NOT ALLOWED: No one is authorized to copy any portion of the podcast content or use the Life after GDPR Podcast name, image or ness for any commercial purpose or use, including without limitation inclusion in any books, e-books, book smaries or synopses, or on a commercial website or social media site (e.g., Facebook, Twitter, Instagram, etc.) that offers or promotes your or another’s products or services without written explicit consent to do so.

Transcripts are based on our best efforts but will contain typos and errors. Enjoy.

[MUSIC SOUND EFFECT BEGINS AND FADES]

[00:00:00] Rick Dronkers: Hey everybody. Thank you for tuning into the Life After GDPR Podcast where we discussed digital marketing in a post GDPR world. In today's episode, I have a returning guest, Petri Mertanen. He was on one of the Measure Camp episodes that we previously released, and Petri reached out to me because he's been working on a interesting technology, especially from a privacy point of view it can help you determine the success of your marketing campaigns and marketing activities without requiring a digital analytics tool.

[00:00:37] Rick Dronkers: So with all that's happening in the digital analytics landscape, that is of course very interesting to explore, and always been working on this for a long time. I remember a Measure Camp presentations of him from years back. So, yeah, I'm keen to see what he's got for us.

[00:00:55] Rick Dronkers: So Petri is a digital analytics consultant helping clients in, Finland and outside of Finland to do well with their data. Let's see what he has in store for us. Petri, welcome to the podcast.

[00:01:09] Petri Mertanen: Thank you Rick. It's good to be back and this time remotely from Finland.

[00:01:14] Rick Dronkers: So we are gonna talk about a potential solution to a cookieless future everybody is talking about. And I remember actually a couple of years ago, I think it's six or seven years ago, at a Measure Camp Amsterdam at a different location you were already giving presentations about media mix modeling and the impact that could have.

[00:01:37] Rick Dronkers: So ou've been working on this topic for a long time.

[00:01:41] Petri Mertanen: It's very interesting topic and you can do. Marketing mix modeling and forecast the total sales of the company, which is of course basically with every marketing activities we try to grow our sales.

[00:02:02] Rick Dronkers: MMM as you will often see it referenced to online, it has already existed maybe even before digital marketing existed, I think it stems from a world where people were doing print advertising, TV advertising and radio advertising, and then try to figure out what was driving their sales is that correct?

[00:02:23] Petri Mertanen: Yeah, it's, it's correct. Basically marketing, mixed modeling has been here for decades. I don't know, from 1960s or 1970s. And it's usually based on linear regression analysis. And like you said, it has been here around quite, a long. It's interesting that we haven't really used the same method in the online environment.

[00:02:55] Rick Dronkers: In this podcast we'll probably discover some of the reasons why it might not apply for, for some situations online, but I think one of the other reasons is that once we got to digital analytics measuring specific users, probably people felt like that was a more accurate way to measure things and a more interesting metric to look at. And maybe the, the modeling got a little bit forgotten about and people focused more on the accurate metrics.

[00:03:26] Petri Mertanen: Yes indeed. I think we thought that now this is the holy grail different kind of digital attributes and models are, they will tell us the truth, how customers, behave. But as we know, we still have some issues when it comes to cookies and a cookie related tracking, different kind of devices, different kind of Browsers and, and so on. We would like to know the truth, but really hard sometimes to find the right method.

[00:04:05] Rick Dronkers: Yeah. I would even say like we're all trying to measure reality, and with digital analytics, we felt like we were pretty close, maybe because it seemed so accurate. But in reality, we were still not like until we stick electrodes into all humans' brains, right? We cannot really measure reality of how a consumer thinks about our product when they're browsing the website.

[00:04:28] Rick Dronkers: So we're always measuring some abstraction. Maybe taking an even little bit more abstracted approach to measurement is better than having like this fake accuracy that digital analytics gave us in some situations.

[00:04:43] Petri Mertanen: of course there have been changes in legislation and we are getting less data than we used to have before and so on. But really mathematical level, and you use a statistical model actually, you can also measure how good that model actually is because then you can predict some sort of outcomes.

[00:05:11] Petri Mertanen: If it's just some sort of conversions, like maybe leads or it can be online sales, transactions. You can predict what will be the outcome. And then when you get the actual data, you can see how accurate the model actually is.

[00:05:34] Petri Mertanen: So it's easy evaluate the goodness of the model.

[00:05:38] Rick Dronkers: So if I dumb it down a bit, let's say I am a, I'm a marketing manager cmo, and I have 1 million in spent, and I want to divide this between Google search brand, Google search, non-brand, and perhaps Facebook ads in Instagram. And in Facebook itself and some other source. Then with this modeling, it will allow me to say if I would allocate 100,000 more to Facebook ads, we expect the end result to be X on your conversions. Is that correct?

[00:06:19] Petri Mertanen: Yes, that is correct. And with the online marketing mixed modeling, we don't need to use any web analytics data at all. We put in all the spend in each channel. So that is our basically input. and we have input values, and then there is this formula and then we predict the outcome that can be, well any, transactions or online sales.

[00:06:54] Petri Mertanen: And then with the model, we can try to really optimize what is the best mix. or, where do you should put the money? Yeah. What, what's the best allocation of the, of the marketing

[00:07:07] Rick Dronkers: budget? Yeah,

[00:07:09] Petri Mertanen: indeed.

[00:07:09] Rick Dronkers: let's take it back a step C. Could you walk me through like from a technical point of view, like the fundamentals of how this system works, right? So on the front end, we have our advertising sources.

[00:07:22] Rick Dronkers: There's data collected and there's of course like, let's take web shop as example, So there's transactions with product categories and revenue and whatever. And there's data collected. H w does the rest of the infrastructure look like?

[00:07:35] Petri Mertanen: You can collect, the data from each advertising channel. It's very good if you have a, like a, marketing data warehouse, if you can put all these spend it let's say to big query. or some somewhere, to the cloud usually. Then you have let's say online sales you have e-commerce system, and you can get the sales data from there.

[00:08:04] Petri Mertanen: And this sales data is not affected to any content restrictions. This is the actual sales data. Get the sales data into pick, query or other cloud infrastructure as well. So basically we have input data, we have output data, and with that data we can basically use the data in the formula and you can do the marketing mix model formula.

[00:08:36] Petri Mertanen: Of course, if you are a data scientist, you may want to use R or some may want to use Python or then we do have MMM specific platforms. In Finland we have this company, our partner called Selfor they do the heavy lifting of the modeling in their platform.

[00:09:00] Petri Mertanen: Or if you are skilled or statistical person, you can use platforms like bigml.com Or some other, big data platform where you can actually do the modeling without coding and of course then you can see the results from the formula. But the next part is when you see the results, then how do you actually optimize different kind of allocations? How do you do different kind of scenarios if we move expand it from Facebook to Google or we do different kind of changes. How that will affect on our sales and these scenarios and the scenario setting.

[00:09:54] Petri Mertanen: And the tool is, it's very interesting because then you can see or think like we want to grow our online sales this much how marketing or advertising money we need and how we should divide or allocate this to different kind of channels.

[00:10:15] Rick Dronkers: I'm wondering, so let's say we have the Facebook data set, right? We have the amount of money spent, we have the amount of clicks, right? And then for Google, we have the same amount of money spent, amount of clicks, and then maybe even separated per campaign, right? Or per ad set or per specific ad. And then on the other hand, we have the conversions.

[00:10:36] Rick Dronkers: So if we cut out analytics like Google Analytics or whatever kind of analytics, normally in analytics, you would be looking at, okay, this session came from Facebook. Now they clicked around a little bit and then they purchased, right? So the that, in that case, last click attribution is to Facebook, And maybe you have multi-touch attribution. So you can see 30 days ago they clicked on Google. 10 days ago, they clicked on Facebook ad and then they, and then today they clicked on Facebook ad again. Then they purchased, and then you might attribute to Facebook and Google.

[00:11:10] Rick Dronkers: But this model you are describing is saying, okay, on one end we have spend data and clicks, which basically would suggest to the model how many people came to the website, right? With how many money was spent and then on the other side, you have amount of transactions.

[00:11:27] Petri Mertanen: Yes. And actually we don't even need the click data. because it's weird that you have trust on the formula and we have to learn away from, let's say sessions. And that's the power of the formula that we can only look at the spend data and we can, of course, we can divide it by campaigns.

[00:11:53] Petri Mertanen: Because if there are campaign, which is, let's say we advertise certain product category, then we can also divide the sales by product category.

[00:12:05] Petri Mertanen: And then we can sort of map these things that when you, we advertised more, to this category, we saw incremental sales in this product. And the one which is really interesting is the incremental sales and the base sales because during the long run, if the, let's say online store is open, it's working.

[00:12:32] Petri Mertanen: There is no errors. There is always some base sales. And sometimes in, digital analytics area, we tend to think. When we have this kind of campaign, all the sales is because of that campaign, but that is, of course, that is not actually true. So, online MMM that it actually discovers the incremental sales of its advertising campaign.

[00:13:04] Rick Dronkers: From a privacy perspective, let's look at that angle. This is of course very interesting because all our tradition or online traditional ways of doing analytics are now under fire because basically they all use a cookie to identify a user or a session and there might be ways around that, but I think it's safe to assume that over the long run, every tool that writes a cookie will at least require consent if it's not forbidden.

[00:13:38] Rick Dronkers: And even if you require consent, then there will be people who opt out. So, the accuracy of our Google Analytics, Piwik Analytics, whatever tool you use goes down if you respect privacy,loss. Right? And if you don't, do workarounds. So then this of course becomes very interesting.

[00:13:58] Rick Dronkers: So what is the reason that not everybody is doing this? So there must be downsides, right? What is the reason not everybody is doing this right now.

[00:14:07] Petri Mertanen: MMA is not for the, the smallest clients or even medium size clients. And of course, you need yeah, certain, amount of volume. Let's say that we need to have a thousand conversions or track transactions per month. Well, even in the medium size, online story that can happen.

[00:14:31] Petri Mertanen: So, bigger companies, they are more than thousand certain conversions and they spend like more than 1 million euros per year for advertis. These are the sort of numbers that we know that there is a possibility to use online. MMM very well. like I said, we don't, need the data collected through the cookies.

[00:14:58] Petri Mertanen: And this is really for some people that. always been in the digital analytics space, it's really hard to understand how does this work actually . You have to be maybe you have to study a little bit, linear recreation, analysis or something you can get a hint of. how we can actually use that in the marketing or advertising area.

[00:15:27] Rick Dronkers: So the immediate thought I had was if you can combine the two, that could be very interesting. So, use your traditional analytics setup to basically verify or maybe help you verify sooner the media mix modeling outcomes, because I can imagine that the media mix modeling outcomes will sometimes take a bit longer. You need more data in order to, reach a conclusion, a result.

[00:15:55] Petri Mertanen: Yeah. back in the days when we were doing like total sales, we used like weekly level data. We had to take like, data for the past two or three years. But now with online MMM we can use daily data and that means that we can see data for the past three to six months.

[00:16:24] Petri Mertanen: Let's say that we take data from the past six months, so we actually have roughly. More than 180 uh, days. So we have, data points, like almost 200 data points. And that's actually something we can work on with. we can do the first model. And of course, like you said we can easily compare that. Online MMM results. For example, with data driven attribution reports or models. And then we basically can see that does these two methods, do they align or are they differences?

[00:17:09] Petri Mertanen: And in the longer run we can see how well the online MMM model predicts the sales. And then when we are doing the optimization, we are doing little bit different allocations. Then we can see that we changed something. So, did our, revenue or online sales, did it grow as predicted or expected or was there something wrong with the model

[00:17:37] Rick Dronkers: Yeah. Cause I can assume the experimentation on top of this, You probably want to eliminate contamination. Let's say you have a website with, maybe four product categories. You probably want to make sure that you do an experiment per product category to make sure that you can actually measure the impact properly and not do like five experiments at the same time, because that will, that makes it harder for this model to figure out what happened.

[00:18:07] Petri Mertanen: That is true that we need some, variation in the data. So it's actually a really good idea. Let's say that we have spent some money for display advertising Then we need to just stop it for a while to get some variation in the data. So the model can evaluate better if there is a change because of this certain input.

[00:18:35] Petri Mertanen: If there is a change in the sales we can actually take some, external factors in the. Like seasonality or even like, a weather or we can use different kind of, factors in the model variables.

[00:18:54] Rick Dronkers: Yeah, that's super interesting. Cause I used to work, one of my clients in the past was they sold a lot of things, but a lot of it was used car sales. there was quite some they had a lot of media campaigns on a lot of platforms online, off. And there we actually did the same without this modeling. But basically we were very careful about when did we stop or start TV advertising or when did we stop or start display advertising. And then by looking at the data, you could actually see the incrementality. But that was a mostly manual action we were doing to kind of have a grasp of like, if we do tv, how much impact does it have on online?

[00:19:34] Petri Mertanen: And now that we have the most of the you know, digital advertising data, it's daily based, it's already online with some sort of,tools like, let's say Supermetrics for BigQuery or something, we can quite easily actually establish the marketing data warehouse. And then we only need the real sales data from somewhere, maybe from the e-commerce backend or somewhere.

[00:20:08] Petri Mertanen: And we do already, the data is there. And then we need just to figure out how we do the modeling. And like I said, there are some MMM specific platforms like Cell Forte. We can make even a, let's say we make, six months, proof of concept or trial period. And then we know, then we can see what is the results to traditional or data driven attribution model.

[00:20:38] Rick Dronkers: Like you already said, people like me who have been working with digital analytics for us, it's hard to. It feels like letting go of control, right? Because you have to trust the model. But the funny thing is that in our world, we also increasingly have to trust the model because data driven attribution, right?

[00:20:59] Rick Dronkers: Is also, usually a black box model that is, trying to tell you what is the truth. and you know, we never really knew the truth to begin with.

[00:21:08] Petri Mertanen: But even if you have like, especially the Google algorithm is black box algorithm, but even if you use MMMs basically platform. Every model is basically done customer by customer. So in that way, you can actually try if there is some sort of variable of outside factor which affect on your sales, and you can basically discuss about these different variables and factors. With the data scientist who actually is responsible for the model and it works. So there is not like, that platform and you try it yourself and that's it.

[00:22:03] Petri Mertanen: No, there is some consult doesn't work with that. So there has to be a data scientist who actually measures that the model is really working and how well it's working.

[00:22:18] Rick Dronkers: If I'm thinking about this for my clients, what are the things they need to have or do or think about. One of the things is you need to, you need to have at least a thousand transactions and at least a million spent, approximately. And higher is probably better, right? More input for, for the model.

[00:22:39] Rick Dronkers: So that's one thing to

[00:22:40] Rick Dronkers: keep in mind. Probably you could also say that you probably want to use multiple marketing sources, right? If you only use Google search ads, you probably don't really need it, right?

[00:22:50] Petri Mertanen: The more there are channels and the more there are variation in the spends, the better.

[00:23:00] Rick Dronkers: Yes. Okay. So that's also interesting. Can you have multiple conversion inputs? So let's say for in, for instance, let's say we have a high price product, right? We sell, I dunno, iPhones, whatever, something expensive, right? Maybe even higher. Where maybe we have a good indicator of success, right? Somebody adds the product to wishlist or you know, some earlier conversion on the website instead of the real purchase we use, like, something that we think is high likely predictor of success. Can we also optimize for that?

[00:23:36] Petri Mertanen: Yeah. we can change the, actually the output, or the predict predicted value in the formula. And if we want to change that and see something else, then transactions or sales, we

[00:23:51] Petri Mertanen: try that. there is actually a one, one case that customer wanted to have, not like the end conversion point. It was a little bit earlier in the process.

[00:24:03] Rick Dronkers: For instance, I have a couple of clients who are in recruiting and then online we only measure people filling out the form, but they only make money when they actually place a candidate somewhere, right? So, there's still a gap between it, and of course you can also get that final conversion in, but that there's also a gap in time between filling out the form online and them being placed.

[00:24:26] Petri Mertanen: But in that case, we have to be careful that are we using the web analytics data? And is it like 80% accurate or do we get the data from the website in the packing system or where we get the real number

[00:24:44] Rick Dronkers: Uh, but in this case, so so let's take recruiting as an example. We could get from the, from the applicant tracking system, the ats, we could get. The amount of form submissions, right? People who fill out the form, who leads, let's call them, I know that, so their marketing spend is focused on getting form submissions, but then of course they don't make money on form submissions they make money way later in the process when somebody gets placed somewhere. But the feedback loop between the form submission and them getting placed is sometimes couple of months maybe. So that's hard to optimize with the couple of months of feedback

[00:25:20] Petri Mertanen: And it's same thing in B2B business, for example, that the B2B businesses get like the more than thousand of leads per month, but the sales comes within two or three months after leads.

[00:25:37] Rick Dronkers: is there a possibility to, I know with B2B sales you have this concept of marketing qualified leads and sales qualified leads, The marketers basically get a bunch of leads in, right? And then the sales people are like, yeah, but half of these leads are shit, I want higher quality leads. if you have this long feedback loop, is there a way to later on add additional information to make the model better. Like, Hey, of these thousand leads that we sent you three months ago, only 500 were good. Right? So learn from that. Is that an option?

[00:26:11] Petri Mertanen: That's actually, quite tricky one. Ahh, have to think about it a little better and deeper. for example, if we get the sales data later on, put that in the formula as an output variable. then, we can compare these two models first.

[00:26:32] Petri Mertanen: We try to predict, the number of leads and then we change the aspect a little bit. We have a different kind of model and we try to Predict the actual sales, but it, we know that it comes a little bit later. but we have to take care of that because we know that there is a time lag.

[00:26:52] Petri Mertanen: In advertising in general, we know that there is some time lag, when they're advertising starts. It takes some time when the advertising takes. These are the interesting really, you know, details what you need to discuss with the data scientist when you are building the modeling, because you can tailor the model. And, it's the same thing that we don't expect the linear growth, that it keeps growing, you know until the rest of the world.

[00:27:28] Petri Mertanen: There are some limitations for market size, for example in Finland, it's a small market, 5.5 million people. it's not worth, Just to advertise, with a lot of money because you cannot reach anymore people.

[00:27:46] Rick Dronkers: Yeah, after a while you reached them all. No, I get that. You've been working with this technology for a while. I remember you were tinkering with like yourself built models in the past. Has the technology really improved a lot in the last couple of. Like to make it simpler for companies to work with these tools.

[00:28:08] Petri Mertanen: I can remember like, it's not that long ago, but I think three, four years ago we did MMN project and there was one data scientist. She came right from the school you know, university. She didn't know basically anything about the marketing details. And I was the expert, in that area.

[00:28:32] Petri Mertanen: First of all, it was really hard to collect all the data we spent hours or I don't know, weeks to get the data in the right format. And then we had to do some data cleaning before we got, into the, model building. And this has changed a lot because most of the stuff is online. You can actually use these really nice connectors or integration tools to get the data into some sort of cloud environment that is ready for modeling. Or maybe there is some sort of small, ETL process in the middle.

[00:29:14] Petri Mertanen: But with the help of these integration tools, the work gets much more faster. And actually when we use the daily level data, then we can really, direct our marketing activities and we can make these different kind of scenarios and test things compared to the previous, way to do things like every six months or even once per year. We were always late. and it was really slow to do like project based MMM.

[00:29:55] Rick Dronkers: Yeah. Yeah, because you want to, of course test, in the end, the value will be in the experiments you do, and for that to work, it has to be a little bit flexible and work with you, so that the biggest increases in that functionality you would say.

[00:30:09] Petri Mertanen: Yeah. Yeah. Like with daily level data, we can, for example, we can see that, yeah, we had this first quarter and this is how we performed and now let's make these changes. And after second quarter we can see that is th e results better or worse. And of course then we can. Maybe the seasonal things in the model and, different kind of things we can improve and develop the model to be better.

[00:30:43] Rick Dronkers: I'm just thinking out loud here, but is it possible for me to get out of the model a. Specific transaction ID and then see what source the model attributed it to, or does it only give aggregates?

[00:31:01] Petri Mertanen: We don't need to go to a transaction level. basically the only level the total sales with certain product categories, even for certain products.

[00:31:16] Rick Dronkers: yeah.

[00:31:16] Petri Mertanen: need to go to transaction level.

[00:31:18] Rick Dronkers: No, but I think also you can't, right? Because in the end you're doing a prediction, so you can't actually, if you would be able to attribute a specific transaction, a unique transaction ID to a source. Cause I was thinking the other way around. I was thinking, can we sometimes when a user does a transaction or becomes a client, they have this dropdown question of like, how did you find us? Right? Like qualitative input, like how did you find us? And what I do at some clients is I compared the what people say, how they found them compared to what our analytics says, how they came to the website, right? So I try to basically figure out if the qualitative metrics.

[00:31:59] Rick Dronkers: Match up with the quantitative metrics. And I was thinking if for this model you could do the same, but I, now that I said it out loud, I think it's impossible because the model will never attribute towards the unique transaction id. It will always be in aggregate groups, right?

[00:32:14] Petri Mertanen: Yeah, that's, true because we don't actually use the individual transactions, in the model. But do you know that well the people are answering to that question that you are asking it can be two things, right? Either, either they are answering wrong or I'm measuring wrong, right? So it can be both problems. But I feel like the qualitative answering So when you ask people, how did you find us? I feel like that usually matches up better with first touch attribution.

[00:32:49] Rick Dronkers: Our, like Google Analytics or whatever usually is more focused towards less touch attribution. But usually when you ask people, how did you find us? They're gonna think about, where did I find you first?

[00:33:00] Petri Mertanen: That may be true. And we do know that usually the, okay, it depends on the attribution model, but in the digital world, we tend to, overrate the touch points. Like if you ever have heard from Google paid consultant saying that need to put some more money.

[00:33:22] Rick Dronkers: Yeah. Yeah.

[00:33:24] Rick Dronkers: And from the Facebook consultant. I think having a model, like having a tool like this, what it also solves is the endless attribution discussion. Because instead of thinking about attribution, it will actually look at what is the additional value of spending this money on A, B, and C, and then make an experiment out of it, and then continuously experiment Hey, okay, the model suggests putting a hundred extra a hundred thousand in Facebook. Let's try that.

[00:33:50] Rick Dronkers: It switches the discussion a little bit because the endless attribution discussion, most people with analytics know, like it's a never ending loop you can get into with clients and it doesn't really solve anything. It doesn't really generate any action.

[00:34:04] Petri Mertanen: And of course, for certain companies, which are, products or services online, but they are, selling them in the brick and mortar stores as well. online MMM is sort of a start. To get the confidence for the MMM and then you can grow that into your I mean, you can get in your total sales, but then you can get in your offline advertising, tv, radio, outdoor, you name it, print I think the online MMM it's fast and it's much, much cheaper way, to experimen the power of MMM.

[00:34:53] Petri Mertanen: And and then you can continue if you see the results, if you can see how accurate that is, if you succeeded, optimizing your marketing mix, then maybe you get the confidence to get it, you know, wider and do the whole or total M MM m with offline advertising and the total sales.

[00:35:18] Rick Dronkers: Exactly. Step by step approach. I'm just thinking about like in a client case, So let's take Facebook for example, right? If we do Facebook advertising and the model suggests to us to throw an extra a hundred thousand in Facebook, as a experiment. is it also best practice or recommended to then not adjust the Facebook campaigns in that period? Because I can imagine like if at the same time your team is also creating new visuals for Facebook or new headlines and new, right? So there's always work being done in the advertising source itself as well.

[00:35:55] Rick Dronkers: So do you have like a best practice of, okay, when we do an experiment of allocating more or less money to a certain source, we also don't adjust the campaigns or do the campaigns need to be at like a certain level before you.

[00:36:09] Petri Mertanen: Of course the spendage has to be in, in certain level, but then. We can split the, the spend. Its, by, certain advertising versions, for example. So we can certainly test the creative part as

[00:36:27] Rick Dronkers: Hmm. Okay. Yeah, that could be interesting. Then of course, you even more volume to be testing at the creative level, right. Because then the differences become very small. So you need probably to be large B2C brand to do that, but that's very interesting. Yeah.

[00:36:43] Petri Mertanen: if there is a scenario that we will put like a hundred thousand Euros more to Facebook you maybe have to talk with the Facebook advertising specialist that, how can we do it or is it possible even

[00:36:57] Rick Dronkers: Yeah, yeah, yeah, yeah. Before you burn it. yeah, of course the model doesn't know the what are the limitations for market size or target group or whatever.

[00:37:07] Petri Mertanen: So it's not like that it's going to automate everythinwe need, human touch in a decision making and planning. Yeah. Yeah, we're not automated way fully yet.

[00:37:22] Petri Mertanen: But most of the research is when it comes to machine learning or ai, we get the best results, when there is, human knowledge and, some sort of modeling or algorithms. Yeah,

[00:37:37] Rick Dronkers: For now, until Skynet takes over.

[00:37:40] Petri Mertanen: Yeah, that. At least my colleague is prepared for, he has everything ready for that. Creating now.

[00:37:48] Rick Dronkers: For the apocalypse. Patrick, thanks. Thanks a lot for sharing this information. I think your dog is in the background. Does he want to join the podcast for a second or no?

[00:37:59] Petri Mertanen: Yeah, I don't know. She wants to go outside.

[00:38:03] Rick Dronkers: We have to cut it short. Where can people find your online or learn more about this? Are you active on LinkedIn, Twitter, or your website? What's the best place to send them to?

[00:38:13] Petri Mertanen: Of course I'm on Linked-In and you can actually find find my latest, presentations about this in slideshare.net/mertanen.

[00:38:26] Rick Dronkers: We'll link to it in the show notes for everybody that's interested. Cool. Thanks a lot for sharing your knowledge.

[00:38:34] Petri Mertanen: Thank you, Rick. Thank you so much to having me.

[00:38:36] [MUSIC SOUND EFFECT BEGINS AND FADES]