How to Get Those CRO Tests Right?

What is a good conversion rate?

Kausambi: Yeah. Yeah, for sure. And, uh, you know, it gives you like a breadth of learning and then of course, there'll be a time you are already, uh, you know, a niche which is CRO, but even within that, of course, you, you got a lot of time to figure that out. A common question that a bunch of, um, I think it's, it's across the industry.

Like everybody's always questioning, like, what is the store conversion that I have. Uh, you know, is it, where's the benchmark? Like where am I with respect to everything around me? Right. So what's really good conversion rate.

Ryan: Yeah, that's a tricky one. That's one of those hot button questions where, uh, most experienced CRS will try to avoid answering it if they can, because the answer is, it depends.

Everything is highly contextual. Um, you know, even a similar product and sort of similar audience, that business is going to have different traffic, acquisition activities. And some of those, those will bring in higher quality traffic for lower quality or. You know, higher intent versus lower intent. Um, but if I had to pin a, just a quick number on it for, for an e-com store, I would say you should be aiming to get into sort of the three to five.

Sort of like sitewide conversion rate, uh, in some cases that might be really ambitious and in some cases that's low. Like we've worked with clients where the people coming in were highly motivated. And so if the conversion rate was typically in sort of the 10 to 20% range, which, you know, a lot of people who run e-comm stores to them, that that would sound impossible, but it does happen.

Factors affecting conversion rates

Kausambi: Got it. Got it. Yeah, definitely. I'm in force for a large part of the world that would be like, you know, uh, goals, goals, goals. Right. Um, but, but, uh, now to what, one interesting differentiation that you mentioned is. The founders or the team or the brand themselves a highly motivated. Right. But what are all apart from that?

What are some of these key factors that actually do remind where in the spectrum you might end up with?

Ryan: Um, well, I think brand plays a large role. So if people are already familiar and having an affinity with your brand, and then they see the ad that triggers them to come to your website and make a purchase.

So it's like that visit is much later in the, you know, awareness and intent funnel. Versus if you're. Sort of new, just kind of prospecting, just paying a lot on, say Facebook ads or LinkedIn ads or Google ads. Um, and just to try to drive traffic that isn't very aware, then, you know, you shouldn't expect a super high conversion rate right off the bat.

So yeah, it kind of depends on the maturity of your marketing and, uh, the brand affinity and awareness stuff like that.

Factors to consider when optimizing conversions

Kausambi: Yeah. Yeah. But, but if they, if someone has to kind of now start stepping back and looking at like, Hey, like how I got to optimize conversions in my store, right. Water in your experience, some of the top three to five kind of metrics that have to come together for them.

And what are, what are these things that they have.

Ryan: Yeah, that's also a pretty complex area. So even when you're talking metrics for conversion optimization, there's, I'd say three layers to it. One being the actual test level metrics, which is, you know, we're running a test based on final conversions in terms of transactions, or, you know, sometimes you don't have the traffic for that then.

So you might shift your strategy towards just getting people to the next step of the funnel, you know, driving more traffic to cart and checkout. And so those, it all works together. So the test level metrics layer one, and then there's. Business impact metrics, which is, if you're looking at the program as a whole, you know, what is this doing for the business?

And typically that's like an ROI calculation, and I'll go into that a bit more because it's, it's also a deep area with lots of problems and potential pitfalls. Uh, and then there's the, what I'd say, project or program level metrics. So assessing the CRO experimentation program itself, um, So, I dunno which one do you think we should start with?

Kausambi: The first one? Let's go one by one. Let's just go one after.

Ryan: Okay. Sure. So yeah, most ideally you want to be A/B testing, running your individual tests based on final conversions, as much as possible, because you want people to actually put their money where their mouth is. Like you can get signals that people might be willing to purchase.

They might say they're willing to purchase, but when it actually comes down to parting with their money, that's a different story. So. You know, transaction conversion rate is a pretty good one for that. I'd say a better one is measuring test based on revenue per user. Because that kind of isolates other other factors, it would include things like AOV in the calculation.

So it's driving a whole bunch of purchases of cheap products, you know, so you can kind of balance those two factors. If you're measuring the revenue for each user, the problem with that is it's a lot more difficult to calculate you. Can't just plug RPU into a typical baby test calculator because you actually need the errors statistics or the air distribution of the test data.

So there are tools out there that can do that, like analytics to look at, for example, um, but you need to set your, your analytics up in advance, even to be able to record that data like a, a basic GA set up, you won't be able to do that. So a bit more complexity, but definitely worth it because you're getting a more holistic view of the effect of the.

Kausambi: Would you want to also think of segmentation when you're thinking about that?

Ryan: Yeah. Yeah, definitely. Um, again, depending on kind of like your volume, everything depends always, I guess, but you know, if you have, if it's a fairly low traffic website and you're just barely able to do some, some A/B testing, like you have a few hundred conversions per month, then segmentation is not really going to help you much because once you slice and dice that test data into smaller segments, there's just not enough there to be confident in the results.

Even if you have statistical significance, but you only have like say 12 versus eight conversions, then it's wow. 50% uplift, but you know, you leave that test running a little longer. You'll get a couple more conversions and. You know, it'd be going up in steps and stuff. So segmenting going to come in and around and say like a thousand transactions a month.

So that once you split up control and variant and look at different segments, like, um, you know, we're talking about browser types or traffic source or looking at new versus returning, uh, those are the typical ones that you look at. So you can get a lot more insight if you can, like they actually have the volume to be.

Costs to consider when running experiments

Kausambi: Got it. Got it. Right. And then let's, let's move to the next phase of, of the metrics that you've got, uh, kind of, uh, you know,

Ryan: Yeah. So yeah, if you're running a bunch of tests, then eventually whether you're in an agency working for clients, or you're doing this in house, someone is going to be asking, okay, what is the payoff of this?

We're spending all this money to pay you to run these experiments. And it costs, it costs money to get them developed, to get them designed. If you're doing them, you know, not just little tiny tweaks in a visual editor, which has its own its own pitfalls. So there's a cost associated with it. And, and you know, this is business.

People want to know that it's actually. Worth it. So the common way to handle that is to do an ROI calculation of a bunch of tests, say you do it on a monthly basis or a quarterly basis where you look at, okay, we ran this test, had a 10% uplift. It was exposed to, you know, 80% of our traffic because it was on the homepage, for example.

Uh, and so you can just kind of multiply, you know, what, what does a 10% uplift due to revenue? Over the course of a certain amount of time, because the idea is, you know, implement the test. You get a little bit of boost can do conversion rate and that lasts for some period of time. So it does get cumulative like that one test gives you a bit of a boost and then you run another test, implement that change.

And so you should be doing this little step function where you're going up and if you have a losing test, you don't implement it. Right. So it doesn't, it doesn't go down for. Um, and so that's pretty common. Uh, and like I said, there's a lot of, there's a lot of problems with it in both directions. So there's, it can overstate things because you know, a lot of people will run a bunch of tests and they get, say 10, 5% uplift and they start smacking them. And then they look at their actual conversion rate. And it's not necessarily following that. So you have this ROI calculation. That's like, look how fantastic we're doing. But the actual store-wide conversion rate, if you're doing it right, it should be going up still.

But you have to think of what would that conversion rate have done if you were not doing anything. So like you said, in the, in the promotion for this on LinkedIn, like CRO isn't a set it and forget it type of thing.

Creating a web store is definitely not a set it and forget it. You can't just build it and leave it there and expect it to have the same performance, it's going to decrease over time.

So, yeah. So the CRO efforts are saving you from a decline as well as giving you a bit of a boost. So there's, there's a that to look at. Um, the other issue is kind of on the others. Other side of it, where. So if I am, if I am communicating ROI in a calculation like that, then I also like to pair it with, here are the experiments that we saved you from implementing because we tested it.

So there was this idea. We thought this was going to work. It didn't, it was a losing experiment. But that has value because if you hadn't tested that, if you'd have just implemented it, then that would have harmed your site performance. So it's good to have both, like, here are the winners and this is the kind of cumulative effect of that over time.

And here are the sort of loss prevention, or that's more like people stealing from a store it's more risk management, I guess, pushing out Ben tested changes. Um, and again, it's all the main issue with this. This is just a model, so don't put too much stock in it. It is, it can be a pretty useful way to get a read on what your experimentation efforts are doing, but, you know, models, don't, don't reflect reality.

And I'd say, there's the one thing that really affects the validity of these ROI calculations is the statistics themselves. So if you take the example of a 10% lift on a test, so we run controlled versus variant for a certain amount of time, varying inches, 10% better. And you've said 95% as your confidence threshold.

So you get statistical significance and most people think that that means there's a 95% chance that the variant is 10%. But it's actually, it's a little bit backwards from that. What, what it means is the 10% lift gives you 90% confidence that the variant is better at all. Like it could, you know, there's confidence intervals involved.

So the reaching significance just means that the confidence intervals. A little bit apart, sometimes they're really apart, but something could be right next to each other almost so, so, you know, it's easy to just say this experiment was 10% winning, uh, and plug that into the ROI calculation, but it could be maybe just a 2% lift and the 10% is just what got you.

The significance for the result.

Kausambi: I think it was your top LinkedIn performing LinkedIn post - a blog that you wrote in your site about how strong type of test is, is like the stopping point of all of this whole, uh, you know, test framework that you set up.

How do you want brands to think about that? Uh, as they're planning out their measurements,

Ryan: The hypothesis side of things? Yeah, well that's, um, I guess maybe I just kind of skipped over that because it's, it's, it's critical to doing any sort of experimentation. And this is one of the, one of the things we do with our clients is we have an air table set up that manages the entire experimentation program, starting from doing the analytics audit to research insights, which you plug in there.

And then there's a test idea. Submission. And we use that as a way to sort of enforce and educate things like having a proper hypothesis for your test. So even if it's just someone's idea, we're going to direct them to the K here's the form. Pick which research is based on. And even right then people would be like, wait, what research?

What do you mean? And so that's like an education tool. We're like, well, even if it's your own opinion and you just looked at the site and you have some idea of a principle or something that could apply, put it into the research form first and then pick that as the basis for your tests. And then also developing a proper hypothesis.

Um, and then this really comes down to like the, the win rates of your program. And so that, that kind of ties into those program level metrics that I, that I mentioned where, uh, if you're running a whole bunch of tests, just based on whatever people's random ideas, like, you know, I don't want to. Courage ideas from coming out, coming in from various parts of the organization.

We definitely encourage that, but we wanted to have that kind of structure so that people will be coming up with good ideas that are based on validated research, because that's, what's actually going to move the needle. That's when you're going to see some really good results and wins that are going to keep snapping on each other and, and the learnings as well.

So that's always an important consider.

Kausambi: Yeah, absolutely. I really wanted you to spend a couple of moments on that too. We have a bunch of questions already coming in. Um, uh, I, I, uh, on the zoom tool, please drop the questions on the chat. Um, uh, what I'm going to do, Ryan is just quickly, uh, the few questions that solidly coming from the streams.

How to drive traffic on your ecommerce store

Question: So this is Liza Lisa, and she's asking, I run an apparel store in the US and have a conversion. 1.2% on average, what's the minimum, uh, traffic that I should be looking at in a 30 day period to increase conversions to two to three. And what can I do apart from running Facebook, Google ads to drive this.

Ryan: Good question. So, so I'm typically focused on what happens to the traffic once it gets to the site. So, you know, in terms of traffic, traffic, exhibition, paid ads on Facebook, uh, pretty common and Google ads. Uh, organic is the longer-term investments. So if you can develop some sort of content strategy to be providing valuable content to your target, your audience over time, that will definitely show up in terms of organic search traffic, in terms of how much traffic you need a.

For what I'd have to kind of do the calculation in my head, but usually ideally we'd have a thousand conversion conversions per month on whatever metric you want to test on. So ideally that's going to be actual transactions. Um, and like we said, at the beginning, you can get by with less than that. Um, if you, if you really want to be doing some testing, You can sort of get by with say a hundred conversions per variation or per variant, but I've kind of cautioned that because the testing on low volume websites tends to be really discouraging and you end up having a lot of inconclusive tests.

And so everyone puts all this effort into coming up with these good ideas and the designer does an awesome design. The developers build something amazing. And then it just says, ah, we don't know if it's a winner or not. And so everyone kind of gets, gets discussed. Pretty bummed out.

Kausambi: But apart from this question is also interesting because, um, at least I was also asking about a two to 3%.

I think that's what she wants to reach. Um, so, and we all know that when your traffic increases traffic quality, most likely it's dropping a little bit. So there's like higher chance of your conversions dipping versus right. So.

Ryan: I guess it depends what the incentives are. I mean, if she's paid based on the actual site conversion rate, then maybe you want to trim off some of the lower performing traffic sources that might not be a, actually a good solution in terms of the wider, wider business context. Um, But yeah, generally like a store that's just kind of starting out with experimentation or even conversion research, uh, experimentation isn't quite viable.

Uh, below 1% is pretty common. Um, around 1% is a pretty typical starting point. And if you can do some good conversion research, like, you know, exit polls are really easy to, to stand up. You don't need a lot of sample signs. You can get by with say a hundred, 150 responses to an exit poll. So if you see that, you know, people abandoning your checkout, then just set up a quick single question.

Well, that says, Hey, what's holding you back from completing your purchase today. And you'll get some, some gold in there pretty quickly. Like people will actually tell you. What's frustrating with them. If people are frustrated on a website and then you give them an outlet to express that frustration that are very grateful for it.

And they're like, you'll get, you'll get a lot of troll responses as well, but you're going to get some, some actionable stuff pretty quick. Um, and so if you can do, do some kind of simple research like that, um, you know, look at heat maps and analytics to see what's happening on the site and then to do some sort of qualitative, uh, surveying like a customer survey.

Um, to understand more about the why of what you see as happening. Um, and even if you can't test it because you don't have the volume, the insights you generate from that type of activity. Almost just implement them directly in batches and see how it goes. Just keep an eye on, on the conversion rate. And then once you get up to that level of testing, then using that, those research as inputs for test ideas, then that's when you're really going to start seeing some traction.

Tips to improve add to cart conversion

Kausambi: Yeah, I hope that helps, uh, at least, uh, the next question is from Harry, uh, any tips on improving? Oh, add to cart conversions.

Ryan: Yeah, I think one, uh, one quick tip would be on mobile. Um, just having the add to cart button visible at all times. So if that's not already set up on your store, have a look on your mobile and just kind of scroll on the product page.

And as soon as the main add to cart, button goes out of you just have a sticky one at the bottom that stays there. So as they're kind of consuming the content, learning more about the product, they don't have to hunt around and try and find the add to cart. Hmm. Yeah, that's a great tip. Actually, the place to any, any, uh, sites not to seek commerce.

Um, all right. Question from Gretta.

Kausambi: How would you say your approach to CRO has changed from when you started to now? And what apps do you recommend for running store experiments?

Ryan: How has my approach changed... It's pretty much always changing, but I guess I haven't paid a lot of attention.

To how, um, I think as a, an inexperienced CRO basically just took the courses and started working on projects right away. There's just that, um, you know, Dunning, Kruger effect curve, where when you're a beginner, you think, you know everything and then the more you learn. The more you realize you don't know and you, then you go through this dip.

I like to think that I'm climbing out the other side now in terms of, cause I know there's a lot that I know that I don't know now. So I think that's a good thing and it's a sign of, okay.

Um, apps. For running experiments, usually it's mostly about the testing tool. Um, the one, one that I like in particular is convert.com. I just think that they're kind of meeting the industry in terms of the innovations and the product itself and handling things like tracking prevention and cookie issues.

Um, and they're just pretty active in the CRO community, but, but generally they're all very similar and I don't. Personally care too much about the testing tool interface. Um, I like to send all the data to GA, uh, and do the analysis there. Um, and so basically you just need the tool to split the traffic, you know, randomize the traffic and assign the control and variant and for targeting.

And so having targeting logic and stuff like that. So it's not too critical. You can work with almost any testing tool, as long as. You know, make sure to check it first, run an AA test and check the calibration. Um, otherwise, you know, Hotjar, I always recommend as kind of like the lightweight, um, you know, customer experience tool, it's pretty, you know, good for heat maps and session recordings.

Um, and for running those onsite pulls that I was talking about. Um, but they've been getting more expensive lately and changing their pricing structure. So kind of looking for alternatives for awhile. I'm telling clients to just use Hotjar for the poles and not bother paying for, you know, getting every single session recorded it in there.

And then using Microsoft clarity for heat maps and session recordings. It's actually a pretty good, pretty good tool. And it's free. I don't know how long it will stay that way, or if they're planning on introducing other tiers to it or something. But, um, but then I noticed that Hotjar just, just changed their pricing model.

So now you can't really use it for free just to do the polls. Uh, as easily as you could before. So that's something I'm kind of trying to find alternatives to, or maybe someone needs to build something.

Kausambi: Yeah. Yeah. If anybody who's listening out there. Um, and, uh, and another, the next question is from  and he's asking what are some of the effective ways to ensure products are easily discovered by users on your store?

Ryan: Yeah, there's lots of ways to surface products. So one of, one of them is, is the main nav. So, you know, that's kind of like one of the main places, places people will look. So you want to make sure that, that you have good, easy to use navigation. Um, but that's just one part of it. Otherwise. You know, the homepage is kind of a logical place where people do the initial kind of segmentation or exploration.

So, you know, having a pretty good overview of, of what your categories are like, if you think of, if you walk into a physical retail store within a couple of seconds of walking in and you kind of can do a quick scan, you can see, okay, there's the men's section and the women's section and there's the kids.

And so, and there'll be a couple of examples right away on displays and stuff like that. So, Try to emulate that experience as much as possible on the website. Just think of them, they're walking into your store. So just make it super easy for them to see what you even have and what are the different areas and, and let them choose the path path like which, which of this is for me, because that's kind of what's happening in the lizard brain.

Part of their psyche is, you know, what is this, why, you know, why am I here? And is this for me? Because the first part of the brain that's making those initial distinctions is very selfish. It's, uh, you know, the mammalian brand evolved a lot later and that's the one that is more caring and, uh, and, and thinks things through a little bit more.

So, so yeah, appeal to those lizard brain, make them make it really super easy for them to see. Um, and then the other thing I would say is. Uh, you know, similar product recommendations on the product page. Um, there's also a lot of tactics you can use with once they have ads added something to a cart, you could do like a pop-up that says, you know, for upsells and cross-sells, um, stuff like that.

Or you could do it in the cart, or you could do it in the checkout. You could do a post-purchase one click upsell. Uh, so there's in that whole sequence of events. There's a lot of. Points where you could experiment with surfacing other product recommendations that are targeted to what they've just added or what they're about to purchase.

Kausambi: Hmm. So in your experience, even if it doesn't really convert into an upseller increasing of the basket size at that point in time, does it actually help them recall in some way, have you noticed some sort of connection.

Ryan: I haven't actually measured it, but, but you definitely could. Um, cause yeah, like these days CRO is not just focused on the purchase.

I guess some people will call it something else. If you're looking at retention rates and reorder rates and stuff like that. But, but you should be, I think it's really important to be looking at those deeper funnel post-purchase metrics and look at the entire customer experience. And, and yeah, if you're, even if they're not buying something at the moment, you suggest it it's it's in their mind.

And so they're going to be aware of it. And so I would, I would expect that you could run experiments like that and see an uptick in routine.

Types of experiment to run on your store

Kausambi: Yeah. Yeah. All right. We are already 10 minutes over time, but that just shows that everybody's super excited to learn from your end. I do have one more question. So I'm going to take a couple of more minutes and I'm, and I've asked, I asked it to most of the folks that you've spoken to in the last seven days before today, what is the top experiment?

Like? You've run a bunch of experiments, but what is something that sticks top of your mind? An anecdote about a brand. That was very interesting. That gave you an aha moment that you didn't think about and any learnings from that. Uh, if you could share. Yeah. Yeah. There was a, an experiment that we did recently for a client where, uh, and this is a really common situation.

They were sending most of their traffic from their paid ads. This was mostly on Facebook, just to a generic landing page. So that's at least one step above what other people do, which is just send it all to the home page or something. That's not even, not even the dedicated landing page, but in this case, it was a bunch of like influencers in the industry that people would be familiar with.

And they were featured in the ad. So they would have say four or five different versions of the ads with a different influential person talking about their experience with this product. And then they just get to get sent to a landing page where it's like, here's the things we offer. You should buy them basically.

So what we wanted to do is convert that into more of a story where they click on the ad that has this person, and then the page they get to is completely focused on that person. So it's like a longer version of the video and the section about like, how do they actually use this product and a quote from them about, um, you know, the benefits that they've seen from doing it and stuff and, and the effect of that.

Was really good. It was over 20% conversion rate increase, which, you know, is, is pretty, pretty tough to get effects that big when you're running a lot of experiments. Like once in a while you get just kind of knock one out of the park, but generally you're pretty happy to get lifts and sort of the five to 10% range, especially in a sort of medium maturity program.

And so it just kind of really confirmed that, um, you know, that people do get kind of hooked in from the edge and. You want to continue that reassurance, people need a lot of handholding and sort of patting on the back. Like, yes, you're in the right place. You know, you've clicked this thing. You're now on this page, it's talking about the same thing that you originally clicked on, right?

And that, that actually doesn't have like a really big impact and people being comfortable to continue because they feel like there's this logical flow in story that's happening.

Kausambi: Hmm. Hmm. Well, I love that example. I think again, something that, uh, you know, any anybody, uh, not just e-commerce friends, but anybody can actually take away from.

From this conversation. Thank you so much. This was super insightful. This was super, super insightful. And, uh, folks, if you have any more questions, please feel free to reach out to Ryan, Ryan, how can they reach out?

Ryan: Ah Linkedin (Ryan Thomas) is the best place. I'm not super active on there. I'm trying to get better at it, but, um, I check it all the time.

So yeah, feel free to reach out and send me a DM. I always like talking about CRO and experimentation. That's super interesting. There's so many different areas within it. And interesting people in the industry and stuff. So yeah, it was connecting with people there.

Kausambi: And of course you can easily find Ryan just type for Koalatative. Uh, Ryan, thank you so much. And that's a wrap for today. Thanks everybody for being, for being on the, on the screen. Thanks for having me. So far, I read them. This was awesome. I love both examples specifically. The last.