For those well-versed in CRO, the narrative is familiar. You conduct a test, celebrate its success, and swiftly implement it as a personalisation until your developers have the chance to hardcode it onto the site. However, as you wait for it to be implemented, updates to the main site may occur, introducing bugs. Depending on the severity of the bug, its impact can be huge, potentially overshadowing the actual benefits shown by the personalisation when it was a test.
As our clients’ optimisation programs expand, managing an ever-growing array of personalisations can quickly become, to put it bluntly, overwhelming. Yet, the stakes are high; should any issues arise due to incompatible code, all eyes turn to us for answers (and rightly so). Given how complex some websites can be, even minor alterations can affect how a personalisation looks. Thus, it’s extremely important to have a method in place for identifying underperforming personalisations before they escalate into significant problems.
Custom audiences are made in GA4 that correspond to users that view a personalisation. We then integrate them into our monitoring platform using the Google Analytics Reporting API. Performance is compared daily to results when the test was live to locate those that are underperforming.
This lets us quickly identify and fix any underlying bugs that impact a personalisation.
One of the forgotten heroes of statistical analysis (yes, I’m a nerd) is the actual impact a test has on a certain metric, otherwise known as a confidence interval. We can use confidence intervals to check a personalisation’s performance daily and look for any discrepancies with what we expect to see. If a personalisation isn’t doing so well for multiple days, that tells us we need to step in and check nothing major is the cause.
In a perfect world, we could just set this up and be done with it. But some questions remain: what if we tested during a promotional or high-traffic period, causing higher than average conversion rates? What if a campaign is happening that causes a metric to drop? What if users forget how to use the internet? Ultimately, a number of variables other than bugs could cause a personalisation to perform lower than expected! We keep this in mind when investigating performance dips and work to isolate the root cause of any issues. This is facilitated by our custom tracking that monitors any errors identified in personalisations (watch out for this post very soon!).
Most times, we can serve up personalisations through the testing platform without any hiccups. But for those rare moments when things do go haywire, we've got this extremely useful tool to spot and fix bugs pronto. So, whether you're on Team Daydot or one of our awesome clients, you can trust that our data's solid and we're pulling out all the right insights for each experiment!