headstone from Google Graveyard

When we last had Melanie Bowles as a speaker in 2019, she lead an informative session on building a sustainable experimentation strategy. Since nothing at all has changed since 2019, we just replayed that talk and then everyone went home. While I’m obviously being facetious — much of the strategies that Melanie laid out in that talk are still very relevant! The landscape has changed a lot since then, from big changes in browser privacy and client-side technology to the shutdown of the most widely used tool in the industry, Google Optimize.

While there’s no clear successor to Optimize, there are many good testing tools out there, including popular options like: AB Tasty, Convert.com, Visual Website Optimizer, Optimizely, etc. Most of these tools do also offer integration with GA4.

Deployment count of tools via BuiltWith data

As you can see from this chart based upon deployments in the top 1M sites, none of these tools are exactly catching fire with popularity. A big reason for that may be that none of them are free for unlimited usage like Optimize was. Melanie also pointed out that A/B testing and similar functionality like feature flagging has in some cases moved into all-in-one suites like Amplitude, Salesforce, etc. The sunset of Optimize can be looked at as a chance to mature our A/B testing practices, and focus them where they can have the most impact.

Melanie also suggested that we embrace AI tools, especially on the ideation side of testing. There’s no substitute for human expertise when building out tests, but it’s certainly not cheating to let ChatGPT come up with some potential variations for your test! Just remember to give the AI as much context as you can. Melanie ran through a quick example which included providing the AI with a customer persona, which you can find in her slides below!

As a new twist for the meetup in 2025, we’re making a donation to a speaker-selected non-profit at each event. Melanie chose to designate Columbus Cultural Orchestra — a program for young people 13-25 to develop their musical skills and enhance diversity in orchestral music — for a $250 donation!

Our first event of 2025 was a book release party for CBUSDAW’s very own Tim Wilson!

If you’ve ever been to a CBUSDAW event before (or listened to his podcast the Analytics Power Hour) you’ll know that Tim has a lot of things to say about analytics. Smart things, funny things, cranky things, etc. To our benefit, he’s organized many of these thoughts together into a book (with co-author Dr. Joe Sutherland) called “Analytics the Right Way: A Business Leader’s Guide to Putting Data to Productive Use“.

This is an excellent book that may be targeted towards “business leaders” with its title, but can also be incredibly useful for analysts themselves in terms of how to think about doing analytics in a productive way. There’s a lot of books out there covering tools, methods, and technology — but Tim and Joe’s book stands out in being about actually using analytics within an organization to further business goals. (NB: this is Jason writing this recap, and not Tim awkwardly hyping his own book in the third person. Also Tim I promise I will get around to writing that Amazon review at some point.)

But this wasn’t just a book signing with free beer, Tim did also give a talk about some of the topics he covered in the book! We had a great crowd, with friends and colleagues of Tim’s coming in from as far as Chicago, Nashville, and Boston.

We also had Jim Gianoglio jump in behind the camera (Tim’s normal job) and get some great action shots:

For our November 2024 event, we brought Josh Silverbauer in from Philly to talk about behavior analytics (in the form of MS Clarity) and marketing analytics (in the form of Piwik PRO) and when you might want to use each one.

Since Josh is well-known for writing parody songs to introduce speakers, here at CBUSDAW we flipped the script on him and opened the event with a surprise parody song about Josh.

We present “It’s Josh Silverbauer” to the tune of “In the Midnight Hour” by Wilson Pickett, sung by Jim Gianoglio and featuring Jason Packer on kazoo.

While Josh is a fan of both Clarity and Piwik PRO, he’s pointed out that he’s not paid anything by either organization… so he’s free to tell it like it is. And “how it is” is that both tools are great additions to any analyst’s arsenal — and with a generous free tier for Piwik PRO and a totally free product with Clarity there’s not much barrier to entry.

Josh pointed out how the two tools can easily used to supplement each other. For example, one could use Piwik PRO to find a particular aggregate group of users that aren’t converting well, and then review those users’ entire sessions with the session recording feature in Clarity. Or check the heatmap of the landing page for that same group.

If you’ve used session recording tools in the past you know that it can be pretty tedious to watch the recordings one-by-one. It’s like, “geez, just click the button already user #23341, it’s RIGHT THERE”.

Microsoft has recently integrated CoPilot into Clarity so it can now help save you from watching a ton of videos and instead can summarize and do some basic analysis for you.

Josh described Piwik PRO as “what Google’s Universal Analytics 2.0 could have been if GA4 didn’t exist”. If you were a serious user of our dearly departed UA you’ll feel right at home in Piwik PRO, and you’ll be pleased to see how well thought-out the platform is.

Josh’s slides:

Ok, so Clarity and Piwik PRO are both pretty cool tools, but what about rock operas?

Josh (and Jason) are releasing Volume 2 of Josh’s epic analytics rock opera entitled “User Journey Volume 2: The Custom Dimension” on November 18th.

You can listen to the first volume, “Universal Sunset” now on Spotify and most other streaming platforms.

Finally, don’t forget to join us next month for our yearly holiday event at the Grandview Theater. No speakers, but there will be a movie this year and you can vote on what you’d like it to be!

Disclaimer: Piwik PRO is a sponsor of CBUSDAW — but they only pay for the monthly pizza, not our (or Josh’s) endorsements.

As analysts, we love to optimize everything that we possibly can — so when we have a speaker that gives us a new way to think about testing we are here for it!

For our October event we had Sanjay Tamrakar talk to us about doing geo-testing. Sanjay covered basic methods like traditional pilot testing, to difference in differences, all the way to the current state-of-the-art with Geolift.

Back in the pre-digital era, Columbus was considered to be a top location in the US to pilot test new products, since its demographics closely matched the country as a whole. This allowed companies to try out new products, but only gave marketers an idea of how well a new product might do nationwide, not how much incremental lift different product variations might give or how different Columbus might perform vs. Chicago or Charleston. This kind of granularity requires more powerful tools.

These days there are much more expedient and statistically rigorous methods to test things, like Geolift. Geolift is a an open-source package from Meta that allows the creation of artificial control groups which we can use to test against treatment groups without having to worry as much about building control groups using user-specific information and the privacy issues which that can bring. Geolift’s synthetic control methods can create control groups by amalgamating different untreated areas whose performance was expected to match the treated areas.

There was even some R code showing up on the big screen, which sadly Tim Wilson missed!

Sanjay was also kind enough to provide us with his slides:

Fresh from another successful DataConnect Conference, Lauren Burke-McCarthy led our July session of Data & Analytics Wednesday talking about how to survive and succeed as a solo practitioner of data science.

Being a “solo practitioner” could mean being the only data scientist on your team, being siloed in some way, or even being a freelance contractor. The strategies that Lauren presented were focused on how to best communicate and set expectations with stakeholders. We’ve all been there when a project has gone off the rails because what a practitioner implemented didn’t match at all what a stakeholder had envisioned. Let’s nip these misalignments in the bud as best we can before they can blossom into fully grown issues.

In fact it turns out many (perhaps most!) of these techniques could work for us in any data-related role we were in. What after all even is a data scientist? Lauren also took a crack at answering that age-old question off the top of her head. To paraphrase her answer, a Data Scientist focuses on models and experiments to make future-looking prediction — vs a Data Analyst works on analysis of current and historical data to identify trends and develop insights. If those two things seem to blur into each other at times, that just shows how Lauren’s advice on processes and communication works for both! Perhaps even those of us who have now added “AI” in our job titles? Could well be…

Looking to learn more about these techniques? Lauren was kind enough to provide us with our slides so you can take a look for yourself:

And, of course, pictures!

Please join us next month when the ever-delightful Matt Gershoff will be in town to discuss how to think purposely about data as we move towards privacy by design.

Our June 2024 meetup featured Dr. Maria Copot from OSU delving into some of the underlying theories behind our favorite A/B testing platforms. Though before we get into the fun math part (yes, it’s fun, don’t look at me like that) — we need to all remember that there needs to be a question behind your experiment. If you don’t have a hypothesis you’re trying to validate, then what’s the point of testing something? Once you’ve got something you want to test, then you can test it, but testing just for the sake of saying how many A/B tests your department ran last year isn’t going to get you where you want to be.

A lot of us have been asked, “is this result statistically significant?” And maybe we’ve even said, “well, the P-value is <0.05 so it’s significant”… But what exactly is a P-value and why is 0.05 the number a big deal? Dr. Copot explained the basics of P-values, including that 0.05 is an arbitrary benchmark, and that it can’t tell you anything about the size of an effect, its validity, or reason behind it. If that still sounds a bit confusing, it’s time to queue the memes about scientists being unable to explain P-values in an intuitive way. We think Dr. Copot’s explanation would be in the top quantile of that distribution at any rate. Even if math is fun, it isn’t always intuitive.

Dr. Copot also talked about sample sizes and power analysis (one such online calculator I’ve used many times here: https://www.evanmiller.org/ab-testing/sample-size.html), but then moved on to talking about Bayesian methods. Traditional A/B tools (like Google Optimize, RIP) have typically used Frequentists methods like we’ve been talking about with P-values. Newer tools have folded in some Bayesian methods, which thankfully are a little more intuitive, if perhaps more mathematically & computationally expensive.

Finally, we talked about how privacy regulations, sampling, and cookie limitations can make doing these kinds of experiments more difficult. One way around these limitations is to use paid platforms like Prolific where you can make your own sample group and run a group of fully consented users through an experiment of your choosing.

Please join us next month when Lauren Burke-McCarthy will talk about how to succeed as a solo data scientist.

Our March event featured Bill Balderaz from Futurety getting us up-to-date on the impending demise of third party cookies. As most of us have heard, third party cookies in Chrome are scheduled to be turned off later this year. What we’re less sure about, is what the heck is coming in to replace these cookies?

Love them or hate them, third party cookies have been a fundamental building block of the $600B+ digital ads ecosystem for years — and this turndown will be one of the biggest changes in the history of of the industry. While browsers such as Safari and Firefox have blocked third party cookies for years, it’s Chrome’s impending Q3 2024 turndown of these cookies that has caught the industry’s attention due to Chrome’s dominant marketshare (64%).

So what comes next? Surely we’re not going back to completely un-targeted “punch the monkey” style ads… One answer is first party data, where instead of relying on a third party cookies tracking our cross-site activity, we rely upon our own customer data. What does that mean in practical terms?

Bill gave us an example of how first party data can be used to build audiences with Futurety’s own product HUCKLE.

Per usual, the CBUSDAW audience had plenty of good questions. As we move towards this (third-party) cookieless future, we’re all  still collectively figuring out what exactly this means and how to balance user privacy with effective marketing solutions.

And, action shots from the event. The main author of this recap neglected to note that he had some fun with the event by doing a LinkedIn poll as a competition between local (real) cookie purveyors. Lion Cub’s won the poll and, as a result, were provided at the event, along with a “cookie consent” form. The event photographer had both a picture of that consent form and edit access to this recap, so he snuck this paragraph in when he added the photo gallery below!

Please join us next month when we have a panel of AI aficionados on tap for an audience-driven talk on the future of data science and AI.

Lots of analysts (the author of this recap included) once believed that multi-touch attribution was the long-awaited answer to John Wanamaker’s famous question about which half of our ad spend was wasted. Except even before we started losing lots of data, there were some serious problems with MTA that we’d been turning a blind eye to. In this new era of cookie restrictions and data privacy regulations, MTA has become even more problematic.

Enter MMM. Jim Gianoglio, founder of Cauzle Analytics, gave us in introduction to marketing mix modeling (MMM) — another way to look at answering this same eternal marketing question about what channels are and aren’t working. MMM has been around for a long time (it was so popular in the ’90s that it was immortalized in multiple songs), but the methods and practices have matured and it can really be useful for a high-level understanding of performance for many organizations.

Using MMM doesn’t mean you have to stop using MTA either! MTA can still give you information on a more granular level, and then MMM can help with overall strategy.

If you’re looking to learn more about MMM, check out Jim’s mailing list, podcast, and Slack at mmmhub.org.

https://www.slideshare.net/JasonPacker/cbusdaw-october-23-marketing-mix-modeling

Google’s search results pages (SERPs) are the lifeblood of site traffic for most companies. Organic plus paid search provides 68% of site traffic on average (with organic being the large majority of that), but Google (and Bing’s) experiments with included generative AI-powered search results may change things up!

Lincoln Rinehart, Sr. Director of SEO at Adept Marketing, helped provide some reassurance to those of us worried about how Google’s inclusion of AI into these SERPs might affect our traffic.

Even though Bing beat Google to the punch on this, and Google seriously flubbed their initial response, Google still dominates search engine market share. Any changes Google makes to their search experience can change the kind of traffic your site receives in a big way.

This kind of change seems pretty concerning, but Lincoln pointed out a number of ways in which it could be good for SEO and good for your site (even if you get less traffic).

• Changes the focus of SEO away from trying to “game the system” and towards how to benefit the users that do come to your site.
• SEO changes from a giant checklist approach to a holistic cross-disciplinary practice.
• Could decrease the amount of junky content created solely for SEO.
• Traffic that does come will be much more qualified.

Additionally, Google says AI-generated content is ok, as long as it’s not made explicitly for the purpose of manipulating rankings.

Lincoln also was careful to point out that this change may not even happen! This is still very much an experiment on Google’s side and they have some significant issues to work out before they decide if (and how) they might release this more widely.

Lincoln has also provided his slides! We promised not to dump all of the text out of the slides as white-on-white text at the bottom of the page.

And, just in case Google decides the presence of images makes a site more authoritative, we’ve got pics, too:

April’s meetup saw Jeewan Singh and Tomy Rhymond of Slalom Columbus give us an introduction into the concept of DataOps.

We all deal with data on a regular basis, though we usually are focused on only a small part of the whole lifecycle process of that data. Thinking more broadly about the whole journey of data, from ingest to insight, requires broadening our horizons past the details of analytical tactics towards a more holistic approach.

How do we better deal with managing that whole lifecycle? That’s where we need to change our mindsets and move towards the kind of agile processes that define DataOps.

The traditional way of dealing with data starts from a fundamental assumption that the process and business requirements are static. A naive idea that once we figure out how to ETL the data and get the business the reporting it needs, that we’re done. Of course we all know that’s not how things work in the real world, thus we need to have methods to deal with this constant change and evolution. We’re still allowed to complain when the business stakeholders change the requirements just when we’re almost done with the project (that’s a basic human right), but perhaps if we’ve built more flexibility into our process from the start that won’t mess us up quite as much.

Automation, continuous integration and delivery, test cases right from the start, infrastructure as code rather than clickops — these are the kinds of things that are at the heart of DataOps. If you’ve ever worked within DevOps you’ll find the approach to be similar, but tweaked and extended for the specific needs of the data world.

In case you missed it, Tomy and Jeewan have also generously provided their slide deck:

https://www.slideshare.net/JasonPacker/dataops-cbusdaw-april-23