For our October event, Bryan Huber walked us through how he’s developed and deployed a custom GPT within his organization. As the Global VP of Digital Marketing and Analytics at Comfort Keepers, his team fields a wide variety of marketing questions from his organizations franchisees at different levels of technical sophistication. To empower those questioners as well as lighten the load on his own team, Bryan developed a custom GPT that leverages the question-answering power of ChatGPT, but also grounds it in his own organization’s best practices and adds some guardrails.

So what is a custom GPT anyways? It’s regular ChatGPT, but with a series of available customizations, including:

• Custom instructions as you would in general ChatGPT, but sharing them across all users of all chats with the custom GPT.
  • This helps your users engineer better prompts, and put them on the right path from the start with each conversation.
  • Instructions can also help control what the custom GPT can do, steering users away from problematic areas.
• Uploading your own documents to a knowledge base.
  • For example, you could make your own internal best practices documentation or research interactive by uploading them to a custom GPT.
  • These uploaded documents serve as a way to ground conversions in your own vetted information and also make those documents searchable.
• Restrict the features users have access to.
  • Bryan shared some examples of egregiously poor marketing images created in ChatGPT. By turning off the image generation feature in the custom GPT, this prevent users from making those images and instead guides them to using the custom GPT to help create marketing text and come up with ideas rather than making slop images that might not follow organizational guidelines.
  • Similarly removing the web search capability can help focus the output on the vetted knowledge base and not just whatever web search can dig up.
• Create “actions”, in the form of external API calls.
  • For example if you wanted up-to-date currency conversion numbers in your custom GPT you could connect to an external API using your own API key and get accurate numbers there rather than relying upon outdated training data or slow web search (which might be disabled in your GPT!)
  • Part of Bryan’s roadmap is to connect the custom GPT to the Google Ads API which allows its users to get detailed real-time information about things like CPC costs of keywords.

All of this for zero additional dollars, as custom GPTs are included on all paid ChatGPT plans!  Please note that on lower-level plans the custom GPTs you create will be public by default and include their conversation data into future OpenAI training data (the latter can be turned off under “Additional Settings” once the GPT is created).

This functionality is not exclusive to OpenAI, Claude offers similar functionality in “Projects” and Google Gemini does in “Gems”.

He also walked us through his journey of rolling out this tool to users, from early adopters to a happy user base of over 300 users.

Bryan also provided us with his slides! Since he’s also an organizer of this event, he would’ve had a stern conversation with himself if he had not.

As always, the crowd had lots of practical questions!

For our September events we welcomed sponsor Piwik PRO to Columbus for not one but two events!

On Wednesday evening we had Jason Packer of Quantable Analytics talking about tracking methods, and Piotr Słonina of Piwik PRO talking about how their product fuses different kinds of tracking methods together in reporting.  Piotr and Marcin Pluskota flew all the way to Columbus from Wrocław, Poland for the event! We’d like to apologize for their flight delays and inform them that a multi-hour delay in Atlanta is, in fact, a rite of passage.

Due to an unforeseen scheduling snafu we had our first “al atrio” presentation in the atrium of Rev1 rather than in our typical room location, but we made it work. Jason and Piotr pitched a double-header of a presentation that covered things like:

• • How can we track anonymized users in a privacy-respectful way, even if those users decline cookies?
    Our answer – by not “tracking” those users in a way that lasts beyond a short period of time, or in ways that could clearly identify a particular user. Jason felt this should probably all be done with cookies, but the regulations around cookies have caused many vendors to look for work-arounds. Some of those work-arounds are more privacy-respectful than others.
  • Where is the line between a session hash (like IP + User-Agent) and browser fingerprinting?
    Our answer – it’s not a distinct line, but tools that use invasive methods and provide durable fingerprints are on the wrong side of it, at least when used for tracking.
  • How do tools like GA4 and Piwik PRO handle these different types of users: logged-in, cookied, and non-cookied?
    Our answer – GA4’s default blended user identity has a tiered hierarchy of user id, cookies, and modeling based upon “cookieless pings”. Piwik PRO has a more flexible solution that uses session hashes and allows individual sites to choose their own adventure when it comes to dealing with the so-called “consent gap”.

Jason’s Slides:

Piotr’s Slides

For those looking to learn even more, we also hosted a follow-up seminar on Thursday. This was the first official Piwik PRO meetup in the US, and we were were proud to have done that in Columbus! Some of the highlights from Thursday included:

• A deeper discussion of the Piwik PRO suite including Tag Manager, CMP, and CDP.
• Discussion of Piwik PRO’s migration away from their freemium model. Pricing now starts at $40/mo, more pricing info here.
• A sneak preview of how Piwik PRO will be integrating the Fraud0 anti-bot system into their platform.
• Delicious food from Brassica, and a very restrained amount of griping about GA4 from the audience.

Still hungry for even more Piwik PRO? Check out the upcoming Piwik PRO day on October 21, a virtual event featuring speakers such as Simo Ahava, Brian Clifton, Steen Rasmussen, as well as CBUSDAW veterans Matt Gershoff and Josh Silverbauer!

Some pics from both events:

Did you know that Data and Analytics Wednesdays were initially started (not in Columbus) as pure socializing/networking events? From the get-go, we’ve always included an educational component in ours because, well, you know, midwestern-purposefulness or something.

We break that format once a year in December with a content-free event. And, in 2025, we tried to break it a second time by having our July event be a group outing to a Columbus Clippers baseball game.

Alas! We failed to go entirely content-free because baseball, after all, is the OG analytics-oriented sport! Bill James! Sabermetrics! Moooonnnnneeeeyyyyybbbbbaaaaalllll!

So we had a little bit of content. Most of the attendees arrived early so that a few members of the Clippers scoring team could pop down to our seats for a little Q&A that was as fascinating as it was informal!

And then we watched the game!

The discussions in our seats (alas!) drastically outperformed the Clippers’ performance on the field. The Louisville Bats were up 7-3 halfway through the 4th inning, and the score remained there for the duration of the contest.

But, with each attendee armed with some Clippers Cash, a koozy, a pass to the Tansky Club behind home plate, and great weather, an excellent time was had by all!

Our June event featured Ash Lewis from Ohio State Linguistics talking about why LLMs give us incorrect information so often, and strategies we can use to reduce this behavior.

While the standard term for this is of course “hallucination”, Ash pointed out that the term “confabulation” more accurately describes what is happening. Hallucination implies that the LLM is incorrectly perceiving something, however what we’re describing is not misperception. It’s AI creating statistically probable, yet incorrect information.

Wikipedia agrees with Ash and described hallucination thusly:

This term draws a loose analogy with human psychology, where hallucination typically involves false percepts. However, there is a key difference: AI hallucination is associated with erroneously constructed responses (confabulation), rather than perceptual experiences.

Of course as any linguist would point out, we don’t get to prescriptively say how language should be used… so we’re pretty much stuck with “hallucination”.

Whatever the term, this happens because everything that an LLM creates is simply what is statistically probable. Output which is also true is coincidental to the process. In other words: it’s guessing about everything, it just happens to be right enough to be very useful.

If you think that this is an issue of the past limited to older models, here’s current example from o4-mini:

During the talk we verified our group’s nerd cred by knowing how this guy found out who his dad was.

So how do we reduce this problem as much as possible? Here’s Ash’s helpful field guide notes:

Our prompt about our group’s history has set ourselves up for failure by breaking most of Ash’s rules.
We made the following mistakes:

• Not breaking down (decomposing) our ask into small components. We didn’t ask for a more granular question, like the year the group started or a list of previous topics, we asked for a whole history.
• Not encouraging the LLM to check its own work, step through its reasoning, or provide sources, or indicate uncertainty. As soon as we follow-up and ask things like, “what is your source for attendance doubling” it will say it has none.
• Not letting the LLM search the web (this is a form of RAG). The o-series models from OpenAI are pretty good at knowing when they should do this, and likely would have done so in this scenario.
• Not getting more than one response. When asked a second time with the same prompt, it said it didn’t have enough information to give a response.

Ash then dug into details about the work she is doing with COSI (the Columbus science museum) creating an AI agent that can help visitors with questions about the museum. This work attempts to limit hallucinations as well as provide the museum a more affordable and privacy-friendly solution than just sending things to ChatGPT.

She also helpfully has provided us with her slides!

As usual — especially when it’s a talk on AI — the crowd had a lot of great questions!

And a few pictures from the event. All totally real. Really. Maybe?

Our May 2025 event was all about communication. Specifically, it was about all the various flavors (or “genres”) of speaking publicly, be that on a conference stage, in a conference room to a group of stakeholders, in a client’s office for a high-stakes pitch, or to a camera on a video call where the participants may or may not have their cameras turned on.

Ruth Milligan and Acacia Duncan, two thirds of the author trio for The Motivated Speaker: Six Principles to Unlock Your Communication Potential walked a 50-strong audience of engaged attendees through an introspective and interactive exercise in identifying their “shaded (ineffective) habits” when it comes to public speaking.

What is a “shaded habit?” It’s something (or multiple things) that every individual has picked up over the course of their lives that feels natural and comfortable even as it gets in the way of effectively communicating.

The bad news? Everyone has them.

The good news? No one was born with any of them, so whatever those habits are for an individual, they can be identified and unlearned (or, at least, sufficiently mitigated).

Ruth and Acacia opened the session with some ripped-from-the-headlines ripped-from-their-clients-with-identifying-details-removed of communication failures and led a discussion with the attendees as to the root causes of those failures. From there, they prompted everyone to think about their own rhetorical style and what they could identify as their shaded habits. Attendees jotted their thoughts on Post-it notes that Ruth (and Tim) collected and grouped for review and discussion:

The most commonly identified habit? Overuse of “filler words”: “um”, “like”, “you know”. How to address it? Breathe! And shorter sentences. With pauses that give the period its due. [artistic license intentionally taken on the preceding sentence fragments. To make a point. Just did it again.]

Other types of shaded habits that came up included: rambling, talking too fast, not minding the clock, going into too much detail, not thinking sufficiently about the audience’s needs (what questions they want answered rather than what information the speaker wants to share), and more!

Some of the shaded habits were diagnosed as being different forms of stress responses, of which there are fundamentally four distinct flavors: fight, flight, freeze, and fawn. The tricky thing about stress responses is that they’re not going to just go away. They’re going to happen. But, by recognizing what our default flavor of stress response is, we can prepare for how to deal with it, be it by lifting heavy weights just before speaking (for real…!) or “grounding” ourselves (anchor feet to the floor, hands palm down on the table if sitting) or repeating a mantra (“This too shall pass” may work, but it can be whatever works for you!).

Following the exercise and discussion, we had a drawing to give away five copies of The Motivated Speaker to lucky audience members and then had a book signing (20% of the proceeds from the book sales went to Sanctuary Night).

As the emcee noted at the start of the meetup, our goal is for every event is for attendees to take something away that they can put into action within a week, and our May event absolutely delivered on that front!

Additional pictures from the meetup are below:

Our April 2025 event featured Ty Henkaline talking about work that he has done with non-profits in Franklin County to help better understand homelessness. Ty has been working with Smart Columbus’ Columbus Community Information Exchange Initiative (CIE) to produce research that utilizes data from the Mid-Ohio Food Collective (MOFC) and the Community Shelter Board (CSB) to help us better understand this growing problem.

As Ben Franklin — for whom our county is named after — famously said, “An ounce of prevention is worth a pound of cure.” No question this is doubly true for homelessness, and providing early warning to agencies that help prevent these crises is a great use of data.

But as Ty pointed out, this data is not always easy to come by. Our existing systems were all built separately, and data integration was never a priority. Sensitive data about at-risk individuals is a challenging arena to work in, and Ty emphasized both the value of having partners that were truly invested in making this system work as well as the potential value of additional data sources.

This “spike chart” was a huge hit with the audience, and shows the following things:

1. A growing increase in services usage (in particular food banks) was a strong leading indicator of a homelessness.
2. With far fewer data sources compared to LA, Franklin County was able to see a very similar effect. How often do you see that in data modeling?
3. Individuals experiencing first-time homelessness continue to need an elevated level of services after the initial crisis. This reinforces the notion that prevention can do a lot to improve the overall load on the system.

As promised, Ty provided us with his slides, which contain lots of links and some calls to action! Try scrolling to navigate the slides, or check out the direct link here.

If you’re interested in helping or learning more, please feel free to message Ty on LinkedIn.

Check out the engaged audience!

headstone from Google Graveyard

When we last had Melanie Bowles as a speaker in 2019, she lead an informative session on building a sustainable experimentation strategy. Since nothing at all has changed since 2019, we just replayed that talk and then everyone went home. While I’m obviously being facetious — much of the strategies that Melanie laid out in that talk are still very relevant! The landscape has changed a lot since then, from big changes in browser privacy and client-side technology to the shutdown of the most widely used tool in the industry, Google Optimize.

While there’s no clear successor to Optimize, there are many good testing tools out there, including popular options like: AB Tasty, Convert.com, Visual Website Optimizer, Optimizely, etc. Most of these tools do also offer integration with GA4.

Deployment count of tools via BuiltWith data

As you can see from this chart based upon deployments in the top 1M sites, none of these tools are exactly catching fire with popularity. A big reason for that may be that none of them are free for unlimited usage like Optimize was. Melanie also pointed out that A/B testing and similar functionality like feature flagging has in some cases moved into all-in-one suites like Amplitude, Salesforce, etc. The sunset of Optimize can be looked at as a chance to mature our A/B testing practices, and focus them where they can have the most impact.

Melanie also suggested that we embrace AI tools, especially on the ideation side of testing. There’s no substitute for human expertise when building out tests, but it’s certainly not cheating to let ChatGPT come up with some potential variations for your test! Just remember to give the AI as much context as you can. Melanie ran through a quick example which included providing the AI with a customer persona, which you can find in her slides below!

As a new twist for the meetup in 2025, we’re making a donation to a speaker-selected non-profit at each event. Melanie chose to designate Columbus Cultural Orchestra — a program for young people 13-25 to develop their musical skills and enhance diversity in orchestral music — for a $250 donation!

Our first event of 2025 was a book release party for CBUSDAW’s very own Tim Wilson!

If you’ve ever been to a CBUSDAW event before (or listened to his podcast the Analytics Power Hour) you’ll know that Tim has a lot of things to say about analytics. Smart things, funny things, cranky things, etc. To our benefit, he’s organized many of these thoughts together into a book (with co-author Dr. Joe Sutherland) called “Analytics the Right Way: A Business Leader’s Guide to Putting Data to Productive Use“.

This is an excellent book that may be targeted towards “business leaders” with its title, but can also be incredibly useful for analysts themselves in terms of how to think about doing analytics in a productive way. There’s a lot of books out there covering tools, methods, and technology — but Tim and Joe’s book stands out in being about actually using analytics within an organization to further business goals. (NB: this is Jason writing this recap, and not Tim awkwardly hyping his own book in the third person. Also Tim I promise I will get around to writing that Amazon review at some point.)

But this wasn’t just a book signing with free beer, Tim did also give a talk about some of the topics he covered in the book! We had a great crowd, with friends and colleagues of Tim’s coming in from as far as Chicago, Nashville, and Boston.

We also had Jim Gianoglio jump in behind the camera (Tim’s normal job) and get some great action shots:

For our November 2024 event, we brought Josh Silverbauer in from Philly to talk about behavior analytics (in the form of MS Clarity) and marketing analytics (in the form of Piwik PRO) and when you might want to use each one.

Since Josh is well-known for writing parody songs to introduce speakers, here at CBUSDAW we flipped the script on him and opened the event with a surprise parody song about Josh.

We present “It’s Josh Silverbauer” to the tune of “In the Midnight Hour” by Wilson Pickett, sung by Jim Gianoglio and featuring Jason Packer on kazoo.

While Josh is a fan of both Clarity and Piwik PRO, he’s pointed out that he’s not paid anything by either organization… so he’s free to tell it like it is. And “how it is” is that both tools are great additions to any analyst’s arsenal — and with a generous free tier for Piwik PRO and a totally free product with Clarity there’s not much barrier to entry.

Josh pointed out how the two tools can easily used to supplement each other. For example, one could use Piwik PRO to find a particular aggregate group of users that aren’t converting well, and then review those users’ entire sessions with the session recording feature in Clarity. Or check the heatmap of the landing page for that same group.

If you’ve used session recording tools in the past you know that it can be pretty tedious to watch the recordings one-by-one. It’s like, “geez, just click the button already user #23341, it’s RIGHT THERE”.

Microsoft has recently integrated CoPilot into Clarity so it can now help save you from watching a ton of videos and instead can summarize and do some basic analysis for you.

Josh described Piwik PRO as “what Google’s Universal Analytics 2.0 could have been if GA4 didn’t exist”. If you were a serious user of our dearly departed UA you’ll feel right at home in Piwik PRO, and you’ll be pleased to see how well thought-out the platform is.

Josh’s slides:

Ok, so Clarity and Piwik PRO are both pretty cool tools, but what about rock operas?

Josh (and Jason) are releasing Volume 2 of Josh’s epic analytics rock opera entitled “User Journey Volume 2: The Custom Dimension” on November 18th.

You can listen to the first volume, “Universal Sunset” now on Spotify and most other streaming platforms.

Finally, don’t forget to join us next month for our yearly holiday event at the Grandview Theater. No speakers, but there will be a movie this year and you can vote on what you’d like it to be!

Disclaimer: Piwik PRO is a sponsor of CBUSDAW — but they only pay for the monthly pizza, not our (or Josh’s) endorsements.

As analysts, we love to optimize everything that we possibly can — so when we have a speaker that gives us a new way to think about testing we are here for it!

For our October event we had Sanjay Tamrakar talk to us about doing geo-testing. Sanjay covered basic methods like traditional pilot testing, to difference in differences, all the way to the current state-of-the-art with Geolift.

Back in the pre-digital era, Columbus was considered to be a top location in the US to pilot test new products, since its demographics closely matched the country as a whole. This allowed companies to try out new products, but only gave marketers an idea of how well a new product might do nationwide, not how much incremental lift different product variations might give or how different Columbus might perform vs. Chicago or Charleston. This kind of granularity requires more powerful tools.

These days there are much more expedient and statistically rigorous methods to test things, like Geolift. Geolift is a an open-source package from Meta that allows the creation of artificial control groups which we can use to test against treatment groups without having to worry as much about building control groups using user-specific information and the privacy issues which that can bring. Geolift’s synthetic control methods can create control groups by amalgamating different untreated areas whose performance was expected to match the treated areas.

There was even some R code showing up on the big screen, which sadly Tim Wilson missed!

Sanjay was also kind enough to provide us with his slides: