If you are an analyst or someone who did some experiment and wondering whether your experiment was successful or not. This tutorial can help you to answer that question.
Why confidence interval?
There are so many kinds of a/b testing, why confidence interval? Because it is easier to compare between experiment and this something we can achieve solely using Tableau. Here is some example, the line chart for the experiment a and c almost overlap each other, so it is difficult for people to know which experiment is faster (faster is better).
Examination of the data.
I generated random data using R where each row represents daily, and each experiment, there are two primary metrics for this tutorial completion rate and average time to complete. You can download the data using this link.
The first step of creating a confidence interval, creating all new calculation.
Let’s start with the first metric, which is average time to complete. Create all this new Calculated Field in Tableau.
- Average Avg Time to Completed, to get average from Avg Time to Completed.
AVG([Avg Time To Completed])
- StDev Avg Time to Completed, to get standard deviation from Avg Time to Completed.
STDEV([Avg Time To Completed])
- Upper CI Avg Time to Completed and Lower CI Avg Time to Completed,these columns to get upper and lower value from Avg Time to Completed.
[Average Avg Time to Completed] + [StDev Time to Completed]
[Average Avg Time to Completed] - [StDev Time to Completed]
- Width CI Avg Time To Completed, this one to get width from Upper CI Avg Time to Completed dan Lower CI Avg Time to Completed.
[Upper CI Avg Time to Completed] - [Lower CI Avg Time to Completed]
Create Confidence Interval Graph.
From these five new calculations, we are ready to create a confidence interval graph. Be prepared, this will be quite long. The first step is:
- Drag Measure Names to Filters and select Lower dan Upper CI Avg Timeto Completed that we created before.
After we filterMeasure Names, next steps are:
- Drag Experiment into Rows and Color.
- Drag Lower CI Avg Time to Completed to the Columns.
- Then change the type of graph to Gantt Bar.
- Drag Measure Values to Columns and using a right-click, activate the Dual Axis.
- Right-click the Lower CI and activate the Synchronize Axis.
- On the Marks tab, choose All and discard Measure Names sothat the color returns based on the experiment.
Almost there, here are some steps to make it perfect:
- On the Marks choose Lower CI.
- Drag Width CI Avg Time To Completed to Size.
- Average Avg Time To Completed to Detail.
- Click Size and reduce it, so the body is smaller than Lower and Upper Confidence Interval.
These steps are to add information about average from that confidence interval.
- Change the tab Data into Analytics.
- Drag Reference Line to the graph.
- Drop Reference Line between Cell and Lower CI.
- There will be a new panel after you dropped it.
- Click Value and change it into Average Avg Time To Completed
The last step is right click in the x-axis and chooses Edit Axis then untick Include Zero. This is the final graph, as we can see, experiment A is faster than C.
If you want we can use this for completion rate, for that we need a new calculation using this:
[Total Completed Booking]/[Total Booking]
The step will be the same. We need to recreate all five new calculation that we did before for Avg Time to Completed.
Here is the final dashboard from this tutorial. I use a parameter to change between these two metrics, time to complete and completion rate.
From this tutorial, we learn about how to create a confidence interval chart and how this chart can give us a quick conclusion.