A/B test segment analysis compares treatment effects within meaningful groups, but small post-hoc slices are exploratory unless the segment and analysis were planned with enough traffic.
Start with the overall result
Confirm that assignment, attribution, and the primary outcome are sound before slicing the data. Segment views cannot rescue an experiment with broken measurement.
Prefer planned segments
Choose device, country, source, or new versus returning status before launch when the hypothesis predicts a different response. Plan enough eligible traffic for the segment if it may drive the decision.
Look for interaction, not two labels
A win in one group and a non-win in another does not automatically prove the groups differ. Compare the estimated treatment effects and uncertainty directly rather than comparing separate confidence badges.
Use exploration to plan the next test
Treat an unexpected segment pattern as a lead, especially when the sample is small or many slices were checked. Run a targeted follow-up experiment before shipping a complex personalized rule.
Run the test in Shopify
Sofcy supports URL, template, theme, and custom JavaScript API experiments.
Start free trial