Compare campaign results without confusing audience mix with performance Skill Workbench · 2026-10-23 · Original fictional teaching material Build a campaign comparison in three steps: choose the denominator, compare like audience groups, then calculate a common-mix scenario and write a defensible reporting note. For a marketing or reporting beginner who can divide small numbers and read a table. A calculator is enough; no analytics account is required. - Calculate a pooled rate from counts and distinguish percentage points from a relative change. - Explain why the observed aggregate and within-segment comparisons can lead in different directions. - Apply the same explicit audience weights to both campaigns and separate that scenario from a causal claim. CASE CONTRACT 1. All visitors, campaigns and outcomes are fictional. Each row is a disjoint group; each visitor can convert at most once in its row. The definitions and observation window are identical within each supplied table. 2. Rate (%) = conversions / visitors × 100. Enter percentages as 15, not 0.15. A pooled campaign rate uses summed conversions divided by summed visitors. 3. A common-mix scenario uses the specified segment weights for both campaigns. Weights add to 100%; it is not the observed campaign total or a forecast. 4. These groups were not randomly assigned. Neither pooled rates nor a weighted comparison establish a causal winner. No uncertainty interval or statistical-significance claim is requested. 1. Choose the denominator before comparing rates Recover counts, segment rates and the observed campaign total. A conversion rate answers a question about a defined group. Before calculating it, identify who was eligible to appear in the denominator, what counts as a conversion and the observation window. A percentage without those definitions is difficult to compare. Here every row supplies both counts, uses one fixed window and contains distinct visitors, so rows can be added without counting a visitor twice. Calculate a segment rate by dividing its conversions by its visitors. To obtain an observed campaign rate, add the conversions and visitors separately, then divide those totals. Averaging two percentages gives equal weight to their groups, even if one group contains four times as many people. That average answers a different question. Keep the original counts beside every derived percentage so someone can reconstruct the calculation. Returning visitors may behave differently from new visitors. If one campaign reaches more returning visitors, its observed total can rise without a comparable improvement inside either group. First describe the audience composition; do not immediately attach a performance explanation. The pilot below is a calculation exercise, not evidence that either fictional campaign would perform the same way for another audience. Use precise units when describing a gap. A move from 12% to 15% is an increase of 3 percentage points; its relative increase is 3 divided by 12, or 25%. Those are two descriptions of the same change. This course asks for rates and percentage-point gaps, so enter the displayed percentage number rather than a decimal proportion or a relative-change figure. Worked example: add the counts, then divide One campaign has 200 new visitors and 20 conversions, plus 50 returning visitors and 20 conversions. The segment rates are 10% and 40%. The observed rate is (20 + 20) / (200 + 50) × 100 = 16%. The simple average, 25%, describes a hypothetical equal mix instead of the 250 observed visitors. Use the pilot counts to calculate the requested measures. The returning-visitor share is returning visitors divided by all visitors in that campaign. Pilot campaign counts campaign | segment | visitors | conversions A | new | 400 | 40 A | returning | 100 | 50 B | new | 200 | 30 B | returning | 300 | 120 YOUR EXERCISE Pilot A new-visitor rate (%): __________________ Pilot A returning-visitor rate (%): __________________ Pilot A observed rate (%): __________________ Pilot B observed rate (%): __________________ Pilot B returning-visitor share (%): __________________ Which campaign has the higher observed pilot rate?: __________________ A: A B: B equal: They are equal Your explanation: Write one sentence naming the denominator for A and another distinguishing B’s observed rate from a causal claim. FEEDBACK AND WORKED SOLUTION A’s segment rates are 10% and 50%; its pooled rate is 90/500 = 18%. B’s pooled rate is 150/500 = 30%, with a 60% returning audience. The observed gap is 12 percentage points, but audience composition and other differences still need examination. Pilot A new-visitor rate (%) → 10. 40 / 400 × 100 = 10%. Pilot A returning-visitor rate (%) → 50. 50 / 100 × 100 = 50%. Pilot A observed rate (%) → 18. A has 90 conversions and 500 visitors: 18%, not the 30% unweighted average. Pilot B observed rate (%) → 30. B has 150 conversions among 500 visitors: 30%. Pilot B returning-visitor share (%) → 60. 300 of B’s 500 visitors are returning: 60%. Which campaign has the higher observed pilot rate? → B. B leads 30% to 18% in the observed total. This answer describes these visitors, not a causal effect. TRANSFER QUESTION If the table instead counted sessions, with some people appearing repeatedly, what word and denominator would need to change in the report? 2. Diagnose a reversal in the aggregate result Compare the observed totals with the same two segments under equal weights. The next dataset is separate from the pilot. Start its calculation again rather than carrying over a pilot percentage. For each campaign, write one line for the new-visitor rate, one for the returning-visitor rate and one for the pooled rate. This separates the observed behavior within a segment from the proportion of the whole audience that the segment represents. In these counts, campaign B has the higher pooled result even though campaign A has the higher rate in each segment. There is no arithmetic contradiction. B has many more visitors in the segment with higher conversion rates, while A has many more in the lower-rate segment. Different weights can reverse the aggregate ranking. A useful report retains both views rather than selecting whichever supports a preferred conclusion. An equal-mix calculation asks a deliberately different question: what rate would these segment rates produce with half the audience in each segment? Multiply each segment rate by 0.5 and add. Use the same weights for A and B. Label the result as a scenario; the calculation does not change the actual visitors or conversions and does not remove every difference between the groups. Audience mix is only one possible explanation to investigate. The campaigns may also differ in timing, placement, eligibility or measurement. An observational comparison cannot isolate those effects by arithmetic alone. NIST describes controlling important nuisance factors through blocks in experimental design; merely grouping this existing table is not equivalent to running such an experiment. A reporting note should identify that next design question without claiming a proven winner. Worked example: keep the audience weights visible Suppose segment rates are 12% for new visitors and 36% for returning visitors. A 50/50 scenario is 0.5 × 12 + 0.5 × 36 = 24%. If the observed audience were 80% new and 20% returning, the corresponding weighted rate would be 16.8%. The segment rates are unchanged; the weights explain the difference. A marketing team wants to call campaign B the winner because its overall conversion rate is higher. These fictional campaign groups were not randomly assigned and reached very different proportions of new and returning visitors. Prepare a short comparison that shows the observed totals, an equal-mix calculation and the claim the evidence supports. Observed campaign groups campaign | segment | visitors | conversions A | new | 900 | 90 A | returning | 100 | 60 B | new | 100 | 8 B | returning | 900 | 450 YOUR EXERCISE Observed rate for A (%): __________________ Observed rate for B (%): __________________ A under a 50/50 audience mix (%): __________________ B under a 50/50 audience mix (%): __________________ Which conclusion is supported?: __________________ b-causes: B caused a higher conversion rate. a-causes: The adjusted rate proves A causes more conversions. mix-matters: The comparison changes with audience mix; this data alone does not identify a causal winner. Your explanation: Write a short reporting note that distinguishes the actual observed rates from the hypothetical equal-mix rates, and names a useful next investigation. The note is not automatically graded. FEEDBACK AND WORKED SOLUTION A: (90 + 60) / (900 + 100) = 15%. B: (8 + 450) / (100 + 900) = 45.8%. Within new visitors A is 10% versus B’s 8%; within returning visitors A is 60% versus B’s 50%. A shared 50/50 mix gives 35% and 29%. This reversal explains why the overall rate alone is an incomplete campaign comparison. Observed rate for A (%) → 15. A has 150 conversions among 1,000 visitors: 150 / 1,000 × 100 = 15%. Observed rate for B (%) → 45.8. B has 458 conversions among 1,000 visitors: 45.8%. Returning visitors form 90% of its audience. A under a 50/50 audience mix (%) → 35. A’s segment rates are 10% and 60%; equal weighting gives 0.5 × 10 + 0.5 × 60 = 35%. B under a 50/50 audience mix (%) → 29. B’s segment rates are 8% and 50%; equal weighting gives 29%. Which conclusion is supported? → The comparison changes with audience mix; this data alone does not identify a causal winner.. B leads in the observed total, but A leads in each segment and in the equal-mix scenario. Neither comparison removes all possible confounding in an observational dataset. TRANSFER QUESTION Which two audience shares would you display beside the pooled rates so a reader can see why the aggregate comparison reverses? 3. Write a comparison for a stated planning audience Use a new 70/30 common mix and describe what the calculation can support. A planning team may have a particular audience in mind rather than an equal mix. The final fixture supplies new counts and requests a scenario with 70% new visitors and 30% returning visitors. Derive each new segment rate from this table first. Applying the old rates to new data, or using each campaign’s observed weights, would silently answer a different question. Apply the common weights to each campaign independently: 0.7 times its new-visitor rate plus 0.3 times its returning-visitor rate. A positive gap defined as A minus B means A is higher in that scenario. If you express the scenario for 1,000 visitors, multiply its percentage by 1,000 and divide by 100. That count is a calculated illustration, not an additional set of observed conversions. Keep three reporting layers visible: observed counts and rates, the common-mix calculation, and the remaining decision. A scenario can show how a chosen audience composition changes a comparison. It cannot guarantee future response, supply uncertainty intervals or prove that creative content caused a difference. Do not promote a tidy point estimate into a budget promise that the data did not measure. For a stronger next investigation, first agree on eligibility, a conversion definition, a fixed observation window and an outcome measure. If a randomized comparison is appropriate and feasible, discuss assigning campaign versions within important audience groups and planning the analysis before results arrive. Sample size, implementation checks and uncertainty need their own work; this worksheet neither performs that design nor authorizes a real campaign change. Worked example: turn a scenario rate into an illustrative count With segment rates of 20% and 40%, a 75% new / 25% returning scenario gives 0.75 × 20 + 0.25 × 40 = 25%. For 800 hypothetical visitors this corresponds to 200 conversions. Write “scenario equivalent” beside 200; do not add it to an observed conversions column. Use only the final table and the specified 70% new / 30% returning mix. Enter A-minus-B as a percentage-point gap. The illustrative count uses 1,000 hypothetical visitors. Final campaign counts campaign | segment | visitors | conversions A | new | 600 | 72 A | returning | 400 | 200 B | new | 300 | 30 B | returning | 700 | 280 YOUR EXERCISE Final A new-visitor rate (%): __________________ Final B returning-visitor rate (%): __________________ A under the 70/30 common mix (%): __________________ B under the 70/30 common mix (%): __________________ Common-mix A minus B (percentage points): __________________ Scenario-equivalent conversions for A per 1,000 visitors: __________________ Which next-step statement is supported?: __________________ guarantee: Guarantee 234 conversions in the next campaign. causal: Declare A the proven causal winner and move all budget. plan-comparison: Report the scenario and investigate a suitably designed comparison before a causal decision. Your explanation: Draft a reporting note with the 70/30 assumption, both scenario rates, the gap’s units, and one concrete item to agree before a randomized comparison. FEEDBACK AND WORKED SOLUTION The final segment rates are A: 12% and 50%; B: 10% and 40%. The 70/30 scenario produces A=23.4%, B=19%, a 4.4-percentage-point gap. A’s scenario equivalent is 234 per 1,000 hypothetical visitors. This is not a forecast or causal estimate; agree on eligibility, outcome and window before planning an experimental comparison. Final A new-visitor rate (%) → 12. 72 / 600 × 100 = 12%. Final B returning-visitor rate (%) → 40. 280 / 700 × 100 = 40%. A under the 70/30 common mix (%) → 23.4. 0.7 × 12 + 0.3 × 50 = 23.4%. B under the 70/30 common mix (%) → 19. 0.7 × 10 + 0.3 × 40 = 19%. Common-mix A minus B (percentage points) → 4.4. 23.4 − 19 = 4.4 percentage points; this is not a relative percentage increase. Scenario-equivalent conversions for A per 1,000 visitors → 234. 1,000 × 23.4 / 100 = 234 hypothetical conversions, not an observed count. Which next-step statement is supported? → Report the scenario and investigate a suitably designed comparison before a causal decision.. A common mix answers a defined arithmetic question. A stronger causal decision needs a suitable design and additional evidence. TRANSFER QUESTION If the intended audience became 30% new and 70% returning, which inputs would stay fixed and which calculation would you rerun? PORTFOLIO REVIEW Keep all three exercise records and explanations. Fixed checks can be scored; explanations are ungraded. Do not infer occupational certification. SOURCE NIST: Randomized block designs https://www.itl.nist.gov/div898/handbook/pri/section3/pri332.htm Background for tracking important nuisance factors and varying the factor of interest within comparable blocks. This observational worksheet is not a randomized block experiment. O*NET 13-1161.00 task 5438 · 08/2026 · Measure the effectiveness of marketing, advertising, and communications programs and strategies. O*NET 13-1161.00 task 5434 · 08/2026 · Prepare reports of findings, illustrating data graphically and translating complex findings into written text. https://www.onetonline.org/link/summary/13-1161.00 O*NET 31.0, USDOL/ETA, CC BY 4.0. Skill Workbench authored the teaching and fictional practice; no endorsement.