Free mini-course · 50 minutes · No account needed
Compare campaign results without confusing audience mix with performance
Build a campaign comparison in three steps: choose the denominator, compare like audience groups, then calculate a common-mix scenario and write a defensible reporting note.
For a marketing or reporting beginner who can divide small numbers and read a table. A calculator is enough; no analytics account is required.
What you will make
A three-part comparison portfolio with observed rates, a segment-aware comparison and a planning scenario. Your written reporting note accompanies the checked calculations.
- Calculate a pooled rate from counts and distinguish percentage points from a relative change.
- Explain why the observed aggregate and within-segment comparisons can lead in different directions.
- Apply the same explicit audience weights to both campaigns and separate that scenario from a causal claim.
The exercise contract
All records, policies and outcomes in this course are fictional. The supplied contract determines the expected answers.
- All visitors, campaigns and outcomes are fictional. Each row is a disjoint group; each visitor can convert at most once in its row. The definitions and observation window are identical within each supplied table.
- Rate (%) = conversions / visitors × 100. Enter percentages as 15, not 0.15. A pooled campaign rate uses summed conversions divided by summed visitors.
- A common-mix scenario uses the specified segment weights for both campaigns. Weights add to 100%; it is not the observed campaign total or a forecast.
- These groups were not randomly assigned. Neither pooled rates nor a weighted comparison establish a causal winner. No uncertainty interval or statistical-significance claim is requested.
Recover counts, segment rates and the observed campaign total.
1. Choose the denominator before comparing rates
A conversion rate answers a question about a defined group. Before calculating it, identify who was eligible to appear in the denominator, what counts as a conversion and the observation window. A percentage without those definitions is difficult to compare. Here every row supplies both counts, uses one fixed window and contains distinct visitors, so rows can be added without counting a visitor twice.
Calculate a segment rate by dividing its conversions by its visitors. To obtain an observed campaign rate, add the conversions and visitors separately, then divide those totals. Averaging two percentages gives equal weight to their groups, even if one group contains four times as many people. That average answers a different question. Keep the original counts beside every derived percentage so someone can reconstruct the calculation.
Returning visitors may behave differently from new visitors. If one campaign reaches more returning visitors, its observed total can rise without a comparable improvement inside either group. First describe the audience composition; do not immediately attach a performance explanation. The pilot below is a calculation exercise, not evidence that either fictional campaign would perform the same way for another audience.
Use precise units when describing a gap. A move from 12% to 15% is an increase of 3 percentage points; its relative increase is 3 divided by 12, or 25%. Those are two descriptions of the same change. This course asks for rates and percentage-point gaps, so enter the displayed percentage number rather than a decimal proportion or a relative-change figure.
Try it yourself
Use the pilot counts to calculate the requested measures. The returning-visitor share is returning visitors divided by all visitors in that campaign.
Pilot campaign counts
| campaign | segment | visitors | conversions |
|---|---|---|---|
| A | new | 400 | 40 |
| A | returning | 100 | 50 |
| B | new | 200 | 30 |
| B | returning | 300 | 120 |
Numbers use the units in each label. Blank answers are incomplete. ID lists can use commas, spaces or semicolons.
Read this exercise’s solution
A’s segment rates are 10% and 50%; its pooled rate is 90/500 = 18%. B’s pooled rate is 150/500 = 30%, with a 60% returning audience. The observed gap is 12 percentage points, but audience composition and other differences still need examination.
Apply the idea
If the table instead counted sessions, with some people appearing repeatedly, what word and denominator would need to change in the report?
Compare the observed totals with the same two segments under equal weights.
2. Diagnose a reversal in the aggregate result
The next dataset is separate from the pilot. Start its calculation again rather than carrying over a pilot percentage. For each campaign, write one line for the new-visitor rate, one for the returning-visitor rate and one for the pooled rate. This separates the observed behavior within a segment from the proportion of the whole audience that the segment represents.
In these counts, campaign B has the higher pooled result even though campaign A has the higher rate in each segment. There is no arithmetic contradiction. B has many more visitors in the segment with higher conversion rates, while A has many more in the lower-rate segment. Different weights can reverse the aggregate ranking. A useful report retains both views rather than selecting whichever supports a preferred conclusion.
An equal-mix calculation asks a deliberately different question: what rate would these segment rates produce with half the audience in each segment? Multiply each segment rate by 0.5 and add. Use the same weights for A and B. Label the result as a scenario; the calculation does not change the actual visitors or conversions and does not remove every difference between the groups.
Audience mix is only one possible explanation to investigate. The campaigns may also differ in timing, placement, eligibility or measurement. An observational comparison cannot isolate those effects by arithmetic alone. NIST describes controlling important nuisance factors through blocks in experimental design; merely grouping this existing table is not equivalent to running such an experiment. A reporting note should identify that next design question without claiming a proven winner.
Try it yourself
A marketing team wants to call campaign B the winner because its overall conversion rate is higher. These fictional campaign groups were not randomly assigned and reached very different proportions of new and returning visitors. Prepare a short comparison that shows the observed totals, an equal-mix calculation and the claim the evidence supports.
Observed campaign groups
| campaign | segment | visitors | conversions |
|---|---|---|---|
| A | new | 900 | 90 |
| A | returning | 100 | 60 |
| B | new | 100 | 8 |
| B | returning | 900 | 450 |
Numbers use the units in each label. Blank answers are incomplete. ID lists can use commas, spaces or semicolons.
Read this exercise’s solution
A: (90 + 60) / (900 + 100) = 15%. B: (8 + 450) / (100 + 900) = 45.8%. Within new visitors A is 10% versus B’s 8%; within returning visitors A is 60% versus B’s 50%. A shared 50/50 mix gives 35% and 29%. This reversal explains why the overall rate alone is an incomplete campaign comparison.
Apply the idea
Which two audience shares would you display beside the pooled rates so a reader can see why the aggregate comparison reverses?
Use a new 70/30 common mix and describe what the calculation can support.
3. Write a comparison for a stated planning audience
A planning team may have a particular audience in mind rather than an equal mix. The final fixture supplies new counts and requests a scenario with 70% new visitors and 30% returning visitors. Derive each new segment rate from this table first. Applying the old rates to new data, or using each campaign’s observed weights, would silently answer a different question.
Apply the common weights to each campaign independently: 0.7 times its new-visitor rate plus 0.3 times its returning-visitor rate. A positive gap defined as A minus B means A is higher in that scenario. If you express the scenario for 1,000 visitors, multiply its percentage by 1,000 and divide by 100. That count is a calculated illustration, not an additional set of observed conversions.
Keep three reporting layers visible: observed counts and rates, the common-mix calculation, and the remaining decision. A scenario can show how a chosen audience composition changes a comparison. It cannot guarantee future response, supply uncertainty intervals or prove that creative content caused a difference. Do not promote a tidy point estimate into a budget promise that the data did not measure.
For a stronger next investigation, first agree on eligibility, a conversion definition, a fixed observation window and an outcome measure. If a randomized comparison is appropriate and feasible, discuss assigning campaign versions within important audience groups and planning the analysis before results arrive. Sample size, implementation checks and uncertainty need their own work; this worksheet neither performs that design nor authorizes a real campaign change.
Try it yourself
Use only the final table and the specified 70% new / 30% returning mix. Enter A-minus-B as a percentage-point gap. The illustrative count uses 1,000 hypothetical visitors.
Final campaign counts
| campaign | segment | visitors | conversions |
|---|---|---|---|
| A | new | 600 | 72 |
| A | returning | 400 | 200 |
| B | new | 300 | 30 |
| B | returning | 700 | 280 |
Numbers use the units in each label. Blank answers are incomplete. ID lists can use commas, spaces or semicolons.
Read this exercise’s solution
The final segment rates are A: 12% and 50%; B: 10% and 40%. The 70/30 scenario produces A=23.4%, B=19%, a 4.4-percentage-point gap. A’s scenario equivalent is 234 per 1,000 hypothetical visitors. This is not a forecast or causal estimate; agree on eligibility, outcome and window before planning an experimental comparison.
Apply the idea
If the intended audience became 30% new and 70% returning, which inputs would stay fixed and which calculation would you rerun?
Save your three-part portfolio
Check all three exercises and include an explanation for each. You may download a portfolio with mismatches for later review; it clearly records which checks passed. Editing an answer or explanation invalidates that exercise’s previous check.
Complete and check all three exercises to prepare your portfolio.
Inspect the portfolio before downloading
Your portfolio will appear here after each exercise has been checked.
Download before leaving. Course answers stay in this page and are cleared on refresh; they are not sent to a server or added to the older course progress tracker.
Source and scope
Occupational task context comes from the O*NET® 31.0 Database, USDOL/ETA, under CC BY 4.0. View the occupation.
Task 5438 · 08/2026 · Measure the effectiveness of marketing, advertising, and communications programs and strategies.
Task 5434 · 08/2026 · Prepare reports of findings, illustrating data graphically and translating complex findings into written text.
Method reference: NIST: Randomized block designs. Background for tracking important nuisance factors and varying the factor of interest within comparable blocks. This observational worksheet is not a randomized block experiment.
Skill Workbench selected this context and authored the teaching, examples, practice rules, data and feedback. USDOL/ETA has not approved, endorsed or tested these additions. O*NET® is a trademark of USDOL/ETA. Matching the fixed checks does not certify occupational competence.