Theory — stratification, self-selection and non-response bias
Updated: 2026-10-05. Covers the gap between “random” and “representative”.
Concept
Three distinct design problems often conflated: stratification/quota design (declaring which population features matter and how much), self-selection bias (who responds to an invitation), and non-response bias (systematic difference between respondents and invitees).
Definitions / competing meanings
- Stratification: partition the population on criteria (gender, age, geography, deprivation, attitudes), set per-stratum quotas, select within strata. The choice of strata encodes a political judgement about which differences matter (SRC-0100, SRC-0102).
- Self-selection bias: opt-in response stages mean participation is not random among invitees; civic-interest skew is observed systematically (SRC-0038 fn.35, SRC-0120).
- Non-response bias: non-respondents differ systematically — e.g. CA UK respondents skewed to highest educational level and required overweighting (SRC-0038 §“Civic lottery”).
- Quota design: hard quota (stratum must hit target) vs soft/distribution approach (maximise fairness across quota-compliant panels — leximin, SRC-0078).
Major theoretical arguments
- Normative: quota design is where “who counts as a relevant difference” gets decided — arguably the most politically consequential technical step in the pipeline (SRC-0107, SRC-0109).
- Empirical: opt-in dependence means representativeness is achieved downstream by stratification, not given by the lottery — stratification corrects for frame and response bias only on measured variables (SRC-0038, SRC-0100).
Practical implications
- Publish the frame, invitation→respondent funnel, and respondent-vs-population profile (CA UK does; SRC-0038).
- Two-stage correction observed: postal invitation without enrolment → oversampled (Ireland 2019 local-government pilot → back to postal, SRC-0017); overweight under-responsive strata (CA UK, SRC-0038).
- Leximin fairness objectives (Panelot) maximise minimum inclusion probabilities given the respondent pool — they cannot correct self-selection that happened earlier (SRC-0078).
Critiques / limitations
- Unmeasured bias: strata correct only for the variables chosen; civic-interest disposition remains uncorrected everywhere in the corpus.
- Quota rigidity: very small strata + dropouts can make hard quotas infeasible → distribution-over-panels methods exist precisely for this (SRC-0078, SRC-0080).
Operationalisation in the corpus
- CA UK: deprivation-weighted invitation + attitudinal stratum + overweighting respondents (SRC-0038).
- Newham: stratification repeated each assembly wave; composition vs population table published (SRC-0048, SRC-0050).
- Sortition Foundation practice: stratify “as far as practicable” language in their published materials (SRC-0054).
Cicada relevance
- Cicada’s comparable records must always log: sampling frame, strata chosen + rationale, funnel (sent/registered/selected/seated), and correction methods. The same record fields (agenda/invitation/strata sections of the case template) already capture this — the theory file justifies keeping them.