Cross-sectional studies

Cross-sectional studies may be used to assess:

• the prevalence of an outcome, defined as:
• the prevalence of various different exposures, defined as:

within a population. They provide a snapshot of the population at a single point in time and can be relatively straightforward to conduct using surveillance data.

Case-control studies

Case-control studies are used to investigate the effects of specific exposures on an outcome. In these studies the outcome of interest has already occurred, individuals who have experienced the outcome are termed “cases”, and are compared to individuals without the outcome, known as “controls”. The odds of being exposed are compared in cases vs. controls via the odds ratio:

Test-negative case-control studies

Test-negative case-control studies are a sub-type of case-control study, commonly used to estimate vaccine effectiveness. These studies aim to minimise bias specifically as a result of test-seeking behaviour by selecting only individuals who test for a disease. As for case-control studies, the cases are individuals who test positive, controls are those who test negative, and again the odds of being exposed are compared between cases and controls. Since individuals are only included if they were tested, the design implicitly conditions on testing, so it remains important to consider whether the factors that lead to testing could differ between exposure groups (see collider bias below).

Cohort studies

In cohort studies participants are selected based on their exposure status and followed up over time, either prospectively or retrospectively using existing records. The incidence of the outcome of interest is compared between individuals with and without the exposure using the relative risk:

R example

The same table can therefore support different summaries depending on the study design. The code below shows how to compute the prevalence of the outcome and exposure, the odds ratio, and the relative risk from a table in R.

study_table <- matrix(
  c(42, 158, 28, 272),
  nrow = 2,
  byrow = TRUE,
  dimnames = list(
    Exposure = c("Yes", "No"),
    Outcome = c("Yes", "No")
  )
)

A <- study_table["Yes", "Yes"]
B <- study_table["Yes", "No"]
C <- study_table["No", "Yes"]
D <- study_table["No", "No"]

study_table

        Outcome
Exposure Yes  No
     Yes  42 158
     No   28 272

round(
  c(
    prevalence_outcome = (A + C) / (A + B + C + D),
    prevalence_exposure = (A + B) / (A + B + C + D),
    odds_ratio = (A * D) / (B * C),
    relative_risk = (A / (A + B)) / (C / (C + D))
  ),
  3
)

 prevalence_outcome prevalence_exposure          odds_ratio       relative_risk 
              0.140               0.400               2.582               2.250 

The same four cell counts therefore answer different questions: prevalence in cross-sectional studies, odds ratios in case-control studies, and risks or risk ratios in cohort studies. Not every measure is valid for every design. For example, in a case-control study the numbers of cases and controls are fixed by the study design, so the prevalence of the outcome and the relative risk cannot be estimated directly from the table, whereas the odds ratio can.

Information bias

Information, or measurement, bias arises due to inaccuracies in data and may be induced by errors or missing information in either the outcome or exposure variables, for example, incorrect coding of a diagnosis. Information bias can be mitigated through careful study design and data collection mechanisms, these could include: double entry of data, and linking information across different systems for validation.

Collider bias

Collider bias, a common form of selection bias, occurs when individuals are selected into an analysis by conditioning on a common “collider” variable (a variable which is influenced by two other variables).

Figure 2 shows an example of collider bias: people who are regular smokers and people at high risk of severe COVID-19 illness are both more likely to be admitted to hospital. By conditioning on hospitalisation a distorted association between smoking and COVID-19 mortality could be induced, and this bias may occur in either a positive or negative direction. For example: a negative association between smoking and COVID-19 severity was reported by more than one early risk-factor study for COVID-19, with a review concluding this result may have been a result of collider bias.

Collider bias is most easily avoided by not conditioning on the collider, but this may be impractical depending on the nature of data collection. In the context of COVID-19, Griffith et al. have suggested several methods for assessing the sensitivity of model results to collider bias, such as inverse probability weighting. Where information is available, the extent of the bias can also be examined by comparing the profile of selected individuals to the wider population of interest, e.g. whether hospitalised individuals tend to be older or more likely to have comorbidities as compared to the general population.

Confounding

Confounding is related to collider bias, but occurs by not conditioning on an explanatory variable. Confounding introduces a “backdoor path”, an additional source of association between the explanatory variable and the outcome, so the measured association cannot be interpreted as a causal effect.

An example of confounding is shown in the causal diagram in Figure 3: during vaccine roll-out older people were more likely to receive a COVID-19 vaccine, but also more likely to experience severe COVID-19. If age group is not adjusted for then a distorted association between vaccination and COVID-19 severity may be induced. Confounding can be avoided by ensuring that factors that might be associated with the outcome are measured and adjusted for.

Epidemic phase bias

Epidemic phase bias occurs in studies with time-varying incidence of infection. This bias is induced by a relationship between the time from infection to symptom onset and an individual’s eventual outcome, e.g. those who go on to die may experience a more rapid onset of symptoms following infection, an average of days sooner. Since estimates must be conditioned on the observed symptom onset date, rather than the unobserved infection date, this relationship can introduce bias into results when an epidemic is in a mode of growth or decline.

Epidemic phase bias is corrected by “shifting” the symptom onset date among those who experience the more severe outcome to be days later, so that the time from infection to symptom onset is uncorrelated with the outcome (the time-to-event of interest remains the same for both outcomes.) As the true value of is typically unknown, sensitivity analysis with differing values of can be used to assess the susceptibility of results to this bias.

R example

The simple sensitivity analysis below shows the basic mechanism in a growing epidemic where severe outcomes are associated with faster symptom onset.

set.seed(2026)

n             <- 1000
# Growing epidemic: infections concentrated on later days
infection_day <- sample(1:30, n, replace = TRUE, prob = 1:30)
severe        <- rbinom(n, 1, 0.15)
# Severe cases present 3 days faster than mild cases
onset_day     <- infection_day + ifelse(severe == 1, 4, 7)

# Sensitivity analysis: shift the severe-case onset date by c days
cutoff <- 30
epb_df <- data.frame(
  shift_days     = 0:3,
  estimated_risk = round(sapply(0:3, function(c) {
    mean(severe[onset_day + c * severe <= cutoff])
  }), 3),
  true_risk = round(mean(severe), 3)
)

epb_df

  shift_days estimated_risk true_risk
1          0          0.183     0.149
2          1          0.170     0.149
3          2          0.160     0.149
4          3          0.152     0.149

ggplot(epb_df, aes(x = shift_days, y = estimated_risk)) +
  geom_line(colour = blog_cols[["red"]], linewidth = 1.1) +
  geom_point(colour = blog_cols[["red"]], size = 3) +
  geom_hline(
    yintercept = epb_df$true_risk[1],
    linetype = "dashed",
    colour = "gray40"
  ) +
  annotate(
    "text",
    x = 0, y = epb_df$true_risk[1],
    label = "True risk", vjust = -0.5, hjust = 0, size = 3.5
  ) +
  scale_x_continuous(name = "Shift (days)", breaks = 0:3) +
  scale_y_continuous(name = "Estimated severe risk")

When no shift is applied, the estimated risk of severe outcomes is biased upwards, because the faster symptom onset among severe cases means they are more likely to have an onset date within the observation window than mild cases. As the shift increases, the estimated risk approaches the true risk, which is recovered when the shift equals the true difference in time to symptom onset (3 days).

Identifiability conditions

Three identifiability conditions are required for an observational study to be treated as a conditionally randomised experiment:

1. Consistency: the values of treatment under comparison correspond to well-defined interventions that, in turn, correspond to the versions of treatment in the data, i.e.  for individuals with .

2. Exchangeability: the conditional probability of receiving every value of treatment depends only on measured covariates . This is alternatively phrased as: conditional on a set of measured covariates , the untreated group, had they been treated, would experience the same average outcome as the treated group, i.e.

   

   or equivalently, , the counterfactual outcome and the observed treatment are independent within the covariate levels . Here, is the notation for independence.

3. Positivity: the probability of receiving every value of treatment conditional on is positive, i.e.

   

Among these conditions, positivity can be checked empirically in observational studies (for example by looking for covariate levels where nobody is treated), whereas consistency relies on treatment being well-defined, and exchangeability relies on the assumption that all predictors of an outcome have been measured. It is usually impossible to guarantee exchangeability, but understanding the potential for biases such as confounding and collider bias (defined in the next post) can inform which covariates to include such that the assumption will be approximately true.

If the three identifiability conditions are fulfilled, a (hypothetical) randomised experiment, known as a target trial, may be emulated using causal inference from observational data, with the average causal effect quantified as .

R example

The simulated example below creates a simple confounded observational dataset where a baseline covariate, , affects both treatment assignment and outcome risk. The crude treated-vs-untreated comparison can then be contrasted with the true average causal effect built into the simulation.

set.seed(2026)

n <- 2000
causal_data <- data.frame(z = rbinom(n, 1, 0.45))

causal_data$y0 <- rbinom(n, 1, plogis(-2.2 + 1.2 * causal_data$z))
causal_data$y1 <- rbinom(n, 1, plogis(-2.8 + 1.2 * causal_data$z))
causal_data$a <- rbinom(n, 1, plogis(-1.1 + 1.4 * causal_data$z))
causal_data$y <- ifelse(causal_data$a == 1, causal_data$y1, causal_data$y0)

crude_rd <- mean(causal_data$y[causal_data$a == 1]) -
            mean(causal_data$y[causal_data$a == 0])
true_ace  <- mean(causal_data$y1 - causal_data$y0)

c(
  crude_risk_difference      = round(crude_rd, 3),
  true_average_causal_effect = round(true_ace, 3)
)

     crude_risk_difference true_average_causal_effect 
                    -0.022                     -0.061 

In this setup the crude comparison is biased because treated individuals are more likely to come from the higher-risk covariate stratum. The crude risk difference (-0.022) understates the protective effect of treatment compared with the true average causal effect (-0.061).

Inverse probability weighting

The concept of inverse probability weighting is to assign weights to each individual in the cohort based on the inverse of the probability of receiving the observed treatment level, conditional on their covariates, . For example, a treated individual, , with a set of covariates would be assigned a weight of .

Conversely, an untreated individual, , with covariates would be assigned a weight of . These weights are used to adjust for potential baseline confounders when estimating the causal effect of treatment, allowing the creation of a pseudo-population where treatment assignment is independent of the observed covariates.

R example

In R, this usually starts by fitting a propensity score model and then constructing the inverse-probability weights directly from the fitted probabilities.

propensity_model <- glm(a ~ z, family = binomial(), data = causal_data)

causal_data$ps <- predict(propensity_model, type = "response")
causal_data$ipw <- ifelse(
  causal_data$a == 1,
  1 / causal_data$ps,
  1 / (1 - causal_data$ps)
)

weighted_risk_treated <- weighted.mean(
  causal_data$y[causal_data$a == 1],
  causal_data$ipw[causal_data$a == 1]
)
weighted_risk_untreated <- weighted.mean(
  causal_data$y[causal_data$a == 0],
  causal_data$ipw[causal_data$a == 0]
)

c(
  ipw_risk_treated = round(weighted_risk_treated, 3),
  ipw_risk_untreated = round(weighted_risk_untreated, 3),
  ipw_risk_difference = round(
    weighted_risk_treated - weighted_risk_untreated,
    3
  )
)

   ipw_risk_treated  ipw_risk_untreated ipw_risk_difference 
              0.094               0.158              -0.064 

The weighted risk difference should move back towards the target-trial estimand by rebalancing the covariate distribution between treatment groups.

Matching

Matching involves constructing a subset of the population in which the distribution of covariates, , is the same for the treated and untreated groups. This is typically achieved by pairing each individual with an individual in the other treatment group who has the same or similar covariate values. The causal effect can be estimated as the average difference in outcomes among the matched pairs. Note that matching only treated individuals to controls estimates the average effect in the treated, whereas matching individuals in both groups estimates the average effect in the whole population.

R example

Since is binary, we can match exactly on . In the example below, each individual is matched at random (with replacement) to an individual in the opposite treatment group with the same value of , so that the matched outcome stands in for the unobserved potential outcome:

match_rd <- function(d) {
  d$y_match <- NA_real_
  for (z_val in 0:1) for (a_val in 0:1) {
    idx  <- which(d$z == z_val & d$a == a_val)
    pool <- d$y[d$z == z_val & d$a != a_val]
    d$y_match[idx] <- pool[sample.int(length(pool), length(idx), replace = TRUE)]
  }
  mean(ifelse(d$a == 1, d$y - d$y_match, d$y_match - d$y))
}

set.seed(2026)
round(match_rd(causal_data), 3)

[1] -0.068

Matching on ensures the comparison group has the same covariate distribution as the treated group, removing the confounding contribution of .

Standardisation

Standardisation involves estimating the counterfactual risk using a weighted average of the risks (or standardisation of the risks) in each covariate level. For example, for a covariate with two levels :

which generalises to:

The standardised mean is known as the g-formula.

R example

In practice, standardisation means fitting an outcome model, predicting every individual’s outcome under both treatment levels, and then averaging those counterfactual predictions across the observed covariate distribution.

outcome_model <- glm(y ~ a + z, family = binomial(), data = causal_data)

risk_if_treated <- predict(
  outcome_model,
  newdata = transform(causal_data, a = 1),
  type = "response"
)
risk_if_untreated <- predict(
  outcome_model,
  newdata = transform(causal_data, a = 0),
  type = "response"
)

c(
  standardised_risk_treated = round(mean(risk_if_treated), 3),
  standardised_risk_untreated = round(mean(risk_if_untreated), 3),
  standardised_risk_difference = round(
    mean(risk_if_treated - risk_if_untreated),
    3
  )
)

   standardised_risk_treated  standardised_risk_untreated 
                       0.096                        0.159 
standardised_risk_difference 
                      -0.063 

Transition probabilities

The probabilities of competing events can be related to a set of covariates via multinomial logistic regression. Define as the set of all competing states after , then the logit of transition to a given state vs. transition to a baseline state is:

with and being regression coefficients for the probability of transition to state .

Meanwhile, parameters of the time to transition distribution can be related to a different (or identical) set of covariates via a log-linear model (for parameters which are constrained to be positive) with coefficients and :

For most parametric time to transition distributions `accelerated failure time’ models are considered, where the covariates affect the rate of progression of time, i.e. the survival function for an individual at time with covariates is related to the baseline survival via:

where is a vector of regression coefficients.

Likelihood contributions

The likelihood contributions are as follows:

• For an individual who enters state and reaches state at time :

• For an individual who is censored in state at time :

  where is the conditional cumulative distribution of the time to transition from state to state , given that this transition occurs.

Discrete and continuous parameter processes

A stochastic process is formally defined as a collection of random variables , indexed by a parameter such as time, which varies in a mathematical index set . The variable is the state of the process at time , and the set of all possible values of is termed the state space.

Two important cases of stochastic processes are discrete parameter processes, when , and continuous parameter processes, when . Both may be restricted to the positive domain, in which case the parameter may be used to represent time, and these processes are termed ‘discrete-time’ and ‘continuous-time’ processes.

Counting processes

In survival analyses, the focus is typically on observing events as they occur over time, forming a class of stochastic process known as a point process. If the number of events that occur over time are counted, then the stochastic process is instead known as a counting process. Counting processes form the basis for multi-state models, and linkage to the theory of martingales (not discussed here) provides a theoretical framework for these models.

Markov process

A stochastic process is called a ‘Markov process’ if the future state of a process depends only on the present state, and not on the sequence of events that preceded it. Equivalently, for a Markov process the conditional probability of a future state given is independent of previous states .

Time-homogeneous Markov processes

Markov processes are said to be ‘time-homogeneous’ if the probability of transition from one state to the next (termed the transition probability) is independent of the time parameter . Equivalently, for denoting the conditional probability of a process being in state at time , given the process was in state at time , a Markov process is time-homogeneous if:

A well-known time-homogeneous Markov counting process is the Poisson process, where events occur randomly and independently of each other. A homogeneous Poisson process is described by its constant intensity parameter , which represents the average rate of events that occur per unit time. For any infinitesimal time interval , as , the probability of exactly one event occurring in this interval is asymptotically .

Multi-state processes

Multi-state processes are discrete-state stochastic processes with a finite state space . In these processes, an individual begins in one state and spends a random, continuously-distributed time in that state before moving to a random next state. Multi-state processes are defined by the initial distribution of the states , and the transition probabilities from state to state during the interval , with , and where represents the history of the process up to time .

Figure 1 shows how a counting process may be built on top of a multi-state process: panel A shows a multi-state process with transitions between two states, and the corresponding counting process for the transition event from State 1 State 2 is shown in panel B.

Covariates

The (potentially time-varying) effect of a given set of covariates, , on the transition intensities can be expressed via a proportional hazards regression model. Typically these combine a parametric baseline hazard, , with log-linear regression:

where is the hazard ratio for the th covariate on the transition.

Sojourn time

The time spent in a state prior to transition is termed the sojourn time. In the case of a time-homogeneous Markov model, the mean sojourn time in state , , is the inverse of the transition intensity for leaving the state:

R example

The msm package can be used to fit continuous-time Markov multi-state models. The cav dataset included in the package contains serial examinations of heart transplant recipients, with states = no coronary allograft vasculopathy (CAV), = mild/moderate CAV, = severe CAV, and = death. Patients were only examined at intermittent clinic visits, so the times of transitions between the first three states are interval-censored, whereas the time of death is known exactly (deathexact = 4). The code below fits a model with a constant intensity for each allowed transition, with non-zero entries of the initial matrix indicating which transitions are allowed.

library(msm)

# Allowed transitions have non-zero starting values for the intensities
Q_init <- rbind(
  c(0,     0.25,  0,     0.25),
  c(0.166, 0,     0.166, 0.166),
  c(0,     0.25,  0,     0.25),
  c(0,     0,     0,     0)
)

cav_fit <- msm(
  state ~ years, subject = PTNUM, data = cav,
  qmatrix = Q_init, deathexact = 4
)

round(qmatrix.msm(cav_fit)$estimates, 3)

        State 1 State 2 State 3 State 4
State 1  -0.170   0.128   0.000   0.043
State 2   0.225  -0.608   0.343   0.040
State 3   0.000   0.131  -0.437   0.306
State 4   0.000   0.000   0.000   0.000

sojourn.msm(cav_fit)

        estimates        SE        L        U
State 1  5.869552 0.3307930 5.255734 6.555057
State 2  1.644897 0.1288274 1.410825 1.917805
State 3  2.287819 0.2743666 1.808595 2.894023

round(pmatrix.msm(cav_fit, t = 5), 3)

        State 1 State 2 State 3 State 4
State 1   0.520   0.139   0.091   0.251
State 2   0.244   0.139   0.181   0.436
State 3   0.061   0.069   0.169   0.701
State 4   0.000   0.000   0.000   1.000

The first output is the estimated transition intensity matrix , with rows summing to zero. The mean sojourn times are the inverse of the negative diagonal entries, giving an estimated mean time of 5.9 years in state 1 before a transition. The final output is the transition probability matrix , for example the probability of a patient who is in state 1 now being in state 4 (dead) after 5 years is estimated to be 0.25.

Cause-specific hazard function

Let be a random variable for the survival time and be a random variable for the cause of failure. The cause-specific hazard function for the th cause, is defined by:

Defining as the cause-specific density function, and as the overall survivor function, the relationship shown in part II of this series still holds in the presence of competing risks:

Cause-specific cumulative incidence

The cause-specific cumulative incidence function, i.e. the probability of failure from cause before time , in the presence of all other risks, is given by:

for , with often written as .

An expression for in terms of the cause-specific hazard function may be derived using the equation above:

R example

We can illustrate the difference between the naive Kaplan-Meier estimate and the Aalen-Johansen estimator for cumulative incidence in a simple example. Consider a dataset of 10 patients with follow-up times and event types (death, discharge, or censoring):

library(survival)

cr_data <- data.frame(
  time   = c( 2,    3,    4,      5,     6,
              7,    8,    9,     10,    11),
  status = c("death", "discharge", "censor", "death", "discharge",
             "death", "censor",    "discharge", "death", "censor")
)

# Naive estimate: treat discharge as ordinary censoring
km_fit <- survfit(Surv(time, status == "death") ~ 1, data = cr_data)

# Aalen-Johansen: with a multi-state outcome, survfit() returns the estimated
# probability of being in each state; the first level is censoring
aj_fit <- survfit(
  Surv(time, factor(status, levels = c("censor", "death", "discharge"))) ~ 1,
  data = cr_data
)

cif_df <- data.frame(
  time   = c(0, aj_fit$time),
  cif_aj = c(0, aj_fit$pstate[, "death"]),
  cif_km = c(0, 1 - summary(km_fit, times = aj_fit$time, extend = TRUE)$surv)
)

round(cif_df, 2)

   time cif_aj cif_km
1     0   0.00   0.00
2     2   0.10   0.10
3     3   0.10   0.10
4     4   0.10   0.10
5     5   0.21   0.23
6     6   0.21   0.23
7     7   0.33   0.38
8     8   0.33   0.38
9     9   0.33   0.38
10   10   0.48   0.69
11   11   0.48   0.69

ggplot(cif_df, aes(x = time)) +
  geom_step(aes(y = cif_km, colour = "Naive (1 - KM)"), linewidth = 1.1) +
  geom_step(aes(y = cif_aj, colour = "Aalen-Johansen"), linewidth = 1.1) +
  scale_colour_manual(
    values = c(
      "Naive (1 - KM)" = blog_cols[["dark"]],
      "Aalen-Johansen" = blog_cols[["red"]]
    )
  ) +
  scale_y_continuous(limits = c(0, NA), name = "Cumulative incidence of death") +
  scale_x_continuous(name = "Follow-up time")

The Aalen-Johansen estimate is equivalent to the sum given above, where is the Kaplan-Meier estimate of surviving all causes. Other packages, such as cmprsk and tidycmprsk, also provide cumulative incidence estimates and Fine-Gray regression.

The naive curve lies above the Aalen-Johansen estimate because discharge is handled as if it were non-informative censoring, even though it removes patients from ever experiencing the event of interest.

Sub-distribution hazard

The sub-distribution hazard is therefore defined as the instantaneous risk of experiencing the event of interest given that the individual has not already experienced this event:

where, as before, is the random variable for the event that occurs.

Fine-Gray regression model

Fine-Gray regression links the sub-distribution hazard, , to the cause-specific cumulative incidence function, , through the relationship:

As with the Cox proportional hazards model, a proportional hazards regression model is assumed, where the hazard of cause at time for individual is:

with being the baseline sub-distribution hazard for cause . Covariate coefficients are estimated by maximising the weighted partial likelihood :

where are inverse probability of censoring weights, which down-weight individuals who have experienced a competing event to account for the probability that they would have been censored had they remained under follow-up, are the set of distinct failure times, is the set of all individuals who are at risk of failure immediately before time (including those who have experienced a competing event), is the covariate vector for an individual who failed at time , and is the covariate vector for the th individual at risk at time . Covariate effects on the sub-distribution hazard may be interpreted as covariate effects on the cumulative incidence of the event of interest.

Frequentist vs Bayesian

In the ‘frequentist’ framework the parameters of a statistical model are considered to have a fixed ‘true’ unknown value to be estimated, and the data are realisations of random variables. The parameters of the model are estimated by maximising the likelihood to obtain the maximum likelihood estimate (MLE), denoted .

Bayesian methods are an alternative to maximum likelihood estimation, used to estimate the probability distribution of the parameters of the model given the data, known as the ‘posterior distribution’. In the Bayesian framework, all quantities (both observable quantities and parameters) are considered as random variables, with observed data being realisations of these random variables. The posterior distribution, , is related to the likelihood function, , via Bayes formula:

Censored and truncated data

Constructing likelihoods for censored or truncated data requires careful consideration of the information provided by each observation. The previous post in this series provided a definition of censoring and truncation, and the right and left censoring and truncation times.

The likelihood components provided by censored observations, assuming that censoring is independent of the event process, are:

• for exactly observed data: information on the probability of the event occurring at this time, i.e. 
• for right-censored observations: information on the survivor function up to a certain time, i.e. 
• for left-censored observations: information on the cumulative incidence up to a certain time, i.e. 
• for interval-censored observations: information on the probability that the event is within this interval, i.e. .

As data may be observed with different levels of censoring or truncation, the likelihood function is constructed from the joint product of each component for each observation :

The equivalent likelihood components for truncated data provide information on conditional probabilities of events, given that the individual was observed:

• for left-truncated observations, where observation is conditional on the event time being at least the left-truncation time :
• for right-truncated observations, where observation is conditional on the event of interest having occurred by the truncation time :
• for interval-truncated observations, where observation is conditional on the event occurring within the truncation interval : .

R example

For a short example of the product-limit construction we can use the survival package in R to compute a Kaplan-Meier estimate from a small dataset of 10 observations, with some censored and some observed event times:

library(survival)

km_data <- data.frame(
  time   = c(2, 3, 4, 4, 5, 6, 7, 8, 10, 10),
  status = c(1, 1, 0, 1, 1, 0, 1, 0,  1,  0)
)

km_fit <- survfit(Surv(time, status) ~ 1, data = km_data)

data.frame(
  time     = km_fit$time,
  n_risk   = km_fit$n.risk,
  n_event  = km_fit$n.event,
  n_censor = km_fit$n.censor,
  survival = round(km_fit$surv, 2)
)

  time n_risk n_event n_censor survival
1    2     10       1        0     0.90
2    3      9       1        0     0.80
3    4      8       1        1     0.70
4    5      6       1        0     0.58
5    6      5       0        1     0.58
6    7      4       1        0     0.44
7    8      3       0        1     0.44
8   10      2       1        1     0.22

km_df <- data.frame(
  time   = c(0, km_fit$time),
  surv   = c(1, km_fit$surv),
  lower  = c(1, km_fit$lower),
  upper  = c(1, km_fit$upper),
  censor = c(0L, km_fit$n.censor)
)

ggplot(km_df, aes(x = time)) +
  geom_step(aes(y = lower), colour = blog_cols[["red"]], linetype = "dotted") +
  geom_step(aes(y = upper), colour = blog_cols[["red"]], linetype = "dotted") +
  geom_step(aes(y = surv), colour = blog_cols[["red"]], linewidth = 1.1) +
  geom_point(
    data = subset(km_df, censor > 0),
    aes(y = surv), shape = 3, size = 3, stroke = 1
  ) +
  scale_y_continuous(limits = c(0, 1), name = "Estimated survival") +
  scale_x_continuous(name = "Follow-up time (days)")

The figure shows the Kaplan-Meier survival curve, with steps at each observed event time and crosses indicating censored observations (including those which coincide with an event time, at days 4 and 10). Survival only steps down at observed event times (rows where n_event > 0), while censored observations reduce the risk set without causing a jump. With only 10 observations the confidence limits are wide.

R example

To illustrate fitting a Cox proportional hazards model, we can use the survival package in R to fit a model to the lung dataset, which contains survival data for patients with advanced lung cancer. We’ll model the hazard of death as a function of age and sex, and check the proportional hazards assumption using Schoenfeld residuals.

library(survival)

# lung: time to death (days) for advanced lung cancer patients
# status: 1 = censored, 2 = dead; sex: 1 = male, 2 = female
cox_fit <- coxph(Surv(time, status == 2) ~ age + sex, data = lung)

round(summary(cox_fit)$coefficients[, c("coef", "exp(coef)", "Pr(>|z|)")], 3)

      coef exp(coef) Pr(>|z|)
age  0.017     1.017    0.065
sex -0.513     0.599    0.002

round(cox.zph(cox_fit)$table[, "p"], 3)

   age    sex GLOBAL 
 0.647  0.106  0.250 

ci <- summary(cox_fit)$conf.int
coef_df <- data.frame(
  term = c("Age (per year)", "Sex (female vs male)"),
  hr   = ci[, "exp(coef)"],
  lo   = ci[, "lower .95"],
  hi   = ci[, "upper .95"]
)

ggplot(coef_df, aes(x = hr, y = term)) +
  geom_vline(xintercept = 1, linetype = "dashed", colour = "gray50") +
  geom_errorbar(
    aes(xmin = lo, xmax = hi),
    orientation = "y", width = 0.1, colour = blog_cols[["dark"]]
  ) +
  geom_point(size = 3, colour = blog_cols[["red"]]) +
  scale_x_log10(name = "Hazard ratio (log scale)") +
  scale_y_discrete(name = NULL)

Older age increases the hazard of death (exp(coef) > 1), although there is only weak evidence of an effect (p = 0.06). Female sex is protective (exp(coef) < 1), and there is strong evidence of this effect (p = 0.002). The Schoenfeld test p-values give a quick check of the proportional hazards assumption: here, there is no strong evidence against it (all p > 0.05), but a non-significant test does not prove that the assumption holds, so the residuals should also be inspected graphically with plot(cox.zph(cox_fit)).

Censoring

Censoring occurs when information about an individual is only known within certain intervals or “censoring times”.

Truncation

While censoring gives us partial information, truncation occurs when information about an individual is completely unobserved, and unavailable at the time of data collection or analysis. This is different from censoring because censored individuals are at least partially observed, while truncated individuals never enter our dataset. As with censoring, left-, right-, and interval-truncation are possible.

• Left truncation: occurs when only event-times which take place after the left-truncation time, , are available. For example, if individuals whose infection occurs prior to the study, , are not included in our dataset.
• Right-truncation: occurs when only event-times which take place before the right-truncation time, , are available. For example, when we have no knowledge of individuals who are at risk but whose event of interest takes place after the study endpoint, .
• Interval-truncation: occurs when only event-times which take place within a specific truncation interval are available for observation. The key difference from left- and right-truncation is that in interval-truncation, both the left and right boundaries of the observation window play a role in determining which event-times are observed.

The cumulative incidence function

Let be the random variable representing the survival time, , for an individual, with survival times assumed to be independent and identically distributed (i.i.d) across individuals, and assume this random variable has a probability distribution with probability density function .

The distribution function of , also known as the cumulative incidence function, is the probability of “failure” before time , defined as:

When there are several possible events, as in part IV of this series, this is extended to cause-specific cumulative incidence functions.

The survivor function

The survivor function gives the probability of surviving (not experiencing the event) beyond time . This is the cornerstone of survival analysis and is defined as:

where is the random variable representing the time until the event occurs.

The survivor function starts at 1 (everyone is “alive” at time 0) and decreases toward 0 as time progresses. It has an intuitive interpretation: means 80% of individuals remain event-free after 6 months.

The hazard function

The hazard function represents the instantaneous rate of experiencing the event at time , given survival up to that point. Think of it as the “risk” at each moment, conditional on having survived so far (although, as a rate, it is not a probability and can exceed 1):

By conditional probability, , so the hazard function can also be expressed as:

This limit is the definition of the derivative of with respect to , and therefore equal to :

Hence the hazard function is related to the survivor function through the relationship:

The reproduction number

A key measure of the spread of a disease, related to the incidence of infection, is the basic reproduction number, denoted . The value measures the expected number of cases directly generated by one case in a population where all individuals are susceptible to infection.

For instance, if for a particular disease then one infected individual will, on average, infect three others. When an infection will spread in a population, whereas when the disease will eventually die out.

Importantly, not every individual within a population will be equally susceptible to infection. Immunity to a pathogen might be acquired through vaccination or recovery from previous infection, both of which can provide a level of protection against future infections. Through widespread vaccination, immunity can be conferred to a large portion of a population, reducing susceptibility and thereby decreasing the number of new cases generated by each case. This is captured by the effective reproduction number, , the expected number of cases directly generated by one case at time in a population where some individuals may no longer be susceptible, or where control measures are in place. Unlike , changes over time.

Incidence and prevalence

Incidence of infection and the prevalence of an infectious disease in a population are generally unobserved, with observable information restricted to test outcomes among subsets of a population. Testing processes may themselves be influenced by unobserved parameters, such as an individual’s propensity to test, and the time from infection to symptom presentation (if symptoms present at all).

To reconstruct the incidence and prevalence of infection from the available testing data, statistical and mathematical modelling techniques which adjust for sampling biases are employed. These techniques combine information from a range of sources to make valid statistical inference about unobserved incidence and prevalence.

Severity

Two common metrics used to understand the severity of infectious diseases and the factors associated with adverse outcomes are the infection fatality risk (IFR) and case fatality rate (CFR). Both are proportions, although the CFR is conventionally called a “rate”:

• Infection Fatality Risk (IFR): The proportion of all infected people who die from the disease (including those whose infections were never detected)
• Case Fatality Rate (CFR): The proportion of confirmed cases who die from the disease

Since infections resulting in death are more likely to be recorded, and mild infections (particularly asymptomatic infections) less likely to be recorded, the CFR will almost always be higher than the IFR. Statistical modelling can be used to relate these measures of severity to underlying characteristics collected through case-surveillance data.

Effectiveness of interventions

Monitoring outcomes among a target group after the introduction of a public health intervention allows for the real-world effectiveness of the intervention to be assessed, and changes over time to be detected. For example, effectiveness estimates might consider reductions in the transmission or acquisition of an infection among a treated group compared to an untreated group. Making these estimates requires reliable testing data, recognition of potential biases, and methodologies which can account for these biases.

Challenges in estimating infectious disease burden

The majority of the data available for infectious disease monitoring relies on observational data, where a population is followed without actively intervening on an exposure in a randomised way. As a result, estimates of infectious disease burden are often considerably complicated by selection bias, which arises when the selected population observed is different from the general population of interest.

Common causes of selection bias in observational data include: testing policies (which may vary nationally and sub-nationally), individual health-seeking behaviour, test availability, clinician decision-making, and detection rates of different tests. Inaccuracies in the available case-surveillance data may also arise: at data input, due to incomplete/inaccurate reporting or misdiagnosis; and at data collection, due to insufficient surveillance infrastructure and data processing methods.

Unless these potential sources of bias are well-understood and accounted for they can affect the validity of estimates. For example, the IFR measure of severity requires complete knowledge of both the number of infections and number of deaths. Particularly in the early stages of a pandemic, when testing is either unavailable or sub-optimal, these quantities may be impossible to measure. The CFR is more straightforward to assess, since confirmed infections are more likely to be recorded, but this measure is highly sensitive to testing practices. This was particularly the case during the initial wave of the COVID-19 pandemic, when biases in case-surveillance led to wide variability in national estimates of the case-fatality rate, from 0.1% to over 25% (World Health Organization, 2020).

Even when knowledge of these multi-faceted biases is available, statistical methodologies must be carefully chosen to properly control for the biases.