lifelines Survival Analysis Documentation

lifelines is a pure Python implementation of survival analysis techniques. It is designed to answer questions about why events occur at specific times under uncertainty (e.g., time-to-event data).

Common use cases include:

• Medical/Actuarial: Measuring lifetimes or disease remission.
• SaaS: Measuring subscriber lifetimes or time to first action.
• Inventory: Analyzing stock-out events relative to demand.
• Sociology: Measuring the duration of political parties, relationships, or marriages.
• A/B Testing: Determining the time taken for different groups to perform an action.

Survival analysis revolves around three interconnected mathematical entities. lifelines provides tools to estimate and transform between these:

1. Survival Function $S(t)$: The probability that the event has not occurred yet by time $t$ (i.e., $Pr(T > t)$). It is a non-increasing function ranging from 0 to 1.
2. Hazard Function $h(t)$: The instantaneous rate of the event occurring at time $t$, given that the event has not occurred until then.
3. Cumulative Hazard Function $H(t)$: The integral of the hazard function over time: $H(t) = \int_0^t h(z) \mathrm{d}z$.

Relationships:

• $S(t) = \exp(-H(t))$
• $H(t) = -\ln(S(t))$
• $h(t) = \frac{-S'(t)}{S(t)}$

In survival analysis, right-censoring occurs when the event of interest (e.g., death, churn, failure) has not yet been observed for some individuals in the study at the time of analysis.

Key points:

• Right-censored individuals: We only know their current lifetime duration, which is strictly less than their actual lifetime.
• Common Pitfall: Ignoring right-censored individuals or simply taking the mean of all observed lifespans (including censored ones) leads to a severe underestimation of the true average lifespan.
• Purpose of Survival Analysis: It is specifically designed to handle these estimations correctly by accounting for the fact that some individuals have not yet experienced the event.

Predicting survival in a time-varying setting is non-trivial because future covariate values are unknown. While CoxTimeVaryingFitter provides prediction methods, they are logically limited.

Users can still compute:

• Hazard values at known observation times.
• The baseline cumulative hazard rate.
• The baseline survival function.

However, predicting future survival requires assuming future covariate values, which may not be accurate.

After fitting, you can access model components via these attributes:

• cph.params_: The coefficients of the covariates.
• cph.baseline_hazard_: The baseline hazard.

Hazard Ratio: The exponentiated coefficient $\exp(\text{coef})$ is the hazard ratio. For a binary covariate (e.g., mar for married), $\exp(\text{coef})$ represents the ratio of hazards between the two levels (e.g., married vs. unmarried).

To perform survival regression, your data must be in a Pandas DataFrame format where each row represents a single observation. The DataFrame should include:

1. Duration column: Denotes the time elapsed for each observation.
2. Event status column (optional): Denotes if the event occurred (typically 1 if the event occurred, 0 if censored).
3. Covariates: Additional variables you wish to regress against.
4. Optional columns: You may also include columns for stratification, weights, or clusters.

Once a regression model (like CoxPHFitter, WeibullAFTFitter, or AalenAdditiveFitter) is fitted, you can predict survival outcomes for specific individuals or covariate sets using the following methods:

• predict_survival_function(X): Returns the estimated survival function for the provided covariates X.
• predict_cumulative_hazards(X): Returns the estimated cumulative hazard function for the provided covariates X.

Note that X must be a set of covariates (e.g., a Pandas Series representing a single observation or a DataFrame of multiple observations).

X = regression_dataset.loc[0]

ax = wft.predict_survival_function(X).rename(columns={0:'WeibullAFT'}).plot()
cph.predict_survival_function(X).rename(columns={0:'CoxPHFitter'}).plot(ax=ax)
aaf.predict_survival_function(X).rename(columns={0:'AalenAdditive'}).plot(ax=ax)

Survival regression is used when you have additional data (covariates like age, country, etc.) in addition to the duration of an observation. It involves regressing these covariates against durations. Traditional linear regression cannot be used because of censoring.

Common models in survival regression include:

• Cox's model
• Accelerated failure models
• Aalen's additive model

All these models attempt to represent the hazard rate $h(t | x)$ as a function of time $t$ and covariates $x$.

If your dataset contains correlated subjects (e.g., subjects appearing multiple times or matched pairs from propensity-score matching), the independent-and-identically-distributed assumption is violated and standard errors will be incorrect.

To adjust for this, use the cluster_col keyword in the .fit() method of CoxPHFitter. Pass the name of a column that contains identifiers for correlated groups. For example, in a matched pair, both subjects in the pair should share the same value in the cluster_col column. Specifying cluster_col automatically invokes the robust sandwich estimator for standard errors (equivalent to setting robust=True).

from lifelines import CoxPHFitter

# Assuming 'id' column contains group identifiers for correlated subjects
cph = CoxPHFitter()
cph.fit(rossi, 'week', 'arrest', cluster_col='id')

When choosing between different parametric models (e.g., Weibull vs. Log-Normal), use the Akaike Information Criterion (AIC). The model with the smallest AIC is generally preferred as it provides the best fit with the fewest parameters.

# Compare AIC values
print(llf.AIC_)
print(lnf.AIC_)
print(wf.AIC_)