Untitled Team — DS105 Group Project

Does rainfall cause storm overflow spills?

Research question: Has rainfall caused storm overflow spills across English water companies between 2021 and 2025?

Short answer: Yes, but not as much as you may think. The results vary a lot by company and there are many other factors such as failing infrastructure and sewage blockages that also have an effect. See the conclusion on the website for more details on this.

Website

View the project website →

Team

Research Design

Data sources

Note on EDM data: EDM Annual Returns are published by the Environment Agency as Excel files and downloaded directly from their open data portal. This is a static bulk download, not an API. It is used as supplementary data to provide annual spill counts. All primary data collection uses the requests library.

Credentials

No API keys are required. Thames Water and Open-Meteo are public APIs. The EA Water Quality API is also public. There is no .env file needed to reproduce this project.

Pipeline

Database schema

project.db — EDM pipeline

thames_water_data.db — Thames Water pipeline

Foreign key: edm_yearly.permit_ref → sample_sites.permit_ref
Foreign key: discharges.location_identifier → monitors.location_identifier

Reproduction Steps

1. Clone the repository

git clone https://github.com/lse-ds105/group-project-untitled-team
cd group-project-untitled-team

2. Install dependencies

pip install pandas numpy matplotlib statsmodels requests pyproj seaborn sqlite3 openpyxl

3. Download EDM data (manual step)

Option A — Google Drive (recommended, skips all API calls): (save time and confusion) Download the full data/ folder from Google Drive: https://drive.google.com/drive/folders/1OREjVw8cHfsD6A5UR2IV2DQ0Qe4bupbG This contains the raw EDM Excel files, all collected Thames Water and Open-Meteo JSON files, and the processed databases. Place the contents in data/raw at the repo root, matching the folder structure shown below. With this option, notebooks 03, 05, and 06 can be skipped entirely — the processed outputs are already present.

Option B - Download EDM data manually and run all notebooks: Download the EDM Annual Returns Excel files for 2021–2025 from the EA open data portal:
https://environment.data.gov.uk/
Search "EDM Annual Return". Place the files in data/raw/ with the exact filenames referenced in 01-clean-edm-data.ipynb. EDM_2021_Storm_Overflow_Annual_Return_-all_water_and_sewerage_companies.xlsx EDM_2022_Storm_Overflow_Annual_Return-all_water_and_sewerage_companies.xlsx EDM_2023_Storm_Overflow_Annual_Return-all_water_and_sewerage_companies.xlsx EDM_2024_Storm_Overflow_Annual_Return-all_water_and_sewerage_companies.xlsx EDM_2025_Storm_Overflow_Annual_Return-all_water_and_sewerage_companies.xlsx EDM_2025_Storm_Overflow_Annual_Return-_all_water_and_sewerage_companies.xlsx

⚠️ Before running NB01, open this file and delete row 1 (a note row the EA inserted above the column headers). Save and close. Without this step NB01 will load the wrong column names for 2025.

4. Run notebooks in order

01-clean-edm-data.ipynb
02-select-sample-sites.ipynb
03-thames-water-collection.ipynb      ← calls Thames Water API, may take ~30 minutes
04-thames-water-processing.ipynb
05_rainfall_collection.ipynb          ← calls Open-Meteo API, may take ~10 minutes
06-sensor-matching.ipynb              ← optional
07-quality-check.ipynb
08-thames-water-rainfall-metric.ipynb
09-thames-water-analysis.ipynb
10_analysis_final.ipynb

5. View the website

Visit the GitHub Pages URL to see it live, or if running locally:

cd website
npm i
npm run build
npm run start

Repository Structure

group-project-untitled-team/
├── .github/workflows             # for deploying to GitHub pages
├── data/
│   ├── raw/                      # gitignored, recommend to be downloaded in the drive.
│   │   ├── edm/                  
│   │   ├── weather/              
│   │   └── Thames-Water/         
│   └── processed/
│       ├── thames_water_locations.csv
│       ├── edm/
│       │   ├── project.db        # EDM + weather + sensor SQLite database
│       │   └── edm_clean.csv     # cleaned EDM data (output of nb 01)
│       ├── Thames-Water/
│       │   └── thames_water_data.db
│       └── website/              # JSON data files for website charts
├── docs/
│   ├── figures/                  # PNG figures and pipeline SVG
│   ├── analysis_methodology.md
│   ├── thames-water-discharge-findings.md
│   └── thames-water-rainfall-metric-findings.md
├── notebooks/
│   ├── 01-clean-edm-data.ipynb
│   ├── 02-select-sample-sites.ipynb
│   ├── 03-thames-water-collection.ipynb
│   ├── 04-thames-water-processing.ipynb
│   ├── 05_rainfall_collection.ipynb
│   ├── 06-sensor-matching.ipynb
│   ├── 07-quality-check.ipynb
│   ├── 08-thames-water-rainfall-metric.ipynb
│   ├── 09-thames-water-analysis.ipynb
│   └── 10_analysis_final.ipynb
├── reflections/
│   ├── MIT22514.md
│   ├── isaac-kirk.md
│   └── asmith1112.md
│   └── JackChilds.md
├── website/                      ← Next.js site (GitHub Pages)
├── Roadmap.md
└── README.md

Key Findings

1. Rainfall clearly drives spills at event level — discharge days have 3.8× more rainfall and 2.1× higher antecedent soil moisture than non-discharge days (Thames Water, 76,848 events)
2. Annual rainfall signal is real but weak — across all nine water companies, how much it rained in a year explains only about 3–4% of the difference in spill counts between sites
3. Which company owns the infrastructure matters more than the weather — knowing which water company a site belongs to is a better predictor of how often it spills than knowing how much it rained that year
4. Some companies spill far more than their weather conditions justify — after stripping out how much it rained, South West Water spills 91.8% more than the national average and Yorkshire Water spills 71.9% more, while Northumbrian Water spills 41% less
5. 58.7% of Thames Water 2025 spills started below the discharge threshold — suggesting that a majority of overflows are driven by ageing and overstretched infrastructure rather than exceptional weather

Limitations

The data is self-reported. Water companies publish their own spill totals and monitor status. We cannot audit those figures here, so we treat them as the published record rather than the final truth.

Thames Water may not be representative of the whole country. They are the only company that publishes event-level historical spill data, which is why the rainfall metric and event-level analysis are possible at all. That makes them unusually useful, but not automatically representative of every company elsewhere in England. It is also worth noting that we faced significant difficulty collecting their historical discharge alerts — the API continuously returned slightly different data, or none at all, for the exact same query moments apart. More detail is in the collection notebook.

Most companies only publish annual aggregates. For the rest of the country, we only see one number per outflow per year. That means we can compare totals, but not the exact timing of each spill. This is the primary reason the annual rainfall signal (Spearman r = 0.14) is so much weaker than the daily signal from Thames Water event data (r = 0.657) — the timing information that drives spills is entirely lost in the annual aggregate.

We do not use water quality sensor readings. We originally planned to use EA water quality sensors to look for pollution spikes as a proxy for spill events. However the sensors are on average much too far from the overflows to be useful (median matched distance 5.86km), and many sit upstream of the outflow rather than downstream. Using them would have produced meaningless results, so this strand of the analysis was abandoned.

Local geology still matters. Soils and geology change how fast rainwater reaches the sewer system. Comparing rainfall against local norms (the 1-sigma regional threshold) reduces that problem, but the adjustment is still imperfect — the thresholds are computed from only 5 years of data rather than a stable long-run climate baseline.

EDM coverage improved over time. Event Duration Monitor coverage was incomplete until the end of 2023. Earlier totals undercount real spill activity, and part of what looks like a sharp 2023 increase is simply the monitoring network catching up rather than a genuine worsening.

Pipeline and method limitations

Circular baseline for the 1-sigma threshold. The regional thresholds used to define dc_1sigma_days are computed from the same 2021–2025 data used in the regression. A 30-year climate normal would avoid this circularity. The 1-sigma metric also achieves Spearman r = 0.139 vs r = 0.190 for the fixed empirical threshold — it is methodologically more defensible but numerically weaker on this dataset.

155 sites from 4,707 nationally. The sample covers roughly 3% of all EDM sites, stratified geographically. 33 of 155 sites have fewer than 5 years of data because monitors were installed at different points during the study period.

OLS on count data. Spill counts are non-negative integers with a heavily right-skewed distribution. A Negative Binomial regression would be more appropriate for overdispersed count data and might produce different significance results.

Small sample per company. With approximately 20 sampled sites per company, individual company estimates in Chart D have wide confidence intervals. None are individually statistically significant. The rankings are consistent with all other analyses but should be treated as indicative.

Year effects cannot be fully controlled. With only 5 annual observations for EDM and 4 for Thames Water, year dummy variables would overfit. A linear year trend is used for Thames Water instead. The Thames Water year trend (adding +9.8% R²) cannot distinguish genuine infrastructure deterioration from expanding monitoring coverage — the jump from 620 events in 2022 to 21,024 in 2023 is consistent with both explanations.

Correlation, not causation. Rainfall is plausibly exogenous, but pipe age, catchment imperviousness, and urban density are omitted variables that affect spill frequency and correlate with regional rainfall patterns. A causal design would require an instrumental variable or a difference-in-differences exploiting known infrastructure investment events.