Duration: 2 days, 15 hours and 6 minutes

Affected Components: Core Router - 01

Apr 17, 19:25:00 GMT+0
Monitoring - We have identified a large-scale volumetric DDoS attack targeting our infrastructure, which is causing intermittent connection drops, timeouts, and performance fluctuations. Mitigation is actively in progress and traffic is being filtered upstream. Some users may continue to experience degraded performance while we work to fully stabilise the service. Further updates will follow as mitigation progresses..

Apr 18, 12:12:09 GMT+0
Monitoring - As part of our initial mitigation, peering was temporarily taken down to help contain the attack. We are now beginning to bring peering back up in stages. Upstream filtering remains active and connections should be stable, though users may notice slightly elevated ping times while peering is restored. Further updates to follow..

Apr 20, 10:31:06 GMT+0
Resolved - This incident has been resolved..

Apr 20, 10:32:36 GMT+0
Postmortem - # Reason for Outage (RFO) - Network Incident, 17 April 2026 ## Summary On **17 April 2026** our network experienced two related DDoS events. A short-duration event at **02:00** caused brief service degradation and was resolved by **02:30**; we now assess this to have been a reconnaissance probe by the same actor. The main attack began at **20:15** the same day, peaking above **2 Tbps**. It employed a carpet bombing strategy, spreading malicious traffic across a wide range of our IP space rather than concentrating on a single host, which reduced the effectiveness of conventional destination-based mitigation. Service stability was restored by **20:41** after we withdrew all peering and transit sessions and routed 100% of traffic via our upstream scrubbing path. The network continued to operate in this scrub-only configuration over the weekend as a precaution, with **full capacity restored at approximately 09:00 on Monday 20 April** once peering and transit sessions were safely reinstated. During the scrub-only period, the network remained fully operational but some customers may have seen sub-optimal routing and modestly elevated latency compared to normal conditions. ## Timeline ### Early hours, 17 April 2026 (precursor event) | Time | Event | | ----- | ------------------------------------------------------------------------------------ | | 02:01 | Smaller-scale attack traffic detected against our network; brief service degradation | | 02:30 | Attack traffic subsides; service restored | Based on attack vectors, targeting, and timing, we now assess this earlier event to have been a reconnaissance probe by the same actor ahead of the main attack approximately 18 hours later. ### Friday 17 April 2026 (main attack) | Time | Event | | ----- | ---------------------------------------------------------------------------------------------- | | 20:15 | Automated monitoring detects anomalous ingress traffic; alerts fire | | 20:17 | Upstream scrubbing engaged; attack confirmed as distributed volumetric | | 20:21 | Traffic peaks above 2 Tbps across multiple prefixes | | 20:22 | All peering and transit sessions withdrawn; 100% of traffic routed via upstream scrubbing path | | 20:35 | Traffic volumes begin to decline; scrubbing effectiveness improves | | 20:41 | Service stability restored; network operating in scrub-only configuration | ### Saturday 18 – Sunday 19 April Network continued to operate in scrub-only configuration. Attack traffic monitored for resumption; none observed at prior scale. ### Monday 20 April 2026 | Time | Event | | ----- | ------------------------------------------------------------------------------------- | | 09:00 | Peering and transit sessions progressively reinstated; full network capacity restored | ## Impact ## Precursor event (02:01 - 02:30, 17 April) A short period of degraded service was observed during this event. Impact was limited due to the smaller scale of the traffic. ### Main attack (20:15 - 20:41, 17 April) The incident caused **network-wide degradation of service**. Customers experienced packet loss, elevated latency and jitter, and intermittent connectivity issues. The network remained online throughout, but performance was significantly impacted during the peak of the attack due to widespread congestion. ### Scrub-only period (20:22 on 17 April - 09:00 on 20 April) From the point at which peering and transit sessions were withdrawn, all inbound traffic transited our upstream scrubbing path. The network remained fully operational, but customers may have observed: * Modestly elevated latency for some destinations due to non-optimal routing * Minor changes in path selection visible via traceroute * No packet loss or connectivity issues attributable to this configuration Full performance was restored once peering and transit were reinstated on Monday morning. ## Attack Characteristics Analysis of the main attack traffic showed: * Peak volume above **2 Tbps** * **Carpet bombing** across multiple prefixes, impacting large portions of our IP space simultaneously * Predominantly **UDP-based flood traffic** (IP protocol 17) * High volumes of **fragmented UDP packets**, increasing processing overhead and reducing mitigation efficiency * Traffic distributed across a **large number of global source IPs and ASNs**, consistent with a distributed botnet * Concurrent targeting of multiple destination IPs within the same subnet The combination of fragmentation and wide distribution is a well-known technique designed to bypass destination-based filtering, increase load on mitigation infrastructure, and reduce the effectiveness of single-path scrubbing. The earlier 02:01 event exhibited similar characteristics at significantly smaller scale, which is consistent with reconnaissance intended to test our detection and response profile before committing the main attack. ## Cause At peak, the scale and distribution of the attack exceeded the capacity of our **single upstream mitigation path**. This is a known architectural limitation that this incident brought into sharp focus. The result was: * Saturation of upstream scrubbing capacity * Upstream packet drops and queueing * Spillover congestion affecting broader network performance Because traffic was spread across many IPs and prefixes, it could not be isolated to a single target and required network-wide mitigation handling. ## Mitigation & Response The attack was detected immediately via automated monitoring. Traffic was initially routed through upstream scrubbing, and filtering rules were adjusted dynamically as the attack pattern evolved. As the attack continued to scale and spill over onto peering and transit ingress paths, a decision was taken at **20:22** to **withdraw all peering and transit sessions** and route 100% of inbound traffic through our upstream scrubbing path. This took the remaining attack traffic off our direct ingress and placed it behind dedicated mitigation capacity, which restored stability within minutes. The network was held in this configuration deliberately over the weekend to ensure the attack had fully subsided before reintroducing direct paths. Peering and transit sessions were progressively reinstated from approximately **09:00 on Monday 20 April**, with full capacity restored shortly thereafter. ## Improvements Implemented Following this incident, we have made immediate changes: * **Additional upstream scrubbing capacity** has been provisioned * **Multi-provider mitigation** is now in place, removing reliance on a single scrubbing path * **Mitigation thresholds and escalation behaviour** under extreme load have been re-tuned, including earlier engagement of secondary capacity * **Correlation of smaller precursor events** with subsequent activity has been strengthened, so probe-style traffic is flagged and tracked proactively ## Conclusion This was a large-scale, highly distributed volumetric attack, preceded earlier the same day by a smaller reconnaissance event, designed to stress both capacity and mitigation systems simultaneously through scale, fragmentation, and distribution. Service stability was restored within 26 minutes of detection by withdrawing peering and transit sessions and placing all traffic behind upstream scrubbing. The network was held in this configuration over the weekend as a precaution, with full capacity reinstated on the morning of Monday 20 April. While the network remained operational throughout, the event exposed limitations in single-path mitigation under extreme conditions. Those limitations have now been addressed. We remain committed to transparency about incidents of this nature and will continue to evolve our defensive posture as attack techniques develop. If you have specific questions about how this incident affected your services, please contact support..

Affected Components: Radius - 02, Radius - 01, Database - 01, Database - 02, Core Router - 01

Apr 17, 08:53:19 GMT+0
Resolved - Just a quick update - earlier today we experienced a large scale targeted DDoS attack. The impact was minimal, and our team handled it quickly. Most users shouldn’t have noticed anything beyond possible brief disruption. Everything is now operating normally, and we’re continuing to keep an eye on things. Thanks for your patience 💙.

Duration: 1 day, 13 hours and 12 minutes

Affected Components: CityFibre

Nov 3, 23:31:25 GMT+0
Identified - The Major Incident Team has identified the root cause of the incident as a fibre break, resulting from a high-voltage power issue that caused the cables to melt. Due to the extent of the damage, engineers are required to remove the existing cable and install a new one. The estimated time for full service restoration is approximately 8 hours; however, some services are expected to restore prior to the completion of the works. The Major Incident Team is closely monitoring the situation to ensure the issue is fully resolved. Next update at: 03:30 03/11/2025.

Nov 4, 08:01:47 GMT+0
Identified - Engineers remain on-site and are continuing with the splicing works. CityFibre’s NOC has confirmed that the majority of services have now been restored. However, due to some delays encountered in the field, full restoration is now expected by approximately 07:30\. The Major Incident Team is closely monitoring the situation to ensure the issue is fully resolved. Next update at: 08:00 04/11/2025.

Nov 3, 22:11:40 GMT+0
Investigating - The Major Incident Team are currently investigating an incident that is impacting a number of services in Edinburgh. CityFibre are working to identify the issue and resolve the issue as quickly as possible. Next update at: 22:30 03/11/2025.

Nov 3, 23:06:54 GMT+0
Identified - The Major Incident Team have identified the cause of the incident as fibre strike and are currently working to restore the services. The ERS engineers are currently on-site to repair the fibre. The Major Incident Team are closely monitoring the situation to ensure the issue is fully resolved. Business Impact: Approximately 3000 FTTH, FTTP and Dark Fibre services are down. Next update at: 23:30 03/11/2025.

Nov 4, 08:02:12 GMT+0
Monitoring - CityFibre believes that all services have now been restored. However, we are currently awaiting confirmation from the on-site engineers that the splicing works have been completed and that they are fully hands-off. Next update at: 08:30 04/11/2025.

Nov 5, 11:23:44 GMT+0
Resolved - This incident has been resolved..

Duration: 10 minutes

Affected Components: Core Router - 01

Oct 27, 23:00:00 GMT+0
Identified - Hey everyone - we’ll be performing a quick maintenance tonight at **23:00** to split our **/23 network into two /24s**. This helps us scale and improve routing for future deployments. During the change, you might notice a **short interruption in connectivity**. Once it’s complete, your router will simply need to **request a new DHCP lease** \- don’t worry, **your IP address won’t change**..

Oct 27, 23:00:01 GMT+0
Identified - Maintenance is now in progress.

Oct 27, 23:10:00 GMT+0
Completed - Maintenance has completed successfully.

Duration: 19 hours and 29 minutes

Affected Components: Core Router - 01

Oct 8, 12:59:25 GMT+0
Investigating - We are currently investigating this incident..

Oct 8, 14:00:20 GMT+0
Monitoring - We’re now monitoring the issue closely, and the reduced peering will remain in place while we make sure everything stays stable. .

Oct 9, 08:28:42 GMT+0
Resolved - The DDoS attack has been fully mitigated. Services are now confirmed stable and performing as expected. We’ve implemented some changes to our DDoS defence system to mitigate similar events in the future. Thanks for your patience while we dealt with this one 💪.

Oct 9, 08:30:40 GMT+0
Postmortem - ### **RFO – DDoS Incident** **Date/Time:** 8th October 2025 **Impact:** Intermittent connectivity and packet loss across parts of the network. **Duration:** \~45 minutes **Status:** Resolved **Summary:** At approximately 14:00, our network began experiencing a large-scale Distributed Denial of Service (DDoS) attack targeting our edge infrastructure. The attack caused intermittent packet loss and service degradation for some customers. **Actions Taken:** * Identified malicious traffic patterns. * Temporarily withdrew our route servers via **LINX** and **LONAP** to isolate and contain the impact. * Engaged automated DDoS filtering which successfully mitigated the attack and restored stability. **Preventative Measures:** We’ve implemented some changes to our DDoS defence system and filtering rules to mitigate this in the future. **Status:** All services are now stable and fully operational..

Oct 8, 13:30:15 GMT+0
Identified - We’re currently being targeted by a DDoS attack which is causing some disruption. Our network team is actively mitigating and things should start stabilising soon. Please bear with us while we get everything back to normal 💪.

Duration: 3 days, 16 hours and 52 minutes

Affected Components: CityFibre

Sep 25, 21:45:03 GMT+0
Investigating - We are seeing reports of slightly increased latency in Scotland. This does not appear to be isolated to Olilo, as there are reports from outside our network as well. The issue has been raised with CityFibre and we’ll provide updates as we receive them..

Sep 29, 14:36:45 GMT+0
Resolved - We’ve run checks on our side, and the increased latency only seems to be affecting certain CityFibre areas (Scotland). Since there’s no actual downtime, this isn’t considered a fault on their end. The best we can do for now is wait for CityFibre to resolve it, likely as part of ongoing upgrades or network improvements..

Duration: 1 hour

Affected Components: Core Router - 01, CityFibre

Sep 23, 22:00:00 GMT+0
Identified - We’ll be carrying out scheduled maintenance on **Tuesday, 23/09/2025 between 23:00 and 00:00**. During this window, an FPC reboot will take place, which may cause a short service interruption. Service will automatically restore once the reboot is complete, and we expect the overall impact to be minimal. This work is an exciting next step in bringing **Openreach customers onto the Olilo network**. Once complete, we’ll be ready to start welcoming you onto our **Core Network**. Thanks for bearing with us while we continue building out the network!.

Sep 23, 22:00:01 GMT+0
Identified - Maintenance is now in progress.

Sep 23, 23:00:00 GMT+0
Completed - Maintenance has completed successfully.

Duration: 30 minutes

Affected Components: Core Router - 01, CityFibre

Sep 14, 23:30:00 GMT+0
Identified - We will be performing planned maintenance on **Monday 15/09/2025** between **00:30 and 01:00**. During this time, an FPC reboot will take place which may cause a brief interruption in service. We expect the impact to be minimal and service will automatically restore once the reboot is complete. Thank you for your patience..

Sep 15, 00:00:00 GMT+0
Completed - Maintenance has completed successfully..

Duration: 16 hours and 33 minutes

Affected Components: CityFibre

Aug 6, 20:49:31 GMT+0
Identified - Major Incident Management can confirm that a restoration plan has been put in place - The restoration strategy involves initiating a new cable route by tapping into the CF Joint. From there, the team will install a new length of cable leading directly to a track joint that will be positioned within one of the designated carriageway chambers. This track joint will serve as an intermediary connection hub, allowing for flexible routing in turn restoring services to the affected customers. Due to the level of work that is required it has been confirmed that this work will carry into the night - Based on this next comms will be published at 1am on the 7th of August Updates to follow at the incident progresses. \*\*Next update at: 01:00 07/08/2025\*\*.

Aug 6, 14:46:49 GMT+0
Investigating - The Major Incident Team are currently investigating an incident that is significantly impacting services in Glasgow and surrounding areas. CityFibre are working to identify the issue and resolve the issue as quickly as possible. **Next update at: 16:30 06/08/2025**.

Aug 6, 15:46:11 GMT+0
Investigating - CityFibre have got engineers on-site and our team will continue to triage the issue. **Next update at: 17:30 06/08/2025**.

Aug 6, 17:50:57 GMT+0
Investigating - Engineers are currently on-site and actively investigating the issue, with all relevant support teams fully engaged. To ensure the teams have the space to focus on resolution efforts, we are managing communications carefully and will provide an update as soon as further information becomes available. Major Incident Management are closely monitoring the situation to ensure the issue is resolved as quickly as possible. **Next update at: 20:00 06/08/2025**.

Aug 6, 19:02:47 GMT+0
Identified - Major Incident Management has held an additional bridge call with our NOC, Planning team, and on-site personnel. It has been confirmed that the issue was caused by third-party civil works taking place in the Glasgow area. The ground team is currently developing a restoration plan to reinstate services between Glasgow and Inverness. Updates to follow at the incident progresses. **Next update at: 22:00 06/08/2025**.

Aug 7, 07:19:29 GMT+0
Resolved - CityFibre is pleased to confirm that the service disruption has been fully resolved as of 03:08 on 07/08/2025. The incident was caused by ongoing third-party civil works in the Glasgow area, which impacted multiple services across our network. We sincerely appreciate your patience and understanding during this time. .

Duration: 6 hours and 1 minute

Affected Components: CityFibre

Aug 5, 11:50:02 GMT+0
Investigating - We have had reports of outages in the Glasgow area. This has been reported to CityFibre and will update once we get confirmation..

Aug 5, 11:55:33 GMT+0
Investigating - CityFibre are aware of an ongoing incident affecting a number of services. Engineers are conducting preliminary investigations to determine the comprehensive Service Impact Analysis. CityFibre apologise for the inconvenience caused. Our main priority is to deliver the levels of service that our customers deserve and as such we have invoked our Major Incident Process. CityFibre are working to affect a full restoration as soon as possible. They will continue to issue regular updates until all services have been restored. Further details will be issued within 15 minutes..

Aug 5, 12:45:41 GMT+0
Identified - The Major Incident Team are currently investigating an incident that is significantly impacting services in Glasgow and surrounding areas. We are working to identify the issue and resolve the issue as quickly as possible. **Next update at: 14:30 05/08/2025**.

Aug 5, 13:36:14 GMT+0
Identified - CityFibre have identified the cause of the incident as a broken fibre, engineers are on-site repairing the fibre to restore the service. CityFibre apologise for any inconvenience. CityFibre are closely monitoring the situation to ensure the issue is fully resolved. **Next update at: 15:30 05/08/2025**.

Aug 5, 14:32:29 GMT+0
Monitoring - CityFibre have now confirmed splicing has been completed. Services are now back online..

Aug 5, 17:50:51 GMT+0
Resolved - This incident has been resolved..

Duration: 7 hours and 34 minutes

Affected Components: CityFibre

Aug 3, 08:51:06 GMT+0
Investigating - We have had reports of outages in the Glasgow area. This has been reported to CityFibre and will update once we get confirmation..

Aug 3, 10:39:40 GMT+0
Investigating - We have contacted CityFibre support again who have said they are unaware of any issues but have raised to the networks team. We are aware of the rumour of damaged cable replacement but we are at liberty to CityFibre confirming this. If this is planned and not been communicated we will be raising a complaint..

Aug 3, 11:46:19 GMT+0
Monitoring - We have had reports that members are now back online. We will keep monitoring this and resolve if we don’t hear anything further. .

Aug 3, 16:25:11 GMT+0
Resolved - As we have had no further reports we will be marking this has resolved..

Duration: 10 hours and 9 minutes

Affected Components: CityFibre

Jul 29, 09:15:41 GMT+0
Investigating - CityFibre are aware of an ongoing incident affecting a number of services. Engineers are conducting preliminary investigations to determine the comprehensive Service Impact Analysis. CityFibre apologise for the inconvenience caused. They are working to affect a full restoration as soon as possible..

Jul 29, 09:30:14 GMT+0
Investigating - The Major Incident Team are currently investigating an incident that is causing a high level of degraded services across Sheffield. They are working to identify the issue and resolve the issue as quickly as possible. **Completed Actions:** \- Major Incident Process Invoked. - CityFibre NOC Engaged. - CityFibre Ground Team Engaged. **Next update at: 11:30 29/07/2025**.

Jul 29, 10:41:22 GMT+0
Investigating - The Major Incident Team held an additional bridge call with internal support teams and can confirm that our field engineers are currently on-site conducting triage testing at the affected location. Once the exact fault is identified using OTDR (Optical Time Domain Reflectometer) diagnostics, ground crews will be mobilised to the suspected break point to initiate repairs and restore degraded customer services. **Current Impact** There is no total outage at this time. However, customers may be experiencing severe latency and reduced service performance. Further updates will follow as restoration efforts progress. **Completed Actions:** \- CityFibre Ground Team at the affected FEX. **Next update at: 13:30 29/07/2025**.

Jul 29, 12:34:50 GMT+0
Identified - CityFibre engineers have conducted an OTDR trace at our Rotherham FEX (rot36), focusing on **Rack 9, ODF Ports 5 and 6**. The trace has identified a possible fault impacting **Port 6**, which is now under detailed investigation. Major Incident Management is actively engaged, and ground teams have been authorised to proceed with repair activities once the fault location is confirmed. Communications have been approved for release as progress is made. There is **no total outage** at this time. However, customers may experience **severe latency** or **degraded service performance**. Ground teams are preparing for repair work following fault confirmation. Major Incident Management will oversee all updates and changes to ensure timely resolution. **Next update at: 15:00 – 29/07/2025**.

Jul 29, 14:38:52 GMT+0
Identified - As of now, restoration efforts are actively progressing. Major Incident Management remains closely engaged and is working in collaboration with support teams to coordinate a comprehensive resolution to the issue. Further updates will be shared as progress continues. **Current Impact** There is no total outage at this time. However, customers may be experiencing severe latency and reduced service performance. **Next update at: 17:30 29/07/2025**.

Jul 29, 19:25:07 GMT+0
Resolved - CityFibre are pleased to inform you that the issue has been resolved. The issue was due to damaged fibre, spare fibre was used to restore service. Thank you for your patience during this time..

Duration: 1 hour

Affected Components: Openreach, CityFibre

Jul 4, 00:00:01 GMT+0
Identified - Maintenance is now in progress.

Jul 4, 00:00:00 GMT+0
Identified - During this window our partner will be introducing 3x new LNS routers in London to support terminating broadband subscribers. As a result they will need to re-balance sessions onto the new LNS routers by disconnecting currently online tails. A brief disconnect and reconnect will be observed during this window for subscribers..

Jul 4, 01:00:00 GMT+0
Completed - Maintenance has completed successfully.

Duration: 17 hours and 48 minutes

Affected Components: CityFibre

Jun 30, 12:59:00 GMT+0
Investigating - CityFibre has declared a **Major Incident** due to a core link failure between London (LON5) and Slough (SLO664). This is affecting all services in the area..

Jun 30, 13:00:00 GMT+0
Investigating - Engineers from both FLM and ERS have been dispatched to investigate. We're currently waiting on ETAs for both teams. At the same time, CityFibre has escalated to the vendor to check for any hardware or software faults in the active network equipment..

Jun 30, 15:40:00 GMT+0
Investigating - We are aware of an ongoing incident affecting a number of services in Slough. Engineers are carrying out preliminary investigations to assess the full service impact. We apologise for the disruption — restoring service is our top priority, and our Major Incident Process remains active. CityFibre is working urgently to achieve a full restoration. Further details will be issued within 15 minutes. Thanks for bearing with us!.

Jun 30, 16:08:00 GMT+0
Investigating - The Major Incident Team has received an ETA for ERS and FLM of 18:45 and 18:40 respectively. In the meantime, the vendor responsible for the active equipment are continuing to investigate the hardware. The Major Incident Team continues to closely monitor the situation and will ensure that all necessary actions are taken to achieve a full resolution..

Jun 30, 18:48:00 GMT+0
Identified - ERS and FLM engineers on-site have identified that the issue lies with the active hardware. The Major Incident Team has engaged the vendor, who is continuing remote diagnostics via the FLM engineer on-site. The vendor has also dispatched a technician, with an ETA of approximately 21:00. The Major Incident Team continues to closely monitor the situation and will ensure that all necessary actions are taken to achieve a full resolution. **Next update at: 21:30 30/06/2025**.

Jun 30, 21:04:31 GMT+0
Identified - The vendor’s technician has arrived on-site and is working with their remote teams and CityFibre’s Technical teams to troubleshoot the issue. The Major Incident Team continues to closely monitor the situation and will ensure that all necessary actions are taken to achieve a full resolution. **Next update at: 23:30 30/06/2025**.

Jun 30, 22:18:08 GMT+0
Identified - The vendor’s technician has determined the fan modules and power modules have failed in the chassis. New fan modules and power modules have been requested and have ETA of 02:00. The Major Incident Team will continue to closely monitor the situation and will ensure that all necessary actions are taken to achieve a full resolution as quickly as possible. **Next update at: 02:30 30/06/2025**.

Jul 1, 06:46:41 GMT+0
Resolved - CityFibre is pleased to confirm that the issue was successfully resolved at 01:14 on 01/07/2025. The issue was resolved by reloading the configuration, restoring full-service functionality. The Major Incident Team will continue to monitor for stability..