Case Study 01 — VPN & Routing
The VPN That Randomly Broke — And Nobody Knew Why
Random disconnections, no clear logs, vendor support found nothing. The real cause was three separate factors that only became critical in combination.
Situation
VPN in place, modern firewall, no obvious errors. Yet users were getting disconnected constantly and randomly. Logs showed no clear fault. Vendor support found nothing. The problem persisted across weeks of attempted fixes.
The client had invested in current-generation firewall hardware specifically because of VPN reliability concerns. The disconnections continued regardless. Increasing frustration, both from end users and from internal IT, was creating pressure to replace equipment that may not have been the problem at all.
What Was Actually Happening
The investigation revealed three distinct problems operating simultaneously. Each one was individually manageable. Together, they created a pattern of failure that looked random but was structurally deterministic.
- Asymmetric routing — inconsistent traffic paths depending on load and timing meant that return packets were taking different paths than outbound packets
- MTU mismatches causing silent packet fragmentation mid-session — packets were being silently dropped rather than fragmented correctly
- Firewall session tables breaking under specific traffic conditions — stateful inspection was losing track of sessions under load
- Each factor individually harmless — together producing systemic instability that appeared random from the outside
The asymmetric routing problem was particularly subtle. Under low load, packets would typically follow consistent paths. Under higher load, routing decisions would shift — and the firewall, designed for stateful inspection, would lose track of established sessions when return traffic arrived on an unexpected interface.
Why Standard Diagnostics Missed It
Each individual component passed its own diagnostics. The firewall reported healthy session tables. Routing looked correct from any given point in the topology. MTU problems weren't visible in average packet analysis because the fragmentation was happening silently — packets were being dropped rather than generating obvious errors.
The combination of factors only became visible through simultaneous analysis across multiple layers: routing tables under actual traffic conditions, real-time session tracking, and MTU path analysis using appropriately-sized test packets.
What FM-NetSec Nordic Did
- Rebuilt routing logic to enforce deterministic traffic paths — eliminating the conditions under which asymmetric routing could occur
- Fixed MTU and MSS handling end-to-end across all relevant interfaces — ensuring correct fragmentation rather than silent drops
- Stabilised firewall session behaviour under varying load conditions — adjusting session table parameters to handle the actual traffic patterns present
Result
- Zero random disconnections following remediation
- Support tickets related to VPN dropped drastically
- Predictable, stable connectivity across all sites
"If your VPN 'randomly fails', it's almost never the VPN."
The hardware was not the problem. The VPN software was not the problem. The issue was in the infrastructure surrounding both — in the routing decisions, the MTU configuration, and the stateful firewall behaviour under real-world load. Replacing equipment would have changed nothing.
Questions About This Case
What caused the VPN disconnections?
Three simultaneous problems: asymmetric routing that shifted under load, an MTU/MSS mismatch causing silent packet drops, and stateful firewall behaviour that lost track of sessions when return traffic arrived on an unexpected interface. Each problem was individually manageable — together they created failures that appeared random.
Why did standard diagnostics miss the issue?
Each component passed its own diagnostics in isolation. The firewall reported healthy session tables. Routing looked correct from any single vantage point. MTU problems were not visible in average packet analysis. The failure pattern only became visible through simultaneous analysis across multiple layers under real traffic conditions.
Was replacing the hardware the right move?
No — and it would have changed nothing. The hardware was functioning as designed. The issue was in the infrastructure surrounding it: routing logic, MTU configuration, and how the stateful firewall handled asymmetric traffic paths.
What can be learned from this case?
Multi-factor problems rarely reveal themselves to standard diagnostics. The right approach is methodical analysis across all relevant layers simultaneously — not sequential elimination of individual components that each appear healthy in isolation.
