↓ Skip to main content
Side Effects
  1. Home projects/

A new control panel for a fiber network

··Index: 6·
Python Fastapi Vue Snmp Networking
Table of Contents

RTK Nowy Sącz (Regionalna Telewizja Kablowa) is a regional cable TV and internet provider that runs a fiber-to-the-home network on GPON, with optical line terminals (OLTs) in the network and an optical network unit (ONU) in every subscriber’s home. Operators manage all of this with a web tool that lists the OLTs and ONUs, reads the live status of an ONU, manages ONU profiles, adds new units and shows the subscriber behind each one.

The tool has been in production since 2014 on Python 2 and Flask. It worked, but any query to the hardware blocked the server, and the code was at the end of the road, so I rebuilt it while the old version kept working.

From Python 2 to an async stack
#

This time the backend is Python 3.13 with FastAPI, asyncpg for the database and PySNMP for the hardware, behind a new versioned API. On the front I used Vue 3 with PrimeVue, vue-query and a typed client generated from the API contract. The old API keeps running next to the new one, so I can switch parts of the system over (or back) separately.

The old Flask API and the new FastAPI with a Vue 3 front end run side by side and both talk to the OLTs.Old, in production since 2014NewPython 2 + Flask APIa query blocks the serverOLT hardwareSNMPVue 3 front endFastAPI, async, versioned APIPython 3.13asyncpgPySNMPdatabaseOLT hardwareside by side
The old Flask API and the new async stack side by side, both reaching the OLTs

Making the hardware fast and safe to ask
#

OLTs answer over SNMP, they’re slow, and it doesn’t take much to overload them. The old code opened a new SNMP session for every call, had no timeouts or retries, and silently cut off tables longer than one bulk read. On top of that, five half-finished SNMP implementations were sitting in the repository. I kept the one with the best architecture, fixed it, and put a chain of protections in front of the hardware:

  • cache-aside with short lifetimes and a short “stale while error” window, so a device that stops answering gets its last known good data served, marked as stale;
  • singleflight deduplication, where identical requests arriving together share one query to the device;
  • a queue per OLT, with a limit on parallel operations and writes going before reads;
  • a circuit breaker per OLT, which gives a failing device a rest and lets the API answer at once instead of hanging;
  • separate answers for an empty table, a timeout, an unavailable device, a full queue and an unknown unit, with errors never cached;
  • a time budget you can do arithmetic on: timeouts, retries and deadlines add up to less than the request limit, and a test checks that they still do.
A request passes a cache, request deduplication, a per-device queue and a circuit breaker before it reaches the OLT hardware.cache hit: no query to the deviceAPI requestcachesingleflightqueueper OLTcircuitbreakerOLTover SNMPshort lifetimes,stale data if thedevice failsidentical requestsshare one querylimit on paralleloperations,writes before readsfailing device getsa rest, API answersat once20 identical requests make 1 query to the devicea warm-cache read is about 50 times faster than a cold read
The protections between an API request and the OLT hardware

Results
#

All numbers are relative, measured on the real hardware or on an SNMP simulator.

  • Reading an ONU’s host table without the old “trigger” writes before it is 1.6 to 3.1 times faster, depending on the OLT, and that’s now the default path.
  • A read from the warm cache is about 50 times faster than a cold read from the device.
  • 20 identical concurrent requests make exactly 1 query to the device, and 100 requests on a warm cache make none.
  • Large tables now come back complete, thanks to a proper continuation loop for bulk reads.

Testing and rollout
#

I built an OLT simulator and an SNMP lab to be able to run everything locally, and integration tests run against simulated devices. Tests that would touch live equipment refuse to start unless you name the target explicitly.

Writes to the database and SNMP writes to the network are separate switches, and both stay off until someone enables them. A new release goes live only after the health check passes, and a single command rolls it back. I also removed credentials and real network addresses from the repository, including all of the git history, and hooks now block them from coming back.

Related