What is SONiC?
Imagine a huge post office with thousands of mail slots. Letters (data packets) arrive all the time, and a very fast machine decides which slot each letter goes into. That machine is the switch chip. It is extremely fast, but it cannot think. It only follows a list of rules.
Someone has to write those rules, keep them up to date when roads close, and check that everything is healthy. That someone is the switch's operating system. For years, every brand of chip came with its own operating system, and it was locked shut.
SONiC is a free, open operating system that can drive switch chips from many different brands. Inside it, a team of small helpers each have one job. One learns routes from neighbours, one watches the fans, one counts packets. They never talk to each other directly. Instead they write notes on one big shared whiteboard (a database). A translator reads the whiteboard and turns the notes into rules the chip understands.
- Post office sorter
- The switch ASIC, the chip that forwards packets
- Helpers
- Docker containers: bgp, teamd, lldp, pmon, snmp…
- Shared whiteboard
- Redis databases (CONFIG_DB, APPL_DB, ASIC_DB, …)
- Translator
- orchagent + syncd, speaking SAI to the chip
Task & purpose
SONiC stands for Software for Open Networking in the Cloud. Microsoft created it to run the Azure data-center network and released it as open source in 2016. It is now governed by the SONiC Foundation under the Linux Foundation.
The problem
Cloud operators run tens of thousands of switches. With a closed vendor OS, every bug fix waits on the vendor, every feature waits on the vendor's roadmap, and every vendor needs its own tools.
The job
SONiC does three things. It runs the control plane (BGP, LACP, LLDP…). It turns what the control plane decides into hardware forwarding tables. And it exposes the switch to automation through CLI, gNMI, REST and telemetry.
The key idea
Disaggregation. Hardware and software are chosen separately. The SAI layer lets one software stack run on many ASICs.
Design priorities
Simple, focused features for the cloud (mostly BGP-based L3 Clos fabrics). Upgrades without dropping traffic. Everything automatable. Standard Linux and Debian underneath.
Repo context: README.md · governance.md · sourcecode.md (map of the sonic-net repositories)
General overview
SONiC is a Debian Linux image. Most of its features run as Docker containers, and the containers coordinate through a Redis database container. The layered view below is the single most useful thing to memorize.
Containers do the work
Each feature is isolated, so it can be restarted, upgraded or turned off on its own.
The database is the bus
All communication goes through Redis tables using a producer/consumer pattern, plus keyspace notifications.
Intent flows down
The chain is CONFIG_DB → APPL_DB → ASIC_DB → hardware. Each step translates to a lower-level language.
State flows up
Counters, link states and health go into COUNTERS_DB and STATE_DB, where the CLI, SNMP and telemetry read them.
The container model
Running each feature in a container provides fault isolation, independent restarts, and an easy way to ship third-party features. systemd manages the containers. Inside each one, supervisord manages the processes.
| Container | Role | Key processes |
|---|---|---|
database | Hosts the Redis server(s). Every other container depends on it. | redis-server |
swss | Switch State Service. Turns intent into SAI objects. | orchagent, portsyncd, neighsyncd, vlanmgrd, intfmgrd, portmgrd… |
syncd | The only process that talks to the vendor SAI and SDK. | syncd (links the vendor libsai) |
bgp | Routing stack. | FRR: bgpd, zebra, staticd; fpmsyncd; bgpcfgd |
teamd | Link aggregation (LACP). | teamd, teamsyncd, teammgrd |
pmon | Platform monitoring: fans, PSUs, thermals, transceivers. | xcvrd, psud, thermalctld, ledd, syseepromd |
lldp, snmp, dhcp_relay, radv | Protocol and management features. | lldpd, lldp_syncd, snmpd + agent… |
gnmi / telemetry, mgmt-framework | Northbound programmatic management. | gNMI server, REST server, translib |
lldp does not bring down forwarding.Read: Container hardening · Optional feature control · Docker-to-host communication
The Redis databases
SONiC keeps its state in several logical Redis databases, one per kind of state. The mapping is defined in database_config.json. By default all of them share one Redis instance, but they can be split across several instances.
| DB (id) | Holds | Written by → Read by | Key separator |
|---|---|---|---|
APPL_DB (0) | Application intent: routes, neighbors, port state | *syncd / *mgrd daemons → orchagent | : |
ASIC_DB (1) | SAI objects and attributes to program | orchagent (via sairedis) → syncd | : |
COUNTERS_DB (2) | Port, queue and ACL statistics | syncd flex-counter → CLI, SNMP, telemetry | : |
LOGLEVEL_DB (3) | Per-daemon log levels | CLI → daemons | : |
CONFIG_DB (4) | Persistent configuration (the source of truth) | CLI, gNMI, config load → *mgrd, orchagent | | |
FLEX_COUNTER_DB (5) | Which counters to poll, and how often | orchagent → syncd | : |
STATE_DB (6) | Observed runtime state: oper status, transceiver info, features | many → many | | |
Example keys: CONFIG_DB: VLAN|Vlan100, APPL_DB: ROUTE_TABLE:10.0.0.0/24, ASIC_DB: ASIC_STATE:SAI_OBJECT_TYPE_ROUTE_ENTRY:{…}. /etc/sonic/config_db.json is a persisted snapshot of CONFIG_DB, loaded at boot.
How producers and consumers communicate
The sonic-swss-common library provides ProducerStateTable and ConsumerStateTable. A producer writes the key into a temporary table, adds the key to a "pending keys" set, and publishes a notification, all atomically in a Lua script. The consumer wakes up, pops a batch of keys, and applies them. Because the pending-keys set holds each key once, repeated updates to the same key before the consumer runs coalesce into one. This is built-in de-duplication.
Read: Multiple DB instances · Multi-namespace DB · ZMQ state tables · example config_db.json
Life of a route
Following one BGP route from the wire to the chip shows almost every design idea in SONiC.
- bgpd receives an UPDATE from a peer, runs best-path selection, and hands the route to zebra.
- zebra installs the route in the kernel and sends it to fpmsyncd over the Forwarding Plane Manager (FPM) channel.
- fpmsyncd writes
ROUTE_TABLE:<prefix>into APPL_DB. It queues writes in a Redis pipeline and flushes them in batches. - orchagent (RouteOrch) consumes the change. It resolves next hops (NeighOrch), builds or reuses ECMP next-hop groups, checks resources, and creates SAI route objects.
- sairedis serializes those SAI calls into ASIC_DB. By default this is asynchronous; in synchronous mode orchagent waits for the result.
- syncd reads ASIC_DB, translates virtual object IDs (VIDs) to real hardware IDs (RIDs), and calls the vendor SAI, which programs the ASIC.
- Optionally, the result is reported back so that BGP only advertises routes that are actually installed in hardware ("suppress FIB pending").
Read: BGP loading optimization · FRR↔SONiC channel · Suppress FIB pending · Orchagent route redesign · Route performance
orchagent, syncd & SAI
orchagent
A single-threaded event loop (select() over many consumer tables). It is organized into "Orch" modules such as PortsOrch, RouteOrch, NeighOrch, AclOrch and VxlanOrch. Tasks that cannot run yet, for example a route whose next hop has not been resolved, stay in a queue and are retried later. The modules depend on each other, so the order in which work is handled matters.
syncd
Owns the vendor SDK process. It keeps the VID↔RID mapping, so the software above uses stable virtual IDs while hardware IDs can change across restarts. It also runs flex-counter polling and forwards hardware notifications, such as port state changes and FDB events, back up to orchagent.
SAI
The Switch Abstraction Interface, an OCP-standard C API (create/remove/set/get for object types like ROUTE_ENTRY and NEXT_HOP_GROUP). Each vendor ships its own libsai. It is an adapter pattern at industry scale.
Failure handling
What happens if the hardware rejects a request, for example because a table is full? The design docs cover several options: retry, report the error back up through APPL_DB/STATE_DB, take a SAI dump for debugging, and mark the component as unhealthy.
Read: SAI failure handling · Dump on SAI failure · Orchagent error handling · Synchronous mode · Notification queue policy
Configuration & management
All configuration ends up in CONFIG_DB. The management interfaces are simply different ways of writing to it.
CLI
A Python click CLI (config …, show …). config save persists CONFIG_DB to config_db.json. config reload reapplies it from disk.
YANG models
SONiC YANG models define the schema of CONFIG_DB. They validate changes before they are written.
Generic Config Update (GCU)
Applies JSON Patch changes to the running config. The patch is split into steps and ordered by YANG dependencies. Checkpoints allow rollback.
gNMI / gNOI / REST
Programmatic management and streaming telemetry. The mgmt-framework translates OpenConfig models into SONiC tables.
ZTP
Zero-Touch Provisioning: a new switch boots, fetches its config and image over DHCP/HTTP, and sets itself up.
Golden config
A single JSON file that describes the whole desired config, even on multi-ASIC systems.
Read: GCU & rollback · JSON Patch ordering · YANG validation · Management Framework · YANG guidelines · ZTP · Golden config
Platform & hardware
Besides the ASIC, a switch has fans, power supplies, sensors, transceivers (optics) and LEDs. The pmon container watches them through a vendor-implemented Python platform API.
Transceivers
xcvrd detects plug and unplug events, reads EEPROM and diagnostic data (CMIS/SFF), and applies media-specific settings.
Thermal & power
thermalctld and psud run policy-driven fan control and PSU monitoring. system-health combines all of it into one status.
Firmware
fwutil handles component firmware upgrades (BIOS, CPLD, FPGA).
Read: pmon enhancement · Transceiver monitor · Media settings · System health · CMIS diagnostics
Reliability & hitless upgrades
| Mode | Data plane impact | How it works |
|---|---|---|
| Cold reboot | Minutes of outage | Full restart. The ASIC is reinitialized. |
| Fast reboot | Under ~30 s outage | Kexec into the new image while preserving key state such as ARP, FDB and CoPP tables. The ASIC is still reset. |
| Warm reboot / warm restart | Target: no packet loss | The ASIC keeps forwarding while the software restarts. State is saved, restored, reconciled, and only the differences are applied. |
Warm restart, step by step
- Freeze. Stop processing new changes and let pending tasks drain.
- Save. Save the Redis databases and the protocol state. BGP graceful restart asks peers to keep forwarding, and LACP timeouts are extended.
- Restart. The new container or image starts, and APPL_DB and ASIC_DB are restored.
- Reconcile. orchagent rebuilds its internal state from APPL_DB. syncd compares the old and new "views" of ASIC_DB and applies only the diff. SAI calls must be idempotent.
- Converge. Protocols resync with peers, and stale entries are cleaned up once timers expire.
Read: SONiC Warmboot · SWSS warm restart · View switch · SAI idempotence · Fast reboot · LACP during warm reboot
Scaling out: multi-ASIC, chassis & VOQ
Multi-ASIC
One box contains several ASICs. Each ASIC gets its own Linux network namespace with its own database, swss, syncd, bgp and teamd. A global host namespace handles management. This is partitioning by physical resource.
Chassis
A modular chassis with a supervisor card and line cards. Each line card runs its own SONiC. Shared state is kept in a chassis-level database on the supervisor.
VOQ
Virtual Output Queues schedule traffic across the fabric. A distributed forwarding design needs a global view of neighbors and interfaces system-wide.
Read: Multi-ASIC HLD · Multi-ASIC DB · VOQ architecture · VOQ HLD · Line card provisioning
Observability
Counters
Flex counters: orchagent registers counter groups in FLEX_COUNTER_DB, and syncd polls them from hardware into COUNTERS_DB at a configurable interval.
Streaming telemetry
A gNMI server supports subscribe (dial-in) and dial-out. High-frequency telemetry pushes sub-second samples straight from the ASIC.
Resources
CRM tracks used and available entries in each hardware table and raises threshold alarms.
Events & logs
Syslog with rate limiting. A structured, YANG-described event/alarm framework. Auto-techsupport and kdump capture evidence when something crashes.
Packet visibility
sFlow sampling, Everflow (mirroring driven by ACLs), drop counters, and watermarks.
Read: Flex counter · gRPC telemetry · HFT · CRM · Event/alarm · Auto-techsupport · sFlow
High availability
ECMP
The simplest form of HA in a Clos fabric: many equal-cost paths. Fine-grained and consistent hashing reduce how many flows move when a path fails.
Dual ToR
A server connects to two top-of-rack switches through a mux cable. The design comes in active/standby and active/active variants, with health probing and switchover.
MCLAG
Two switches act as one LAG endpoint. They sync state with ICCP (iccpd).
SmartSwitch HA
DPU pairs with a dedicated HA manager (hamgrd). The design covers flow-state sync, liveness through BFD, and split-brain handling.
Read: Fine-grained ECMP · Dual ToR A/S · Dual ToR A/A · MCLAG · SmartSwitch HA
Security
Control-plane protection
CoPP applies per-trap-group policers so that packet floods cannot overwhelm the CPU. Storm control limits broadcast, unknown-unicast and multicast traffic.
Access
AAA with TACACS+ and RADIUS, password hardening, and SSH configuration.
Supply chain
Secure boot, signed images (secure upgrade), and FIPS-validated crypto.
Least privilege
Container hardening removes --privileged and unneeded capabilities from containers.
Read: CoPP · Storm control · TACACS+ · Secure upgrade · Container hardening
Build & extensibility
The sonic-buildimage repository builds one image per ASIC vendor (sonic-broadcom.bin, sonic-mellanox.bin, sonic-vs.img for the virtual switch, and others). Builds are containerized, cached and reproducible. The Application Extension framework turns SONiC into a platform for third-party features: a package manager for Docker images, with manifests, dependencies and versioning.
Read: Reproducible build · Build enhancements · App extension HLD · Versioning strategy · Kubernetes support
New frontiers
DASH & SmartSwitch
Disaggregated API for SONiC Hosts. It offloads SDN services (VNET, ACLs, load balancing) onto DPUs, with millions of connection flows per card.
PINS
P4 Integrated Network Stack. An SDN controller programs the switch through P4Runtime, while SAI stays underneath.
AI fabrics
Ethernet scale-up for AI clusters, adaptive routing (ARS), packet trimming, and link-level retry.
Read: DASH HLD · PINS HLD · AI scale-up · Adaptive routing
Glossary
| ASIC | The switching chip that forwards packets in hardware at terabits per second. |
|---|---|
| SAI | Switch Abstraction Interface: a vendor-neutral C API for programming ASICs. |
| SWSS | Switch State Service: the container holding orchagent and the *syncd/*mgrd daemons. |
| orchagent | Translates APPL_DB and CONFIG_DB intent into SAI objects in ASIC_DB. |
| syncd | Applies ASIC_DB to the hardware through the vendor SAI. Polls counters. |
| VID / RID | Virtual object ID (stable, used in software) and real object ID (from hardware). |
| FPM | Forwarding Plane Manager: the channel that carries routes from zebra to fpmsyncd. |
| ECMP | Equal-Cost Multi-Path: spreads flows across several next hops by hashing. |
| CoPP | Control Plane Policing: rate limits for traffic sent to the CPU. |
| CRM | Critical Resource Monitoring: tracks hardware table usage. |
| ToR / Clos | Top-of-Rack switch / a multi-stage leaf-spine data-center topology. |
| gNMI / gNOI | gRPC Network Management / Operations Interface. |
Full doc index: sonic_docs_toc.md · doc/ folder