At a glance
Make one switch programmable
It turns routing decisions into hardware forwarding tables on a single box, and it can restart itself while packets keep flowing.
The data is packets in flight. If state is lost, it is relearned from neighbors.
Make thousands of disks act as one
It spreads objects across many servers, keeps replicas consistent, and heals itself when disks or hosts fail.
The data is the user's bytes. If state is lost, it is gone for good.
| SONiC | Ceph | |
|---|---|---|
| Domain | Network operating system for data-center switches (L2/L3) | Unified distributed storage: object (RGW, S3/Swift), block (RBD), file (CephFS) |
| Origin | Microsoft for Azure, open-sourced in 2016 through OCP | Sage Weil's PhD research at UC Santa Cruz (mid-2000s). Later Inktank, then Red Hat, then IBM |
| Governance | SONiC Foundation (Linux Foundation) and a TSC (governance.md) | Ceph Foundation (Linux Foundation) and a Ceph leadership team |
| Unit of deployment | A whole OS image that owns the switch (installed with ONIE) | Daemons on general-purpose Linux servers |
| Scope of one instance | One box (or one chassis/multi-ASIC system) | One cluster of up to thousands of nodes |
| Data plane | Fixed-function ASIC hardware, programmed through SAI | Software on CPUs (OSD daemons), writing to raw devices |
| Core state store | Redis, in memory | BlueStore on raw disk, with RocksDB for metadata |
| Coordination | Shared-database pub/sub inside the box. Routing protocols between boxes | Paxos-based monitor quorum. Daemons message each other directly |
| Consistency | Eventual (network convergence) | Strong for RADOS writes (all replicas acknowledge) |
| Main languages | C++ (swss, sairedis), Python (CLI, pmon, config), Go (gNMI) | C++ (core daemons), Python (mgr modules, cephadm), TypeScript (dashboard) |
| License | Mostly Apache 2.0. Bundled GPL components such as FRR and the Linux kernel | Mostly LGPL 2.1/3 |
| Releases | Twice a year, named by date (e.g. 202405, 202411) | Roughly yearly named stable releases (e.g. Reef, Squid) with point releases |
Architectures side by side
Both are collections of role-specific daemons around a source of truth. The difference is where that source of truth lives and how far it reaches.
Protocol daemons write intent into Redis. orchagent turns it into SAI objects, and syncd programs the ASIC. Packets never touch SONiC software, except for control traffic sent to the CPU. Life of a route →
The client hashes the object name to a placement group (PG). CRUSH maps the PG to an ordered list of OSDs. The client writes to the primary OSD, which replicates the write and acknowledges once every replica has committed. Every byte passes through Ceph software, and no central server looks up where data lives.
What they do similarly
Many of the patterns are the same on both sides, even though the two systems solve different problems.
Both exist to separate the software from proprietary hardware appliances, so operators can pick hardware and software independently.
Role-specific processes, each restartable on its own. Both are now deployed as containers (SONiC Docker images; Ceph via cephadm or Rook).
Desired state is declared in one authoritative place. Other daemons watch it and reconcile toward it.
Both are control loops. They compare desired state with actual state and repair the difference, and both must handle state that is partially stale after a restart.
A pluggable backend behind a stable interface. The difference is who implements it: hardware vendors for SAI, Ceph itself for ObjectStore.
Both place work with a deterministic hash instead of a lookup table, and both try to minimize movement when a member joins or leaves (consistent hashing).
Failure detection based on heartbeats and timeouts, tuned to avoid flapping.
HEALTH_OK/WARN/ERR with named checksMany signals rolled up into one health status for operators.
Both expose metrics to external monitoring systems.
Both put a key-value store at the center of their internal state, but they choose opposite durability profiles.
Open governance under a Linux Foundation-hosted foundation, with vendors both collaborating and competing.
Contrasting design decisions
The same design question, answered in opposite ways. In each row, the reasoning follows from what the system protects: forwarding continuity for SONiC, data durability for Ceph.
| Decision | SONiC chose… because… | Ceph chose… because… |
|---|---|---|
| Where state lives | In memory (Redis). Almost all state can be recomputed from config plus neighbors, and speed matters more than persistence. | On disk (BlueStore) with checksums. The state is the product, so losing it is unacceptable. |
| How components talk | Through a shared database (pub/sub tables). Loose coupling, state that is easy to inspect, and simple restarts. | Direct messaging (msgr2) between daemons, plus versioned maps. A central database would bottleneck thousands of nodes. |
| Consistency | Eventual. The network converges, and a short inconsistency (a microloop) is tolerable. | Strong. A write is acknowledged only after every replica commits, so a read never returns lost data. |
| Consensus | None inside, and none between switches. Each box decides independently, so failures stay isolated. | Paxos among monitors. The whole cluster must agree on one authoritative map. |
| Placement | Per-flow hash in the ASIC (ECMP), recomputed at every hop. | Per-object CRUSH in the client, aware of failure domains (host, rack, row). |
| Reaction to failure | Reroute immediately (milliseconds). Nothing needs copying; you just stop using the failed path. | Wait, then re-replicate slowly. A down OSD is only marked out after a timeout (about 10 minutes by default), and recovery is throttled so that moving data does not overwhelm client I/O. |
| Upgrades | Per-box image swap with warm or fast reboot. The ASIC keeps forwarding while the software restarts (reliability). | Rolling, daemon by daemon, orchestrated by cephadm. The cluster stays up because replicas cover the daemon that is restarting. |
| Hardware abstraction | Vendors implement SAI (often as closed binaries). SONiC cannot build forwarding silicon itself. | Ceph implements BlueStore itself on raw block devices. Disks are generic, so Ceph owns the whole stack. |
| Config validation | YANG models, JSON Patch with rollback (GCU). Standard models help with multi-vendor automation. | Typed options in the monitor config DB (ceph config set), with per-daemon overrides. One implementation, so a custom schema is sufficient. |
| Scale strategy | Scale out the network: add more switches (Clos). Each SONiC instance stays small. | Scale out the cluster: add more OSDs to one logical system. A single instance can grow to exabytes. |
| Threading model | orchagent is a single-threaded event loop. Simple and deterministic, with known throughput limits. | The classic OSD uses thread pools. Crimson (Seastar) moves to a shard-per-core design for NVMe speeds. |
Decision deep dives
Open any decision to see why each project chose what it did, the pros and cons on both sides, and where to find the evidence in each codebase. Ceph source paths refer to the ceph/ceph monorepo.
1. Where does state live?SONiC: in-memory RedisCeph: on-disk BlueStore + RocksDB
Why SONiC chose this
Almost everything in a switch's state is derived. It comes from persisted config (config_db.json), from protocols relearning routes and neighbors, and from rediscovering the hardware. Redis started as a fast in-memory hub for passing data between processes, and route programming needs a high write rate. The durable truth lives elsewhere: in the config file, in neighboring routers, and in the operator's controller.
- Very fast reads and writes
- Easy to inspect with
redis-cli - A restart gives a clean slate, so corrupted state doesn't survive a reboot
- Limited by memory on switch CPUs, which are often small
- One Redis instance becomes a hot spot under route churn
- Persistence has to be built specially: warm reboot needs explicit save and restore
Evidence: multi_database_instances.md · SONiC_Warmboot.md
Why Ceph chose this
The data is the product. Ceph's earlier backend, FileStore, sat on top of a local filesystem (usually XFS) and paid for it twice: every write went to a journal and then to the filesystem, and POSIX semantics added overhead. BlueStore replaced it by writing data directly to the raw device and keeping metadata in RocksDB (on BlueFS), with checksums on every block and optional inline compression.
- Durable, with end-to-end checksums that catch bit rot
- No double-write penalty for large writes
- Efficient copy-on-write clones for RBD snapshots
- Owning a storage engine is a huge, long-term engineering commitment
- RocksDB compaction causes latency spikes
- Memory tuning (
osd_memory_target) matters a lot
Evidence: src/os/bluestore/ · src/kv/ (RocksDB wrapper)
Bottom line: each project chose based on whether losing the state is recoverable or catastrophic.
2. How do components communicate?SONiC: shared-DB pub/subCeph: direct messaging + maps
Why SONiC chose this
SONiC combines components from many teams and vendors, written in C++, Python and Go. A shared database is a language-neutral contract: any daemon can crash, restart and resync by reading its tables, with no need to track its peers. The producer/consumer table library in sonic-swss-common standardizes the pattern. When Redis became too slow on hot paths, the project added a ZMQ channel rather than giving up the model.
- Loose coupling and independent restarts
- All state can be observed in one place
- Works across languages
- An extra hop and serialization on every update
- Redis becomes a bottleneck
- Table schemas are implicit contracts that drift (YANG was added later)
- No ordering across tables, so orchagent has to handle dependencies
Evidence: ZMQ producer/consumer · SONiC Guide: databases
Why Ceph chose this
With thousands of daemons, a central database on the data path would be a bottleneck and a single point of failure. OSDs talk to each other directly for replication, heartbeats and recovery. The only centralized pieces are the small, versioned cluster maps held by the monitors. Messaging goes through the async messenger (msgr2 protocol, with optional encryption and compression).
- Scales horizontally
- No central bottleneck on the data path
- Low latency
- Live state is spread across daemon memory, so debugging means admin sockets and dumps
- Wire-protocol compatibility must be managed across versions
- A client with an old map epoch must fetch the new map and retry
Evidence: src/msg/async/ · src/mon/ (map services)
Bottom line: a shared bus is the best choice inside one box and the worst choice across thousands.
3. What consistency model?SONiC: eventualCeph: strong
Why SONiC chose this
IP networks are eventually consistent by nature. Each router converges on its own, and packets keep flowing on the old state while the new state is installed. Blocking until everything agrees would drop packets. Inside the box, orchagent writes to ASIC_DB asynchronously by default to maximize throughput. Synchronous mode was added later so that hardware failures reach the caller.
- Maximum availability and throughput
- Degrades gracefully
- Transient microloops and blackholes
- BGP can advertise routes that aren't in hardware yet
- It's hard to know when a change is "done"
Evidence: synchronous mode · suppress FIB pending (closes the advertise-before-install gap)
Why Ceph chose this
Storage consumers such as databases on RBD volumes and applications on CephFS assume read-after-write consistency, and CephFS needs near-POSIX semantics. Ceph uses primary-copy replication. The primary OSD for a placement group orders every write, and the client is acknowledged only after all replicas in the acting set have committed.
- A simple model for clients
- No write conflicts to resolve
- Safe for databases
- Write latency is set by the slowest replica
- I/O blocks when a PG has fewer than
min_sizereplicas - Strong consistency across sites is too slow, so multi-site RGW and RBD mirroring are asynchronous
Evidence: src/osd/PrimaryLogPG.* · src/osd/PeeringState.*
Bottom line: CAP in practice. SONiC chooses availability because a dropped packet is retried anyway. Ceph chooses consistency because a lost write is permanent.
4. Is there consensus?SONiC: noneCeph: Paxos (MON quorum)
Why SONiC chose this
Each switch runs independently, so a software bug or bad config affects one box, not the fabric. Routing protocols reach agreement on reachability without a quorum. Fleet-wide intent lives in the operator's controllers. Where two boxes do need to coordinate (MCLAG, dual ToR), they use targeted protocols between exactly two peers.
- No quorum to lose
- Failures stay isolated to one box
- Simpler software
- No global view from inside SONiC
- Fleet coordination must be built externally
- Peer pairs can split-brain and need specific mitigations
Evidence: MCLAG (ICCP) · dual ToR
Why Ceph chose this
Ceph needs one authoritative answer to "which OSDs are up, and where does each PG live?" Without it, two OSDs could both act as primary and accept divergent writes. Monitors (usually 3 or 5) agree on the maps using Paxos. The maps are small and change rarely, and data I/O never goes through the monitors, so the cost of consensus is acceptable.
- A single source of truth
- Safe failover decisions
- Split-brain protection for data
- A majority of monitors must be up
- The monitor store grows during long recoveries
- Monitor placement needs care
Evidence: src/mon/Paxos.* · src/mon/OSDMonitor.*
Bottom line: SONiC avoids shared fate between switches, and Ceph needs a shared truth about where data lives. Both are right for their domain.
5. How is work placed?SONiC: ECMP hash in the ASICCeph: CRUSH in the client
Why SONiC chose this
The forwarding decision is made per packet, in nanoseconds, in silicon. Only a local, stateless hash is fast enough. The hash fields and algorithms are features of the ASIC, which SONiC exposes through SAI. Fine-grained ECMP was added to limit flow reshuffling when members change.
- Zero lookup latency
- Stateless
- Scales with the hardware
- Hash polarization across tiers
- Large "elephant" flows collide, so load is uneven
- Limited control. Membership changes reshuffle flows unless resilient hashing is used.
Evidence: hash design · fine-grained ECMP
Why Ceph chose this
A central lookup table for billions of objects would be a bottleneck and a single point of failure. With CRUSH, any client computes where an object lives from the cluster map alone. The map encodes the hierarchy (host, rack, row, data center), device weights and rules such as "3 replicas in different racks". Placement groups add a layer of indirection, so rebalancing moves whole PGs instead of individual objects.
- No lookup bottleneck
- Aware of failure domains
- Deterministic and decentralized
- Statistical imbalance, so a balancer (upmap) is needed
- PG counts need tuning (the autoscaler helps)
- Map changes trigger data movement, and exact placement is hard to reason about by hand
Evidence: src/crush/ · src/pybind/mgr/balancer/ · src/pybind/mgr/pg_autoscaler/
Bottom line: both avoid lookup tables. SONiC computes per packet with no memory of past decisions, while Ceph computes per object and follows the cluster's topology.
6. How do they react to failure?SONiC: reroute nowCeph: wait, then heal slowly
Why SONiC chose this
Losing a path loses no data. The faster traffic stops using a dead link, the fewer packets are dropped. BFD detects failures in under a second, the ASIC removes the ECMP member, and prefix-independent convergence (PIC) avoids reprogramming every route.
- Recovery in under a second
- No data to move
- Flapping links cause churn, so damping is needed
- Aggressive timers raise the risk of false positives
Evidence: BGP PIC · link event damping
Why Ceph chose this
Moving data is expensive and risky, and most failures are transient, such as a reboot or a brief network blip. An OSD is marked down quickly, and I/O moves to the surviving replicas. It is only marked out, which triggers re-replication, after a timeout. Recovery and backfill are throttled so they don't starve client I/O. Operators set noout during maintenance.
- Avoids moving terabytes needlessly
- Client performance is protected during recovery
- The cluster runs degraded (fewer copies) during the wait, so a second failure is riskier
- Many throttles to tune
- Recovering large disks takes hours to days
Evidence: src/osd/ (recovery and backfill) · src/common/options/ (for example mon_osd_down_out_interval, osd_max_backfills)
Bottom line: the cost of reacting sets the speed. Rerouting is nearly free, so SONiC reacts at once. Re-replication is expensive, so Ceph waits.
7. How are upgrades done?SONiC: per-box image swap + warm rebootCeph: rolling daemon restarts
Why SONiC chose this
A switch has no redundancy inside itself to hide a restart. The choices are to shift traffic away (TSA) and reboot, or to use warm reboot, which keeps the ASIC forwarding while the software restarts. Swapping the whole image (A/B images, sonic-installer) makes each version an atomic unit that is easy to roll back.
- Atomic versions
- Roll back by booting the previous image
- Hitless when warm reboot works
- Warm reboot is complex, and every feature must support save, restore and reconcile
- Fragile for some feature combinations
- A one-line fix still means shipping a whole image
Evidence: warm-reboot open issues · Reliable TSA · fast reboot
Why Ceph chose this
A Ceph cluster is redundant by design, so one daemon at a time can restart while its peers carry the load. The upgrade order is monitors, then managers, then OSDs, then MDS and RGW. cephadm automates it and checks ok-to-stop first. Daemons at N and N−1 interoperate during the upgrade window, negotiated through feature bits.
- No downtime
- Fine-grained
- Uses redundancy the cluster already has
- Mixed-version windows need compatibility code
- Large clusters take a long time to upgrade
- Each OSD restart causes peering and brief latency blips
Evidence: src/cephadm/ · src/pybind/mgr/cephadm/ (upgrade logic) · qa/suites/upgrade/
Bottom line: a system with no internal redundancy has to make restarts hitless (warm reboot). A system with redundancy can simply restart a piece at a time.
8. How is hardware abstracted?SONiC: vendors implement SAICeph: owns the storage engine
Why SONiC chose this
Switch ASICs are proprietary, and vendors will not open their SDKs. SAI, standardized through OCP, gives SONiC a stable C API while vendors keep their IP. This is what made a multi-vendor, open network OS commercially possible.
- Portability across vendors
- A competitive hardware market
- Vendors invest in their own SAI implementations
- A lowest-common-denominator API, plus vendor extensions
- Bugs in closed binaries are hard to debug
- Behavior varies by platform, so conformance testing is needed
Evidence: SAI failure handling · conformance guideline
Why Ceph chose this
Disks and SSDs are commodity parts with standard, open interfaces (block devices, NVMe). The value is in the storage engine, and owning it lets Ceph guarantee durability and tune performance. For very fast NVMe, Ceph is building a new engine (SeaStore) and a new OSD (Crimson) rather than depending on vendors.
- Full control
- Consistent behavior on any hardware
- Open to debug end to end
- A huge engineering burden
- Using hardware offloads (DPUs, computational storage) needs per-vendor work
- Rewrites take years
Evidence: src/os/ObjectStore.h · src/crimson/ · src/erasure-code/ (plugin interface)
Bottom line: each project abstracts the layer it can't control (SONiC) or chooses to own (Ceph).
9. How is configuration modeled?SONiC: YANG + JSON PatchCeph: typed options in the MON config DB
Why SONiC chose this
Network operators expect industry-standard models (YANG, OpenConfig, gNMI and NETCONF) for automation that spans several vendors. CONFIG_DB's schema was loose at first, which allowed bad writes. YANG validation was added to reject them, and Generic Config Update made partial changes safe and reversible.
- Interoperable
- Machine-validated
- A large tooling ecosystem
- YANG is verbose and complex
- Models must stay in sync with how the tables are really used
- The OpenConfig↔SONiC translation layer adds code
Evidence: YANG validation · GCU & rollback · YANG guidelines
Why Ceph chose this
Ceph is a single implementation, and most of its configuration is tunables: thousands of options with types, defaults and levels (basic, advanced, dev) declared in the source. Moving config into the monitors' central store removed the drift between ceph.conf files on different hosts.
- Simple and self-documenting (
ceph config help) - Consistent across the cluster
- Overrides per daemon and per host
- A huge number of options, so it's hard to know which matter
- No industry-standard model
- Some changes need a daemon restart
Evidence: src/common/options/*.yaml.in · src/mon/ConfigMonitor.*
Bottom line: standards matter when many vendors must interoperate. A single implementation can use its own schema.
10. How does it scale?SONiC: many small instancesCeph: one large logical system
Why SONiC chose this
Data-center networks scale out with Clos fabrics made of many identical switches, and the complexity lives in routing protocols and automation. Each SONiC instance stays small. Even multi-ASIC and chassis systems partition into per-ASIC namespaces instead of growing one big process.
- Failures stay isolated
- Linear scaling
- Commodity boxes
- Managing thousands of boxes happens outside SONiC
- Config drift between boxes
- Every box must be upgraded individually
Evidence: multi-ASIC HLD · VOQ architecture
Why Ceph chose this
Users want one pool of capacity with one namespace. Ceph clusters grow by adding OSDs to the same logical system, and data rebalances automatically.
- One capacity pool
- Automatic balancing
- A single point of management
- Cluster-wide blast radius: a bad CRUSH change can move all the data
- Operators often run several clusters to limit that risk
Evidence: src/crush/ · src/pybind/mgr/ (balancer, orchestrator)
Bottom line: SONiC scales by adding instances, while Ceph scales by growing one instance.
11. What threading model?SONiC: single-threaded orchagentCeph: thread pools → Crimson shard-per-core
Why SONiC chose this
orchagent processes tables that depend on each other: ports, then interfaces, then neighbors, then routes. A single event loop gives deterministic ordering and much simpler code, and SAI calls went through one channel anyway. Throughput has been improved around the loop, with bulk SAI APIs, pipelining, ZMQ and async stages, rather than by making it multi-threaded.
- Simple and deterministic
- Easy to reason about and debug
- A throughput ceiling at very large route scale
- One slow handler delays every other table
Evidence: orchagent route redesign · BGP loading optimization
Why Ceph chose this
An OSD serves many PGs and needs to use all its cores. The classic OSD uses sharded operation queues and thread pools. On fast NVMe, locks and context switches came to dominate, so Crimson rebuilds the OSD on Seastar: a shard-per-core, share-nothing design built on futures.
- Multi-core throughput
- Crimson aims for lower, more predictable latency per I/O
- Locking complexity and hard-to-reproduce bugs (classic OSD)
- Crimson is a rewrite that has taken years
Evidence: src/osd/scheduler/ · src/crimson/osd/
Bottom line: SONiC can stay simple because the ASIC does the heavy lifting. Ceph's CPU is its data plane, so it must scale across cores.
What SONiC could benefit from Ceph
Ideas from Ceph that would fit SONiC's constraints. Each item has a rough fit rating: high fit means it could be adopted with little conflict, partial fit means it needs adapting.
Continuous scrubbing of state high fit
Ceph scrubs (compares replica metadata) and deep-scrubs (verifies data checksums) in the background on a schedule, so silent corruption is caught before it matters.
For SONiC: a general scheduled consistency checker across CONFIG_DB → APPL_DB → ASIC_DB → hardware for every table, not only routes. Mismatches would be reported as structured health events. Some one-off checkers already exist, and the event framework could carry the results.
Named health checks with severity and mute high fit
Ceph reports health as HEALTH_WARN with stable codes (for example OSD_DOWN, PG_DEGRADED). Each code has documentation and remediation steps, and operators can mute a known issue for a set time.
For SONiC: system-health could adopt stable codes with documentation and muting, which would make fleet automation and alert routing simpler.
"ok-to-stop" safety APIs for maintenance high fit
Ceph answers "can I safely restart this daemon or host without losing availability?" before an operator acts (ok-to-stop, noout flags).
For SONiC: a standard pre-upgrade check such as "is traffic drained? are peers redundant? is warm reboot supported for the current feature set?" Traffic shift-away already exists (Reliable TSA). Exposing it as a queryable safety gate is the Ceph idea.
Shard-per-core processing (Crimson) partial fit
Crimson avoids locks by pinning shards of work to cores and passing messages between them.
For SONiC: partition orchagent's work by table or by prefix range across cores to raise route-programming throughput. The obstacle is dependencies between Orch modules (routes depend on neighbors, which depend on ports), which is why the work is serialized today.
A cluster-level manager and orchestrator partial fit
Ceph's MGR plus cephadm manage many nodes as one system: rolling upgrades, adding hosts, and one dashboard.
For SONiC: this is usually done outside SONiC by each operator's own fleet tooling. An optional, open "fabric manager" reference built on gNMI/gNOI would lower the barrier for smaller operators. It must stay optional, because SONiC deliberately avoids coupling switches together (see irreconcilable).
Versioned state with epochs partial fit
Every Ceph map change bumps an epoch, and daemons can tell exactly which version they are acting on.
For SONiC: tag CONFIG_DB changes with a monotonic generation number so that consumers, telemetry and GCU can reason about "has change N been applied everywhere in the box?"
Upgrade-path testing at scale high fit
Ceph's teuthology framework runs large test suites, including upgrades from older releases on real clusters.
For SONiC: sonic-mgmt already has testbeds. Standard N-1 → N warm-upgrade suites for every feature, run across vendors, would make it much safer to rely on warm reboot.
What Ceph could benefit from SONiC
A standard hardware-offload API partial fit
SAI made "one software stack, many chips" work for switches. DASH is extending the same idea to DPUs and SmartNICs.
For Ceph: a vendor-neutral offload interface for erasure coding, compression, encryption, checksums or NVMe-oF on DPUs and computational storage, so that acceleration does not mean vendor lock-in. The partial fit is because Ceph's advantage is running anywhere in software, so offload must stay optional.
Restart the control plane without disturbing the data plane partial fit
SONiC's warm restart saves state, restarts, reconciles, and applies only the differences, while forwarding never stops.
For Ceph: restarting an OSD today triggers PG peering and a short degraded state. State-preserving restarts (hand in-flight state to the new process, then reconcile) could shorten upgrade windows. It only partly fits, because in Ceph the data plane is the software process.
Inspectable state in a shared database high fit
In SONiC, almost any question about current state can be answered by reading a Redis table, with no special tools.
For Ceph: much of the live state sits in daemon memory and is reached through admin sockets and per-daemon dump commands. A uniform, queryable state-export schema, even as read-only telemetry, would make debugging and automation easier.
Standard models: YANG, gNMI, OpenConfig partial fit
SONiC exposes config and state through industry-standard models, so tools from many vendors work with it.
For Ceph: storage has fewer model standards (SNIA Swordfish is one), but a schema-first management API with streaming subscriptions, rather than scrape-only metrics, would help large operators.
Formal HLD templates and a conformance program high fit
SONiC requires HLDs to follow a template that includes warm-boot impact, restrictions and a test plan. It also defines conformance guidelines for distributions.
For Ceph: a required "upgrade and recovery impact" section in design proposals, and a conformance program for downstream distributions and appliances.
Awareness of the network high fit
Ceph performance depends heavily on the network: replication traffic, recovery storms, and tail latency. SONiC exposes buffer, drop, PFC and queue telemetry (HFT).
For Ceph: use switch telemetry to throttle recovery when congestion appears, or to build CRUSH failure domains from real topology discovered through LLDP.
Irreconcilable differences
These are not preferences. Each follows directly from what the system exists to do, and changing one would turn the system into something else.
1. Soft state vs hard state
SONiC can throw away its state and relearn it from config and neighbors. Ceph's state is the only copy of someone's data.
Consequence: SONiC optimizes for restarting fast, and Ceph optimizes for never losing a write. SONiC can use in-memory Redis with no replicas. Ceph must write to disk, checksum, and replicate. Neither choice works for the other system.
2. Data plane in silicon vs in software
SONiC's forwarding runs on fixed-function ASICs driven by vendor SDKs that are often closed. Ceph's I/O path is its own open code on CPUs.
Consequence: SONiC can never be fully open or fully debuggable at the lowest layer, and features depend on what the chip supports. Ceph can change anything, but it pays CPU cost for every byte. Achieving line rate at Tbps requires silicon, and durable general-purpose storage requires software.
3. No consensus vs consensus
Network resilience comes from each switch deciding for itself. A distributed routing protocol converges without a quorum. Ceph needs a Paxos quorum to agree on one authoritative cluster map.
Consequence: adding a shared consensus layer across switches would couple their fates, so one bug or partition could take down a whole fabric. Removing consensus from Ceph would allow split-brain writes and data loss. Each system's key safety property comes from the opposite choice.
4. Opposite reactions to failure
A switch reroutes in milliseconds because dropping a path costs nothing. Ceph deliberately waits before re-replicating, because moving terabytes on a false alarm is itself an outage.
Consequence: the timers and tuning philosophies cannot be shared: aggressive detection for SONiC, patient and throttled recovery for Ceph.
5. Owning the box vs living on the box
SONiC is the whole operating system image (installed through ONIE). It owns the kernel, drivers and platform. Ceph is an application stack that runs on whatever Linux the operator chooses.
Consequence: SONiC must ship platform drivers, fan control and firmware tooling for every device (platform). Ceph never has to, but it also cannot control the host's kernel or tuning.
6. Ecosystem shape: multi-vendor plug-ins vs one reference implementation
SONiC is a shared core plus many vendor-supplied pieces (SAI, SDK, platform modules), spread across many repositories. Ceph is one vendor-neutral codebase that downstream vendors package.
Consequence: SONiC's behavior can differ between platforms, so it needs conformance testing. Ceph behaves the same everywhere, but it has to implement everything itself. The multi-repo vs monorepo split (below) follows from this.
How the repositories differ
| SONiC | Ceph | |
|---|---|---|
| Layout | Many repos under sonic-net. sonic-buildimage ties them together with submodules. This SONiC repo is docs and governance only (sourcecode.md). | Monorepo ceph/ceph: src/ (daemons, libraries, mgr modules), doc/, qa/ (test suites). |
| Design docs | HLDs as Markdown in this repo's doc/ folder, one folder per feature, using a template. | Developer docs in doc/dev, plus design discussion on mailing lists, trackers and developer summits. |
| Build | A Docker-based image build per ASIC platform (sonic-<vendor>.bin) plus a virtual switch image (reproducible build). | CMake. Packages (deb/rpm) and container images. |
| Testing | sonic-mgmt: pytest + PTF against physical and virtual testbeds. Unit tests in each repo, and VS-based tests. | Unit tests, make check, and teuthology integration and upgrade suites in qa/. |
| Issue tracking | GitHub issues and PRs in each repo. | GitHub PRs plus a Redmine tracker. |
| Release model | Date-named branches twice a year. Vendors and operators often maintain their own downstream branches. | Named major releases with long-lived stable branches and point releases. |
Interview angle
Ceph maps to…
S3-like object storage: RGW, replication vs EC, scrubbing.
Key-value store: RocksDB, strong consistency.
Consistent hashing: CRUSH and placement groups.
Digital wallet: consensus (Paxos vs Raft).
SONiC maps to…
Message queue: producer/consumer tables, batching.
Rate limiter: CoPP.
Metrics & alerting: flex counters, CRM.
Payment reconciliation: warm-restart reconcile.
Practice prompts
- Why does Ceph wait before re-replicating a failed disk, while a switch reroutes instantly? Which would you choose for a cache tier, and why?
- SONiC keeps its state in memory and Ceph keeps it on disk. For each one, describe what breaks if you swap the choice.
- Design a consistency checker for SONiC based on Ceph's scrubbing. What would you compare, how often, and what do you do when something does not match?
- CRUSH and ECMP both avoid lookup tables. What does each give up by doing so?