A turnstile for your storage cluster
There is a moment in every feature where you have to decide what the code is actually about. For MicroCeph remote replication, the obvious answer was “it is about talking to a remote cluster.” That answer is wrong, and building on it would have hurt.
Here is the thing that took me a while to see: replication is not really a data problem at the control-plane layer. The data movement is Ceph’s job, and Ceph is very good at it. What MicroCeph owns is much smaller and much more annoying. It owns the question what is this resource allowed to do right now.
That is a state machine. So we wrote one.
The turnstile
The canonical toy example of a finite state machine is a coin-operated turnstile. Two states, locked and unlocked. Two inputs, coin and push. Drop a coin into a locked turnstile and it unlocks. Push an unlocked turnstile and you walk through, and it locks behind you.
What makes the turnstile a good teaching example is not the happy path. It is the boring edges. What happens when you push a locked turnstile? Nothing, but it is a defined nothing. What happens when you drop a coin into an already-unlocked turnstile? You lose a coin, and the turnstile stays unlocked. It does not crash. It does not enter some fourth secret state. Every combination of state and input has an answer, and the answers are written down in one place.
That last property is the whole point, and it is exactly what I wanted for replication.
What we were avoiding
Picture the version of this feature that does not have a state machine. You have a pool. It might have mirroring enabled. It might not. Someone asks to configure a replication property on it. So you write:
if !isMirroringEnabled(pool) {
return fmt.Errorf("mirroring not enabled")
}Fine. Then someone asks for status. Same guard. Then promote, then demote, then list. Each one grows its own version of that check, and they drift, because they always drift. Then a user asks to disable replication on a pool where replication is already disabled, and now you have to decide: is that an error, or is it a no-op? Whatever you decide, you decide it in that function, and the person writing the next function will decide it differently.
Six months later the honest answer to “what happens if I promote a pool that is not replicating?” is “read all seven handlers and find out.”
The rules were never wrong, exactly. They were just scattered. A state machine does not add intelligence, it just puts the rules in one file where you can look at them.
Two states, seven triggers
MicroCeph’s replication FSM is built on qmuntal/stateless, a Go port of the .NET Stateless library. The states are as plain as the turnstile’s:
const (
StateDisabledReplication ReplicationState = "replication_disabled"
StateEnabledReplication ReplicationState = "replication_enabled"
StateInvalidReplication ReplicationState = "replication_invalid"
)Two working states. The third is a rejection value rather than a state the machine ever occupies: the CephFS handler returns it when a request names a resource type that does not make sense for CephFS, and the API layer bails on the error before the machine is ever constructed. Worth knowing if you go reading, because the name suggests a third circle of the diagram and there isn’t one.
The triggers are the operations a user can ask for: enable, disable, configure, list, status, promote, demote. Seven verbs, two states. Without a state machine that is fourteen combinations, each of which someone has to think about, and most of which nobody does.
The machine itself is small enough to read in one sitting:
// Configure transitions for disabled state.
newFsm.Configure(StateDisabledReplication).
Permit(constants.EventEnableReplication, StateEnabledReplication).
OnEntryFrom(constants.EventDisableReplication, disableHandler).
InternalTransition(constants.EventListReplication, listHandler).
InternalTransition(constants.EventDisableReplication, disableHandler).
InternalTransition(constants.EventPromoteReplication, promoteHandler).
InternalTransition(constants.EventDemoteReplication, demoteHandler)
// Configure transitions for enabled state.
newFsm.Configure(StateEnabledReplication).
Permit(constants.EventDisableReplication, StateDisabledReplication).
OnEntryFrom(constants.EventEnableReplication, enableHandler).
InternalTransition(constants.EventConfigureReplication, configureHandler).
InternalTransition(constants.EventListReplication, listHandler).
InternalTransition(constants.EventStatusReplication, statusHandler).
InternalTransition(constants.EventPromoteReplication, promoteHandler).
InternalTransition(constants.EventDemoteReplication, demoteHandler)That is the entire policy of the feature. You can read it out loud.
Permit versus InternalTransition
This distinction is the part I would most want a reader to take away, because it is where the design earns its keep.
Permit means this trigger moves you to a different state. There are exactly two of those: enable takes you from disabled to enabled, disable takes you back. That is the turnstile, and it is deliberately boring.
InternalTransition means this trigger runs a handler but leaves you where you are. Listing does not change whether replication is on. Neither does asking for status. These are the pushes against a turnstile that is already unlocked: real work happens, the state does not move.
Splitting these two apart is what stops the boolean-flag mess from creeping back in. Once every operation has to declare which kind it is, you cannot accidentally write a status call that mutates state, because there is no place to put that code where it would not look wrong.
Look at what falls out of it in the disabled state. Configure and status are simply absent, because configuring a replication property on a resource that is not replicating is meaningless. List, promote and demote are present, because they are meaningful even when replication is off. No guard clauses required. Absence is the guard.
And notice the answer to the question I posed earlier, the one about disabling something that is already disabled:
InternalTransition(constants.EventDisableReplication, disableHandler)Disable appears in the disabled state as an internal transition. So it is a no-op, not an error, and it is idempotent by construction. That decision is now a single line in a table rather than a convention I have to remember to apply in the next handler I write.
When a user tries something the table does not allow, they get one consistent answer, from one place:
func unhandledTransitionHandler(_ context.Context, state stateless.State, trigger stateless.Trigger, _ []string) error {
return fmt.Errorf("REPFSM: operation: %s is not permitted at %s state", trigger, state)
}One error message, defined once, instead of seven hand-written variations that each phrase it slightly differently.
Something similar applies to observability, with a caveat I got wrong the first time I described it:
func logTransitionHandler(_ context.Context, t stateless.Transition) {
logger.Infof("REPFSM: Event(%s), SrcState(%s), DstState(%s)", t.Trigger, t.Source, t.Destination)
}Grep REPFSM and you get every state change a resource went through, written from one place instead of sprinkled through handlers. The caveat: stateless does not fire OnTransitioning for internal transitions. Only the real state changes are logged, which is to say enable and disable. The other five operations pass through silently.
For an audit trail of state changes that is arguably correct behaviour, and it is what the callback is named after. But if you want to know that someone ran a status check at 3am, this is not where you will find out, and I would rather say so than let you discover it during an incident.
The payoff: one machine, many workloads
Everything above would be a tidy but modest win if replication meant RBD and only RBD. It does not, and this is where the shape of the design pays for itself.
Notice that the state machine itself never mentions RBD. Not the states, not the triggers, not the transition table. The machine knows about replication, not about what is being replicated. The workload-specific part lives behind an interface:
type ReplicationHandlerInterface interface {
PreFill(ctx context.Context, request types.ReplicationRequest) error
GetResourceState() (ReplicationState, error)
EnableHandler(ctx context.Context, args ...any) error
DisableHandler(ctx context.Context, args ...any) error
ConfigureHandler(ctx context.Context, args ...any) error
StatusHandler(ctx context.Context, args ...any) error
// Cluster wide Operations (don't require any pool/image info.)
ListHandler(ctx context.Context, args ...any) error
PromoteHandler(ctx context.Context, args ...any) error
DemoteHandler(ctx context.Context, args ...any) error
}And picking one is a map lookup:
func GetReplicationHandler(name string) ReplicationHandlerInterface {
// Add RGW and CephFs Replication handlers here.
table := map[string]ReplicationHandlerInterface{
"rbd": &RbdReplicationHandler{},
"cephfs": &CephfsReplicationHandler{},
}
rh, ok := table[name]
if !ok {
return nil
}
return rh
}RBD mirroring and CephFS mirroring are genuinely different underneath. Different daemons, different Ceph commands, different notions of what a “resource” even is. What they share is the lifecycle: something is either replicating or it is not, and the set of sensible operations depends on which.
So the FSM holds the lifecycle, and the handler holds the mechanics. Adding a workload means writing a handler and adding a line to that map. You do not touch the state machine, which means you cannot break replication for the workloads that already worked. That is not a small thing when the feature is the one people reach for after their primary site catches fire.
Where the seams show
I would be selling you something if I stopped there, because the second workload is exactly where this design started to strain.
The transition table is global. It says configure, promote and demote are legal operations in the enabled state, and it says that for every workload, because it does not know which workload it is serving. CephFS mirroring does not implement those three. So the CephFS handler has to say no at runtime:
func (rh *CephfsReplicationHandler) ConfigureHandler(...) error {
return fmt.Errorf("%s not implemented for cephfs", types.ConfigureReplicationRequest)
}Three hand-written refusals, living inside a handler. Which is, if you are keeping score, precisely the scattered-guard pattern I opened this post complaining about. It did not go away. It got pushed from seven places down to three, and from the common path out to the edges, but it is still there.
The honest read is that one FSM cannot express “these operations exist, but only for some workloads.” You would need per-workload transition tables for that: let each handler describe its own legal moves and build the machine from that description. That is a real design, and it would delete those three error strings. It is also more machinery than the problem currently justifies, so it has not been built.
There is a similar leak in the other direction. The RBD path treats “I could not determine the state” as disabled rather than surfacing an error, which is the more dangerous of the two defaults if you are relying on it during a failover. Worth knowing about if you are reading this code with operational intent.
RGW is the obvious third workload. Whether it arrives with per-workload tables or three more polite error strings is a decision someone gets to make.
What I would tell past me
The instinct when you get a feature request like “add remote replication” is to start at the bottom, with the Ceph commands, because that is the part that feels like real work. The commands are real work. But they are also the part Ceph documentation already covers, and the part that changes for every workload.
The part that does not change, the part that is genuinely yours to design, is the tiny question of what is allowed when. Get that written down in one place, in a form you can read out loud, and the rest is plumbing you can add to safely for years.
The seams I described above do not change that conclusion, they sharpen it. Even where the abstraction leaks, the leak is visible: three named methods returning three explicit errors, in one file, easy to find and easy to delete when someone builds the better version. That is what you are actually buying with a state machine. Not the absence of awkward cases, but knowing where they all live.
Two states. It really is just a turnstile.
If you want the code, it lives in microceph/ceph/replication.go. I also talked through the operator-facing side of this at Cephalocon 2024, if you would rather watch than read.