Know what is being distributed.
Independent jobs, replicated inference services and a model partitioned across machines have different network and scheduling requirements. Select the strategy before choosing the interconnect.
Treat membership as state.
The runtime needs to distinguish an enrolled node from an available node and an available node from one executing useful work. Establish how identity, health and capacity are reported.
Plan for partial failure.
A node can stop, a link can degrade, or an action can finish without its acknowledgement arriving. Recovery should reconcile actual state before retrying consequential work.
Validate the complete system.
Measure realistic workload performance with the intended storage and network. Include contention, restart and recovery scenarios. Record the configuration with the result.