The goal of Solr's AutoScaling framework is for search clusters to be able to grow to a trillion documents without much human intervention.
The first part of the talk covers AutoScaling framework concepts. We'll talk about AutoScaling Policies and Preferences, the AutoScaling API and event triggers.
Further, we’ll discuss practical use-cases to keep the cluster healthy and performing optimally, complete with fault tolerance.
For example, we'll cover how to achieve these scenarios by utilizing the framework.
- Effectively managing disk space by setting triggers and sending out alerts.
- Maintaining a minimum replication factor when nodes go down. We'll also use rules to make sure the replicas are spread out, thus maximizing fault tolerance.
- Scaling out replicas to serve more traffic by setting thresholds. The thresholds could be latency or QPS based. We could also run it as schedulers to better serve peak load.
- Move replicas around to balance load across the cluster.
- Indexing triggers: Are shards getting too large? Support for auto shard splits etc.