How To Globally Register Processes In Elixir
The dangers of the Single Global Process
August 12, 2019
There are a few things in the Elixir/Erlang ecosystem that I consider required reading. To spawn, or non to spawn? by Saša Jurić is definitely one of them. If you haven't read it, you lot need to. Information technology'll alter the mode yous call up about building elixir applications.
Seriously go read it.
That post flipped the elixir communities' idea of proficient design on its head and for a adept reason. Modeling the domain with pure functions is a powerful approach and one that nosotros should strive for when we tin can.
Merely, there was one pattern that emerged that I remember has been misapplied as a universal solution. That blueprint is what I've been calling - for lack of a ameliorate proper noun - the "single global process" pattern or SGP for short. You've probably seen this design. Its the i where you do this elegant, functional domain modeling and so put it in a long-running procedure somewhere, effectively turning the procedure into a write-through enshroud. In Saša's post, the single procedure is the RoundServer.
I don't call up Saša intended to promote this design. I wasn't there when he was writing information technology, but I always thought that using a single RoundServer to manage a round was somewhat incidental. The unique process was an implementation particular, and the disquisitional point was to model your domain with functions.
Because here's the matter. The SGP blueprint introduces a ton of problems. Problems that you, dearest reader, are going to need to solve. In my experience, the SGP is one of the about intricate patterns you tin innovate to your system despite being one of the easiest to build. I'm going to exercise my all-time to convince y'all of this by enumerating several of the problems that you'll face every bit well as some potential solutions.
The Setup
To drive these points dwelling house, we need a motivating case. I want to focus on the runtime concerns, so I'm going to reduce our problem domain to a simple counter. Here's our functional core:
defmodule Counter exercise def new ( initial \\ 0 ) do %{ ops: [], initial: initial } end def incr (%{ ops: ops } = counter ) do %{ counter | ops: [{ :incr , 1 } | ops ]} end def decr (%{ ops: ops } = counter ) do %{ counter | ops: [{ :decr , 1 } | ops ]} end def count (%{ ops: ops , initial: init }) exercise Enum . reduce ops , init , fn op , count -> example op do { :incr , val } -> count + val { :decr , val } -> count - val end end end cease To update a count we "cons" increment and decrement operations onto a growing list. When we want to find the actual count, we fold over the list either incrementing or decrementing starting from some initial value.
For the server, we'll utilise a GenServer.
defmodule CounterServer do use GenServer def start_link ( opts ) do GenServer . start_link ( __MODULE__ , opts , name: Keyword . go ( opts , :proper noun )) end def increment ( name ) do GenServer . call ( proper noun , :incr ) terminate def decrement ( name ) do GenServer . telephone call ( name , :decr ) end def count ( name ) practice GenServer . call ( proper name , :get_count ) terminate def init ( _opts ) do { :ok , Counter . new ()} cease def handle_call ( :incr , _from , counter ) do { :respond , :ok , Counter . incr ( counter )} end def handle_call ( :decr , _from , counter ) exercise { :answer , :ok , Counter . decr ( counter )} end def handle_call ( :get_count , _from , counter ) do { :reply , Counter . count ( counter ), counter } end terminate The server is equally trivial. It manages the lifecycle of a counter in response to calls. Unique counter servers tin can exist started similar then:
CounterServer . start_link ( name: :some_fancy_counter ) With these pieces finished, nosotros accept a practiced starting signal to discuss bug with this design.
The trouble
What we have and so far seems pretty adept. And if all you really needed were a simple, in-memory counter, this would probably do the flim-flam. Simply this is a contrived domain that I've intentionally kept simple, so I can focus on other things. Typically the land we bargain with is essential. The data that makes up nearly visitor'due south core domain is not ephemeral. People will make decisions based on these data that we're working with. That means these data needs to be persisted. So let's add persistence. If persisting a counter to a database bothers you, then you lot can tell yourself that the counter is used to neb clients for the use of your API or something.
The typical recommendation for persistence is to initialize the process with state from the database. When the process receives a bulletin to write or update its internal country. These updates are applied to our internal data and then persisted. Utilizing the SGP pattern, any read requests can come directly out of retentivity saving yourself the database roundtrip. Permit'south update our Counter Server to reflect this change:
defmodule CounterServer do use GenServer def start_link ( opts ) do GenServer . start_link ( __MODULE__ , opts , name: Keyword . get ( opts , :name )) stop def increase ( name ) do GenServer . call ( proper name , :incr ) end def decrement ( name ) do GenServer . call ( proper noun , :decr ) end def count ( proper name ) do GenServer . call ( proper noun , :get_count ) end def init ( _opts ) do information = %{ counter: nil , name: opts [ :name ], } { :ok , data , { :keep , :load_state }} end def handle_continue ( :load_state , information ) do { :ok , initial } = get_from_db ( information . proper name ) { :noreply , %{ data | counter: Counter . new ( initial )}} finish def handle_call ( :incr , _from , data ) do new_counter = Counter . incr ( data . counter ) :ok = put_in_db ( data . name , new_counter ) { :reply , :ok , %{ information | counter: new_counter }} terminate def handle_call ( :decr , _from , data ) do new_counter = Counter . decr ( data . counter ) :ok = put_in_db ( information . name , new_counter ) { :reply , :ok , %{ data | counter: new_counter }} end def handle_call ( :get_count , _from , data ) exercise { :reply , Counter . count ( information . counter ), data } finish finish When the counter process starts, we load the state in a handle_continue callback. If nosotros receive an increment or decrement message, nosotros update the counter and shove the new counter into the database. Reads are returned from our in-retention representation saving united states that database call.
This seems great! And information technology is great - right up until you need to run on more than one node. In that location are systems in the world that tin can get by running a single node and perhaps your company is 1 of them. Simply about companies cease up needing to run more than than one node at some betoken whether for resiliency or to handle scale. Most of the time, we run both of these nodes behind a load balancer.
We've hit our first problem. The counter we've created is node-local. Assuming that nosotros showtime these counters on demand, it's simply a matter of time before we end upward with duplicate counters on both of our nodes.
If this situation occurs, and so we take a high likelihood of returning incorrect counts from memory because we tin can't know if some other node has updated the counter in the database. Nosotros as well have a high probability of overwriting the previously stored values, which ways we have a high likelihood of losing data.
A quick aside about data integrity. Inconsistent data problems are some of the evilest bugs yous'll encounter when working with distributed systems. Bugs like these suck because in that location's never a good indication that something is going wrong at the moment it's happening. At that place'due south no crash or stack trace to look at. Unless you're constantly monitoring your data integrity, the only fashion y'all'll find out that y'all have an event is when you stop upwards charging a customer 10,000 dollars or -100 dollars or NaN dollars. There are ways to build eventually consistent systems. Simply that has to exist a conscious selection.
Some fractional solutions
There are a few partial solutions to this problem. The ones that I see virtually people reach for are either persistent connections, pasty sessions, or some combination of the two. Unfortunately, none of these really eliminate the possibility of starting the aforementioned counter on two nodes, primarily if more than one user can interact with a counter at a time. Additionally, introducing sticky sessions is a fast way to end up with "hot" nodes due to unfair distribution of work. However, if you can use sticky sessions and are willing to give upwardly on some levels of consistency than this might work for you.
Another partial solution is always to use Compare and Bandy (CAS) operations when updating the database. Assuming your database implements a CAS correctly, you lot can eliminate the possibility of trampling information. But you will withal render wrong values until yous do a write or detect some other fashion to go an update from the database.
Neither of these solutions entirely solves the trouble, but in conjunction with each other, they might be adept enough for your use case.
Distributed Erlang volition save us all.
Distribution is e'er the solution that's begging for a trouble. And there's no better set of challenges than the ones introduced past an SGP. Then let's indulge ourselves and walk down this path for a chip.
I'll assume that we've found a fashion to observe and connect our nodes together. Now that nosotros've washed that we need to annals our counters beyond the cluster. For this instance, I'g going to use :global because it's built into OTP and easy to utilize. Just the failures I'k near to describe are not limited to :global. You can induce these aforementioned failures with virtually any of the procedure registries that exist in elixir and erlang.
Converting our procedure to use the global registry is direct forward. We demand to alter the start_link function.
defmodule CounterServer practise def start_link ( opts ) do proper noun = Keyword . get ( opts , :name ) GenServer . start_link ( __MODULE__ , opts , name: { :global , proper noun }) terminate end Now when nosotros desire to access our counter, we'll exist able to find it globally regardless of what box we're continued to.
This solution will work well, at least until we encounter the e'er-present specter of distributed systems; the netsplit.
The netsplit bogeyman
Netsplit is a catch-all discussion that probably gets tossed around to much. I know I've been guilty of it. Colloquially it's used to describe any and all faults that you lot could see in a distributed system. The odds of seeing a netsplit will depend on your cluster size and the reliability of your network. You lot're much more likely to encounter faults in a 60 node cluster running in kubernetes on amazon'due south crappy network then if yous're running 2 bare-metallic boxes hard-lined into each other sitting in a co-lo somewhere. But if you run a system for long enough, you'll eventually encounter faults. When you see those faults, you need to have a plan for handling inconsistent state - even if that plan is "Fuck information technology who cares."
Unfortunately, the SGP doesn't lend itself to graceful recovery afterwards a netsplit. Let's talk through some of the issues.
If your boxes accept a netsplit, at a loftier level, it ways that 1 of two things has happened: a node has shut down expectedly or unexpectedly, or the nodes accept disconnected from each other just are all still running. The pull a fast one on is that from a single node's betoken of view you can't really deduce which it was.
Unless the nodes have a consistent way to talk about cluster membership - Raft, Paxos or similar - all an individual node tin can really know is "I tin can no longer talk to these N nodes, and I don't know why." This has a secondary event which happens to brand our lives even harder: Deployments and scaling events tin start to look identical to netsplits. So while real partitions might exist rare, deployments may induce the same failures.
During a partition, the two nodes may not exist able to talk. But that doesn't mean that they aren't reachable from a client. A customer may issue a request, and that asking may get load counterbalanced to either node. If the request happens to state on the node that holds the counter, so nosotros're OK. Merely if the asking ends up on the node that doesn't hold the counter, we accept problems. As I described in a higher place the node can't know if the counter process is truly gone. The default solution is to presume that the procedure doesn't be and start up a new one. We're back to having ii counter processes running on divide nodes once more!
When the partition heals, we'll need to reconcile which counter is the canonical one. Past default :global discards one at random. Other registries such as Horde requite you more control over this reconciliation process. Merely you'll still need to take care in how you reconcile this land.
Node monitors volition not relieve you
One of the means that we tin can effort to solve this is by using node monitors.
:net_kernel . monitor_nodes ( true , [ :nodedown_reason ]) Calling this function in a GenServer will cause node events to exist sent to the process as messages. Unfortunately, this isn't really enough to know the state of your cluster. From whatsoever nodes view it's merely non possible to tell if another node has left for good, been autoscaled abroad, or been asunder because it couldn't go along upwards with health checks.
So how can we solve this? I'm going to focus on three solutions. Only there are many others and several variants of each. Hopefully, these volition give you lot some full general ideas.
Consistent Hashing and Oracles
Consistent hashing is my default way to solve this problem and is also the most naive. The basic scheme is that you'll utilize a consistent hashing algorithm to decide what node a given counter procedure lives on (Discord has a robust library for this). 1 caveat to this approach is that during a netsplit your counter process may not be reachable and thus volition exist unavailable for the elapsing of the dissever.
The other caveat is that you demand a way to specify the canonical set of nodes in your cluster. We know that we can't reliably use node events so we'll need to solve cluster direction another way.
The most straightforward method is to utilize static clusters. If you lot demand to add new nodes to your cluster, you bring up an entirely new cluster with the new nodes. Once everything is upwardly you tin redirect traffic to the new cluster, and you shut down the old cluster. Plain, this increases the fourth dimension information technology takes to deploy and calibration, simply if you take reasonable scaling needs, this tin work well.
If you need to exist able to change the cluster size dynamically, then you're going to have to do more than work to control your deploy process. I way to automate this is to outcome a specific ClusterChange RPC to all nodes in the cluster. If you go with this route, then you need to ensure that you publish RPCs to each node straight instead of relying on the nodes internal distribution. The reason y'all tin't rely on the nodes to propagate cluster changes is that if yous try to change the cluster during a division, you tin can end upward in a situation where only half of the cluster knows near the change.
A 3rd solution is to use an external, consequent store to manage cluster country. Most ofttimes, this ways using something similar ETCD or Zookeeper, which tin can provide high availability, consistent lookups.
In any of these scenarios autoscaling, at least as we typically think of information technology, is off the table. You'll need to invest serious time into your deployment and cluster management to pull this off.
CRDTs everywhere
Another style to solve the SGP inconsistency problem is to use CRDTs everywhere. You'll nonetheless get incorrect data during a netsplit, and when persisting information to a durable store, you'll have to take care not to overwrite data. To avoid this problem, you lot'll need to merge the data in your process with the data in the database and then replace what's in the database using a CAS functioning.
CRDTs have a lot of fantabulous qualities to them, but you demand to ensure you are using them correctly. Information technology'south entirely possible to misuse a CRDT and end up in an inconsistent state. Consistency is non a composable property of software. I also strongly propose you not to build your own CRDTs (other than for fun) and instead use something like LASP in production.
If yous practise make up one's mind to get down the CRDT route, you lot need to be aware that your states may diverge at some point. Each process and the database will take its ain view of the earth, and those views may all be different. In the fullness of fourth dimension, you may converge dorsum to a steady-state. But you might not. This is more than common in large clusters where your data has a high rate of change. But this divergence tin go far very difficult to reason well-nigh the state of the world.
Just don't use the SGP
Nearly of these problems become away if you lot simply don't use a unmarried global process to hold your land. This doesn't mean that you lot give up on modeling your application as pure functions. Instead, it ways giving upward on maintaining those information in a long-running process. It'southward an easy blueprint to reach for and it feels elegant. Only you really need to step back and ask yourself why y'all're doing this and if you're ready to solve all of the additional problems this solution will bring. If you only need a enshroud of values than maybe yous're better off replicating your state to some subset of your nodes instead or building a cache in an ETS table.
If you lot're trying to serialize side-furnishings and so maybe you're amend off relying on idempotency and the consistency guarantees of your database. If you like these "event-sourcey" semantics, perhaps you lot can utilize something like Maestro or Datomic to solve the consistency problems for you. There are means to maintain similar semantics without incurring the same issues.
When is an SGP ok?
In that location are ever tradeoffs when edifice software, and there are times when an SGP is a reasonable solution to the problem at hand. For me, this is when the procedure is curt-lived and will mutate no external state. In fact, the other 24-hour interval, I needed to build a search feature. For the feature to piece of work, we needed to gather information from lots of downstream sources and bring together it all together. The searching was washed equally the user typed so rather than do the information fetching every time the search query was inverse slightly, I did it once, shoved the country in a process, and then was able to quickly search through everything I had already plant with very few additional calls required. This design worked quite well because I wasn't trying to execute mutations from inside the process. If the procedure didn't receive a message within 15 seconds, it shut itself back downward.
I too think SGPs are fine if you genuinely don't care nigh the consistency of the data you're manipulating. That sorta land isn't the norm in my feel. Simply information technology does be and modeling it in a process this way is reasonable.
Conclusion
I want to re-emphasize that everything in Saša'due south post is fantabulous and I concord with it. If you can design your problem such that you're more often than not manipulating data with pure functions that generally leads to robust systems. I call up the problem is how the community has begun to apply the ideas in that post and not the post itself. I'thou also not trying to say that in that location'due south never whatsoever place for an SGP. My goal is to demonstrate that while like shooting fish in a barrel to build and conceptually very elegant, the SGP is 1 of the more complicated patterns you lot can add to your organization. I personally call up its a model that is overused in elixir and I don't believe information technology should be the default choice. Y'all need to take a good reason to put your state in a single, unique process. Otherwise, yous're probably better served by relying on a database to help enforce your data consistency.
Most chiefly if you do decide that an SGP is a correct solution for your use case I want you lot to be aware of the kinds of problems that you'll confront and the types of solutions that yous'll demand to work through. If you lot're prepared to practice that and its a meaningful use of your companies time then it can work well for yous. But you demand to accept articulate-eyes to run across what you're facing.
A huge thanks to José Valim, Lance Halvorsen, Jeff Weiss, Greg Mefford, Neil Menne and others for reviewing this post.
How To Globally Register Processes In Elixir,
Source: https://keathley.io/blog/sgp.html
Posted by: haaghictir.blogspot.com
0 Response to "How To Globally Register Processes In Elixir"
Post a Comment