For all ye SOA and networking-inclined

[cryowizard]

Our app uses JINI for discovery. For better or worse, that's what we use. Our app is all SOA, so dispersed services must have a way of finding each other on different boxes, preferably without pre-defining hosts and ports. Here's how it works -- we start many registrars, one on each box. Distributed services and models on each box register themselves with the local registrar. When a JINI registrar comes up, it uses a well-known IP multicast address (224.0.1.x) to communicate with other registrars and discover what models or services they have. This way, if a service on box A requires access to a service on box B, the registrar on box A will be able to provide access to the service on box B because it spoke to the registrar on box B and knows everything it has (and vice versa). The idea behind using multicast is that you dont have to know how many registrars are there and where are they running -- they are supposed to find each other due to the broadcast nature of the discovery traffic. Once the registrar finds the required service, a direct TCP connection is established from one service to another. JINI registrar has no knowledge of the services themselves, nor does it care. In other words, UDP mulitcast is only used when the registrar services discover each other.

Now, we have two 3-node clusters covering our universe (VCS and GFS, if you care). The clusters are on different subnets. The way company networking people set it up, multicast on the "well-known" address JINI uses will not work by default between different subnets. That multicast traffic has to be routed between subnets, and comany networking group uses different well-known multicast addresses for IP multicast routing. Now, given that JINI is a SUN package and the multicast addresses they use are hardcoded (I checked their source code), the networking people set up a special multicast rendezvous point for us so that the traffic routes properly between subnets our clusters are on, to enable our app to work using the JINI address.

Nobody remembered that being done (do you feel a disaster coming up?)

That was years ago. The hardware that hosted that rendezvous point was situated in our old building...and after 5 years has been turned off last weekend as a part of the building data center decommissioning...

You can guess the rest -- the services and models on two clusters failed to find each other during morning startup (of course on my watch), and a god-aweful shitty mess ensued. We made it before the open, but just barely -- by moving the essential services to one cluster since it is not affected by lack of multicast routing.

Quite educational that outage has been. We are now moving away from multicast to unicast discovery -- every registrar will know a list of other registrars so that it can query them for services directly using TCP, which has no routing problems. This isn't so cool as a blind multicast discovery -- but fuck me if I am going through that "market data can't see static data!" crap again.

Comments

[cryowizard.livejournal.com]

Unlike you, I ain't no network engineer. I just have network monkeys tell me what's what.

The list of registrars boils down to the list of hosts across my universe, which is not difficult to maintain, given you have a procedure for that.

And lets leave my clusters alone, shall we :)