Re: How far can you cluster (SCS) unsupported over a WAN?

From: Keith Parris (keithparris_NOSPAM_at_yahoo.com)
Date: 08/31/05


Date: Wed, 31 Aug 2005 15:58:32 GMT

Alan Greig wrote:
> After reading through the OpenVPN docs I see I can use it to bridge (all
> protocols not just IP) a simulated LAN over IP. This immediately makes
> me think of attempting to bring a cluster up between two simulated VAXes
> across the Internet.

Interesting.

> Now I know people have done this before with physical network bridging
> over high speed WANs and I am sure I did know the limitations at one
> time but have forgotten. So the question is what sort of rtt and
> throughput will I need before something really bad happens? Any ballpark
> figures?

At long distances, the round-trip time is dominated by latency due to
the speed of light across the distance.

In terms of official support, the OpenVMS Cluster Software SPD
(http://h18000.www1.hp.com/info/SP2978/SP2978PF.PDF) says inter-node
distances of up to 150 miles (about 250 km) are supported
out-of-the-box, or 500 miles (about 800 km) if you purchase the Disaster
Tolerant Cluster Services package. 10 megabits is minimum supported
bandwidth. Latency should not be "excessive". Packet retransmission rate
due to packet loss or corruption should be less than 1 in 1000.

But your title says "unsupported", which gives the latitude for much
more fun in the discussion.

Based on my observastions, I believe the cluster code became much more
tolerant in terms of bandwidth and packet loss after the adaptive
retransmit timing and congestion-control improvements introduced at
V6.0, and I think that's why we've subsequently been able to see
multiple customer reports of a cluster running 'successfully' over
things like a 56-kilobit link.

We're aware of one disaster-tolerant VMS cluster operating at a distance
of 3,000 miles. In testing in the early '90s, distances of up to 12,000
miles were tested (response times were very slow, as one might expect).

In my group's labs (Multivendor Systems Engineering), we have a Layer 2
Tunneling Protocol (L2TP) tunnel set up between a couple of Cisco
routers so we can send SCS traffic over IP, and we have a
100-millisecond delay introduced on that IP link using a Shunra network
emulator box. We have a 7-node cluster running, split across this link.
This equipment was being staged for the hands-on workshop "Long-distance
HP OpenVMS Clusters" slated for the (now-postponed) HP Technology Forum
in New Orleans (our thoughts and prayers are with the folks there).

In HP's labs at OpenVMS Engineering in Nashua we're doing some testing
of Oracle Server applications in a simulated long-distance cluster using
some neat new boxes from LightSand which can bridge both Fibre Channel
AND LAN traffic (including SCS) over IP, with distance simulated in that
test configuration by delay introduced by an AdTech box.

So the chances of a test cluster running as you describe are quite good.
Please let us know the results of your testing.



Relevant Pages

  • Cluster: ESA spacecraft flying closer than ever for better science (Forwarded)
    ... After weeks of manoeuvres, Samba and Tango, two of ESA's four Cluster ... neutral, spread over large distances. ... spacecraft, as these processes operate at different scales in nature. ...
    (sci.space.news)
  • Re: clustering points up to a sample size
    ... > Given a set of tuples make each one a cluster of size one. ... closest pair by scanning, ... recomputing distances, n-1 times. ... doing nearest neighbor linkage"), ...
    (sci.math)
  • Re: plot cluster of points without knowing the coordinates.
    ... Is it possible to plot a cluster of points in matlab in 2D given that ... For example we have the set of points with distances: ... The distance p1-p3=5 defines a circle centered at p1=of radius 5. ... Do additional points help resolve ambiguity? ...
    (comp.graphics.algorithms)
  • Re: Civilizations in star clusters
    ... Although deep within a dense globular cluster, stellar distances are ... average distances might be say .5 light year and the planetary lifetime ...
    (sci.space.policy)
  • 400 Mile Clustering??
    ... We are currently running a two-site FDDI cluster over approx 5 miles - ... All nodes have one vote except the usually live node ... I intend to research the newer features of volume shadowing, ...
    (comp.os.vms)