Saturday, June 7, 2008

RE: Reliability in of the cloud

The topic of reliability on the cloud can be fairly broad.

Just in this thread were discussing the issues of:

 

  1. Continues high availability
    1. In a relatively stateless job distribution scenario (Map/Reduce)
    2. Data sources and databases
  2.  Consistency – specifically how do we ensure consistency between separate networks and clusters

 

Many others refers to reliability in the cloud with reference to Amazon S3 and the complexity involved in setting up databases on the cloud.

Other would refer to reliability from a different perspective i.e. what happens if the cloud is down?

 

Each of those items deserve an entire discussion.

As Gavan noted many of those issues has been address by some of the data-grid providers (I happen to represent GigaSpaces). The reason is simple: The nature of the grid applications that are using those products in the financial industry have the requirements that I would view as a superset to those of cloud. The demand in this type of applications is to address reliability, consistency, performance and scalability at the same time. Unlike google and other solution that is commonly used for addressing similar requirements compromising on consistency and/or reliability was not an option.

 

As I noted earlier I'm not going to be able to cover how all this is achieved, instead I'll cover some of the core principles.

 

How do we address consistency and avoid split-brain scenario between separate clusters or data-centers:

Split-brain scenario starts when we maintain more then one copy of the data across separate network segments (data centers is just an example for such scenario).

The question is how do we ensure consistency when an update can happen simultaneously on each of the copies. If there is a network failure both can succeed and once the network connection is re-established we can get into inconsistent situation that we can't recover from.

 

There are various patterns for dealing with this scenario:

  1. Conflict resolution – we decide (based on certain algorithm which update wins) – as you can imagine one of the main drawbacks of this approach is that it tends to be very application specific scenario and is error prone. It also requires manual intervention to resolve such conflict.
  2. Master site – In this approach all updates goes to a central location which becomes the master of all updates. – This approach address consistency aspect but lead to scaling and performance issues.

 

So what is the solution?

      Take the master site and partition it between the sites – in this way each site will act as the master for certain part of the data.

      All sites can maintain local copy of the data for read purposes. That means that the data is always available in case of a network failure.

      In this way we can ensure consistency (there is central owner per data item through the entire cluster), scalability (through replication and partitioning), performance (replication can be made local).

 

Now let's look at the Amazon S3 issues.

One of the simplest solution that fits nicely with the cloud is to decouple the persistency layer from the application and use memory resources as the system of records.

I wrote a lengthy post targeted more specifically to MySQL but the same principles that I decided in that post applies here as well.

http://natishalom.typepad.com/nati_shaloms_blog/2008/03/scaling-out-mys.html

 

There is also an interesting opensource project that is using this pattern and built an In-memory data-grid (GigaSpaces in this specific case) synchronized with Amazon SimpleDB.

http://www.openspaces.org/display/EDS/External+Data+Source+by+Amazon+SimpleDB

 

You could use the same pattern to load data from your own local-site i.e. keep the data persistent at your own site and use the data-grid as the system of record for applications running in the cloud.

The data-grid will be responsible for keeping the data-grid in-sync with the local database. You can control the rate in which those two entities will be synchronized based on the performance, network latency and reliability requirements through configuration.

 

From obvious reasons I can speak more on GigaSpaces then I can speak of the other data-grid products such as Gemstone and Coherence but I believe that the principles are similar while the underlying implementation can still be very different. What I can say safely at this stage is that unlike pure data-grid products we see data-grid as a component in broader solution which we refer to as a Scale-out-application server (The equivalent of google AppEngine but for Java,.Net and C++).

 

Nati Shalom

CTO GigaSpaces

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

From: cloud-computing@googlegroups.com [mailto:cloud-computing@googlegroups.com] On Behalf Of Gavan Corr
Sent: Friday, June 06, 2008 3:58 PM
To: cloud-computing@googlegroups.com
Subject: Re: Reliability in of the cloud

 

There are a number of commercial data caching solutions in the market, Gemstone, Coherence (now from Oracle) and Gigaspaces, and to a lesser extent terracotta. of those, Gemstone is the only one I have seen successfully deployed in a large scale multi site environment to ensure consistency of data between multiple sites, and to do reliable failover if a node or a center fails. Hadoop is gaining interest but not there yet...

Gavan

 

 

 

On Jun 5, 2008, at 9:00 PM, Khazret Sapenov wrote:



Alan,

If you are talking about Hadoop, then high availability is not inherent in it yet (but maybe it changed recently).
As far as I know, while there is Secondary Name Node provided (that resides in another data center) there's no guarantee of real time switch of Job Tracker/Name Node/Task Tracker/Data Nodes of DC A to Job Tracker/Name Node/Task Tracker/Data Nodes of DC B.

 

cheers


--
Khaz Sapenov,
Director of Research & Development
Enomaly Labs

US Phone: 212-461-4988 x5
Canada Phone: 416-848-6036 x5
E-mail: khaz@enomaly.net
Get Linked in> http://www.linkedin.com/in/sapenov

On Thu, Jun 5, 2008 at 8:41 PM, Alan Ho <karlunho@yahoo.ca> wrote:

I guessed that about google app engine too.

Things get really interesting when you need to do election leader decisions across data centers. E.g. If you are doing a big map-reduce task in one data center, it goes down, so you want to finish the task in another data center.

How does one transfer the task ? Is it even worth solving ?

Alan Ho

From: Reuven Cohen <ruv@enomaly.com>
Sent: June 05, 2008 10:03 AM
To: cloud-computing@googlegroups.com
Subject: Re: The Business of Building Clouds

From what I've seen of Google App Engine, they distribute your python code to dozens of servers and then use some kind of round robin to spread the load. Nothing ground breaking.

r/c

On Thu, Jun 5, 2008 at 12:59 PM, wyim wyim <wingmanyim@hotmail.com> wrote:



In regards to failover, does Google App Engine have some sort of a LoadBalancer API?
 
thanks
Wayne Yim

From: stuartcharlton@gmail.com

Subject: Re: The Business of Building Clouds

Date: Thu, 5 Jun 2008 09:07:12 -0700



 

On 5-Jun-08, at 8:35 AM, Alan Ho wrote:

 

Picking a provider that has data center failover is critical - but it does mean that you write your application in a way that can failover gracefully. Cloud providers need to provide the base infrastructure to do so OR constrain the user to a particular programming paradigm (like the limitations of google app engine)

 

That's a very astute observation, Alan.    Constraining an architecture to induce certain properties (guarantees?) is likely the right approach.  

 

 Though I wonder if AppEngine is a bit too "Nanny-ish" that limit its audience in ways that don't really impact the big picture qualities.   

 

For example, the choice of Python was easy because it was a standard Google language, but that doesn't seem to be inherently a more applicable language than say C#, Java or Ruby.

 

 

I expect in the future that cloud computing systems will provide the concept of "cloud events" in case of major datacenter failures. I just don't see any way round it.

 

I wonder if Google actually provides this sort of failover for AppEngine today.   Certainly, they could, though they provide no such guarantees at the moment.

 

As for "cloud events" - yup.   In the traditional data centre, it's likely SNMP or JMX traps.   On the cloud, it's not entirely clear if/where SNMP would play.   Or WS-Man.   Or something newer (?).

 

Cheers

Stu

 

 

 




--
--

Reuven Cohen
Founder & Chief Technologist, Enomaly Inc.
www.enomaly.com :: 416 848 6036 x 1
skype: ruv.net // aol: ruv6

blog > www.elasticvapor.com
-
Get Linked in> http://linkedin.com/pub/0/b72/7b4



Visit our website at http://www.nyse.com
*****************************************************************************
Note: The information contained in this message and any attachment to it is privileged, confidential and protected from disclosure. If the reader of this message is not the intended recipient, or an employee or agent responsible for delivering this message to the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly prohibited. If you have received this communication in error, please notify the sender immediately by replying to the message, and please delete it from your system. Thank you. NYSE Euronext, Inc. 

 



--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups "Cloud Computing" group.
To post to this group, send email to cloud-computing@googlegroups.com
To unsubscribe from this group, send email to cloud-computing-unsubscribe@googlegroups.com
For more options, visit this group at http://groups.google.ca/group/cloud-computing?hl=en
-~----------~----~----~----~------~----~------~--~---
Posted by Unknown at 2:59 PM

No comments:

Post a Comment

Subscribe to: Post Comments (Atom)