Re: [dist-obj] CORBA implementation ideas

[Ron Resnick <resnick@actcom.co.il>]

Michiel Perdeck wrote:
> 
> -----Original Message-----
> From:   Paul Benninghoff [SMTP:benning@cse.ogi.edu]
> Sent:   Monday, December 14, 1998 12:43 PM
> To:     Steve Vinoski
> Cc:     Ron Resnick; dist-obj@cs.caltech.edu
> Subject:        Re: [dist-obj] CORBA implementation ideas
> .....   Heck, when I first read the
> excellent post from Rocky Stewart (cross posted from the corba
> newsgroup I think) on this sort of thing it was a major eye-opener, to
> put it mildly  .....
> [Michiel Perdeck -->]  I would be interested to read this post! Could you or
> Rocky Stewart send it to me?
> Thanks, Michiel Perdeck
> ==========================================
> Michiel Perdeck
> CMG Finance div. Advanced Technology
> michiel.perdeck@cmg.nl -- (+31) (0)20 5033000
> mobile: (+31) (0)6 22.78.32.09


I'm attaching the "Rocky Stewart" post which Paul refers to
below. It's Rocky's original post to comp.object.corba
from about a year ago, along with some commentary that
I inserted when I re-posted it to dist-obj. I agree with Paul:
I think Rocky's experiences, and his very eloquent summary
of them, are well worth repeating in this space.

<admin aside>
I'm sure we've all seen plenty of "CORBA implementation ideas"
headers in our inboxes. Some of us have undoubtedly gone
to auto-kill for this thread. Others are taking advantage
of a chance to soak up some opinions about CORBA, and related
topics, which we can use in our job settings, etc.

One thing I'd like to point out is that this is supposed to be
friendly, professional, and non-combative. I think it's great
that Steve Vinoski has taken so much time to address the
CORBA-naysayer community, and offer us his reasoned and
experienced view. He clearly holds a minority position in
this particular crowd, which makes his efforts all the
more challenging - thanks Steve! Folks - this isn't
a witch hunt, and even if it were, Steve is the *last* person
in the CORBA community who should be hunted ;). He's certainly
not the one to blame for all the tales of woe we are hearing...

OK, back to your CORBA thread in progress..
</admin aside>

Rocky's post from a year ago, with my commentary from that
time period interspersed (still contains my sig line from
when I was at IBM):


> From: rstew31177@aol.com (RStew31177)
> Newsgroups: comp.object.corba
> Subject: Re: Large CORBA systems
> Date: 2 Sep 1997 15:28:45 GMT
> 
> >Ofer Shaked wrote:
> >> 
> >> Hi,
> >> 
> >> Has anyone successfully created large CORBA based systems (preferebly
> >> with Orbix) ?
> >> 
> >> I am talking about systems holding at least 100,000-200,000 objects in a
> >> single process, and that are dynamic i.e. have to allocate and free
> >> objects frequently and with a rapid response time.
> >> 
> >> Thanks,
> >>         Ofer.
> >> 
> 
> Large-Scale CORBA Applications
> 
> I have noticed several questions in this news group about how to build
> large-scale applications on CORBA. Consequently, I thought I would take the
> time to give a short response describing my experiences and lessons learned
> building large systems on Orbix. I am currently implementing my fourth such
> system. This latest system will have more than 100 million objects,
> thousands of interactive clients and a mainframe as a batch client. This
> system will support 3 millions transactions per day on the objects. If this
> is the scale you are interested in then my comments below may be helpful to you.
> 
> Lesson number 1:
> Do not try to make every object a CORBA object. This would not scale up
> due to the significant overhead (in space) that this would incur. Also, it
> takes a lot of time to create a CORBA object compared to non-CORBA C++
> objects, so if you need this to be fast avoid over using CORBA. Contrary to
> popular belief, fined-grained CORBA objects are not the way to go for large
> systems. I prefer more of a component-oriented architecture with "service"
> interfaces rather than fine-grained object interfaces. One way to provide a
> gateway to large numbers of objects is via the DSI, but my experience with
> the Orbix DSI does not give me any confidence in this particular
> implementation of it. Another way is to provide distribution adapters that
> allow any C++ object to become distributable at run-time.
> 
> Lesson Number 2:
> You need a fast object persistence mechanism. If you are mapping objects
> to a relational database then get ready to do some manual, and fancy, cache
> management to get good performance. Also, consider using Smart Proxies on
> the client side if possible. I have recently been using the Persistence
> product and have discovered the need to do some fairly sophisticated cache
> prefetching and object batching in order to get ANYTHING CLOSE TO GOOD
> performance. I have also used object databases - they have their own set of
> problems.

Sounds like something David McCusker would advocate. Get the 
caching & persistence strategies as close to the front lines as you
can (i.e. where the objects are really invoked & used).
Plug for IronDoc here?

> Lesson Number 3:
> Inter-object relationships must be navigated at the database level. The
> OODB or RDBMS mapping software will require this anyway, so why add
> complexity and poor performance by expressing the same relationships via
> the COSS Relationship Service. I have tried this approach and was amazed at
> how slow this was.
> 
> Lesson Number 4:
> Forget using Object References for large-scale systems. If a client needs
> to interacts with a million objects on your servers then get ready for huge
> clients, as the Object Reference is no lightweight object. Do some
> arithmetic before using these to determine if they will fit into your
> typical client address space. The alternative is to refer to objects using
> a lightweight OID (integer based) that can be compared efficiently.
> 
> Lesson Number 5:
> For clients that do require large, dynamically updated collections, you
> will need a service that subscribes to update notifications and manages the
> update of the collection on the client side. So, your architecture better
> include the ability to recognize when objects (both CORBA and non-CORBA)
> change state and the ability to advertise this state change to others in an
> efficient manner. I would avoid the OrbixTalk product like the plauge at
> this point. I am very disappointed that Iona dumped such poor quality on
> us. By the way, the COSS Event Service is not very useful in this role. It
> may be used as lower-level plumbing in a Publish/Subscribe architecture,
> but I would avoid using it directly. See my paper in the February, 1997
> issue of C++ Report for a discussion of this topic.

Anyone seen this? I'll have a look here in our library for it.
The comments on OrbixTalk resonate with what I've heard elsewhere.

> Lesson Number 6:
> DII is slow, so do not depend on it for anything that needs to go fast.
> This may be a problem with the Orbix DII, so I would do some performance
> testing on your favorite ORB before a decision is made not to use it. I
> have found that requests using the Orbix DII are 8 times slower than using
> the static interface. Also note that due to a major bug in Orbix, DII
> cannot be used when client and server are collocated.

Gee, not an Orbix fan, huh? The '8 times slower' figure - benchmarked
by whom? And presumably that includes the call marshalling that the
app. has to do in DII to form the arg list. So, we expect _some_
overhead
(although 8x sounds excessive). Also, does that include lookups in an
IR to find the service you want? That would mean additional network
calls, so could be expected to drastically reduce performance. But I
wouldn't expect IR lookups to be included in a straight static/DII
performance comparison.
 
> Lesson Number 7:
> Large servers (those having lot's of objects in them) require memory
> management. You need to pay careful attention to the life cycles of your
> objects, including your distribution adapters (see lesson 1). If you are
> relying on the client to destroy your objects on the server side, then you
> will require distributed garbage collection to clean up WHEN your clients
> fail to do so. In Orbix, all active objects must be in memory. This
> obviously does not scale up if you expect large numbers of active objects,
> so get ready to enter the memory management business.
> 
> Lesson Number 8:
> Large systems require BOTH load balancing and the ability to partition
> objects physically over several computers. Neither of these capabilities is
> supported by the current implementation of Orbix. Although, I would expect
> a new product from Iona for providing some sort of load balancing
> capability in the near future. As far as partitioning the objects, you will
> have to architect your system so that this is possible. Basically, what you
> will require is a work dispatcher and a policy for distributing the work.

Heh. We were doing stuff like this in Apollo. Remember Mark?
 
> Lesson Number 9:
> Distributed systems are MUCH LESS RELIABLE than centralized ones (due to
> additional failure modes) unless you take deliberate actions to design and
> implement fault tolerance capabilities from the get-go. Orbix does not help
> you here; indeed, I know of no commercially available ORB that does. So,
> get ready to start thinking hard about the failure modes you want to be
> able to handle and provide a means to do so.

Well of course I just have to jump in on this one :-). I was
all set to jump on the capitalized words above, till I read the
'unless'.
And of course this is the key. Dist. systems increase the likelihood of
single component failures simply by adding many more things that can
statistically go wrong. But when you have a failure in a well designed
dist system, you actually have *better* chances of something valuable
staying alive than you would in a purely centralized system.
Don't put all your eggs in one basket, and all that.....

So, the objective for good FT design should be loosely coupled dist
systems with efficient state replication, and uniform access to all
replicated objects.

Are there commercial ORBs that support this sort of thing?
Well, there was Orbix+Isis,
but it looks pretty moribund (was it officially canned? I suspect so.)
I don't know if you'd consider Electra to be 'commercial', but people
are apparently using it in commercial settings.

> Lesson Number 10:
> Large systems must be manageable. That is, there needs to be a way to
> start them up, shut them down, monitor them, distribute new releases, etc.
> This must be thought out as it impacts the designs of the servers in your
> system. Tools are becoming available to help with this, but there is still
> much work on your part to be done.
> 
> Lesson Number 11:
> Large systems require transactional semantics. Be careful, though, when
> you introduce OTS to the mix. This complicates development, testing, and
> administration of the deployed system. Also, designs are impacted; indeed,
> even exception-handling idioms must be modified. I have a new paper that
> discusses some of these issues that I would be happy to distribute (for
> review) to those who request it. You also better make sure that OTS
> integrates with your other technologiesFor example, your persistence
> apporach better support the X/Open XA interface or you are in for a big
> surprise. Pay particular attention to integration with you security
> approach. This must be integrated with the OTS in order to provide secure
> transactional capability.
> 
> Lesson Number 12:
> Connection-oriented protocols do not scale up to support thousands (or
> even hundreds) of clients per server. Orbix uses a connection-oriented
> approach between client and server. You better do some scale testing before
> you commit to any particular number of clients over 10. If you do need to
> support lot's of clients concurrently, then get ready to design a way to
> manage many servers of the same type serving those clients, or use some
> messaging service to support them asynchronously.

Heh. Is comment here necessary? Full agreement here.
 
> Lesson Number 13:
> Do not pass large amounts of data through CORBA calls. Use CORBA to
> broker database handles and pass the data via a database.

Yikes. I can understand this one as a performance necessity given
current ORB designs, and the data passing needs of typical tiered
apps. This is one of the main reasons I think we have to start moving
to highly distributed peered designs.

> Lesson Number 14:
> Be diligent about avoiding deadlock situations in your design.
> 
> Lesson Number 15:
> Understand and document the tradeoffs you are dealing with when designing
> a large system:
> Dependability
> Performance (Space, Time, and Scale)
> Complexity
> Technical Risk
> Flexibility (Be Specific About What, Where, How, When, And Who); That is,
> idendify the variation points in the system.
> Cost (Time And Money, Including Maintenance Cost)
> Reuse
> Etc.

This is very true, but just about impossible in typical large projects
I've seen. Most stakeholders in the project (customers, various
levels of management, etc.) have only vague notions of the tradeoffs
involved, and typically insist as categorical requirements that 
all must be maximally satisfied simultaneously. Your well documented
lists will just confuse them. :-( 

> Lesson Number 16:
> Don't be the first to use a new technology unless it is strategically
> vital to your business - not using it is perceived by your market as failure.

Aw, ya gotta live a little ;-)


Then, in another post, the mysterious Rocky told us a bit more
about himself:
 
> ==========
> 
> >Folks:
> >
> >     Does anyone know if OrbixTalk (multicast) support
> >     event correlation ?? 
> >
> >     thanks in advance,
> >     billa
> 
> It does NOT. TCSI (see www.tcsi.com) does provide an integration of Orbix
> with the In Charge event correlator from SMARTS. Tell them I sent you and
> you may get a discount!  (I designed it for them)
> 
> Rocky Stewart

TCSI is well known to some of us - it's the 'other' Teknekron
(as opposed to TIBCO).

---------
Ron Resnick                       IBM: ron_resnick@vnet.ibm.com
IBM Haifa Research Lab HRL        world: resnick@interlog.com
MATAM 31905 Haifa, Israel        
http://www.interlog.com/~resnick/ron.html
Tel: 972-4-829-6491

"A shadow is the ..
... mobile, persistent, distributed, self-descriptive,
self-contained, pro-active, cooperative, trustworthy, 
object-oriented, software analog of some real world object"
-- Sandor Spruit
============================================================================
To unsubscribe and other requests, mailto:dist-obj-help@unity.cs.caltech.edu 
Archives, FAQ, etc.       http://www.distributedcoalition.org/mailing_lists/