In this paper we presentperformancemeasurementsin a cluster environment. First, we briefly explain our version of optimistic concurrency control and load balance.Then we compareperformanceand speed-upof a CORBA- and an AspectC++-implementation of a distributed object server for pessimistic and optimistic concurrency control.
Object-oriented databasesrepresent an implementation tool for data repositories of
some specializedapplications,which often processnot only selectedattributesof
objects but complete objects.When using relational databasesa,ttributesof many objects
can be processedvery quickly if they are storedas columns in one table. Processingof
completeobjects storedin a relational databasemeansto synthesizeobjectsfrom many
tableswhere they are spreadbecauseof the necessarynormalization of relations.There
is a rule of thumb that if it is necessaryto join more than six tablesto answer a query,
an object-orienteddatabasecould bring advantages[
CASE tools are usually built as a chain of editors that have a common data
repository,in which all input and outputdataof editorsare storedas fine-grainedobjects[1].
We havebuilt a CASE tool with an object-orienteddatarepository and have found that
its performance is not sufficient especially for multi-user environment. This was our
motivation to investigatepossibilities of parallel and distributed object servers.In the
next stepwe built an object serverfor the parallel computerParsytecand have analyzed
its performance [
The restof the paperis organizedasfollows. In section2 we discussrelatedwork. In section3 we briefly explainwhich cross-cuttingconceptsof concurrencywe modelled as aspectsin our proposedand implementedoptimistic concurrencycontrol method. In section4 our new versionof the optimistic concurrencycontrol for clusterenvironment is presented.Section 5 describesour approachto load balancing.In the last sectionwe presentsomemeasurementresultsand conclude.
Empirical comparisonsbetween optimistic and pessimistic concurrency control have
been published in [
We have not found quantitativemeasurementresultsthat would be relevantfor our shared-nothingcluster systemthat should work as a datarepository of a CASE tool.
The applicability of aspect-orientedtechniquesto databasesystemsis known and
was already investigated,e.g. in [
We concentratedon concurrencycontrol as a cross-cuttingconcernat DBMS-level. For most systemsthe dominant dimension of decomposition is transactionprocessing and resource management.This leads to a scattering of code related to concurrency control.
In OPAS (Objecroriented PArallel Server [
Using AOP both TM and Validation Manager (VM, conceptual component for OCC) could be implementedin a separatemanner and then put togetherby the weaver AspectC++ in the preprocessingphase.An aspectcapturesthe acquisition of an object
PerformanceComparisonof Distributed Object ServerImplementations 41 through an advice and storesthe starttime of access.The completion of an object access is intercepted,too. The correspondingadvice (oin-point model of AOP) determinesthe end time and accessmode. It calls the VM for validation using the collectedinformation and processesthe validation result.We do not explain aspect-orientedprogramming in more details becauseit is not the main topis of this paper.
Isolation is one of the ACID properties.It protects concurrently running transactions from seeingeach others' uncommitted data,which could lead to inconsistency.Serializability is the correctnesscriterion for isolation. It statesthat the concurrentexecution of severaltransactionsmust be equivalentto at leastone serial execution [?].
A common approachto isolation is locking, which basically meansthat locks are granted to transactionsto protect a resourcefrom concurrent access.One of the main drawbacksof locking is the need to resolve deadlocks,especially global deadlocksin distributedenvironments.Another disadvantageis that for somespecialapplication the probability of a concurrent accessis very small and the locking overhead decreases performance.
A non-locking and thereforedeadlock-freeapproachto isolation is optimistic concurrcncycontrol (OCC). It actson thc assumptionthatconflictsarerare,so that locking causesonly unnecessaryoverhead.Transaction execution is divided into the phases read, validate and write. In the read phasetransactionsare executedwithout intervention of thc OCC. They arevalidateda posterioriin thevalidationphase,whereconflicts with other transactionsare detected.Conflicts are resolvedby rolling back an involved transaction.Successfullyvalidatedtransactionsreachthe write phaseand their changes are made persistentand visible to others.
Existing validation schemes could be subdivided into backward (BOCC) and forward (FOCC) oriented methods. BOCC takes only already finished transactionsinto account whereasFOCC validatesagainstrunning ones.It is essential for both to be able to determinethe read and write setsof transactions.
We integratedOCC into our object repository OPAS and thereforereplacedthe existing solutionusingtwo phaselocking and pre-claiming.In the applicationdomain of CASE systemsconflicts are unlikely. The validation is done in a distributedmanner. For this reasononly the BOCC approachwas applicable,becausethereis no efficient way to determine all currently running transactions.The validation authority is partitioned among all nodesand accommodatedto the datadistribution in order to maximize locality. An object is validatedon the node where it is stored.
Local serializability is achieved as follows. The start time, end time and mode (read,written)of an object accessarc usedfor validation.A conflict is occurredif the given interval overlapswith anotherof an already successfullyvalidatedchangetransaction.
This test is our original extension of the BOCC validation algorithm. Traditional validation algorithms consideronly object accessin the associatedread and write sets without the concrete point in time of the access.As a consequence,these algorithms mark certaintransactionschedulesasconflicting althoughtheschedulesareserializable. The presentedvalidation schemaconsidersthe concretepoints in time of object access. Therefore it is inscnsitiveto false conflicts whereasthe test itself remainssimple. A prerequisite for the validation schemais the existenceof synchronizedclocks within the cluster computer. This drawback is acceptablebecausethe local system clocks of the the clusternodesareregularly synchronizedin order to run the Andrew File System (AFS).
But the given critcrion for local serializability doesnot guaranteea sufficient level of isolation. The validation order of sub-transactionsbelonging to different global transactions could differ. To reachglobal serializability transactionnumbers(TAN) were introduced.They are unique, linearly orderedand thereforecomparable.For eachobject the TANs of the alreadyvalidated transactionslast read and last written are stored.A conflict with respectto global serializabilityhasoccurredif thereadobjectwas meanwhile written (own TAN less than the TAN of the object for last write access)or the written object was meanwhile read (own TAN lessthan the TAN of the object for last read access).This procedureproducesthe same validation order on all nodes and guarantees global serializability.
The validation is done on a basisof sub-transactionsin the read phase.Thus there is no global validation phase.For this reasonalready validated changesare uncertain and therefore must be protected until the write phase is reached.This is done with locks on changedobjects.We were able to integratethe validation into the two phase commit protocol to reducecommunication. Comparedto the former locking combined with pre-claiming the new approachoffers a higher degreeof parallelism.The achieved performanceis presentedin section6.
Load distribution has been well-investigated in operating systemstheory. In centralized realizationsonly a single instancemakes decisions concerning load distribution, whereasa distributedrealization lacks this instance.Load is distributedby initial placement and could be combined with migration schemes,the movement of an already placed load. The processingtime of a parallelizedrequest is determined by the processing time of the slowest sub-operation.Execution skew refers to the variance in processingtime and often goesback to data skew.Therefore data allocation has a great impact on load distribution especially for shared-nothingsystems.It determineshow much data locality could be utilized during requestexecution.
In OPAS data allocation is done on a per-object basis without replication. A hash function assignsan object to a certain node. Furthermore, client requestsare divided into sub-operationsin order to utilize intra transactionparallelism.Thesesub-operations are initially placed using a static and centralizedschema,and are assignedto the node holding the associateddata for the purposeof maximizing locality. They are executed non-preemptivelywithout migration.
With increasingnumber of processingnodesthe hasheddataallocation schemalead to data skew. Becauseof the tight coupling between allocation and sub-operationprocessing,executionskew arose,too.
Performance Comparison of D istributed Object Server Implementations 43 First of all, we replacedthe hashedallocation by a round robin strategyin order to achievea broad object distribution andminimize dataskew.Furthermore,we introduced a two level load sharing.On the flrst level, sub-operationswere scheduledin a central and dynamic way. Initial placementdecisionsaremadeusing a load index and capacity of the executionunits. On the secondlevel thereis a randomizedworkstealing.It could be classifiedas a distributedload sharingapproach.The basicidea is that underloaded nodessendstealmessagesto other randomly selectedparticipants.Nodes which receive stealmessageswhile they areoverloadedsenda certainnumber of sub-operationsto the initiator. We presentthe resultsof the performancetestsin section6.
The first implementationParsytec-OPASwasdesignedfor theparallel computerParsytec
GClPowerPlusl28 (shared-disk)[
Using a sequentialmachine, object componentswill be searchedafter each other.
When using a parallel machine object componentswhich have the common father can
be searchedin parallel. This kind of parallelity will be called as the
intra-objectparallelity. In data processingtypical for CASE tools there are frequently used operations
like GET OBJECT and SHOW OBJECT. This means,that all componentsof
askedaggregatedobjectshave to be found on the disk, in the object buffer or in the pagebuffer
[
To compareperformanceof both new implementationswith the old systemperformance we usedthe samedata.
The systemhad been ported to the "Chemnitzer Linux Cluster (CLiC)", which has the following features:528 nodes,each has 512 MB RAM and20 GB local disk, 800 MHz frequency,fast Ethemet (100 MBits/s) nodecommunication,throughputmaximal 128GBit/s, minimal latency100prs.
Parsytec-OPASwas adapted for the shared-nothingenvironment of CLiC using
CORBA [
The speed-upsfor both implementationsfor intra-object parallelism are shown in Fig. 2 and 3. The performanceof CORBA-OPAS is slightly better,although the graph shapesmainly correspond.Compared to Parsytec-OPASboth CLiC implementations perform better.The exchangedconcurrencycontrol mechanismprovidesan explanation 1 0 0 050()()(] l..ooqooc)
t. o o --='numl:er of irrst rtrcr ic.rr trr:it s pet - < - - - " - 1 3 r o * ntrrnber_ of Eransact:iotr rnarlaqerc for the arisendifference.The former implementation usedlocking combined with preclaiming, whereasthe latterbenefitsfrom optimisticconcurrencycontrol.
We comparedthe performanceof the two concurrencycontrol mechanisms(locking and OCC) as used in MPI-OPAS. The outcome is as follows. Optimistic concurrency control outperforms the locking in almost every measuredpoint. For the seriesusing fewer instruction units, the speed-upis about 20% higher on the average,whereasthe maximum speed-upis increasedup to 30%. Concerningthe serieswith a greateramount of instruction units, the increaseof the speed-upturned out to be smaller and levelled off at about l}Yo on the average.The results indicate that OCC not only increasesthe actual systemperformance,but also raisesthe inherentpotential for parallelization.
Fig. 4 and 5 show the achievedresultsconcerninginter-objectparallelism for CORBAOPAS and MPI-OPAS. Both implementations performed almost identically. Particularly for a greateramount of clients the speed-upincreasednearly linearly. Due to optimistic concurrencycontrol and load distribution, performancecould be increased.In former versions like Parsytec-OPAS,but also CLiC-OPAS versions with pessimistic concurrency control, speed-updid not exceed 15. The dependencybetween processor nodesand speed-upwas not linear, but logarithmic in theseversions.
Although the speed-upsfor both CLiC implementations are almost identical, the runtime differs greatly. Fig. 6 presentsthe runtime curve for rhe CORBA-OPAS. MpIOPAS is about 15 -25 times fasterthan CORBA-OPAS asshown in Fig. 7. It is already very well known that the usageof middleware like CORBA causesoverhead,but we did not expectthat the overheadratio would be that high.
Furthermore,it is interestingthat the impact of applied optimizations on the performance was different for the two versions.CORBA-OPAS did benefit greatly from load sharing.Its maximum speed-upcould be increasedbyaI% from 22.4 to 31.6 (seeFig. 4). Initial placementof objects was not consideredwith respectto CORBA-OPAS and thereforehad beenleft unchansed.
Performance Comparison of Distributed Object Server Implementations
The benefitsfrom dynamic load sharingas mentionedin section5 were disproportionally smaller for MPI-OPAS. In terms of numbersthey are 3.8% on the averageand L}To at the maximum. Whereasthe initial placementof objectshad a strongereffect on the performance, the round robin strategyreduced data skew and raised the speed-up by 32% from25.4 to the maximum of 33.4 asdisplayedin Fig. 5.
For MPI-OPAS, we measuredthe impact of concurrency control on performance for inter-object parallelism. The result is that OCC outperformedpre-claiming locking. In direct comparisonspeed-upfor OCC is about 60% higher.Thereby the ratio between the speed-upsdid increasefor a greaternumber of processingnodes.Consequentlythis behavior causesthe almost linear speed-uprising illustrated in Fig. 5. Furthermore, we measuredthe influence of synchronizationconflicts on performance.The speed-up decreasedwhile increasingthe conflict rate.Without conflicts,the maximum speed-up ! 10! number of cLients ---i)n transaction
I number of managers number of clienls was 33.4. At a conflict rate of | 7o (resp.5 7o, 10 7o,20 Vo)the maximum speed-up decreasedto 30.8 (resp.29.0,2'7.4, 25.5).
We believe that our resultscan be usedfor performanceand speed-upestimation of proposedcluster systemsand also for the choice of implementation technology.