Monday, October 15, 2012

JVM Tuning for Low Latency High Throughput on Multi Core Linux Box

Recently, I have been tuning JVM parameters to achieve low latency & high throughput on HotSpot VM. For server side Java application, low latency is a very important requirement. In many cases, Java garbage collector pause is the serious threat. Under a very strict SLA, let's say response time under 100ms or less, it requires good amount of engineering effort to make the application behave under high stress environment .  

In my experience, before any JVM parameters tuning, I have spent lots of time to identify and remove thread contention points in the application. My development environment is 4 core Linux machine, JDK 1.6,  JBoss, and required SLA less than 100ms. There are 600 plus different JVM parameters. This article is about tuning GC for low latency for server side Java  applications, and the focus will be on those parameters that have bigger impact on the achieving low latency & high throughput.


    
$ /usr/java/jdk1.6.0_29/bin/java -XX:+PrintFlagsFinal -version 
     


The Java heap is divided into three main sections: Young Generation, Old Generation and the Permanent Generation.











Young Generation: The Eden Space of the Young Generation holds all the newly created objects. When this section fills, the Scavenge Garbage Collector clears out of memory all objects that are unreferenced. Objects that survive this scavenge moved to the "From" Survivor Space. The Survivor Space is a section of the Young Generation for these intermediate‐life objects. It has two equally‐sized subspaces "To" and “From” which are used by its algorithm for fast switching and cleanup. Once the Scavange GC is complete, the pointers on the two spaces are reversed: "To" becomes "From" and "From" becomes "To".


Old Generation: Once an object survives a given number of Scavenge GCs, it is promoted (or tenured) from the "To" Space to the Old Generation. Objects in this space are never garbage collected except in the two cases: Full Garbage Collection or Concurrent Mark‐and‐Sweep Garbage Collection. If the Old Generation is full and there is no way for the heap to expand, an Out‐of‐Memory error (OOME) is thrown and the JVM will crash. 


Permanent Generation: The Permanent Generation is where class files are kept. These are the result of compiled classes and jsp pages. If this space is full, it triggers a Full Garbage Collection. If the Full Garbage Collection cannot clean out old unreferenced classes and there is no room left to expand the Permanent Space, an Out‐of‐ Memory error (OOME) is thrown and the JVM will crash. 



HotSpot JVM may use one of 6 combinations of garbage collectors listed below.


Young collector
Old collector
JVM option
Serial (DefNew)
Serial Mark-Sweep-Compact
-XX:+UseSerialGC
Parallel scavenge (PSYoungGen)
Serial Mark-Sweep-Compact (PSOldGen)
-XX:+UseParallelGC
Parallel scavenge (PSYoungGen)
Parallel Mark-Sweep-Compact (ParOldGen)
-XX:+UseParallelOldGC
Serial (DefNew)
Concurrent Mark Sweep
-XX:+UseConcMarkSweepGC
-XX:-UseParNewGC
Parallel (ParNew)
Concurrent Mark Sweep
-XX:+UseConcMarkSweepGC
-XX:+UseParNewGC
G1
-XX:+UseG1GC




A List of Stop the World Pauses:

  • Young space collections 
  • Full GCs – All collectors 
  • System GCs – Called via JMX or the application 
  • CMS Initial Mark Phase 
  • CMS Remark Phase 
  • CMS Concurrent Mode Failure



In my experiment, CMS gives the best results for low latency & high throughput. Here is summary of what I have learned. 

  1. JVM tuning is application specific. In depth knowledge of the application will help. And one needs to take a holistic when tuning. 
  2. Young Collections are fast and efficient. It is important to give objects the opportunity to die young. Smaller Young Space helps. 
  3. CMS is concurrent and requires CPU and it will compete with the application during collections.
  4. CMS fragments the Old Space and it makes Object allocations are more complicated. 
  5. Sizing the heap correctly is critical. Undersized heaps will make CMS work overtime, and worse it would cause CMS Concurrent Mode Failure. 
  6. Sizing the young ratio is important: 1) Size the survivor spaces appropriately 2) Configure the Tenuring Threshold appropriately  
  7. CMS to wait for a Young GC before starting. 

Here is the list of the recommended JVM settings for low latency & high throughput:


-server
-Xms2048m 
-Xmx2048m 
-XX:+UseConcMarkSweepGC 
-XX:+UseParNewGC
-XX:+AggressiveOpts
-XX:+CMSParallelRemarkEnabled
-XX:+CMSScavengeBeforeRemark
-XX:+UseCMSInitiatingOccupancyOnly
-XX:CMSInitiatingOccupancyFraction=65
-XX:CMSWaitDuration=300000
-XX:GCTimeRatio=19
-XX:NewSize=128m
-XX:MaxNewSize=128m
-XX:PermSize=64m
-XX:MaxPermSize=64m
-XX:SurvivorRatio=88
-XX:TargetSurvivorRatio=88
-XX:MaxTenuringThreshold=15
-XX:MaxGCMinorPauseMillis=1
-XX:MaxGCPauseMillis=5
-XX:+HeapDumpOnOutOfMemoryError
-XX:HeapDumpPath=./gc_heap_dump/
-XX:+PrintGCDateStamps
-XX:+PrintGCDetails
-XX:+PrintTenuringDistribution
-Xloggc:./gc_log.log




Compare with 600 plus parameters to play with, this is a much shorter list. I hope you like this post, and enjoy your journey.

Sunday, September 23, 2012

Object oriented programing technique to replace if...then...else

If...then...else block could get out of control if used in many levels deep. The block becomes hard to maintain.


Here is one example:
    
static public String handleReturnString2 (String stringData) {
     
     if (stringData==null) {
      return "No handler found!";
     } else if (stringData.contains("ERROR") ) {
      return "Error case handled!";
     } else if (stringData.contains("FAILED") ) {
      return "Failed case handled!";
     } else if (stringData.contains("SKIPPED") ) {
      return "Skipped case handled!";
     } else if (stringData.contains("PASSED") ) {
      return "Passed case handled!";
     }
     
     return "No handler found!";
    }



Here is one way to replace it. There is trick to use Pattern match to replace string.contains():
   

import java.util.HashMap;
import java.util.Map;
import java.util.regex.Matcher;
import java.util.regex.Pattern;


// object oriented way to handle if then else  
public class SwitchOnSubstring {

 // define all ReturnCode enum
 public static enum ReturnCode {
  PASSED, SKIPPED, FAILED, ERROR; 
  
  // build pattern string, for string contains  
  static String getAllStringPattern() {
   StringBuilder sb = new StringBuilder(16);
   boolean firstone = true;
   for (ReturnCode p : ReturnCode.values()) {
    if (firstone) {
     firstone = false;
    } else {
     sb.append("|");
    }
    sb.append(p.name());
   }
   return sb.toString();
  }
 }
 
 // define handler interface 
 static private interface CaseHandler {
  public String handleReturn ();
 }

 static private class PassedCaseHandler implements CaseHandler {
  public String handleReturn () {
   return "Passed case handled!";
  }
 }


 // default handler
 final private static CaseHandler HandlerNone = new CaseHandler() {
  public String handleReturn () {
   return "No handler found!";
  }
 };
 
 final private static Pattern ReturnCodePattern = Pattern.compile( ReturnCode.getAllStringPattern());
 
 // use java regex to match string containing "PASSED" or "SKIPPED" "FAILED" "ERROR" to enum PASSED, SKIPPED, FAILED, ERROR;
 static private ReturnCode matchStringToReturnCode (String stringData)  {
  String strRequest = null;
  Matcher matcher = ReturnCodePattern.matcher(stringData);
  if (matcher.find()) {
   if (matcher.start()>=0 && matcher.end()>matcher.start()) {
    strRequest = stringData.substring(matcher.start(), matcher.end());
   }
  }
  return  strRequest==null? null : ReturnCode.valueOf(strRequest);
 }

 // setup the map structure to map enum PASSED, SKIPPED, FAILED, ERROR to their corresponding handlers 
 static private Map<ReturnCode, CaseHandler> mapHandlers = new HashMap<ReturnCode, CaseHandler> () {
  private static final long serialVersionUID = -80L;
  {
   // ErrorCaseHandler for string contains "ERROR"
            put(ReturnCode.ERROR , new CaseHandler() {
          public String handleReturn () {
           return "Error case handled!";
          }
         });

   // FailedCaseHandler for string contains "FAILED"
            put(ReturnCode.FAILED , new CaseHandler() {
          public String handleReturn () {
           return "Failed case handled!";
          }
         });

   // SkippedCaseHandler for string contains "SKIPPED"

            put(ReturnCode.SKIPPED , new CaseHandler() {
          public String handleReturn () {
           return "Skipped case handled!";
          }
         });
            
   // PassedCaseHandler for string contains "PASSED"
            put(ReturnCode.PASSED, new CaseHandler() {
          public String handleReturn () {
           return "Passed case handled!";
          }
         });
           
        }
    };


    // main method to handle the string 
    // if string contains PASSED, return PassedCaseHandler's result
    // if string contains ERROR, return ErrorCaseHandler's result
    // if string contains FAILED, return FailedCaseHandler's result
    // if string contains SKIPPED, return SkippedCaseHandler's result
    static public String handleReturnString (String stringData) {
     ReturnCode code = stringData==null ? null : matchStringToReturnCode (stringData);
     CaseHandler handle = ( code==null ) ? HandlerNone : mapHandlers.get(code);
     return handle.handleReturn() ;
    }
    

}



If you want the source, you can get it from github and perform the following step to compile & test:
   

git clone https://github.com/leizou123/SwitchOnSubstring.git

cd SwitchOnSubstring

mvn compile test 

Saturday, May 19, 2012

Project Euler is awesome, part 2

Project Euler is cool,  awesome. Solutions to the problems force me to come with efficient algorithms.

Here is Problem # 12:

The sequence of triangle numbers is generated by adding the natural numbers. So the 7th triangle number would be 1 + 2 + 3 + 4 + 5 + 6 + 7 = 28. The first ten terms would be:

1, 3, 6, 10, 15, 21, 28, 36, 45, 55, ...

Let us list the factors of the first seven triangle numbers:

 1: 1
 3: 1,3
 6: 1,2,3,6
10: 1,2,5,10
15: 1,3,5,15
21: 1,3,7,21
28: 1,2,4,7,14,28
We can see that 28 is the first triangle number to have over five divisors.

What is the value of the first triangle number to have over five hundred divisors?

Here is the Java solution to this problem:

public class Problem012 {
 
 static private long findPrimeFactors(long number, Long[] primelist) {
     int numOfFactors = 1;
     int exponent;
     long remain = number;
  
     for (int i = 0; i < primelist.length; i++) {
         if (primelist[i] * primelist[i] > number) {
             return numOfFactors * 2;
         }
  
         exponent = 1;
         while (remain % primelist[i] == 0) {
             exponent++;
             remain = remain / primelist[i];
         }
         numOfFactors *= exponent;
  
         //If there is no remainder, return the count
         if (remain == 1) {
             return numOfFactors;
         }
     }
     return numOfFactors;
 }
 
 
 public static void findNumber(int count) { 
  
  long startTime = System.nanoTime();
  
  long number = 1;
  long i = 2;
  long cnt = 0;
  long factorCount1 = 2;
  long factorCount2 = 2;
  
  Long[] primes = runEratosthenesSieve(1000);
  
  while (cnt < count) {
   if (i % 2 == 0) {
    factorCount2 = findPrimeFactors(i + 1, primes);
    cnt = factorCount2 * factorCount1;
   } else {
    factorCount1 = findPrimeFactors((i + 1) / 2, primes);
    cnt = factorCount2*factorCount1;
   }
   i++;
  }
  number = i * (i - 1) / 2;
  long elapsedTime = (System.nanoTime() - startTime) / 1000000; // ms
  
  System.out.println("The number with five hundred divisors is " + number + " time took to find it : " + elapsedTime + " ms");
 }
 
 
 public static void findNumberV2(int count) { 
  
  long startTime = System.nanoTime();
  
  long number = 1;
  long i = 2;
  long cnt = 0;
  
  Long[] primes = runEratosthenesSieve(1000);
  
  while (cnt < count) {
   number += i++;
   cnt = findPrimeFactors(number, primes);
  }

  long elapsedTime = (System.nanoTime() - startTime) / 1000000; // ms
  
  System.out.println("The number with five hundred divisors is " + number + " time took to find it : " + elapsedTime + " ms");
 }

 static public Long[] runEratosthenesSieve(int upperBound) {
  ArrayList<Long> listPrim = new ArrayList<Long> ();
  
  int upperBoundSquareRoot = (int) Math.sqrt(upperBound);
  boolean[] isComposite = new boolean[upperBound + 1];
  for (int m = 2; m <= upperBoundSquareRoot; m++) {
   if (!isComposite[m]) {
            //System.out.print(m + " ");
    listPrim.add(new Long(m));
    for (int k = m * m; k <= upperBound; k += m)
             isComposite[k] = true;
   }
  }
     
  for (int m = upperBoundSquareRoot; m <= upperBound; m++) {
   if (!isComposite[m]) {
    listPrim.add(new Long(m));
   }
  }
  return listPrim.toArray(new Long[listPrim.size()]) ;
 }


 

 public static void main(String[] args) { 

  findNumber(500);
  findNumberV2(500);
 }
}




When you run the code, you will find out the method findNumber() is more efficient than findNumberV2()

Sunday, May 13, 2012

Project Euler is cool, awesome

Project Euler is cool,  awesome. Solutions to the problems force me to come with efficient algorithms.

Here is one example:

The first known prime found to exceed one million digits was discovered in 1999, and is a Mersenne prime of the form 269725931; it contains exactly 2,098,960 digits. Subsequently other Mersenne primes, of the form 2p1, have been found which contain more digits.

However, in 2004 there was found a massive non-Mersenne prime which contains 2,357,207 digits: 28433*2^7830457+1.

Find the last ten digits of this prime number.

Here is the code:
  public static void main(String[] args) {
   long start = System.nanoTime() ;
   long value = 1;
   long LIMIT = 10000000000l;
   for (int i = 1 ; i <=7830457 ; i++) {
    value = value * 2;
    value =value%LIMIT;
   }
   value = value * 28433;
   value =value%LIMIT;
   value = value + 1 ;
   value =value%LIMIT;
   
   long end = System.nanoTime() ;
   System.out.println("result = " + value);
   System.out.println("computation time = " + ( (end - start) / 1000 ) + " ms");
  }

See, the solution is short and very efficient.

Here is another one:

Triangle, pentagonal, and hexagonal numbers are generated by the following formulae:

Triangle T(n)=n(n+1)/2 1, 3, 6, 10, 15, ...
Pentagonal P(n)=n(3n-1)/2 1, 5, 12, 22, 35, ...
Hexagonal H(n)=n(2n-1) 1, 6, 15, 28, 45, ...

It can be verified that T(285) = P(165) = H(143) = 40755.

Find the next triangle number that is also pentagonal and hexagonal.

/**

http://projecteuler.net/problem=45
 
Triangle, pentagonal, and hexagonal numbers are generated by the following formulae:

Triangle   T(n)=n(n+1)/2   1, 3, 6, 10, 15, ...
Pentagonal   P(n)=n(3n-1)/2 1, 5, 12, 22, 35, ...
Hexagonal   H(n)=n(2n-1)   1, 6, 15, 28, 45, ...

It can be verified that T(285) = P(165) = H(143) = 40755.

Find the next triangle number that is also pentagonal and hexagonal.

 * 
 * @author lei zou
 *
 */
public class Problem045 {
 
 private static final BigInteger TWO = new BigInteger("2");

 static public BigInteger[] setTriangleArray(int base, BigInteger[] triangles) {
  for (int i=1; i<=triangles.length; i++) {
   BigInteger n = new BigInteger( Integer.toString(i+base) );
   BigInteger n2 = new BigInteger( Integer.toString( (i+base)+1) ) ;
   triangles[i-1] = n.multiply(n2);
   triangles[i-1] = triangles[i-1].divide(TWO);
  }
  //System.out.println("Triangle is " + Triangle[Triangle.length-1] );
  return triangles;
 }

 static public BigInteger[] setPentagonalArray(int base, BigInteger[] pentagonals) {
  for (int i=1; i<=pentagonals.length; i++) {
   BigInteger n = new BigInteger( Integer.toString(i+base) );
   BigInteger n2 = new BigInteger( Integer.toString( ( 3* (i+base) -1 ) )  ) ;
   pentagonals[i-1] = n.multiply(n2);
   pentagonals[i-1] = pentagonals[i-1].divide(TWO);
  }
  //System.out.println("Pentagonal is " + pentagonals[pentagonals.length-1] );
  return pentagonals;
 }

 static public BigInteger[] setHexagonalArray(int base, BigInteger[] hexagonals) {
  for (int i=1; i<=hexagonals.length; i++) {
   BigInteger n = new BigInteger( Integer.toString(i+base) );
   BigInteger n2 = new BigInteger( Integer.toString( ( 2* (i+base) -1 ) )  ) ;
   hexagonals[i-1] = n.multiply(n2);
   
  }
  return hexagonals;
 }

 
 public static void main(String[] args) {
  int limit = 1000;
  BigInteger[] Triangle = new BigInteger[limit];
  BigInteger[] Pentagonal = new BigInteger[limit];
  BigInteger[] Hexagonal = new BigInteger[limit];
  
  setTriangleArray(0, Triangle);
  setPentagonalArray(0, Pentagonal);
  setHexagonalArray(0, Hexagonal);

  int cnt=0;
  int upperTriangle=limit, upperPentagonal=limit, upperHexagonal=limit;
  int lowTriangle=0, lowPentagonal=0, lowHexagonal=0;
  int i=0, j=0, k=0;
  BigInteger max = new BigInteger("0");
  while (cnt<3) {
   while (i<upperTriangle && j<upperPentagonal && k<upperHexagonal) {
    if (Triangle[i-lowTriangle].equals(Pentagonal[j-lowPentagonal]) 
      && Pentagonal[j-lowPentagonal].equals(Hexagonal[k-lowHexagonal]) ) {
    
     System.out.print( (cnt+1) + "th" + " number is " + Triangle[i-lowTriangle] );
     System.out.println(" Triangle index: " + (++i) + " Pentagonal index: " + (++j) + " Hexagonal index: " + (++k) );
     cnt++;
    
    }
    max = Triangle[i-lowTriangle].compareTo( Pentagonal[j-lowPentagonal] ) > 0  ? Triangle[i-lowTriangle] : Pentagonal[j-lowPentagonal]; 
    max = Hexagonal[k-lowHexagonal].compareTo(max) > 0  ? Hexagonal[k-lowHexagonal] : max;
    // 
    while (i<upperTriangle && Triangle[i-lowTriangle].compareTo(max) <0  ) i++;
    while (j<upperPentagonal && Pentagonal[j-lowPentagonal].compareTo(max)<0  ) j++;
    while (k<upperHexagonal && Hexagonal[k-lowHexagonal].compareTo(max)<0  ) k++;
   }
   //System.out.println("Triangle: " + (i) + " Pentagonal: " + (j) + " Hexagonal: " + (k) + " max " + max );
   if (i==upperTriangle) { 
    lowTriangle = i;
    setTriangleArray(i, Triangle);
    upperTriangle+=Triangle.length;
   }
   if (j==upperPentagonal) {
    lowPentagonal = j;
    setPentagonalArray(j, Pentagonal);
    upperPentagonal += Pentagonal.length;
   }
   if (k==upperHexagonal) {
    lowHexagonal = k;
    setHexagonalArray(k, Hexagonal);
    upperHexagonal += Hexagonal.length;
   }
  }
 }
}



Not sure that is the best solution. But it does solve the problem.

Friday, April 27, 2012

Different ways to calculate Fibonacci number - recursive, dynamic, memorize, and multi-threading

Calculating Fibonacci number is an interesting computer  science problem.

fib (n) = fib (n-1) + fib(n-2).

There are many ways to implement the Fibonacci calculator. Following are Java implementations.


1. Recursion. Looks pretty, just like math formula. Lots of stack frames get created. Expensive to run. Hard to mix with multi-threading
 /**
  * calculate Fibonacci number through Recursion.
  * 
  * Recursion. Looks pretty, just like math formula. Lots of stack frames get created. Expensive to run.
  *   
  * @param number
  * @return
  */
    static long getFibViaRecursion(long number) {
     if (number==0 || number==1)
      return number;
     else 
      return getFibViaRecursion(number-1) + getFibViaRecursion(number-2);
    }

    static void testFibViaRecursion () {
  long start = System.currentTimeMillis();
     System.out.println("getFibViaRecursion(10) - " + getFibViaRecursion(10) );
     System.out.println("getFibViaRecursion(20) - " + getFibViaRecursion(20) );
     System.out.println("getFibViaRecursion(40) - " + getFibViaRecursion(40) );
  long end = System.currentTimeMillis() - start;
  System.out.println("FibViaRecursion took: " + end + "ms");
    }


2. Straight Dynamic. Short in this case, easy to follow, fast.
 /**
  * calculate Fibonacci number with straight forward dynamic method. 
  * 
  * Straight Dynamic. Short in this case, easy to follow, fast.  
  * 
  * @author lei zou
  *
  */
    static long getFibViaDynamic(long number) {
     long val1 = 0;
     long val2 = 1;
     long sum = val1 + val2;
     for (long i=1; i<number; i++) {
      sum = val1 + val2;
      val1 = val2;
      val2 = sum;
     }
     return sum;
    }

    static void testFibViaDynamic () {
  long start = System.currentTimeMillis();
     System.out.println("getFibViaDynamic(10) - " + getFibViaDynamic(10) );
     System.out.println("getFibViaDynamic(20) - " + getFibViaDynamic(20) );
     System.out.println("getFibViaDynamic(50) - " + getFibViaDynamic(50) );
  long end = System.currentTimeMillis() - start;
  System.out.println("testFibViaDynamic took: " + end + "ms");
    }



3. Dynamic memorize with Hashtable - cool, less calculation, higher memory requirement.

    
    static void testDynamicFibViaHashTable () {
  long start = System.currentTimeMillis();
     System.out.println("new DynamicFibViaHashTable().getFib( new Long(10) ) - " + new DynamicFibViaHashTable().getFib( new Long(10) ) );
     System.out.println("new DynamicFibViaHashTable().getFib( new Long(20) ) - " + new DynamicFibViaHashTable().getFib( new Long(20) ) );
     System.out.println("new DynamicFibViaHashTable().getFib( new Long(50) ) - " + new DynamicFibViaHashTable().getFib( new Long(50) ) );
  long end = System.currentTimeMillis() - start;
  System.out.println("testDynamicFibViaHashTable took: " + end + "ms");
    }

    /**
     * 
     * calculate Fibonacci number through memorize (Hashtable)
     * 
     * Dynamic memorize with Hashtable, cool, less calculation, higher memory requirement.     
     * 
     * @author lei zou
     *
     */
 static public class DynamicFibViaHashTable {
  private Hashtable<Long,Long> memorize = new Hashtable<Long,Long>() {
   {
    put(new Long(0),new Long(0));
    put(new Long(1),new Long(1));
   }
  };

     Long getFib(Long number) {

         if (memorize.containsKey(number))  return memorize.get(number); // Return it.
         
         Long val1 = (memorize.containsKey(number-1)) ? memorize.get(number-1) : getFib( number-1);
         Long val2 = (memorize.containsKey(number-2)) ? memorize.get(number-2) : getFib( number-2);

         // The number hasn't been calculated, so let's calculate it.
         Long val = val1+val2;

         // Store the value for later use.
         memorize.put(number, val);
         return val;
     }
 }




4. Dynamic memorize with HashMap - cool, less calculation, higher memory requirement, faster than Hashtable. This is because that Hashtable is synchronized.
 /**
  * calculate Fibonacci number through memorize (HashMap)
  * 
  */
 static void testDynamicFibViaHashMap () {
  long start = System.currentTimeMillis();
     System.out.println("new DynamicFibViaHashMap().getFib( 10 ) - " + new DynamicFibViaHashMap().getFib( 10 ) );
     System.out.println("new DynamicFibViaHashMap().getFib( 20 ) - " + new DynamicFibViaHashMap().getFib( 20 ) );
     System.out.println("new DynamicFibViaHashMap().getFib( 50 ) - " + new DynamicFibViaHashMap().getFib( 50 ) );
  long end = System.currentTimeMillis() - start;
  System.out.println("testDynamicFibViaHashMap took: " + end + "ms");
    }
 
 static public class DynamicFibViaHashMap {

  private Map<Integer, BigInteger> memoize = new HashMap<Integer, BigInteger>();

  public DynamicFibViaHashMap() {
    memoize.put(0, BigInteger.ZERO);
    memoize.put(1, BigInteger.ONE);
  }

  public BigInteger getFib(final int index) {

   if (!memoize.containsKey(index)) {
    BigInteger val1 = memoize.containsKey(index-1) ? memoize.get(index-1) : getFib(index - 1);
    BigInteger val2 = memoize.containsKey(index-2) ? memoize.get(index-2) : getFib(index - 2);
    memoize.put(index, val1.add(val2));
   }

     return memoize.get(index);
    }
 }


5. Dynamic memorize with ConcurrentHashMap in multi-threaded - less calculation, not necessary to speed up the calculation because thread creation is expensive.

 static void testMultiThreadDynamicFib () {
  long start = System.currentTimeMillis();
  MultiThreadDynamicFib proc = new MultiThreadDynamicFib();
     System.out.println("proc.getFib( 10 ) - " + proc.getFib( 10 ) );
     System.out.println("proc.getFib( 20 ) - " + proc.getFib( 20 ) );
     System.out.println("proc.getFib( 50 ) - " + proc.getFib( 50 ) );
     proc.shutDown();
  long end = System.currentTimeMillis() - start;
  System.out.println("testMultiThreadDynamicFib took: " + end + "ms");
    }

 /**
  * calculate Fibonacci number through memorize (ConcurrentHashMap) with multi-threads
  * 
  * @author lei zou
  *
  */
 static public class MultiThreadDynamicFib  {

  private final ExecutorService threads = Executors.newCachedThreadPool();
  
  private ConcurrentHashMap<Integer, BigInteger> memorize = new ConcurrentHashMap<Integer, BigInteger>();

  public void shutDown () {
   threads.shutdown();
  }
  private Callable<BigInteger> createTask (final int index) {
   return new Callable<BigInteger>() {
    @Override
    public BigInteger call() throws Exception {
     return splitFibNumberAtIndex(index);
    }
   };
  }

  public BigInteger getFib(final int index) {
   if (memorize.containsKey(index)) return memorize.get(index);
   
      FutureTask<BigInteger> task = new FutureTask<BigInteger>(createTask(index));
      threads.submit(task);
      BigInteger val =  null;
   while ( val==null && ! task.isDone() ) {
       try {
        val = task.get();
    } catch (InterruptedException e) {
     e.printStackTrace();
    } catch (ExecutionException e) {
     e.printStackTrace();
    } 
      }

   if (val!=null) {
    memorize.putIfAbsent(index, val);
   }
      return val;
     }
  
  public MultiThreadDynamicFib() {
   memorize.put(0, BigInteger.ZERO);
   memorize.put(1, BigInteger.ONE);
  }

  
  public BigInteger splitFibNumberAtIndex(final int index) {

   if (!memorize.containsKey(index)) {
    BigInteger val1 = memorize.containsKey(index-1) ?memorize.get(index-1) :  getFib(index - 1);
    BigInteger val2 = memorize.containsKey(index-2) ?memorize.get(index-2) :  getFib(index - 2);
    memorize.putIfAbsent(index, val1.add(val2));
   }
     return memorize.get(index);
    }
 }
 




Here is the main method:

    public static void main(String [] args) {

     testFibViaDynamic ();
     testFibViaRecursion ();
     testDynamicFibViaHashTable() ;
        testDynamicFibViaHashMap ();     
        testMultiThreadDynamicFib ();
    }

getFibViaDynamic(10) - 55
getFibViaDynamic(20) - 6765
getFibViaDynamic(50) - 12586269025
testFibViaDynamic took: 1ms

getFibViaRecursion(10) - 55
getFibViaRecursion(20) - 6765
getFibViaRecursion(40) - 102334155
FibViaRecursion took: 1086ms

new DynamicFibViaHashTable().getFib( new Long(10) ) - 55
new DynamicFibViaHashTable().getFib( new Long(20) ) - 6765
new DynamicFibViaHashTable().getFib( new Long(50) ) - 12586269025
testDynamicFibViaHashTable took: 6ms

new DynamicFibViaHashMap().getFib( 10 ) - 55
new DynamicFibViaHashMap().getFib( 20 ) - 6765
new DynamicFibViaHashMap().getFib( 50 ) - 12586269025
testDynamicFibViaHashMap took: 3ms

proc.getFib( 10 ) - 55
proc.getFib( 20 ) - 6765
proc.getFib( 50 ) - 12586269025
testMultiThreadDynamicFib took: 71ms

Enjoy the journey.

Thursday, April 19, 2012

Monopoly Game OOD and Implementation Framework with Java


I played speedy chess for fun a while back; to be more precise, two previous jobs ago. That was fun. Never played Monopoly in my life. OOD for the Monopoly is an interesting problem. The game itself is very loosely defined. Players can define their own rules?!

I am here to make an attempt to crack OOD of the game. 

Inheritance modeling is not so typical. To begin with, we have House, Hotel, FreePackingLot and Chance as the board slots. It's a bit stretch to use simplistic "is-a" to define all relationship. It makes sense to me to use Java's interface as the base (interface Square), House, Hotel, FreePackingLot and Chance all implement Square interface. 

Dice is an excellent candidate for a immutable object. Enum is perfect fit for dice face implementation. 

Command pattern can be utilized to capture players' activities, ie buy property, pay rent, make a move after roll a dice. The RequestFactory is used to generate Action based on game rules and/or player's playing strategy. A new Action could be generated (triggered) right after the completion of the first one (a player landed on a Square and decide to buy it). 

The pattern Strategy is a good fit for Card implementation. Each Card has different task or a set of tasks. 

MonopolyBoard consists of Squares. I used the Vistors pattern to update Square after each action is performed. Update is generated from RequestFactory, for example, rent double for Blue colored properties since a Player occupied all Blue colored property. 

The most interesting part of the Framework is the rule implementation. I don't have full grasp of all Monopoly rules. One reasonable thing that we do is to implement each rule as an object, which generates the Action and Update based on the input - player's status, and chain the rule together based on priorities/preferences. Rule can be defined as game rule (task play must perform) or player's playing strategy (buy the first free property landed on wherever available).

GameFacade is a bit over simplified: each player takes turn and play. An Action gets generated and a Update created right after the completion of the Action.

I feel this framework can meet most of requirement, given that  I have ZERO playing experience.

Here is the Framework in Java: 


GameFacade.java
package com.lei.oop.monopolygame;

import java.util.ArrayList;
import java.util.List;


/**
 * 
 * GameFacade for playing monopoly game
 * 
 * @author lei zou
 *
 */
final public class GameFacade {
 
 private static GameFacade self;
 
 private MonopolyBoard board;
 private List<Player> players;
 private List<Card> cards;
 
 private GameFacade() { }
 
 public void reset () {
  String boardFile = System.getProperty("monopoly_game.xml");
  board = new MonopolyBoard(boardFile);
  players = loadPlayers ("monopoly_player_list.xml");
  cards = loadCards ("monopoly_game_cards.xml");
 }
 
 public void play () {
  boolean noWinner=true;
  int playerIdx = 0;
  while (noWinner) {
   Player currentPlayer = players.get( (playerIdx++) % players.size() );
   Dice dice = Dice.rollDice();
   currentPlayer.addDice(dice);
   ActionPerform action = RequestFactory.generationNextAction(currentPlayer);
   while (action!=null) {
    SquareUpdate update = RequestFactory.generationNextUpdate (action);
    this.board.update(update);
    action = RequestFactory.generationNextAction(currentPlayer);
   }
   // check winner 
  }
  
 }

 static public GameFacade getInstance() {
  if (self==null) self = new GameFacade();
  return self;
 }
 
 private List<Player> loadPlayers (String fileName) {
  List<Player> list = new ArrayList<Player> ();
  return list;
 }

 private List<Card> loadCards (String cardFileName) {
  List<Card> list = new ArrayList<Card> ();
  list.add(new Card("card_one", new Card.CardActionOne() ));
  list.add(new Card("card_two", new Card.CardActionTwo() ));
  // ... 
  return list;
  
 }

 public MonopolyBoard getBoard() {
  return board;
 }

 public void setBoard(MonopolyBoard board) {
  this.board = board;
 }

}




Action.java
package com.lei.oop.monopolygame;

/**
 * Action class for the Monopoly game
 * 
 * Command design pattern is utilized to encapsulate the request 
 * 
 * @author lei zou
 *
 */
public class Action implements ActionPerform {
 private Player[] actors; 
 private Square currentPosition;
 private Square newPosition;
 private ActionUpdate callback;
 private boolean status; 

 // Purchase property 
 static public class PurchaseProperty extends Action implements ActionPerform {
  public PurchaseProperty (Square square, Player[] players, ActionUpdate callback) {
   super(square, players, callback);
  }
  public void execute () {
   // buy the property
   update ();
  }

 }

 // Pay rent 
 static public class PayRent extends Action implements ActionPerform {
  public PayRent (Square square, Player[] players, ActionUpdate callback) {
   super(square, players, callback);
  }
  public void execute () {
   // pay rent
   update ();
  }

 }

 // Pay tax 
 static public class PayTax extends Action implements ActionPerform {
  public PayTax (Square square, Player[] players, ActionUpdate callback) {
   super(square, players, callback);
  }
  public void execute () {
   // pay tax
   update ();
  }

 }

 // Take a card  
 static public class DrawACard extends Action implements ActionPerform {
  public DrawACard (Square square, Player[] players, ActionUpdate callback) {
   super(square, players, callback);
  }
  public void execute () {
   // buy the property
   update ();
  }

 }


 // Make a move 
 static public class MakeAMove extends Action implements ActionPerform {
  private int stepsForward;
  public MakeAMove (Square square, Player player, int steps) {
   super(square, new Player[] { player}, player );
   stepsForward = steps;
  }

  public void execute () {
   // move to the next square 
   this.setNewPosition ( GameFacade.getInstance().getBoard().nextSquare(this) );
   update ();
  }
 }

 
 // Action base class 
 public Action (Square square, Player[] players, ActionUpdate callback) {
  this.currentPosition = square;
  this.actors = players;
  this.callback = callback;
 }
 
 public void execute () { }
 
 public void update () { 
  if (callback!=null)  callback.update(this);
 }

 public Square getCurrentPosition() { return currentPosition; }

 public void setCurrentPosition(Square currentPosition) { this.currentPosition = currentPosition; }

 public Square getNewPosition() { return newPosition; }

 public void setNewPosition(Square newPosition) { this.newPosition = newPosition;}
 
 
}




ActionPerform.java
package com.lei.oop.monopolygame;

/**
 * 
 * ActionPerform Interface 
 * 
 * @author lei zou
 *
 */
public interface ActionPerform {

 void execute ();
 
}



ActionUpdate.java
package com.lei.oop.monopolygame;

/**
 * 
 * ActionUpdate Interface 
 * 
 * @author lei zou
 *
 */

public interface ActionUpdate {
 void update(Action command);
}


Asset.java
package com.lei.oop.monopolygame;

import com.lei.oop.monopolygame.ColorGroup.COLOR;

/**
 * Asset class, which has value, player can purchase it or need to pay rent 
 *  
 * @author lei zou
 *
 */

public abstract class Asset implements Square {
 private String name;
 private int purchasePrice;
 private int rentPrice;
 final private int position;
 final private COLOR color;
 private Participant owner;
 
 public Asset (int position) {
  this(position, null); 
 }
 
 public Asset (int position, COLOR color) {
  this.position = position; 
  this.color = color;
 }

 public String getName() { return name; }
 public void setName(String name) { this.name = name; }
 public int getRentPrice() { return rentPrice; }
 public void setRentPrice(int rentPrice) { this.rentPrice = rentPrice; }
 public int getPurchasePrice() { return purchasePrice; }
 public void setPurchasePrice(int purchasePrice) { this.purchasePrice = purchasePrice; }
 public int getPosition() { return position; }
 public COLOR getColor() {return color;}
 
}




Banker.java
package com.lei.oop.monopolygame;

/**
 * 
 * Banker class 
 * 
 * @author lei zou
 *
 */
public class Banker extends  Participant implements  ActionUpdate {
 
 private int balance;
 
 
 public Banker (String name) {
  super(name);
 }
 
 public void update(Action command) {
  // update the Banker after the command action is performed.  
 }
 
}


Card.java
package com.lei.oop.monopolygame;

/**
 * 
 * Card class for monopoly game
 * 
 * Strategy pattern is utilized to implement the instructions from the Card
 * 
 * @author lei zou
 *
 */
public class Card implements CardAction {
 
 static public class CardActionOne implements CardAction {
  public CardActionOne() {};
  public void followInstruction () {
   System.out.println("do CardActionOne");
  }
 }
 static public class CardActionTwo implements CardAction {
  public CardActionTwo() {};
  public void followInstruction () {
   System.out.println("do CardActionTwo");
  }
 }

 private String name;
 private CardAction action;

 
 public Card(String name, CardAction action) {
  super();
  this.name = name;
  this.action = action;
 }
 
 public void move() {
  action.followInstruction();
 }
 public String getName() {
  return name;
 }
 public void setName(String name) {
  this.name = name;
 }
 public CardAction getAction() {
  return action;
 }
 public void setAction(CardAction action) {
  this.action = action;
 }
 
 public void followInstruction () {
  action.followInstruction();
 }
 
}


CardAction.java
package com.lei.oop.monopolygame;

/**
 * Card Action interface 
 * 
 * @author lei zou
 *
 */
public interface CardAction {
 
 void followInstruction () ;

}



Chance.java
package com.lei.oop.monopolygame;

import com.lei.oop.monopolygame.ColorGroup.COLOR;

/**
 *  Chance class, it implements Square  
 * 
 * @author lei zou
 *
 */
public class Chance implements Square {

 final private int position; 
 final private COLOR color = null;
 
 
 public Chance(int position) {
  super();
  this.position = position;
 }

 public void playerLanded  (Player p) {
  // ... 
 }

 public int getPosition() {
  return position;
 }

 public ColorGroup.COLOR getColor() {
  return color;
 }

 public void update (SquareUpdate rule) {
  rule.update(this);
 }

}

ColorGroup.java
package com.lei.oop.monopolygame;

import java.util.HashMap;
import java.util.Map;


/**
 * 
 * ColorGroup class 
 * 
 * @author lei zou
 *
 */
final public class ColorGroup {

 static public enum COLOR { BLUE(1), YELLOW(2), RED(3), GREEN(4), BLACK(5), WHITE(6);
  private final int value; 
  COLOR(int val) { this.value = val; } 
  public int getValue() { return value; }
 }

 private static final Map<Integer, COLOR> IntToColorMap = new HashMap<Integer, COLOR>();
 
 static {
  for (COLOR type : COLOR.values()) IntToColorMap.put(type.value, type);
 }
  
 public static COLOR getColor (int idx) {
  return IntToColorMap.get(new Integer(idx));
 }
}

Dice.java
vp2c01b-dhcp68:LeiJava lei$ sed 's/\&/\&/g;  s/\"/\"/g; s//\>/g;' ./src/main/java/com/lei/oop/monopolygame/Dice.java   | sed "s/\'/\'/g;" 
package com.lei.oop.monopolygame;


import java.util.HashMap;
import java.util.Map;

/**
 * 
 * Dice class, roll a dice and return a dice object 
 * 
 * @author lei zou
 *
 */
final public class Dice {

 static public enum DiceFace { ONE(1), TWO(2), THREE(3), FOUR(4), FIVE(5), SIX(6);
  static public int FaceDifference = SIX.value - ONE.value;
  private final int value; 
  DiceFace(int val) { this.value = val; } 
  public int getValue() { return value; }
 }
 private final DiceFace faceValueOne;
 private final DiceFace faceValueTwo;

 public Dice (final DiceFace faceValue1, final DiceFace faceValue2) {
  faceValueOne = faceValue1;
  faceValueTwo = faceValue2;
 }
 
 private static final Map<Integer, DiceFace> IntToDiceFaceMap = new HashMap<Integer, DiceFace>();
  static {
   for (DiceFace type : DiceFace.values()) {
    IntToDiceFaceMap.put(type.value, type);
     }
 }


 final public DiceFace getFaceValueOne () { return faceValueOne;}
 final public DiceFace getFaceValueTwo () { return faceValueTwo;}
 
 
 @Override
 public String toString() {
  StringBuilder builder = new StringBuilder();
  builder.append("Dice [faceValueOne=").append(faceValueOne)
    .append(", faceValueTwo=").append(faceValueTwo).append("]");
  return builder.toString();
 }
 static public Dice rollDice () {
  int valOne = (int) ( (double)DiceFace.ONE.value + Math.random() * DiceFace.FaceDifference ) ;
  int valTwo = (int) ( (double)DiceFace.ONE.value + Math.random() * DiceFace.FaceDifference ) ;
  return new Dice( IntToDiceFaceMap.get(valOne), IntToDiceFaceMap.get(valTwo));
 }
 
 public static void main(String[] args) throws Exception {
  Dice d = Dice.rollDice();
  System.out.println(d);
 }
}
vp2c01b-dhcp68:LeiJava lei$ 


FreePackingLot.java
package com.lei.oop.monopolygame;

import com.lei.oop.monopolygame.ColorGroup.COLOR;

/**
 * FreePackingLot class, implements Square
 * 
 * @author lei zou
 *
 */
public class FreePackingLot implements Square {

 final private int position;
 
 public FreePackingLot (int pos) {
  this.position = pos;
 }
 
 public void playerLanded (Player p) { }
 
 public int getPosition() {
  return position;
 }
 
 public COLOR getColor () {
  return null;
 }
 
 public void update (SquareUpdate rule) {
  
 }

}


GameRule.java
package com.lei.oop.monopolygame;

/**
 * 
 * GameRule class to implement the monopoly game rules. 
 * 
 * @author lei zou
 *
 */
public class GameRule extends RequestGeneration {
 
 public GameRule (RequestGeneration successor) {
  super(successor);
 }

 public ActionPerform generationNextAction (Participant p) {
  
  ActionPerform action = null;
  
  // 
  return action;
  
  
 }
}


GameStrategy.java
package com.lei.oop.monopolygame;

import java.util.ArrayList;
import java.util.List;

/**
 * GameStrategy to implement the monopoly game playing strategy 
 * 
 * @author lei zou
 *
 */
public class GameStrategy extends RequestGeneration {

 static private List<GameStrategy> listGameStrategy = new ArrayList<GameStrategy>(); 
 
 public GameStrategy (RequestGeneration successor) {
  super(successor);
 }

 public ActionPerform generationNextAction (Participant p) {
  
  ActionPerform action = null;
  
  // if multiple  
  return action;
  
  
 }
}


MonopolyBoard.java
package com.lei.oop.monopolygame;

import java.util.ArrayList;
import java.util.HashMap;
import java.util.List;
import java.util.Map;

import com.lei.oop.monopolygame.ColorGroup.COLOR;



/**
 * 
 * MonopolyBoard class, which is for the monopoly game board.   
 * 
 * @author lei zou
 *
 */
final public class MonopolyBoard {
 
 final private List<Square> squares;
 final private Map<COLOR, Square> ColoredPropertyMap;
   
 public MonopolyBoard(String fileName) {
  
   // load the board square from system property
  squares = new ArrayList<Square>();
  // load the colored properties   
  ColoredPropertyMap = new HashMap<COLOR, Square>();
 }

   
 public Square nextSquare (Action move) {
  Square next = null;
  Action.MakeAMove nextMove = (Action.MakeAMove) move;
  if (nextMove!=null) {
   
  }
  return next;
 }
 
 public void update (SquareUpdate update) {
  for (Square square : squares) {
   square.update(update);
  }
 }
}



Participant.java
package com.lei.oop.monopolygame;

/**
 * Participant abstract class 
 * 
 * @author lei zou
 *
 */
public abstract class Participant {
 private String name;
 public Participant (String name) {
  this.name = name;
 }
 public String getName() {return name;}
 public void setName(String name) {this.name = name;}

}


Player.java
package com.lei.oop.monopolygame;


import java.util.ArrayList;
import java.util.List;

/**
 * Player class 
 *  
 * @author lei zou
 *
 */
public class Player extends Participant implements  ActionUpdate {
 
 private int money;
 private Square currentSquare;
 private List<Square> properties = new ArrayList<Square>();
 private List<Dice> rolledDices = new ArrayList<Dice>();
 private List<Card> cards = new ArrayList<Card>();

 public Player (String name) { super(name);}
 
 void addDice (Dice dice) {
  rolledDices.add(dice);
 }

 public boolean pay (int amount) {
  this.money -= amount;
  return (money>0)? true:false;  
 }
 public void collect (int amount) {
  this.money += amount;
 }

 public int getMoney() {
  return money;
 }

 public void setMoney(int money) {
  this.money = money;
 }

 public Square getCurrentSquare() {
  return currentSquare;
 }

 public void setCurrentSquare(Square currentSquare) {
  this.currentSquare = currentSquare;
 }
 
 public void update(Action command) {
  // callback to update after the action is perform
 }

}



Property.java
package com.lei.oop.monopolygame;

import com.lei.oop.monopolygame.ColorGroup.COLOR;


/**
 * 
 * Property: rent, purchase price
 * 
 * @author lei zou
 *
 */
public class Property extends Asset implements Square {
 private int rent;

 static public class Hotel extends Property implements Square {
  public Hotel (int position) {
   super(position);
  }
 }

 public Property (int position) {
  super(position);
 }
 
 public Property (int position, COLOR color) {
  super(position, color);
 }
 public void playerLanded (Player p) {
  // ... 
 }
 
 public int getRent() { return rent; }
 
 public void setRent(int rent) { this.rent = rent; }

 public void update (SquareUpdate rule) {
  rule.update(this);
 }

}


RequestFactory.java
package com.lei.oop.monopolygame;

import java.util.HashMap;
import java.util.Map;


/**
 * RequestFactory to generate Actions and Updates
 * 
 * @author lei zou
 *
 */

public class RequestFactory {

 static abstract class AbstractFactory {
  abstract ActionPerform createAction(Participant p);
  abstract SquareUpdate createUpdate(ActionPerform a);
 }
 
 static private class ConcreteFactory extends AbstractFactory {
  static private RequestGeneration requestGenerator = loadRequestGenerator();
  static private UpdateGeneration updateGenerator = loadUpdateGenerator();
  ActionPerform createAction(Participant p) {
   RequestGeneration proc = requestGenerator;
   ActionPerform action = proc.generationNextAction(p);
   proc = proc.next();
   while (action==null && proc!=null) {
    // create a new request action 
    action = proc.generationNextAction(p);
   }
   return action; 
  }
  
  SquareUpdate createUpdate(ActionPerform a) {
   
   UpdateGeneration proc = updateGenerator;
   SquareUpdate update = proc.generateNextUpdate(a);
   proc = proc.next();
   while (update==null && proc!=null) {
    // create new update request to update the board 
    update = proc.generateNextUpdate(a);
   }
   
   return update;
  }
  static private RequestGeneration loadRequestGenerator() {
   RequestGeneration first = null;
   // 
   return first;
  }
  
  static private UpdateGeneration loadUpdateGenerator() {
   UpdateGeneration first = null;
   // 
   return first;
   
  }
 }
 
 private static final Map<String, AbstractFactory> FactoryMap = new HashMap<String, AbstractFactory>();
 static {
  FactoryMap.put("Request", new ConcreteFactory());
 }
 
 private static AbstractFactory pf=FactoryMap.get("Request");
 
 static public ActionPerform generationNextAction (Participant p) {
  return pf.createAction(p);
 }
 
 static public SquareUpdate generationNextUpdate (ActionPerform a) {
  return pf.createUpdate(a);
 }

}





RequestGeneration.java
package com.lei.oop.monopolygame;

/**
 * Rule chain to generate action request 
 * 
 * @author lei zou
 *
 */
public abstract class RequestGeneration {
 
 private RequestGeneration successor;
 
 public RequestGeneration (RequestGeneration successor) {
  this.successor = successor;
 }
 public RequestGeneration next () { return successor; }
 public abstract ActionPerform generationNextAction (Participant p);
 public RequestGeneration getSuccessor() { return successor; }
 public void setSuccessor(RequestGeneration successor) { this.successor = successor; }
 
}


Square.java
package com.lei.oop.monopolygame;

import com.lei.oop.monopolygame.ColorGroup.COLOR;


/**
 * Square interface, it represent the slot on the board. 
 * 
 * @author lei zou
 *
 */
public interface Square {

 public void playerLanded (Player p);
 public int getPosition();
 public COLOR getColor ();
 public void update (SquareUpdate rule);
}



SquareUpdate.java
package com.lei.oop.monopolygame;

/**
 * interface SquareUpdate to update the board squares
 * 
 * Vistors pattern is used here to update Square for example double the rent, update the owner 
 * 
 * @author lei zou
 *
 */
public interface SquareUpdate {
 
 void update (Property property);
 void update (Asset asset);
 void update (Square square);
}


SquareUpdateDoubleRent.java
package com.lei.oop.monopolygame;

import com.lei.oop.monopolygame.ColorGroup.COLOR;

/**
 * 
 * double the rent for the asset square 
 * 
 * @author lei zou
 *
 */
public class SquareUpdateDoubleRent implements SquareUpdate {

 final private COLOR color;
 public SquareUpdateDoubleRent (COLOR color) {
  this.color = color;
 }
 
 public void update (Property property) {
  if (property.getColor()!=null && this.color!=null && property.getColor()==this.color)
   property.setRent(property.getRent()*2);
 }

 public void update (Asset asset) {
 }
 
 public void update (Square square) {
 }

}

UpdateGeneration.java
package com.lei.oop.monopolygame;

/**
 *  Rule chain for generate updates 
 * 
 * @author lei zou
 *
 */

public abstract class UpdateGeneration {

 private UpdateGeneration successor;
 
 public abstract UpdateGeneration next ();
 public abstract SquareUpdate generateNextUpdate(ActionPerform a);
 public UpdateGeneration getSuccessor() { return successor; }
 public void setSuccessor(UpdateGeneration successor) { this.successor = successor; }
 

}



Well, that is it. Enjoy the Journey!

Tuesday, April 10, 2012

Simple Thread-Safe Bounded Queue Implementation in Java

I was asked to implement a thread-safe bounded queue for a pattern of producer and customer. The implementation involves ReentrantLock and Condition; it's straight forward. The testing of the queue requires a bit more work. For verification,  I created create a thread pool of size 2. One for consumer, one for producer.

Here is the BoundedQueue code:


public class BoundedQueue <T> {

    private final T[] buffer;
    private final int capacity;
 
    private int front;
    private int rear;
    private int count;
 
    private final Lock lock = new ReentrantLock();
 
    private final Condition notFull = lock.newCondition();
    private final Condition notEmpty = lock.newCondition();

    /**
     * 
     * @param capacity is the fixed queue capacity 
     */
    public BoundedQueue (int capacity) {
        super();
        this.capacity = capacity;
        buffer = ((T[])new Object[capacity]);
    }

    /**
     * 
     * @return the object in the front of the queue 
     * 
     * @throws InterruptedException
     */
    public T get() throws InterruptedException {
        lock.lock();
 
        try {
            while (count == 0) {
                notEmpty.await();
            }
 
            T result = buffer[front];
            front = (front + 1) % capacity;
            count--;
            notFull.signal();
            return result;
        } finally {
            lock.unlock();
        }
    }

    /**
     * return object of the subclass 
     * 
     * @param <R>
     * @param type
     * @return
     * @throws InterruptedException
     */
    public <R extends T> R get(final Class<R> type) throws InterruptedException {
     return (R)this.get();
    }

    /**
     * put the object in the rear of the queue
     * @param data
     * @throws InterruptedException
     */
    public void put(final T data) throws InterruptedException {
        lock.lock();
        try {
            while (count == capacity) {
                notFull.await();
            }
 
            buffer[rear] = data;
            rear = (rear + 1) % capacity;
            count++;
 
            notEmpty.signal();
        } finally {
            lock.unlock();
        }
    }
 
}

Here is the JUnit test code for BoundedQueue:
public class BoundedQueueTest {
 static public class QueueConsumer implements Runnable, Callable<Long> {
  private BoundedQueue<Integer> queue;
  private long sleepTime;
  private int itemCount;
  private List items = new LinkedList<Integer>(); 
  public QueueConsumer (BoundedQueue<Integer> que, int itemCount, long sleepTime) {
   this.queue = que;
   this.itemCount = itemCount;
   this.sleepTime = sleepTime;
  }
 
  @Override
  public Long call() throws Exception {
   long sum = 0;
   try {
    for ( int i=0; i<itemCount; i++) {
     Integer obj = this.queue.get();
     Integer compareToObj = new Integer(i);
     items.add(obj);
     Assert.assertEquals(obj, compareToObj);
     //System.out.println("item - " + obj);
     sum++;
     Thread.sleep(sleepTime);
    }
   } catch (InterruptedException e) {
    e.printStackTrace();
   }

   return sum;
  }

  
  public void run () {
   try {
    long sum = this.call();
   } catch (Exception e) {
    e.printStackTrace();
   }
  }
 }

 static public class QueueProducer implements Runnable, Callable<Long> {
  private BoundedQueue<Integer> queue;
  private int itemCount;
  private long sleepTime;
  private Integer[] items; 
  public QueueProducer (BoundedQueue<Integer> que, int itemCount, long sleepTime) {
   this.queue = que;
   this.itemCount = itemCount;
   this.sleepTime = sleepTime;
   this.items = new Integer[itemCount];
   for ( int i=0; i<itemCount; i++) {
    this.items[i] = new Integer(i);
   }
  }

  @Override
  public Long call() throws Exception {
   long sum = 0;
   try {
    for ( int i=0; i<itemCount; i++) {
     this.queue.put(this.items[i]);
     sum++;
     Thread.sleep(sleepTime);
    }
   } catch (InterruptedException e) {
    e.printStackTrace();
   }
   return sum;
  }
  public void run () {
   try {
    call();
   } catch (Exception e) {
    e.printStackTrace();
   }
  }
 }


 @Test
 public void doFastConsumerBoundedQueueTest() {

  try {
   test(2, 100, 1000, 10, 8);
  } catch (InterruptedException e) {
   e.printStackTrace();
  } catch (ExecutionException e) {
   e.printStackTrace();
  }
 }

 @Test
 public void doFastProducerBoundedQueueTest() {

  try {
   test(2, 100, 1000, 10, 15);
  } catch (InterruptedException e) {
   e.printStackTrace();
  } catch (ExecutionException e) {
   e.printStackTrace();
  }
 }

 
 private void test(final int threadCount, int queueSize, int itemCound, long produceSleepTime, long consumerSleepTime) throws InterruptedException, ExecutionException {
  BoundedQueue<Integer> queue = new BoundedQueue<Integer>(queueSize);
  Callable<Long> task1 = new QueueProducer(queue, itemCound, produceSleepTime);
  Callable<Long> task2 = new QueueConsumer(queue, itemCound, consumerSleepTime);
  
  List<Callable<Long>> tasks = new ArrayList<Callable<Long>>(); 
  tasks.add(task1);
  tasks.add(task2);
  
  ExecutorService executorService = Executors.newFixedThreadPool(2);
  
  List<Future<Long>> futures = executorService.invokeAll(tasks);
  List<Long> resultList = new ArrayList<Long>(futures.size());

  for (Future<Long> future : futures) {
   resultList.add(future.get());
  }
  
  executorService.shutdown();
 }
 
 public static void main(String args[]) {
  new BoundedQueueTest().doFastProducerBoundedQueueTest();
  
 }

}

Enjoy the journey.

Thursday, March 29, 2012

Improvement over BigTop's TestHadoopExamples.groovy with YAML and TestRunHadoopExamples.groovy

Thanks for feedbacks from Wing Yew and Roman, I was set out to improve

./bigtop-tests/test-artifacts/hadoop/src/main/groovy/org/apache/bigtop/itest/hadoopexamples/TestHadoopExamples.groovy

using YAML, Groovy and existing BigTop's itest infrastructure. YAML is better than XML to capture the list of Shell commands.

I have made the improvement on the following two areas:

1. Move the commands inside TestHadoopExamples.groovy to YAML file, so that we need NOT compile the code if add/change the test cases.

2. Introduce the comparator to compare output with the expected to verify the commands' execution. For example, when calculating Pi, make sure the end result matched with the known number.

The code has been tested on ubuntu 10.04 at AWS, and jira submitted.


Here are the test cases in TestHadoopExamples.groovy:
 
  static Map examples =
    [
        pi                :'20 10',
        wordcount         :"$EXAMPLES/text $EXAMPLES_OUT/wordcount",
        multifilewc       :"$EXAMPLES/text $EXAMPLES_OUT/multifilewc",
//        aggregatewordcount:"$EXAMPLES/text $EXAMPLES_OUT/aggregatewordcount 5 textinputformat",
//        aggregatewordhist :"$EXAMPLES/text $EXAMPLES_OUT/aggregatewordhist 5 textinputformat",
        grep              :"$EXAMPLES/text $EXAMPLES_OUT/grep '[Cc]uriouser'",
        sleep             :"-m 10 -r 10",
        secondarysort     :"$EXAMPLES/ints $EXAMPLES_OUT/secondarysort",
        randomtextwriter  :"-Dtest.randomtextwrite.total_bytes=1073741824 $EXAMPLES_OUT/randomtextwriter"
    ];


Here is the YAML content that has all test cases covered.
 

- !!org.apache.bigtop.itest.hadoopexamples.BigTopIntegrationTest
  integrationTestCommandList:
  - !!org.apache.bigtop.itest.hadoopexamples.BigTopTestCommand {command: hadoop jar $HADOOP_HOME/hadoop-examples-*.jar pi 5 5,
    commandComparator: echo "Pi is 3.68", comparatorClass: org.apache.hadoop.cli.util.SubstringComparator}
  postTestCommandList: []
  preTestCommandList: []
  testDesc: calculate pi using hadoop MR
  testName: calculate pi
- !!org.apache.bigtop.itest.hadoopexamples.BigTopIntegrationTest
  integrationTestCommandList:
  - !!org.apache.bigtop.itest.hadoopexamples.BigTopTestCommand {command: hadoop jar $HADOOP_HOME/hadoop-examples-*.jar wordcount /wordcount /wordcount_out,
    commandComparator: null, comparatorClass: null}
  - !!org.apache.bigtop.itest.hadoopexamples.BigTopTestCommand {command: mkdir ./wordcount_out,
    commandComparator: null, comparatorClass: null}
  - !!org.apache.bigtop.itest.hadoopexamples.BigTopTestCommand {command: hadoop fs -get /wordcount_out/* ./wordcount_out,
    commandComparator: null, comparatorClass: null}
  - !!org.apache.bigtop.itest.hadoopexamples.BigTopTestCommand {command: hadoop fs -rmr /wordcount,
    commandComparator: null, comparatorClass: null}
  - !!org.apache.bigtop.itest.hadoopexamples.BigTopTestCommand {command: hadoop fs -rmr /wordcount_out/,
    commandComparator: null, comparatorClass: null}
  postTestCommandList:
  - !!org.apache.bigtop.itest.hadoopexamples.BigTopTestCommand {command: 'cat ./wordcount_out/*
      | grep  Roman | sed ''s/[^0-9.]*\([0-9.]*\).*/\1/''', commandComparator: cat wordcount/* | grep -c Roman,
    comparatorClass: org.apache.bigtop.itest.hadoopexamples.ExtactComparatorIgnoreWhiteSpace}
  preTestCommandList:
  - !!org.apache.bigtop.itest.hadoopexamples.BigTopTestCommand {command: rm -rf ./wordcount,
    commandComparator: null, comparatorClass: null}
  - !!org.apache.bigtop.itest.hadoopexamples.BigTopTestCommand {command: rm -rf ./wordcount_out,
    commandComparator: null, comparatorClass: null}
  - !!org.apache.bigtop.itest.hadoopexamples.BigTopTestCommand {command: mkdir ./wordcount,
    commandComparator: null, comparatorClass: null}
  - !!org.apache.bigtop.itest.hadoopexamples.BigTopTestCommand {command: 'curl http://www.meetup.com/HandsOnProgrammingEvents/events/53837022/
      | sed -e :a -e ''s/<[^>]*>//g;/</N;//ba'' | sed ''s/&nbsp//g'' | sed ''s/^[
      \t]*//;s/[ \t]*$//''  | sed ''/^$/d'' | sed ''/"http[^"]*"/d'' > ./wordcount/content',
    commandComparator: null, comparatorClass: null}
  - !!org.apache.bigtop.itest.hadoopexamples.BigTopTestCommand {command: hadoop fs -mkdir /wordcount,
    commandComparator: null, comparatorClass: null}
  - !!org.apache.bigtop.itest.hadoopexamples.BigTopTestCommand {command: hadoop fs -put ./wordcount/* /wordcount,
    commandComparator: null, comparatorClass: null}
  testDesc: count word in Hadoop MR
  testName: count word in MR
- !!org.apache.bigtop.itest.hadoopexamples.BigTopIntegrationTest
  integrationTestCommandList:
  - !!org.apache.bigtop.itest.hadoopexamples.BigTopTestCommand {command: hadoop fs -rmr examples-output/wordcount,
    commandComparator: null, comparatorClass: null}
  - !!org.apache.bigtop.itest.hadoopexamples.BigTopTestCommand {command: hadoop jar $HADOOP_HOME/hadoop-examples-*.jar wordcount examples/text examples-output/wordcount,
    commandComparator: null, comparatorClass: null}
  postTestCommandList:
  - !!org.apache.bigtop.itest.hadoopexamples.BigTopTestCommand {command: 'hadoop
      fs -cat  examples-output/wordcount/part* | grep "Commission" | sed ''s/[^0-9]*\([0-9]\+\).*/\1/''
      | tr ''\n'' '' '' | sed "s/\(^.*$\)/\1\n/" | sed ''s/^[[:space:]]*//;s/[[:space:]]*$//''
      | sed -e ''s/ /+/g'' | bc', commandComparator: hadoop fs -cat  examples/text/* | grep -c "Commission",
    comparatorClass: org.apache.bigtop.itest.hadoopexamples.ExtactComparatorIgnoreWhiteSpace}
  preTestCommandList: []
  testDesc: countword in TestHadoopExamples.groovy
  testName: countword in MR
- !!org.apache.bigtop.itest.hadoopexamples.BigTopIntegrationTest
  integrationTestCommandList:
  - !!org.apache.bigtop.itest.hadoopexamples.BigTopTestCommand {command: hadoop fs -rmr examples-output/multifilewc,
    commandComparator: null, comparatorClass: null}
  - !!org.apache.bigtop.itest.hadoopexamples.BigTopTestCommand {command: hadoop jar $HADOOP_HOME/hadoop-examples-*.jar multifilewc examples/text examples-output/multifilewc,
    commandComparator: null, comparatorClass: null}
  postTestCommandList:
  - !!org.apache.bigtop.itest.hadoopexamples.BigTopTestCommand {command: 'hadoop
      fs -cat  examples-output/multifilewc/part* | grep "Ambassadors" | sed ''s/[^0-9]*\([0-9]\+\).*/\1/''
      | tr ''\n'' '' '' | sed "s/\(^.*$\)/\1\n/" | sed ''s/^[[:space:]]*//;s/[[:space:]]*$//''
      | sed -e ''s/ /+/g'' | bc', commandComparator: hadoop fs -cat  examples/text/* | grep -c "Ambassadors",
    comparatorClass: org.apache.bigtop.itest.hadoopexamples.ExtactComparatorIgnoreWhiteSpace}
  preTestCommandList: []
  testDesc: multifilewcin TestHadoopExamples.groovy
  testName: multifilewc test in MR
- !!org.apache.bigtop.itest.hadoopexamples.BigTopIntegrationTest
  integrationTestCommandList:
  - !!org.apache.bigtop.itest.hadoopexamples.BigTopTestCommand {command: hadoop fs -rmr examples-output/grep,
    commandComparator: null, comparatorClass: null}
  - !!org.apache.bigtop.itest.hadoopexamples.BigTopTestCommand {command: 'hadoop
      jar $HADOOP_HOME/hadoop-examples-*.jar grep examples/text examples-output/grep   ''[Cc]uriouser''',
    commandComparator: null, comparatorClass: null}
  postTestCommandList:
  - !!org.apache.bigtop.itest.hadoopexamples.BigTopTestCommand {command: 'hadoop
      fs -cat examples-output/grep/part* | sed ''s/[0-9]*//g'' | sed ''s/Curiouser/curiouser/g''',
    commandComparator: echo "curiousercuriouser", comparatorClass: org.apache.bigtop.itest.hadoopexamples.ExtactComparatorIgnoreWhiteSpace}
  preTestCommandList: []
  testDesc: grep in TestHadoopExamples.groovy
  testName: grep in MR
- !!org.apache.bigtop.itest.hadoopexamples.BigTopIntegrationTest
  integrationTestCommandList:
  - !!org.apache.bigtop.itest.hadoopexamples.BigTopTestCommand {command: hadoop jar $HADOOP_HOME/hadoop-examples-*.jar sleep -m 10 -r 10,
    commandComparator: null, comparatorClass: null}
  postTestCommandList: []
  preTestCommandList: []
  testDesc: sleep in TestHadoopExamples.groovy
  testName: sleep in MR





Here is the content of TestRunHadoopExamples.groovy:
 


/**
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements.  See the NOTICE file
* distributed with this work for additional information
* regarding copyright ownership.  The ASF licenses this file
* to you under the Apache License, Version 2.0 (the
* "License"); you may not use this file except in compliance
* with the License.  You may obtain a copy of the License at
* <p/>
* http://www.apache.org/licenses/LICENSE-2.0
* <p/>
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/

package org.apache.bigtop.itest.hadoopexamples

import java.util.Map;

import org.junit.Ignore
import org.junit.Test
import org.junit.runner.RunWith;
import org.junit.BeforeClass
import org.junit.runners.Parameterized.Parameters;

import static org.junit.Assert.assertTrue
import static org.junit.Assert.assertNotNull


import org.apache.bigtop.itest.junit.OrderedParameterized;
import org.apache.bigtop.itest.shell.Shell;


import org.apache.commons.logging.LogFactory
import org.apache.commons.logging.Log

import org.apache.hadoop.cli.util.ComparatorBase

import org.apache.bigtop.itest.junit.OrderedParameterized
import org.junit.runners.Parameterized.Parameters
import org.junit.runner.RunWith

import org.yaml.snakeyaml.Yaml

import org.apache.hadoop.conf.Configuration
import org.apache.bigtop.itest.JarContent



@RunWith(OrderedParameterized.class)
class TestRunHadoopExamples {
 
 static private Log LOG = LogFactory.getLog(TestRunHadoopExamples.class);
 static private String TEST_CASE_FILE_NAME = "./bigtop-testcases.yaml";
 //static private String TEST_CASE_FILE_NAME = "/home/ubuntu/bigtop/bigtop-tests/test-execution/smokes/hadoop/bigtop-testcases.yaml";
 static private String TEST_CASE_NAME_PREFIX = 'TestRunHadoopExamples';
 
 static private Shell sh = new Shell("/bin/bash -s");

 private static final String HADOOP_HOME = System.getenv('HADOOP_HOME');
 private static final String HADOOP_CONF_DIR = System.getenv('HADOOP_CONF_DIR');
 private static String hadoopExamplesJar = JarContent.getJarName(HADOOP_HOME, 'hadoop.*examples.*.jar');
 static {
  assertNotNull("HADOOP_HOME has to be set to run this test", HADOOP_HOME);
  assertNotNull("HADOOP_CONF_DIR has to be set to run this test", HADOOP_CONF_DIR);
  assertNotNull("Can't find hadoop-examples.jar file", hadoopExamplesJar);
 }
 private static Configuration conf;
 private static String HADOOP_OPTIONS;
 private static final String EXAMPLES = "examples";
 private static final String EXAMPLES_OUT = "examples-output";
  
 
 private String testName;
 private String testCaseString;
 

 private String stripOutLeadingBracket (String str) {
  if (str==null) return str;
  if (str.length()<2) return str;
  if (str.startsWith("[") && str.endsWith("]")) {
   return str.substring(1,str.length()-1)
  } else {
   return str;
  }
 }

 private static List<BigTopIntegrationTest> loadBigTopIntegrationTestCases (String fileName) {
  String fileContents = new File(fileName).text
  List<BigTopIntegrationTest> testCaseList2 = new Yaml().load(fileContents)
 }
 
 @BeforeClass
 static void setUp() {
  String skipSetup = System.getProperty('bigtop.itest.skip.setup');
  if (skipSetup!=null && skipSetup.length()>0)
   return;
  
  LOG.info("Start setUp") 
  conf = new Configuration();
  conf.addResource('mapred-site.xml');
  HADOOP_OPTIONS = "-fs ${conf.get('fs.default.name')} -jt ${conf.get('mapred.job.tracker')}";
  // Unpack resource
  JarContent.unpackJarContainer(TestRunHadoopExamples.class, '.' , null)
  
  sh.exec("hadoop fs $HADOOP_OPTIONS -test -e $EXAMPLES");
  if (sh.getRet() == 0) {
   sh.exec("hadoop fs $HADOOP_OPTIONS -rmr -skipTrash $EXAMPLES");
   assertTrue("Deletion of previous $EXAMPLES from HDFS failed", sh.getRet() == 0);
  }
  sh.exec("hadoop fs $HADOOP_OPTIONS -test -e $EXAMPLES_OUT");
  if (sh.getRet() == 0) {
   sh.exec("hadoop fs $HADOOP_OPTIONS -rmr -skipTrash $EXAMPLES_OUT");
   assertTrue("Deletion of previous examples output from HDFS failed", sh.getRet() == 0);
  }
  
  // copy test files to HDFS
  sh.exec("hadoop fs $HADOOP_OPTIONS -put $EXAMPLES $EXAMPLES",
    "hadoop fs $HADOOP_OPTIONS -mkdir $EXAMPLES_OUT");
   assertTrue("Could not create output directory", sh.getRet() == 0);
 }
  
 
 @Parameters
 public static Map<String, Object[]> generateTests() {
  Map<String, Object[]> res = [:];
  List<BigTopIntegrationTest> testList = loadBigTopIntegrationTestCases (TEST_CASE_FILE_NAME);
  int count=1;
  
  for (BigTopIntegrationTest test : testList) {
   def nowCal = Calendar.instance
   String casename = "$TEST_CASE_NAME_PREFIX-$nowCal.time-$count"
   Object[] args = [ casename, new Yaml().dump(test) ]
   res.put( casename, args)
   count++;
  }
  return res;
 }
  
 public TestRunHadoopExamples (String name, String testDetail ) {
  testName = name;
  testCaseString = testDetail;
  displayMessage (["Test case name - $testName, args - $testCaseString"], false)
 }
 
 private void displayMessage (def message, boolean error) {
  if (message!=null) {
   if (error) 
    message.each() { LOG.error "${it}" };
   else 
    message.each() { LOG.info "${it}" };
  }
 }

 public boolean runExample(BigTopIntegrationTest test) {
  boolean success = true
  
  for ( BigTopTestCommand command: test.getCommandList() ) {
   displayMessage (["Shell command ["  + command.getCommand() + "]"], false);
   sh.exec(command.getCommand());
   String stdout = sh.getOut();
   String shReturnCode = sh.getRet()
   String shStdErr = sh.getErr();
   
   if ( command.getComparatorClass() !=null && command.getCommandComparator()!=null && command.getCommandComparator().trim().length()>0) {
    ["ComparatorClass - " + command.getComparatorClass(), "CommandComparator - " + command.getCommandComparator(), "Shell CommandComparator ["  + command.getCommandComparator() + "]"].each() { LOG.info "${it}" };
    sh.exec(command.getCommandComparator());
    String expectedOutput = sh.getOut();
    displayMessage (["CommandComparator return code is $shReturnCode, Output is $expectedOutput"], false);

    String comparatorClassName = command.getComparatorClass();
    ComparatorBase compare = BigTopIntegrationTestFacade.getComparatorClass(comparatorClassName);
    def resultDisplay = []
    if (compare==null) {
     resultDisplay.add("Error! No such ComparatorClass - $comparatorClassName");
     success = false;
    } else {
     if (stdout.length()>=2 && expectedOutput.length()>=2 ) {
      boolean ret = compare.compare( stripOutLeadingBracket (stdout) , stripOutLeadingBracket(expectedOutput) );
      resultDisplay = (ret) ? ["SUCCESS! actual output - $stdout, expected - $expectedOutput, compare class - $comparatorClassName" ] : ["FAIL! actual output - $stdout,  expected - $expectedOutput, compare class - $comparatorClassName"] 
      if (!ret) success = false
     } else {
      resultDisplay.add("Error! No output to compare. ");
      success = false;
     }
    }
    displayMessage (resultDisplay, success);

   } else {
    def resultDisplay = (sh.getRet()==0) ? ["Command return code - $shReturnCode, Output - $stdout" ] : ["Command return code - $shReturnCode,  Output - $stdout, Error output is $shStdErr" ]
    displayMessage (resultDisplay, false)
   }
  }

  return success
 }
  
 @Test
 public void testHadoopMapReduceExample() {
  LOG.info( "testHadoopMapReduceExample() - " + testName);
  
  BigTopIntegrationTest test = new Yaml().load(testCaseString)

  LOG.info("Test case name [" + test.getTestName() + "]");
  LOG.info("Test case description [" + test.getTestDesc() + "]");
  LOG.info("Test case details - " + test.toString());

  assertTrue("Test succeed : ", runExample(test));
 }
 
 public static void main(String[] args) {
  
 }
 
}

Enjoy the journey!

Sunday, March 18, 2012

Build Apache BigTop 0.3.0 at AWS

Well, after spending about 8 hrs, I finally got BigTop 0.3.0 to build on AWS.  Less painful than my BigTop 2.0 build experience. Most of time, is just wait, watch the screen scrolling. To make the experience more enjoyable, I put together a Youtube music playlist and let it roll while waiting.  

I put down some notes here, to remind myself or anyone who cares to travel through the same path.  

1. Select a right AMI (ami-31bc7758) and choose a right AWS configuration (large) are worth the time saved (credits go to Doug and Ron)

2. Need to install JDK 6.x and JDK 5.x.

3. Need to install Maven 3.x

4. Need to install apache-forrest-0.8

5. Need to follow Bikramjit's note.

6. Need to set  MAVEN_OPTS="-Xms1024m -Xmx2048m"

7. Here is my environment variables for BigTop build:



After the build, I did the test on the Hadoop install.

Here are my conf settings:

mapred-site.xml:


hdfs-site.xml:



core-site.xml:


To perform BigTop smoke test, I have done the following steps:

$cp ./build/hadoop/deb/hadoop-1.0.
1/src/test/org/apache/hadoop/cli/testConf.xml /home/ubuntu/bigtop/bigtop-tests/test-execution/smokes/hadoop/target/clitest_data/testConf.xml
$cd ~/bigtop/bigtop-tests/test-artifacts
$mvn install

$cd ~/bigtop/bigtop-tests/test-execution/common
$mvn install


$cd ~/bigtop/bigtop-tests/test-execution/conf
$mvn install

$cd ~/bigtop/bigtop-tests/test-execution/smokes/hadoop

add the followings to pom.xml

<dependency>
<groupId>org.apache.cxf</groupId>
<artifactId>cxf-rt-frontend-jaxrs</artifactId>
<version>2.5.0</version>
</dependency>

$mvn -Dhadoop.log.dir=target/build/test/logs verify > lei4.output

Note: I run mvn verify, it failed pretty bad, then I dig into the code and see what are needed to set. That was the result. 

Here is the content of the output file:


You can see that for cli.TestCLI:
# Tests pass: 152 (98%)
# Tests fail: 2 (1%)

I can not figure out why those 2 test cases failed.

org.apache.bigtop.itest.hadoopexamples.TestHadoopExamples passed:
Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 491.765 sec

org.apache.bigtop.itest.hadoopsmoke.TestHadoopSmoke passed:
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 89.924 sec



Enjoy!