Sunday, March 18, 2012

Build Apache BigTop 0.3.0 at AWS

Well, after spending about 8 hrs, I finally got BigTop 0.3.0 to build on AWS.  Less painful than my BigTop 2.0 build experience. Most of time, is just wait, watch the screen scrolling. To make the experience more enjoyable, I put together a Youtube music playlist and let it roll while waiting.  

I put down some notes here, to remind myself or anyone who cares to travel through the same path.  

1. Select a right AMI (ami-31bc7758) and choose a right AWS configuration (large) are worth the time saved (credits go to Doug and Ron)

2. Need to install JDK 6.x and JDK 5.x.

3. Need to install Maven 3.x

4. Need to install apache-forrest-0.8

5. Need to follow Bikramjit's note.

6. Need to set  MAVEN_OPTS="-Xms1024m -Xmx2048m"

7. Here is my environment variables for BigTop build:



After the build, I did the test on the Hadoop install.

Here are my conf settings:

mapred-site.xml:


hdfs-site.xml:



core-site.xml:


To perform BigTop smoke test, I have done the following steps:

$cp ./build/hadoop/deb/hadoop-1.0.
1/src/test/org/apache/hadoop/cli/testConf.xml /home/ubuntu/bigtop/bigtop-tests/test-execution/smokes/hadoop/target/clitest_data/testConf.xml
$cd ~/bigtop/bigtop-tests/test-artifacts
$mvn install

$cd ~/bigtop/bigtop-tests/test-execution/common
$mvn install


$cd ~/bigtop/bigtop-tests/test-execution/conf
$mvn install

$cd ~/bigtop/bigtop-tests/test-execution/smokes/hadoop

add the followings to pom.xml

<dependency>
<groupId>org.apache.cxf</groupId>
<artifactId>cxf-rt-frontend-jaxrs</artifactId>
<version>2.5.0</version>
</dependency>

$mvn -Dhadoop.log.dir=target/build/test/logs verify > lei4.output

Note: I run mvn verify, it failed pretty bad, then I dig into the code and see what are needed to set. That was the result. 

Here is the content of the output file:


You can see that for cli.TestCLI:
# Tests pass: 152 (98%)
# Tests fail: 2 (1%)

I can not figure out why those 2 test cases failed.

org.apache.bigtop.itest.hadoopexamples.TestHadoopExamples passed:
Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 491.765 sec

org.apache.bigtop.itest.hadoopsmoke.TestHadoopSmoke passed:
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 89.924 sec



Enjoy!

No comments:

Post a Comment