Build and Run Hadoop Java program with Ant and Maven

Recently I got a chance to experiment with ant/maven to build and run Hadoop Java program. I don't have time to read the manuals, and here is what I did on Ant and Maven scripts to build and run a modified WordCount program. 


I have hadoop-0.20.205.0 installed at /usr/local/hadoop20 and environment variable is set accordingly. The build target is to build jar file LeiBigTop-1.1.jar.


Here is the build.xml:

<project name="LeiBigTop" basedir="." default="build.jar"> <property environment="env" /> <property name="hadoop" value="${env.HADOOP_HOME}" /> <property name="hbase" value="${env.HBASE_HOME}" /> <property name="dest.dir" value="target" /> <property name="dest.dir.classes" value="${dest.dir}/classes" /> <property name="dest.dir.release" value="${dest.dir}/release" /> <property name="target.name" value="LeiBigTop" /> <property name="target.version" value="1.1" /> <property name="target.jar" value="${target.name}-1.1.jar" /> <path id="run.class.path"> <pathelement location="." /> <fileset dir="${hbase}/"> <include name="hbase-0.90.5.jar" /> </fileset> <fileset dir="${hbase}/lib/"> <include name="zookeeper-3.3.2.jar" /> </fileset> <fileset dir="${hadoop}/lib/"> <include name="*.jar" /> </fileset> <fileset dir="${hadoop}/"> <include name="*.jar" /> </fileset> <pathelement location="${dest.dir.release}/${target.jar}" /> </path> <path id="build.class.path"> <pathelement location="." /> <fileset dir="${hbase}/"> <include name="hbase-0.90.5.jar" /> </fileset> <fileset dir="${hbase}/lib/"> <include name="zookeeper-3.3.2.jar" /> </fileset> <fileset dir="${hadoop}/lib/"> <include name="*.jar" /> </fileset> <fileset dir="${hadoop}/"> <include name="*.jar" /> </fileset> </path> <target name="clean"> <delete dir="${dest.dir}" /> </target> <target name="compile"> <echo>+---------------------------------------------------------+</echo> <echo>+ Compiling +</echo> <echo>+---------------------------------------------------------+</echo> <echo>${hadoop}</echo> <echo>${hbase}</echo> <echo></echo> <javac srcdir="src" destdir="${dest.dir.classes}"> <classpath refid="build.class.path" /> </javac> </target> <target name="prepare"> <mkdir dir="${dest.dir}" /> <mkdir dir="${dest.dir.classes}" /> <mkdir dir="${dest.dir.release}" /> <mkdir dir="${dest.dir.classes}/META-INF" /> </target> <target name="build.jar" depends="clean,prepare,compile"> <jar basedir="${dest.dir.classes}" destfile="${dest.dir.release}/${target.jar}" /> </target> <target name="runWordCounts" depends="build.jar" description="run the default test"> <exec executable="bin/hadoop" dir="/usr/local/hadoop20"> <arg value="jar"/> <arg value="${basedir}/${dest.dir.release}/${target.jar}"/> <arg value="com.lei.bigtop.hadoop.wordcount.CountWordsV2"/> <arg value="/Users/lei/work/hadoop-example/input"/> <arg value="/Users/lei/work/hadoop-example/output"/> </exec> </target> <target name="runWordCountsWithArg" depends="build.jar" description="run the default test"> <exec executable="bin/hadoop" dir="/usr/local/hadoop20"> <arg value="jar"/> <arg value="${basedir}/${dest.dir.release}/${target.jar}"/> <arg value="${mainclass}"/> <arg value="${input}"/> <arg value="${output}"/> </exec> </target> </project>

Assuming you have Hadoop running correctly. You can run the following to build and run the com.lei.bigtop.hadoop.wordcount.CountWordsV2. 


lei:LeiBigTop lei$ ant -version
Apache Ant(TM) version 1.8.2 compiled on December 20 2010


lei:LeiBigTop lei$ ant


lei:LeiBigTop lei$ ant runWordCounts 


Here is the pom.xml.  



<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd"> <modelVersion>4.0.0</modelVersion> <groupId>com.lei.bigtop</groupId> <artifactId>LeiBigTop</artifactId> <version>1.1</version> <packaging>jar</packaging> <name>LeiBigTop</name> <url>http://maven.apache.org</url> <properties> <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding> </properties> <dependencies> <dependency> <groupId>junit</groupId> <artifactId>junit</artifactId> <version>3.8.1</version> <scope>test</scope> </dependency> <dependency> <groupId>org.apache.hadoop</groupId> <artifactId>hadoop-core</artifactId> <version>0.20.205.0</version> </dependency> </dependencies> <build> <sourceDirectory>src</sourceDirectory> <directory>target</directory> <plugins> <plugin> <groupId>org.apache.maven.plugins</groupId> <artifactId>maven-compiler-plugin</artifactId> <version>2.3.2</version> <configuration> <source>1.6</source> <target>1.6</target> </configuration> </plugin> <plugin> <artifactId>maven-antrun-plugin</artifactId> <executions> <execution> <phase>verify</phase> <configuration> <tasks> <!-- Place any Ant task here. You can add anything you can add between <target> and </target> in a build.xml. --> <exec executable="bin/hadoop" dir="/usr/local/hadoop20"> <arg value="jar"/> <arg value="${project.basedir}/target/${project.build.finalName}.${project.packaging}"/> <!-- <arg value="/Users/lei/Documents/workspace/LeiBigTop/target/LeiBigTop-1.1.jar"/> --> <arg value="com.lei.bigtop.hadoop.wordcount.CountWordsV2"/> <arg value="/Users/lei/work/hadoop-example/input"/> <arg value="/Users/lei/work/hadoop-example/output"/> </exec> </tasks> </configuration> <goals> <goal>run</goal> </goals> </execution> </executions> </plugin> </plugins> </build> </project>

lei:LeiBigTop lei$ mvn -version

Apache Maven 3.0.3 (r1075438; 2011-02-28 09:31:09-0800)

Maven home: /usr/local/apache-maven-3.0.3

Java version: 1.6.0_29, vendor: Apple Inc.

Java home: /System/Library/Java/JavaVirtualMachines/1.6.0.jdk/Contents/Home

Default locale: en_US, platform encoding: MacRoman

OS name: "mac os x", version: "10.6.8", arch: "x86_64", family: "mac"

lei:LeiBigTop lei$ mvn clean compile package verify




Enjoy.