Apache Bigtop Week 10, Testing. Unit Testing Programming in the small vs. Programming in the large...

Apache Bigtop

Week 10, Testing

dougc [at] gmail25 dot com 2012 All Rights Reserved

Unit Testing

• Programming in the small vs. Programming in the large

Parlante’s link: codingbat.com• unit tests for programming in the small• Apache rule: Before submitting patch to

Hadoop Component, pass and verify all component unit tests.


Unit Testing

• Hadoop Unit Tests installed in bigtop• Great reference:

http://www.cloudera.com/blog/2008/12/testing-hadoop/

• Run tests on downloaded hadoop-0.20.205.0: ant test• Where are the bigtop shims for hadoop-

0.20.205.0/1.0/.22? For hive/pig?• Other shims are available but don’t work, have to pick at

build time. In latest relese. Hive 0.8.1. Pig in 0.9.2.





Hadoop Unit Testing


Bigtop Hadoop Unit Testing


Bigtop Unit Test Symlink

• Symlink src: ubuntu@ip-10-116-217-28:/usr/lib/hadoop$ sudo ln -s /usr/src/hadoop /usr/lib/hadoop/src


Bigtop Unit Test Permission

• sudo chmod 757 /usr/lib/hadoop, /usr/lib/hadoop/bin, /usr/lib/hadoop/sbin


Bigtop Unit Tests

• If running in AWS, setup Screen– sudo apt-get install screen screen-profiles screen-

profiles-extras– Type screen, will see clear terminal window, start

ant test, ctrl-a ctrl-d, logout, login again, type screen -r

• Ron’s fix: Modify /etc/hostname


Standalone/Bigtop Hadoop Unit Tests Results

• Standalone: Logs for each test under ~/hadoop-0.20.205.0/build/test

• Bigtop: /usr/lib/hadoop/build/test


Hadoop Mods to get Integration Tests to run (repeat)

• Copy testConf.xml: sudo cp /usr/src/hadoop/test/org/apache/hadoop/cli/testConf.xml /home/ubuntu/bigtop-0.2.0-incubating/bigtop-tests/test-execution/smokes/hadoop/target/clitest_data/

• https://issues.cloudera.org/browse/DISTRO-44• Add Jackson dependency to pom.xml<dependency> <groupId>org.codehaus.jackson</groupId> <artifactId>jackson-mapper-asl</artifactId> <version>1.9.3</version></dependency>

https://issues.cloudera.org/browse/DISTRO-44

https://issues.cloudera.org/browse/DISTRO-44


Bigtop Hadoop Integration Tests

• Running single integration test: mvn –Dit.test=org.apache.bigtop.itest.hadooptests.CLASS verifyExample: mvn -Dit.test=org.apache.bigtop.itest.hadooptests.TestTestCLI verify


Standalone Hbase Unit Tests

• ~/hbase-0.9.2/mvn –P localTest • Running a single unit test: mvn test -

Dtest=org.apache.hadoop.hbase.TestHServerAddress


Bigtop Hbase Unit Tests• Don’t exist? Put in /usr/src/hbase like Hadoop and use Groovy shell to run? • Project to get Hbase unit tests working in bigtop? Partition hbase unit tests

into categories. One approach to issue requests, look at internal state and verify. Another approach only use public APIs, r/w to Hbase. Partition into 2 categories. MiniHbase mock objects in a single JVM process can be used in Bigtop. Different bugs in distributed mode vs. MiniMr/DFSCluster. Write this up as a project.

• PIG uses same test artifacat from unit test for bigtop.• Missing pom goals• Use

– org.apache.bigtop.itest.JUnitUtils.groovy. For annotation support in Junit4/groovy. – org.apache.bigtop.itest.junit.OrderedParameterized.java; extension of Junit, Junit has

all tests are stateless, order doesn’t matter. Tests are not stateless in bigtop, ordering requires run stages, specify which run stage; simple ints with ordering. By default are in run stage 0, if have tests case annotated -1 run stage will execute this first.


Org.apache.bigtop.itest.pmanager questions

• PackageManager/Abstract Class• What is DEBPackage.groovy, ManagedPacakge.groovy,

RPMPackage.groovy for? • AptCmdLinePackageManager.groovy allows apt-get commands in

Groovy? • YumCmdLinePackageManager, RPMPackage,

ZypperCmdLinePackageManager• Bigtop spends time on packaging like apt-get install, no existing Java

APIs to do this, install packages using Java Api. Used internally for Jenkins testing, tests in test-artifacts/package. Manifest driven in xml files for what is expected from package, files with xxx permissions, check and verify paths and permission. If you are introducing a new package you are responsible for this abstract class testing.


Bigtop Hbase Integration Tests• Bigtop-2.0-incubating/bigtop-tests/test-execution/smokes/hbase/mvn

verify• /home/ubuntu/bigtop-0.2.0-incubating/system/TestLoadAndVerify.java • // private static final long NUM_TO_WRITE_DEFAULT =

100*1000;private static final long NUM_TO_WRITE_DEFAULT = 10;

• //private static final int NUM_TASKS = 200;• //private static final int NUM_REDUCE_TASKS = 35; private static final int NUM_TASKS=2; private static final int NUM_REDUCE_TASKS=2;


Bigtop Hbase Integration Results


Pig/HiveMahout/Oozie/Flume Unit Tests

• ant test or mvn test• Project: mavenize Hive:hard • Project: Pig, easier? • ~/hive-0.7.1/src/build.xml• ~/pig-0.9.2/build.xml• mahout-0.6-src, ~/mahout-distribution-0.6; mvn test; install

core and src, 2 subdirectories with same name ~/mahout-distribution-0.6/mahout-distribution-0.6/pom.xml

• git clone https://github.com/yahoo/oozie.git; mvn test• git clone https://github.com/cloudera/flume.git; mvn test

https://github.com/yahoo/oozie.git

https://github.com/yahoo/oozie.git

https://github.com/cloudera/flume.git

https://github.com/cloudera/flume.git


Pig-0.9.2 unit test Results


Bigtop Pig Integration Tests

• Problem with mvn artifact…


Hive-0.7.1 Notes

• Hive Unit Tests install own version of Hadoop• ~/hive-0.7.1/src/build/hadoopcore/• Remove test

TestHadoopThriftAuthBridge20S.java. Cant connect to Thrift Server, socket timeout > 6x.


Hive-0.7.1 Unit Tests


Bigtop Hive Integration Tests

• Follow Pig format, work with Hive Unit Test Authors. Hive integration project suggestion. Hive/Pig on top of M/R with custom language. With a compiler, the unit tests have input/expected output. Hive unit tests are SQL code and verification afterwards. Was hard to retrofit vs. real cluster. Took *.SQL files from Hive and dumping them in Bigtop to take SQL files and compare actual/expected. Can you reuse the same test artifacts for unit tests and bigtop integration tests. Convert Hive Unit tests


Mahout Results


Flume Unit Test Results


Bigtop Flume Integration Tests

• Transition to FlumeNG. NG lost features from Flume. Too early.


Oozie Unit Test


Bigtop Oozie Integration Test

• What to set oozie_url to? http://localhost:11000/oozie

• Start the oozie service.. • Runs only Oozie examples.jar. Project to create

workflow for oozie. Integration testing on cluster needed here. Actions, to broaden data interfaces. Email actions, sqoop action. Good project for J2EE developers.


Bigtop Scripts mod

• Where to modify the bigtop install scripts to fix this?


Command line vs. Eclipse

• Create a Java Project, test programs using HDFS/Hive/Pig, etc… 2 ways to run the files, command line or in Eclipse.


Command Line vs. Eclipse

• This may be important when debugging in cluster and pseudo-distributed mode.

• Cluster loads the 3 conf/ files, core-site.xml, hdfs-site.xml , mapred-sire.xml. Some of the parameters are embedded..

• Java Code may not properly init these params for cluster operation. Sometimes hard to debug


Hadoop CLI

• Command line uses bin/hadoop jarfilename.jar ClassName args

• Did this when running Pi from hadoop-xxx-examples.jar, test programs under jar


Hadoop CLI

• Set absolute path for log4j.propertiesPropertyConfigurator

.configure("/Users/dc/Documents/ workspace/log4j.properties");• Properties files are outside of the jar. Web

search for adding log4j.properties to jar are incorrect. Web search for setting class path are incorrect.


Eclipse Console Output 1/214:03:08,263 INFO TestHDFS:34 - Yo I am logger!!!created new jobconffinished setting jobconf parametersgenerateSampeInpuIf inputDirectory:file:/tmp/MapReduceIntroInput exists:true14:03:08,588 INFO TestHDFS:59 - isEmptyDirectory14:03:08,595 INFO TestHDFS:65 - num file status:414:03:08,596 INFO TestHDFS:75 - file:///tmp/MapReduceIntroInput is not empty14:03:08,596 INFO TestHDFS:80 - A non empty file file:///tmp/MapReduceIntroInput/asdf.txt was found14:03:08,597 INFO TestHDFS:46 - The inputDirectory file:/tmp/MapReduceIntroInput exists and is either a file or a non empty directory14:03:08,598 INFO TestHDFS:111 - Generating 3 input files of random data, each record is a random number TAB the input file name


Eclipse Console Output 2/214:03:25,076 INFO JobClient:589 - Map input records=1514:03:25,076 INFO JobClient:589 - Reduce shuffle bytes=014:03:25,076 INFO JobClient:589 - Spilled Records=3014:03:25,077 INFO JobClient:589 - Map output bytes=30314:03:25,077 INFO JobClient:589 - Total committed heap usage (bytes)=42500096014:03:25,077 INFO JobClient:589 - Map input bytes=30214:03:25,077 INFO JobClient:589 - SPLIT_RAW_BYTES=35814:03:25,078 INFO JobClient:589 - Combine input records=014:03:25,078 INFO JobClient:589 - Reduce input records=1514:03:25,078 INFO JobClient:589 - Reduce input groups=1514:03:25,078 INFO JobClient:589 - Combine output records=014:03:25,079 INFO JobClient:589 - Reduce output records=1514:03:25,079 INFO JobClient:589 - Map output records=1514:03:25,079 INFO TestHDFS:235 - The job has completed.14:03:25,079 INFO TestHDFS:241 - The job completed successfully.


Command Line Output21:44:36,765 INFO TestHDFS:35 - Yo I am logger!!!created new jobconffinished setting jobconf parametersgenerateSampeInpuIf inputDirectory:file:/tmp/MapReduceIntroInput exists:true21:44:36,983 INFO TestHDFS:60 - isEmptyDirectory21:44:36,990 INFO TestHDFS:66 - num file status:421:44:36,991 INFO TestHDFS:76 - file:///tmp/MapReduceIntroInput is not empty21:44:36,991 INFO TestHDFS:81 - A non empty file file:///tmp/MapReduceIntroInput/asdf.txt was found21:44:36,992 INFO TestHDFS:47 - The inputDirectory file:/tmp/MapReduceIntroInput exists and is either a file or a non empty directory21:44:36,992 INFO TestHDFS:112 - Generating 3 input files of random data, each record is a random number TAB the input file name21:44:36,999 WARN NativeCodeLoader:52 - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable21:44:37,007 INFO TestHDFS:169 - The job output directory file:/tmp/MapReduceIntroOutput exists and is not a directory and will be removed21:44:37,022 INFO TestHDFS:235 - Launching the job.


Command Line Output 2/2

21:44:53,523 INFO JobClient:589 - SPLIT_RAW_BYTES=35821:44:53,524 INFO JobClient:589 - Combine input records=021:44:53,524 INFO JobClient:589 - Reduce input records=1921:44:53,531 INFO JobClient:589 - Reduce input groups=1921:44:53,531 INFO JobClient:589 - Combine output records=021:44:53,531 INFO JobClient:589 - Reduce output records=1921:44:53,531 INFO JobClient:589 - Map output records=1921:44:53,532 INFO TestHDFS:237 - The job has completed.21:44:53,532 INFO TestHDFS:243 - The job completed successfully.


M/R Idioms

• What you get for free in M/R• Sorting• Duplicate Detection• Design Pattern Notes: Object Churn, Thread

Safety in Mappers– ThreadLocal vs. Atomic Ivars vs. Locks


M/R Idioms

• Hadoop Partitioner, multiple output files or 1 output file. – Job.setNumReduceTasks(1) same as merge sort– Default HashPartitioner– Create own for filtering, e.g. sending all keys which

start with a common prefix to one specific file.


HDFS Idioms

• Serialization: Writable I/F order of magnitude performance improvement

• HDFS Block R/W• JobTrackers/TaskTrackers/NameNodes. Each

file operation directly goes to the NN.


Integration Testing SmokeHadoop Component, /usr/lib

Tests Exist under bigtop-tests

Hbase Yes IncrementalPELoad.java, TestHBaseCompression.java, TEstHBasePigSmoke.groovy, TestHBaseSmoke.java, TestHFileOutpuFormat.java, TestLoadIncrementalHFiles.java

Hive Yes HiveBulkScriptExecutor.java, IntegrationTestHiveSmokeBulk.groovy, TestHiveSmokeBulk.groovy, TestJdbcDriver.java

Pig No Yes, in Hbase

Zookeeper No Part of components

Mahout No


Integration Testing SmokeHadoop Component Test Code exists Program name

Whirr No, not needed?

SQOOP Yes IntegrationTestSqoopHive.groovy, IntegrationTestSqoopHbase.groovy

Flume Yes TestFlumeSmoke.groovy

Hadoop Yes TestCLI.groovy, TesthadoopSmoke, TestHadoopExamples

Package Test Yes PackageTestCommon.groovy,

Hue Yes, part of package test

Oozie Yes TestOozieSmoke.groovy, StateVerifierZookeeper.groovy


Lab #4 Create an integration test

• Groovy runtime allows shell commands which allow you to use the scripts inside the components saving debugging time for the classpaths and environment files

• Alternatively use Java libraries, DFSCluster, MiniMRCluster, reverse engineer the env. vars settings, sequence of commands to run, Start from a HDFS file system then work way up to Bigtop Component


From Lab #3

• Working map reduce program, run them using mvn verify. Have to make sure HDFS/Hadoop is running first


Create a Mahout artifact dir and child pom


Groovy Test Code

• Assumes HDFS running• Configuration conf = new Configuration();• conf.addResource('mapred-site.xml’)• Shell sh = new Shell("/bin/bash -s");sh.exec("hadoop fs –mkdir /tmp/test”,“hadoop fs –copyFromLocal one /tmp/test/one”, “hadoop fs –cat /tmp/test/one”);


Future Labs

• Integrate unit testing into Bigtop• More integration testing• Integrate different versions of Hadoop

Components(Hbase, Hive, etc) into Bigtop• Mavenize an ant centric Hadoop Component,

Pig, Hive• Puppet Lab; Bigtop puppet code used in CDH4

to deploy/test• Deploying and testing in cluster

Date post:	24-Dec-2015
Category:	Documents
Upload:	andra-mitchell
View:	216 times
Download:	0 times