|
|||||||||
PREV CLASS NEXT CLASS | FRAMES NO FRAMES | ||||||||
SUMMARY: NESTED | FIELD | CONSTR | METHOD | DETAIL: FIELD | CONSTR | METHOD |
java.lang.Objectcascading.flow.planner.FlowPlanner
cascading.flow.hadoop.planner.HadoopPlanner
public class HadoopPlanner
Class HadoopPlanner is the core Hadoop MapReduce planner.
Notes: Custom JobConf propertiescopyJobConf(java.util.Map, org.apache.hadoop.mapred.JobConf)
on a map properties object before constructing a new HadoopFlowConnector
.
A better practice would be to set Hadoop properties directly on the map properties object handed to the FlowConnector.
All values in the map will be passed to a new default JobConf instance to be used as defaults for all resulting
Flow instances.
For example, properties.set("mapred.child.java.opts","-Xmx512m");
would convince Hadoop
to spawn all child jvms with a heap of 512MB.
Field Summary |
---|
Fields inherited from class cascading.flow.planner.FlowPlanner |
---|
assertionLevel, debugLevel, properties |
Constructor Summary | |
---|---|
HadoopPlanner()
|
Method Summary | |
---|---|
Flow |
buildFlow(FlowDef flowDef)
Method buildFlow renders the actual Flow instance. |
static void |
copyJobConf(Map<Object,Object> properties,
JobConf jobConf)
Method copyJobConf adds the given JobConf values to the given properties object. |
static JobConf |
createJobConf(Map<Object,Object> properties)
Method createJobConf returns a new JobConf instance using the values in the given properties argument. |
static boolean |
getNormalizeHeterogeneousSources(Map<Object,Object> properties)
Method getNormalizeHeterogeneousSources returns if this planner will normalize heterogeneous input sources. |
void |
initialize(FlowConnector flowConnector,
Map<Object,Object> properties)
|
protected Tap |
makeTempTap(String name)
Method makeTemp ... |
static void |
setNormalizeHeterogeneousSources(Map<Object,Object> properties,
boolean doNormalize)
Method setNormalizeHeterogeneousSources adds the given doNormalize boolean to the given properites object. |
Methods inherited from class cascading.flow.planner.FlowPlanner |
---|
createElementGraph, failOnGroupEverySplit, failOnLoneGroupAssertion, failOnMissingGroup, failOnMisusedBuffer, handleExceptionDuringPlanning, handleJobPartitioning, handleJoins, handleNonSafeOperations, insertTempTapAfter, verifyAssembly, verifyCheckpoints, verifyPipeAssemblyEndPoints, verifySourceNotSinks, verifyTaps, verifyTraps |
Methods inherited from class java.lang.Object |
---|
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait |
Constructor Detail |
---|
public HadoopPlanner()
Method Detail |
---|
public static void copyJobConf(Map<Object,Object> properties, JobConf jobConf)
properties
- of type MapjobConf
- of type JobConfpublic static JobConf createJobConf(Map<Object,Object> properties)
properties
- of type Map
public static void setNormalizeHeterogeneousSources(Map<Object,Object> properties, boolean doNormalize)
properties
- of type MapdoNormalize
- of type booleanpublic static boolean getNormalizeHeterogeneousSources(Map<Object,Object> properties)
properties
- of type Map
public void initialize(FlowConnector flowConnector, Map<Object,Object> properties)
initialize
in class FlowPlanner
public Flow buildFlow(FlowDef flowDef)
FlowPlanner
buildFlow
in class FlowPlanner
protected Tap makeTempTap(String name)
FlowPlanner
makeTempTap
in class FlowPlanner
|
|||||||||
PREV CLASS NEXT CLASS | FRAMES NO FRAMES | ||||||||
SUMMARY: NESTED | FIELD | CONSTR | METHOD | DETAIL: FIELD | CONSTR | METHOD |