cascading.flow.hadoop.planner

## Class HadoopPlanner

- java.lang.Object
- - cascading.flow.planner.FlowPlanner<HadoopFlow,org.apache.hadoop.mapred.JobConf>
  - - cascading.flow.hadoop.planner.HadoopPlanner

- Direct Known Subclasses:
    Hadoop2MR1Planner

      public class HadoopPlanner
      extends FlowPlanner<HadoopFlow,org.apache.hadoop.mapred.JobConf>

  Class HadoopPlanner is the core Hadoop MapReduce planner used by default through a `FlowConnector` sub-class.

  Notes:

  Custom JobConf properties
  A custom JobConf instance can be passed to this planner by calling `copyJobConf(java.util.Map, org.apache.hadoop.mapred.JobConf)` on a map properties object before constructing a new `FlowConnector` sub-class.

  A better practice would be to set Hadoop properties directly on the map properties object handed to the FlowConnector. All values in the map will be passed to a new default JobConf instance to be used as defaults for all resulting Flow instances.

  For example, `properties.set("mapred.child.java.opts","-Xmx512m");` would convince Hadoop to spawn all child jvms with a heap of 512MB.

- - ### Nested Class Summary

    - ### Nested classes/interfaces inherited from class cascading.flow.planner.FlowPlanner

      `FlowPlanner.TempTapElementFactory`

  - ### Field Summary

    Fields
    Modifier and Type | Field and Description
    --- | ---
    `static String` | `PLATFORM_NAME`

    - ### Fields inherited from class cascading.flow.planner.FlowPlanner

      `checkpointTapRootPath, defaultAssertionLevel, defaultDebugLevel, defaultProperties, TRACE_PLAN_PATH, TRACE_PLAN_TRANSFORM_PATH, TRACE_STATS_PATH`

  - ### Constructor Summary

    Constructors
    Constructor and Description
    `HadoopPlanner()`

  - ### Method Summary

    Methods
    Modifier and Type | Method and Description
    --- | ---
    `protected void` | `checkPlatform(org.apache.hadoop.conf.Configuration conf)`
    `void` | `configRuleRegistryDefaults(RuleRegistry ruleRegistry)`
    `static void` | `copyJobConf(Map<Object,Object> properties, org.apache.hadoop.mapred.JobConf jobConf)` Method copyJobConf adds the given JobConf values to the given properties object.
    `static void` | `copyProperties(org.apache.hadoop.mapred.JobConf jobConf, Map<Object,Object> properties)` Method copyProperties adds the given Map values to the given JobConf object.
    `protected HadoopFlow` | `createFlow(FlowDef flowDef)`
    `static org.apache.hadoop.mapred.JobConf` | `createJobConf(Map<Object,Object> properties)` Method createJobConf returns a new JobConf instance using the values in the given properties argument.
    `org.apache.hadoop.mapred.JobConf` | `getDefaultConfig()`
    `URI` | `getDefaultURIScheme(Tap tap)`
    `FlowStepFactory<org.apache.hadoop.mapred.JobConf>` | `getFlowStepFactory()`
    `PlannerInfo` | `getPlannerInfo(String registryName)`
    `PlatformInfo` | `getPlatformInfo()`
    `URI` | `getURIScheme(Tap tap)`
    `void` | `initialize(FlowConnector flowConnector, Map<Object,Object> properties)`
    `protected Tap` | `makeTempTap(String prefix, String name)`

    - ### Methods inherited from class cascading.flow.planner.FlowPlanner

      `buildFlow, createFlowElementGraph, getAssertionLevel, getDebugLevel, getDefaultProperties, getFlowNodeFactory, handleExceptionDuringPlanning, makeTempTap, resolveAssemblyPlanners, resolveTails, verifyAllTaps, verifyAssembly, verifyCheckpoints, verifyPipeAssemblyEndPoints, verifyResult, verifyResultInternal, verifySourceNotSinks, verifyTaps, verifyTraps`

    - ### Methods inherited from class java.lang.Object

      `clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait`

- - ### Field Detail

    - #### PLATFORM_NAME

          public static final String PLATFORM_NAME

      See Also:
        Constant Field Values

  - ### Constructor Detail

    - #### HadoopPlanner

          public HadoopPlanner()

  - ### Method Detail

    - #### copyJobConf

          public static void copyJobConf(Map<Object,Object> properties,
                         org.apache.hadoop.mapred.JobConf jobConf)

      Method copyJobConf adds the given JobConf values to the given properties object. Use this method to pass custom default Hadoop JobConf properties to Hadoop.
      Parameters:
        `properties` - of type Map
        `jobConf` - of type JobConf

    - #### createJobConf

          public static org.apache.hadoop.mapred.JobConf createJobConf(Map<Object,Object> properties)

      Method createJobConf returns a new JobConf instance using the values in the given properties argument.
      Parameters:
        `properties` - of type Map
      Returns:
        a JobConf instance

    - #### copyProperties

          public static void copyProperties(org.apache.hadoop.mapred.JobConf jobConf,
                            Map<Object,Object> properties)

      Method copyProperties adds the given Map values to the given JobConf object.
      Parameters:
        `jobConf` - of type JobConf
        `properties` - of type Map

    - #### getPlannerInfo

          public PlannerInfo getPlannerInfo(String registryName)

      Specified by:
        `getPlannerInfo` in class `FlowPlanner<HadoopFlow,org.apache.hadoop.mapred.JobConf>`

    - #### getDefaultConfig

          public org.apache.hadoop.mapred.JobConf getDefaultConfig()

      Specified by:
        `getDefaultConfig` in class `FlowPlanner<HadoopFlow,org.apache.hadoop.mapred.JobConf>`

    - #### getPlatformInfo

          public PlatformInfo getPlatformInfo()

      Specified by:
        `getPlatformInfo` in class `FlowPlanner<HadoopFlow,org.apache.hadoop.mapred.JobConf>`

    - #### initialize

          public void initialize(FlowConnector flowConnector,
                        Map<Object,Object> properties)

      Overrides:
        `initialize` in class `FlowPlanner<HadoopFlow,org.apache.hadoop.mapred.JobConf>`

    - #### configRuleRegistryDefaults

          public void configRuleRegistryDefaults(RuleRegistry ruleRegistry)

      Overrides:
        `configRuleRegistryDefaults` in class `FlowPlanner<HadoopFlow,org.apache.hadoop.mapred.JobConf>`

    - #### checkPlatform

          protected void checkPlatform(org.apache.hadoop.conf.Configuration conf)

    - #### createFlow

          protected HadoopFlow createFlow(FlowDef flowDef)

      Specified by:
        `createFlow` in class `FlowPlanner<HadoopFlow,org.apache.hadoop.mapred.JobConf>`

    - #### getFlowStepFactory

          public FlowStepFactory<org.apache.hadoop.mapred.JobConf> getFlowStepFactory()

      Specified by:
        `getFlowStepFactory` in class `FlowPlanner<HadoopFlow,org.apache.hadoop.mapred.JobConf>`

    - #### getDefaultURIScheme

          public URI getDefaultURIScheme(Tap tap)

    - #### getURIScheme

          public URI getURIScheme(Tap tap)

    - #### makeTempTap

          protected Tap makeTempTap(String prefix,
                        String name)

      Specified by:
        `makeTempTap` in class `FlowPlanner<HadoopFlow,org.apache.hadoop.mapred.JobConf>`

Copyright © 2007-2015 Concurrent, Inc. All Rights Reserved.