cascading.platform.hadoop

## Class HadoopPlatform

- java.lang.Object
- - cascading.platform.TestPlatform
  - - cascading.platform.hadoop.BaseHadoopPlatform<JobConf>
    - - cascading.platform.hadoop.HadoopPlatform

-     public class HadoopPlatform
      extends BaseHadoopPlatform<JobConf>

  Class HadoopPlatform is automatically loaded and injected into a `PlatformTestCase` instance so that all *PlatformTest classes can be tested against Apache Hadoop.

  This platform works in three modes.

  Hadoop standalone mode is when Hadoop is NOT run as a cluster, and all child tasks are in process and in memory of the "client" side code.

  Hadoop mini cluster mode where a cluster is created on demand using the Hadoop MiniDFSCluster and MiniMRCluster utilities. When a PlatformTestCase requests to use a cluster, this is the default cluster. All properties are pulled from the current CLASSPATH via the JobConf.

  Lastly remote cluster mode is enabled when the System property "mapred.jar" is set. This is a Hadoop property specifying the Hadoop "job jar" to be used cluster side. This MUST be the Cascading test suite and dependencies packaged in a Hadoop compatible way. This is left to be implemented by the framework using this mode. Additionally these properties may optionally be set if not already in the CLASSPATH; fs.default.name and mapred.job.tracker.

- - ### Field Summary

    Fields
    Modifier and Type | Field and Description
    --- | ---
    `static org.apache.hadoop.hdfs.MiniDFSCluster` | `dfs`
    `static MiniMRCluster` | `mr`

    - ### Fields inherited from class cascading.platform.hadoop.BaseHadoopPlatform

      `configuration, fileSys, logger, properties`

    - ### Fields inherited from class cascading.platform.TestPlatform

      `CLUSTER_TESTING_PROPERTY, numGatherPartitions, numMappers, numReducers, PLATFORM_PREFIX`

  - ### Constructor Summary

    Constructors
    Constructor and Description
    `HadoopPlatform()`

  - ### Method Summary

    Methods
    Modifier and Type | Method and Description
    --- | ---
    `JobConf` | `getConfiguration()`
    `FlowConnector` | `getFlowConnector(Map<Object,Object> properties)`
    `FlowProcess` | `getFlowProcess()`
    `Integer` | `getNumMapTasks(Map<Object,Object> properties)`
    `Integer` | `getNumReduceTasks(Map<Object,Object> properties)`
    `void` | `setNumMapTasks(Map<Object,Object> properties, int numMapTasks)`
    `void` | `setNumReduceTasks(Map<Object,Object> properties, int numReduceTasks)`
    `void` | `setUp()`

    - ### Methods inherited from class cascading.platform.hadoop.BaseHadoopPlatform

      `copyFromLocal, copyToLocal, getDelimitedFile, getDelimitedFile, getDelimitedFile, getHiddenTemporaryPath, getLongComparator, getPartitionTap, getProperties, getStringComparator, getTap, getTestConfigDefScheme, getTestFailScheme, getTextFile, isHDFSAvailable, isMapReduce, remoteExists, remoteRemove, safeFileName, setNumGatherPartitions, setNumMappers, setNumReducers, tearDown`

    - ### Methods inherited from class cascading.platform.TestPlatform

      `getApplicationJar, getDelimitedFile, getDelimitedFile, getDelimitedFile, getDelimitedFile, getDelimitedFile, getFlowConnector, getGlobalProperties, getName, getNumGatherPartitions, getNumGatherPartitionTasks, getNumMappers, getNumReducers, getTabDelimitedFile, getTabDelimitedFile, getTextFile, getTextFile, getTextFile, getTextFile, isDAG, isUseCluster, setNumGatherPartitionTasks, setUseCluster, supportsGroupByAfterMerge`

    - ### Methods inherited from class java.lang.Object

      `clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait`

- - ### Field Detail

    - #### dfs

          public static transient org.apache.hadoop.hdfs.MiniDFSCluster dfs

    - #### mr

          public static transient MiniMRCluster mr

  - ### Constructor Detail

    - #### HadoopPlatform

          public HadoopPlatform()

  - ### Method Detail

    - #### getFlowConnector

          public FlowConnector getFlowConnector(Map<Object,Object> properties)

      Specified by:
        `getFlowConnector` in class `cascading.platform.TestPlatform`

    - #### setNumMapTasks

          public void setNumMapTasks(Map<Object,Object> properties,
                            int numMapTasks)

      Overrides:
        `setNumMapTasks` in class `cascading.platform.TestPlatform`

    - #### setNumReduceTasks

          public void setNumReduceTasks(Map<Object,Object> properties,
                               int numReduceTasks)

      Overrides:
        `setNumReduceTasks` in class `cascading.platform.TestPlatform`

    - #### getNumMapTasks

          public Integer getNumMapTasks(Map<Object,Object> properties)

      Overrides:
        `getNumMapTasks` in class `cascading.platform.TestPlatform`

    - #### getNumReduceTasks

          public Integer getNumReduceTasks(Map<Object,Object> properties)

      Overrides:
        `getNumReduceTasks` in class `cascading.platform.TestPlatform`

    - #### getConfiguration

          public JobConf getConfiguration()

      Specified by:
        `getConfiguration` in class `BaseHadoopPlatform<JobConf>`

    - #### getFlowProcess

          public FlowProcess getFlowProcess()

      Specified by:
        `getFlowProcess` in class `cascading.platform.TestPlatform`

    - #### setUp

          public void setUp()
                     throws IOException

      Specified by:
        `setUp` in class `cascading.platform.TestPlatform`
      Throws:
        `IOException`

Copyright © 2007-2015 Concurrent, Inc. All Rights Reserved.