cascading.tap.hadoop

## Class Hfs

- java.lang.Object
- - cascading.tap.Tap<Configuration,RecordReader,OutputCollector>
  - - cascading.tap.hadoop.Hfs

- All Implemented Interfaces:
    FlowElement, ScopedElement, FileType<Configuration>, Traceable, Serializable
  Direct Known Subclasses:
    Dfs, Lfs, TempHfs

      public class Hfs
      extends Tap<Configuration,RecordReader,OutputCollector>
      implements FileType<Configuration>

  Class Hfs is the base class for all Hadoop file system access. Hfs may only be used with the Hadoop `FlowConnector` sub-classes when creating Hadoop executable `Flow` instances.

  Paths typically should point to a directory, where in turn all the "part" files immediately in that directory will be included. This is the practice Hadoop expects. Sub-directories are not included and typically result in a failure.

  To include sub-directories, Hadoop supports "globing". Globing is a frustrating feature and is supported more robustly by `GlobHfs` and less so by Hfs.

  Hfs will accept `/*` (wildcard) paths, but not all convenience methods like `jobConf.getSize` will behave properly or reliably. Nor can the Hfs instance with a wildcard path be used as a sink to write data.

  In those cases use GlobHfs since it is a sub-class of `MultiSourceTap`.

  Optionally use `Dfs` or `Lfs` for resources specific to Hadoop Distributed file system or the Local file system, respectively. Using Hfs is the best practice when possible, Lfs and Dfs are conveniences.

  Use the Hfs class if the 'kind' of resource is unknown at design time. To use, prefix a scheme to the 'stringPath'. Where `hdfs://...` will denote Dfs, and `file://...` will denote Lfs.

  Call `HfsProps.setTemporaryDirectory(java.util.Map, String)` to use a different temporary file directory path other than the current Hadoop default path.

  By default Cascading on Hadoop will assume any source or sink Tap using the `file://` URI scheme intends to read files from the local client filesystem (for example when using the `Lfs` Tap) where the Hadoop job jar is started. Subsequently Cascading will force any MapReduce jobs reading or writing to `file://` resources to run in Hadoop "standalone mode" so that the file can be read.

  To change this behavior, `HfsProps.setLocalModeScheme(java.util.Map, String)` to set a different scheme value, or to "none" to disable entirely for the case the file to be read is available on every Hadoop processing node in the exact same path.

  When using a MapReduce planner, Hfs can optionally combine multiple small files (or a series of small "blocks") into larger "splits". This reduces the number of resulting map tasks created by Hadoop and can improve application performance.

  This is enabled by calling `HfsProps.setUseCombinedInput(boolean)` to `true`. By default, merging or combining splits into large ones is disabled.

  Apache Tez planner does not require this setting, it is supported by default and enabled by the application manager.
  See Also:
    Serialized Form

- - ### Field Summary

    Fields
    Modifier and Type | Field and Description
    --- | ---
    `protected String` | `stringPath` Field stringPath

  - ### Constructor Summary

    Constructors
    Modifier | Constructor and Description
    --- | ---
    `protected` | `Hfs()`
    `protected` | `Hfs(Scheme<Configuration,RecordReader,OutputCollector,?,?> scheme)`
    `` | `Hfs(Scheme<Configuration,RecordReader,OutputCollector,?,?> scheme, String stringPath)` Constructor Hfs creates a new Hfs instance.
    `` | `Hfs(Scheme<Configuration,RecordReader,OutputCollector,?,?> scheme, String stringPath, SinkMode sinkMode)` Constructor Hfs creates a new Hfs instance.

  - ### Method Summary

    Methods
    Modifier and Type | Method and Description
    --- | ---
    `protected void` | `applySourceConfInitIdentifiers(FlowProcess<? extends Configuration> process, Configuration conf, String... fullIdentifiers)`
    `boolean` | `createResource(Configuration conf)`
    `boolean` | `deleteChildResource(Configuration conf, String childIdentifier)`
    `boolean` | `deleteChildResource(FlowProcess<? extends Configuration> flowProcess, String childIdentifier)`
    `boolean` | `deleteResource(Configuration conf)`
    `long` | `getBlockSize(Configuration conf)` Method getBlockSize returns the `blocksize` specified by the underlying file system for this resource.
    `long` | `getBlockSize(FlowProcess<? extends Configuration> flowProcess)` Method getBlockSize returns the `blocksize` specified by the underlying file system for this resource.
    `String[]` | `getChildIdentifiers(Configuration conf)`
    `String[]` | `getChildIdentifiers(Configuration conf, int depth, boolean fullyQualified)`
    `String[]` | `getChildIdentifiers(FlowProcess<? extends Configuration> flowProcess)`
    `String[]` | `getChildIdentifiers(FlowProcess<? extends Configuration> flowProcess, int depth, boolean fullyQualified)`
    `protected static boolean` | `getCombinedInputSafeMode(Configuration conf)`
    `protected FileSystem` | `getDefaultFileSystem(Configuration configuration)`
    `URI` | `getDefaultFileSystemURIScheme(Configuration configuration)` Method getDefaultFileSystemURIScheme returns the URI scheme for the default Hadoop FileSystem.
    `protected FileSystem` | `getFileSystem(Configuration configuration)`
    `String` | `getFullIdentifier(Configuration conf)`
    `String` | `getIdentifier()`
    `protected static String` | `getLocalModeScheme(Configuration conf, String defaultValue)`
    `long` | `getModifiedTime(Configuration conf)`
    `Path` | `getPath()`
    `int` | `getReplication(Configuration conf)` Method getReplication returns the `replication` specified by the underlying file system for this resource.
    `int` | `getReplication(FlowProcess<? extends Configuration> flowProcess)` Method getReplication returns the `replication` specified by the underlying file system for this resource.
    `long` | `getSize(Configuration conf)`
    `long` | `getSize(FlowProcess<? extends Configuration> flowProcess)`
    `static Path` | `getTempPath(Configuration conf)`
    `URI` | `getURIScheme(Configuration jobConf)`
    `protected static boolean` | `getUseCombinedInput(Configuration conf)`
    `boolean` | `isDirectory(Configuration conf)`
    `boolean` | `isDirectory(FlowProcess<? extends Configuration> flowProcess)`
    `protected String` | `makeTemporaryPathDirString(String name)`
    `protected URI` | `makeURIScheme(Configuration configuration)`
    `TupleEntryIterator` | `openForRead(FlowProcess<? extends Configuration> flowProcess, RecordReader input)`
    `TupleEntryCollector` | `openForWrite(FlowProcess<? extends Configuration> flowProcess, OutputCollector output)`
    `void` | `resetFileStatuses()` Method resetFileStatuses removes the status cache, if any.
    `boolean` | `resourceExists(Configuration conf)`
    `protected void` | `setStringPath(String stringPath)`
    `protected void` | `setUriScheme(URI uriScheme)`
    `void` | `sinkConfInit(FlowProcess<? extends Configuration> process, Configuration conf)`
    `void` | `sourceConfInit(FlowProcess<? extends Configuration> process, Configuration conf)`
    `protected void` | `sourceConfInitAddInputPath(Configuration conf, Path qualifiedPath)`
    `protected void` | `sourceConfInitComplete(FlowProcess<? extends Configuration> process, Configuration conf)`
    `protected static void` | `verifyNoDuplicates(Configuration conf)`

    - ### Methods inherited from class cascading.tap.Tap

      `commitResource, createResource, deleteResource, equals, flowConfInit, getConfigDef, getFullIdentifier, getModifiedTime, getNodeConfigDef, getScheme, getSinkFields, getSinkMode, getSourceFields, getStepConfigDef, getTrace, hasConfigDef, hashCode, hasNodeConfigDef, hasStepConfigDef, id, isKeep, isReplace, isSink, isSource, isTemporary, isUpdate, openForRead, openForWrite, outgoingScopeFor, prepareResourceForRead, prepareResourceForWrite, presentSinkFields, presentSourceFields, resolveIncomingOperationArgumentFields, resolveIncomingOperationPassThroughFields, resourceExists, retrieveSinkFields, retrieveSourceFields, rollbackResource, setScheme, taps, toString`

    - ### Methods inherited from class java.lang.Object

      `clone, finalize, getClass, notify, notifyAll, wait, wait, wait`

- - ### Field Detail

    - #### stringPath

          protected String stringPath

      Field stringPath

  - ### Constructor Detail

    - #### Hfs

          protected Hfs()

    - #### Hfs

          @ConstructorProperties(value="scheme")
          protected Hfs(Scheme<Configuration,RecordReader,OutputCollector,?,?> scheme)

    - #### Hfs

          @ConstructorProperties(value={"scheme","stringPath"})
          public Hfs(Scheme<Configuration,RecordReader,OutputCollector,?,?> scheme,
                                        String stringPath)

      Constructor Hfs creates a new Hfs instance.
      Parameters:
        `scheme` - of type Scheme
        `stringPath` - of type String

    - #### Hfs

          @ConstructorProperties(value={"scheme","stringPath","sinkMode"})
          public Hfs(Scheme<Configuration,RecordReader,OutputCollector,?,?> scheme,
                                        String stringPath,
                                        SinkMode sinkMode)

      Constructor Hfs creates a new Hfs instance.
      Parameters:
        `scheme` - of type Scheme
        `stringPath` - of type String
        `sinkMode` - of type SinkMode

  - ### Method Detail

    - #### getLocalModeScheme

          protected static String getLocalModeScheme(Configuration conf,
                                  String defaultValue)

    - #### getUseCombinedInput

          protected static boolean getUseCombinedInput(Configuration conf)

    - #### getCombinedInputSafeMode

          protected static boolean getCombinedInputSafeMode(Configuration conf)

    - #### setStringPath

          protected void setStringPath(String stringPath)

    - #### setUriScheme

          protected void setUriScheme(URI uriScheme)

    - #### getURIScheme

          public URI getURIScheme(Configuration jobConf)

    - #### makeURIScheme

          protected URI makeURIScheme(Configuration configuration)

    - #### getDefaultFileSystemURIScheme

          public URI getDefaultFileSystemURIScheme(Configuration configuration)

      Method getDefaultFileSystemURIScheme returns the URI scheme for the default Hadoop FileSystem.
      Parameters:
        `configuration` - of type JobConf
      Returns:
        URI

    - #### getDefaultFileSystem

          protected FileSystem getDefaultFileSystem(Configuration configuration)

    - #### getFileSystem

          protected FileSystem getFileSystem(Configuration configuration)

    - #### getIdentifier

          public String getIdentifier()

      Specified by:
        `getIdentifier` in class `Tap<Configuration,RecordReader,OutputCollector>`

    - #### getPath

          public Path getPath()

    - #### getFullIdentifier

          public String getFullIdentifier(Configuration conf)

      Overrides:
        `getFullIdentifier` in class `Tap<Configuration,RecordReader,OutputCollector>`

    - #### sourceConfInit

          public void sourceConfInit(FlowProcess<? extends Configuration> process,
                            Configuration conf)

      Overrides:
        `sourceConfInit` in class `Tap<Configuration,RecordReader,OutputCollector>`

    - #### verifyNoDuplicates

          protected static void verifyNoDuplicates(Configuration conf)

    - #### applySourceConfInitIdentifiers

          protected void applySourceConfInitIdentifiers(FlowProcess<? extends Configuration> process,
                                            Configuration conf,
                                            String... fullIdentifiers)

    - #### sourceConfInitAddInputPath

          protected void sourceConfInitAddInputPath(Configuration conf,
                                        Path qualifiedPath)

    - #### sourceConfInitComplete

          protected void sourceConfInitComplete(FlowProcess<? extends Configuration> process,
                                    Configuration conf)

    - #### sinkConfInit

          public void sinkConfInit(FlowProcess<? extends Configuration> process,
                          Configuration conf)

      Overrides:
        `sinkConfInit` in class `Tap<Configuration,RecordReader,OutputCollector>`

    - #### openForRead

          public TupleEntryIterator openForRead(FlowProcess<? extends Configuration> flowProcess,
                                       RecordReader input)
                                         throws IOException

      Specified by:
        `openForRead` in class `Tap<Configuration,RecordReader,OutputCollector>`
      Throws:
        `IOException`

    - #### openForWrite

          public TupleEntryCollector openForWrite(FlowProcess<? extends Configuration> flowProcess,
                                         OutputCollector output)
                                           throws IOException

      Specified by:
        `openForWrite` in class `Tap<Configuration,RecordReader,OutputCollector>`
      Throws:
        `IOException`

    - #### createResource

          public boolean createResource(Configuration conf)
                                 throws IOException

      Specified by:
        `createResource` in class `Tap<Configuration,RecordReader,OutputCollector>`
      Throws:
        `IOException`

    - #### deleteResource

          public boolean deleteResource(Configuration conf)
                                 throws IOException

      Specified by:
        `deleteResource` in class `Tap<Configuration,RecordReader,OutputCollector>`
      Throws:
        `IOException`

    - #### deleteChildResource

          public boolean deleteChildResource(FlowProcess<? extends Configuration> flowProcess,
                                    String childIdentifier)
                                      throws IOException

      Throws:
        `IOException`

    - #### deleteChildResource

          public boolean deleteChildResource(Configuration conf,
                                    String childIdentifier)
                                      throws IOException

      Throws:
        `IOException`

    - #### resourceExists

          public boolean resourceExists(Configuration conf)
                                 throws IOException

      Specified by:
        `resourceExists` in class `Tap<Configuration,RecordReader,OutputCollector>`
      Throws:
        `IOException`

    - #### isDirectory

          public boolean isDirectory(FlowProcess<? extends Configuration> flowProcess)
                              throws IOException

      Specified by:
        `isDirectory` in interface `FileType<Configuration>`
      Throws:
        `IOException`

    - #### isDirectory

          public boolean isDirectory(Configuration conf)
                              throws IOException

      Specified by:
        `isDirectory` in interface `FileType<Configuration>`
      Throws:
        `IOException`

    - #### getSize

          public long getSize(FlowProcess<? extends Configuration> flowProcess)
                       throws IOException

      Specified by:
        `getSize` in interface `FileType<Configuration>`
      Throws:
        `IOException`

    - #### getSize

          public long getSize(Configuration conf)
                       throws IOException

      Specified by:
        `getSize` in interface `FileType<Configuration>`
      Throws:
        `IOException`

    - #### getBlockSize

          public long getBlockSize(FlowProcess<? extends Configuration> flowProcess)
                            throws IOException

      Method getBlockSize returns the `blocksize` specified by the underlying file system for this resource.
      Parameters:
        `flowProcess` -
      Returns:
        long
      Throws:
        `IOException` - when

    - #### getBlockSize

          public long getBlockSize(Configuration conf)
                            throws IOException

      Method getBlockSize returns the `blocksize` specified by the underlying file system for this resource.
      Parameters:
        `conf` - of JobConf
      Returns:
        long
      Throws:
        `IOException` - when

    - #### getReplication

          public int getReplication(FlowProcess<? extends Configuration> flowProcess)
                             throws IOException

      Method getReplication returns the `replication` specified by the underlying file system for this resource.
      Parameters:
        `flowProcess` -
      Returns:
        int
      Throws:
        `IOException` - when

    - #### getReplication

          public int getReplication(Configuration conf)
                             throws IOException

      Method getReplication returns the `replication` specified by the underlying file system for this resource.
      Parameters:
        `conf` - of JobConf
      Returns:
        int
      Throws:
        `IOException` - when

    - #### getChildIdentifiers

          public String[] getChildIdentifiers(FlowProcess<? extends Configuration> flowProcess)
                                       throws IOException

      Specified by:
        `getChildIdentifiers` in interface `FileType<Configuration>`
      Throws:
        `IOException`

    - #### getChildIdentifiers

          public String[] getChildIdentifiers(Configuration conf)
                                       throws IOException

      Specified by:
        `getChildIdentifiers` in interface `FileType<Configuration>`
      Throws:
        `IOException`

    - #### getChildIdentifiers

          public String[] getChildIdentifiers(FlowProcess<? extends Configuration> flowProcess,
                                     int depth,
                                     boolean fullyQualified)
                                       throws IOException

      Specified by:
        `getChildIdentifiers` in interface `FileType<Configuration>`
      Throws:
        `IOException`

    - #### getChildIdentifiers

          public String[] getChildIdentifiers(Configuration conf,
                                     int depth,
                                     boolean fullyQualified)
                                       throws IOException

      Specified by:
        `getChildIdentifiers` in interface `FileType<Configuration>`
      Throws:
        `IOException`

    - #### getModifiedTime

          public long getModifiedTime(Configuration conf)
                               throws IOException

      Specified by:
        `getModifiedTime` in class `Tap<Configuration,RecordReader,OutputCollector>`
      Throws:
        `IOException`

    - #### getTempPath

          public static Path getTempPath(Configuration conf)

    - #### makeTemporaryPathDirString

          protected String makeTemporaryPathDirString(String name)

    - #### resetFileStatuses

          public void resetFileStatuses()

      Method resetFileStatuses removes the status cache, if any.

Copyright © 2007-2015 Concurrent, Inc. All Rights Reserved.