Overview | Package | Class | Tree | Deprecated | Index | Help
PREV CLASS NEXT CLASS | FRAMES NO FRAMES
SUMMARY: NESTED | FIELD | CONSTR | METHOD | DETAIL: FIELD | CONSTR | METHOD

## cascading.tap.hadoop
Class Hfs

    java.lang.Object
      cascading.tap.Tap<JobConf,RecordReader,OutputCollector>
          cascading.tap.hadoop.Hfs

All Implemented Interfaces:
  FlowElement, FileType<JobConf>, Serializable

Direct Known Subclasses:
  Dfs, Lfs, TempHfs

    public class Hfs

extends Tap<JobConf,RecordReader,OutputCollector>
implements FileType<JobConf>

Class Hfs is the base class for all Hadoop file system access. Hfs may only be used with the `HadoopFlowConnector` when creating Hadoop executable `Flow` instances.

Paths typically should point to a directory, where in turn all the "part" files immediately in that directory will be included. This is the practice Hadoop expects. Sub-directories are not included and typically result in a failure.

To include sub-directories, Hadoop supports "globing". Globing is a frustrating feature and is supported more robustly by `GlobHfs` and less so by Hfs.

Hfs will accept `/*` (wildcard) paths, but not all convenience methods like `getSize(org.apache.hadoop.mapred.JobConf)` will behave properly or reliably. Nor can the Hfs instance with a wildcard path be used as a sink to write data.

In those cases use GlobHfs since it is a sub-class of `MultiSourceTap`.

Optionally use `Dfs` or `Lfs` for resources specific to Hadoop Distributed file system or the Local file system, respectively. Using Hfs is the best practice when possible, Lfs and Dfs are conveniences.

Use the Hfs class if the 'kind' of resource is unknown at design time. To use, prefix a scheme to the 'stringPath'. Where `hdfs://...` will denote Dfs, and `file://...` will denote Lfs.

Call `setTemporaryDirectory(java.util.Map, String)` to use a different temporary file directory path other than the current Hadoop default path.

By default Cascading on Hadoop will assume any source or sink Tap using the `file://` URI scheme intends to read files from the local client filesystem (for example when using the `Lfs` Tap) where the Hadoop job jar is started, Tap so will force any MapReduce jobs reading or writing to `file://` resources to run in Hadoop "standalone mode" so that the file can be read.

To change this behavior, `HfsProps.setLocalModeScheme(java.util.Map, String)` to set a different scheme value, or to "none" to disable entirely for the case the file to be read is available on every Hadoop processing node in the exact same path.

Hfs can optionally combine multiple small files (or a series of small "blocks") into larger "splits". This reduces the number of resulting map tasks created by Hadoop and can improve application performance.

This is enabled by calling `HfsProps.setUseCombinedInput(boolean)` to `true`. By default, merging or combining splits into large ones is disabled.

See Also:
  Serialized Form

Field Summary
`protected String` | `stringPath` Field stringPath
`static String` | `TEMPORARY_DIRECTORY` Deprecated. see `HfsProps.TEMPORARY_DIRECTORY`

Constructor Summary
`protected` | `Hfs()`
`` | `Hfs(Fields fields, String stringPath)` Deprecated.
`` | `Hfs(Fields fields, String stringPath, boolean replace)` Deprecated.
`` | `Hfs(Fields fields, String stringPath, SinkMode sinkMode)` Deprecated.
`protected` | `Hfs(Scheme<JobConf,RecordReader,OutputCollector,?,?> scheme)`
`` | `Hfs(Scheme<JobConf,RecordReader,OutputCollector,?,?> scheme, String stringPath)` Constructor Hfs creates a new Hfs instance.
`` | `Hfs(Scheme<JobConf,RecordReader,OutputCollector,?,?> scheme, String stringPath, boolean replace)` Deprecated.
`` | `Hfs(Scheme<JobConf,RecordReader,OutputCollector,?,?> scheme, String stringPath, SinkMode sinkMode)` Constructor Hfs creates a new Hfs instance.

Method Summary
`protected void` | `applySourceConfInitIdentifiers(FlowProcess<JobConf> process, JobConf conf, String... fullIdentifiers)`
`boolean` | `createResource(JobConf conf)` Method createResource creates the underlying resource.
`boolean` | `deleteResource(JobConf conf)` Method deleteResource deletes the resource represented by this instance.
`long` | `getBlockSize(JobConf conf)` Method getBlockSize returns the `blocksize` specified by the underlying file system for this resource.
`String[]` | `getChildIdentifiers(JobConf conf)` Method getChildIdentifiers returns an array of child identifiers if this resource is a directory.
`protected FileSystem` | `getDefaultFileSystem(JobConf jobConf)`
`URI` | `getDefaultFileSystemURIScheme(JobConf jobConf)` Method getDefaultFileSystemURIScheme returns the URI scheme for the default Hadoop FileSystem.
`protected FileSystem` | `getFileSystem(JobConf jobConf)`
`String` | `getFullIdentifier(JobConf conf)` Method getFullIdentifier returns a fully qualified resource identifier.
`String` | `getIdentifier()` Method getIdentifier returns a String representing the resource this Tap instance represents.
`protected static String` | `getLocalModeScheme(JobConf conf, String defaultValue)`
`long` | `getModifiedTime(JobConf conf)` Method getModifiedTime returns the date this resource was last modified.
`Path` | `getPath()`
`int` | `getReplication(JobConf conf)` Method getReplication returns the `replication` specified by the underlying file system for this resource.
`long` | `getSize(JobConf conf)` Method getSize returns the size of the file referenced by this tap.
`static String` | `getTemporaryDirectory(Map<Object,Object> properties)` Deprecated. see `HfsProps`
`static Path` | `getTempPath(JobConf conf)`
`URI` | `getURIScheme(JobConf jobConf)`
`protected static boolean` | `getUseCombinedInput(JobConf conf)`
`boolean` | `isDirectory(JobConf conf)` Method isDirectory returns true if the underlying resource represents a directory or folder instead of an individual file.
`protected String` | `makeTemporaryPathDirString(String name)`
`protected URI` | `makeURIScheme(JobConf jobConf)`
`TupleEntryIterator` | `openForRead(FlowProcess<JobConf> flowProcess, RecordReader input)` Method openForRead opens the resource represented by this Tap instance for reading.
`TupleEntryCollector` | `openForWrite(FlowProcess<JobConf> flowProcess, OutputCollector output)` Method openForWrite opens the resource represented by this Tap instance for writing.
`boolean` | `resourceExists(JobConf conf)` Method resourceExists returns true if the path represented by this instance exists.
`protected void` | `setStringPath(String stringPath)`
`static void` | `setTemporaryDirectory(Map<Object,Object> properties, String tempDir)` Deprecated. see `HfsProps`
`protected void` | `setUriScheme(URI uriScheme)`
`void` | `sinkConfInit(FlowProcess<JobConf> process, JobConf conf)` Method sinkConfInit initializes this instance as a sink.
`void` | `sourceConfInit(FlowProcess<JobConf> process, JobConf conf)` Method sourceConfInit initializes this instance as a source.
`protected void` | `sourceConfInitAddInputPath(JobConf conf, Path qualifiedPath)`
`protected void` | `sourceConfInitComplete(FlowProcess<JobConf> process, JobConf conf)`
`protected static void` | `verifyNoDuplicates(JobConf conf)`

Methods inherited from class cascading.tap.Tap
`commitResource, createResource, deleteResource, equals, flowConfInit, getConfigDef, getFullIdentifier, getModifiedTime, getScheme, getSinkFields, getSinkMode, getSourceFields, getStepConfigDef, getTrace, hasConfigDef, hashCode, hasStepConfigDef, id, isEquivalentTo, isKeep, isReplace, isSink, isSource, isTemporary, isUpdate, openForRead, openForWrite, outgoingScopeFor, presentSinkFields, presentSourceFields, resolveIncomingOperationArgumentFields, resolveIncomingOperationPassThroughFields, resourceExists, retrieveSinkFields, retrieveSourceFields, rollbackResource, setScheme, taps, toString`

Methods inherited from class java.lang.Object
`clone, finalize, getClass, notify, notifyAll, wait, wait, wait`

Field Detail

### TEMPORARY_DIRECTORY

    @Deprecated
    public static final String TEMPORARY_DIRECTORY

  Deprecated. see `HfsProps.TEMPORARY_DIRECTORY`
  Field TEMPORARY_DIRECTORY

  See Also:
    Constant Field Values

### stringPath

    protected String stringPath

  Field stringPath

Constructor Detail

### Hfs

    protected Hfs()

### Hfs

    @ConstructorProperties(value="scheme")
    protected Hfs(Scheme<JobConf,RecordReader,OutputCollector,?,?> scheme)

### Hfs

    @Deprecated
    @ConstructorProperties(value={"fields","stringPath"})
    public Hfs(Fields fields,
                                                     String stringPath)

  Deprecated.

  Constructor Hfs creates a new Hfs instance.

  Parameters:
    `fields` - of type Fields
    `stringPath` - of type String

### Hfs

    @Deprecated
    @ConstructorProperties(value={"fields","stringPath","replace"})
    public Hfs(Fields fields,
                                                     String stringPath,
                                                     boolean replace)

  Deprecated.

  Constructor Hfs creates a new Hfs instance.

  Parameters:
    `fields` - of type Fields
    `stringPath` - of type String
    `replace` - of type boolean

### Hfs

    @Deprecated
    @ConstructorProperties(value={"fields","stringPath","sinkMode"})
    public Hfs(Fields fields,
                                                     String stringPath,
                                                     SinkMode sinkMode)

  Deprecated.

  Constructor Hfs creates a new Hfs instance.

  Parameters:
    `fields` - of type Fields
    `stringPath` - of type String
    `sinkMode` - of type SinkMode

### Hfs

    @ConstructorProperties(value={"scheme","stringPath"})
    public Hfs(Scheme<JobConf,RecordReader,OutputCollector,?,?> scheme,
                                          String stringPath)

  Constructor Hfs creates a new Hfs instance.

  Parameters:
    `scheme` - of type Scheme
    `stringPath` - of type String

### Hfs

    @Deprecated
    @ConstructorProperties(value={"scheme","stringPath","replace"})
    public Hfs(Scheme<JobConf,RecordReader,OutputCollector,?,?> scheme,
                                                     String stringPath,
                                                     boolean replace)

  Deprecated.

  Constructor Hfs creates a new Hfs instance.

  Parameters:
    `scheme` - of type Scheme
    `stringPath` - of type String
    `replace` - of type boolean

### Hfs

    @ConstructorProperties(value={"scheme","stringPath","sinkMode"})
    public Hfs(Scheme<JobConf,RecordReader,OutputCollector,?,?> scheme,
                                          String stringPath,
                                          SinkMode sinkMode)

  Constructor Hfs creates a new Hfs instance.

  Parameters:
    `scheme` - of type Scheme
    `stringPath` - of type String
    `sinkMode` - of type SinkMode

Method Detail

### setTemporaryDirectory

    @Deprecated
    public static void setTemporaryDirectory(Map<Object,Object> properties,
                                                        String tempDir)

  Deprecated. see `HfsProps`

  Method setTemporaryDirectory sets the temporary directory on the given properties object.

  Parameters:
    `properties` - of type Map
    `tempDir` - of type String

### getTemporaryDirectory

    @Deprecated
    public static String getTemporaryDirectory(Map<Object,Object> properties)

  Deprecated. see `HfsProps`

  Method getTemporaryDirectory returns the configured temporary directory from the given properties object.

  Parameters:
    `properties` - of type Map
  Returns:
    a String or null if not set

### getLocalModeScheme

    protected static String getLocalModeScheme(JobConf conf,
                                               String defaultValue)

### getUseCombinedInput

    protected static boolean getUseCombinedInput(JobConf conf)

### setStringPath

    protected void setStringPath(String stringPath)

### setUriScheme

    protected void setUriScheme(URI uriScheme)

### getURIScheme

    public URI getURIScheme(JobConf jobConf)

### makeURIScheme

    protected URI makeURIScheme(JobConf jobConf)

### getDefaultFileSystemURIScheme

    public URI getDefaultFileSystemURIScheme(JobConf jobConf)

  Method getDefaultFileSystemURIScheme returns the URI scheme for the default Hadoop FileSystem.

  Parameters:
    `jobConf` - of type JobConf
  Returns:
    URI

### getDefaultFileSystem

    protected FileSystem getDefaultFileSystem(JobConf jobConf)

### getFileSystem

    protected FileSystem getFileSystem(JobConf jobConf)

### getIdentifier

    public String getIdentifier()

  Description copied from class: `Tap`
  Method getIdentifier returns a String representing the resource this Tap instance represents.

  Often, if the tap accesses a filesystem, the identifier is nothing more than the path to the file or directory. In other cases it may be a an URL or URI representing a connection string or remote resource.

  Any two Tap instances having the same value for the identifier are considered equal.

  Specified by:
    `getIdentifier` in class `Tap<JobConf,RecordReader,OutputCollector>`
  Returns:
    String

### getPath

    public Path getPath()

### getFullIdentifier

    public String getFullIdentifier(JobConf conf)

  Description copied from class: `Tap`
  Method getFullIdentifier returns a fully qualified resource identifier.

  Overrides:
    `getFullIdentifier` in class `Tap<JobConf,RecordReader,OutputCollector>`
  Parameters:
    `conf` - of type Config
  Returns:
    String

### sourceConfInit

    public void sourceConfInit(FlowProcess<JobConf> process,
                               JobConf conf)

  Description copied from class: `Tap`
  Method sourceConfInit initializes this instance as a source.

  This method maybe called more than once if this Tap instance is used outside the scope of a `Flow` instance or if it participates in multiple times in a given Flow or across different Flows in a `Cascade`.

  In the context of a Flow, it will be called after `FlowListener.onStarting(cascading.flow.Flow)`

  Note that no resources or services should be modified by this method.

  Overrides:
    `sourceConfInit` in class `Tap<JobConf,RecordReader,OutputCollector>`
  Parameters:
    `process` - of type FlowProcess
    `conf` - of type Config

### verifyNoDuplicates

    protected static void verifyNoDuplicates(JobConf conf)

### applySourceConfInitIdentifiers

    protected void applySourceConfInitIdentifiers(FlowProcess<JobConf> process,
                                                  JobConf conf,
                                                  String... fullIdentifiers)

### sourceConfInitAddInputPath

    protected void sourceConfInitAddInputPath(JobConf conf,
                                              Path qualifiedPath)

### sourceConfInitComplete

    protected void sourceConfInitComplete(FlowProcess<JobConf> process,
                                          JobConf conf)

### sinkConfInit

    public void sinkConfInit(FlowProcess<JobConf> process,
                             JobConf conf)

  Description copied from class: `Tap`
  Method sinkConfInit initializes this instance as a sink.

  This method maybe called more than once if this Tap instance is used outside the scope of a `Flow` instance or if it participates in multiple times in a given Flow or across different Flows in a `Cascade`.

  Note this method will be called in context of this Tap being used as a traditional 'sink' and as a 'trap'.

  In the context of a Flow, it will be called after `FlowListener.onStarting(cascading.flow.Flow)`

  Note that no resources or services should be modified by this method. If this Tap instance returns true for `Tap.isReplace()`, then `Tap.deleteResource(Object)` will be called by the parent Flow.

  Overrides:
    `sinkConfInit` in class `Tap<JobConf,RecordReader,OutputCollector>`
  Parameters:
    `process` - of type FlowProcess
    `conf` - of type Config

### openForRead

    public TupleEntryIterator openForRead(FlowProcess<JobConf> flowProcess,
                                          RecordReader input)
                                   throws IOException

  Description copied from class: `Tap`
  Method openForRead opens the resource represented by this Tap instance for reading.

  `input` value may be null, if so, sub-classes must inquire with the underlying `Scheme` via `Scheme.sourceConfInit(cascading.flow.FlowProcess, Tap, Object)` to get the proper input type and instantiate it before calling `super.openForRead()`.

  Note the returned iterator will return the same instance of `TupleEntry` on every call, thus a copy must be made of either the TupleEntry or the underlying `Tuple` instance if they are to be stored in a Collection.

  Specified by:
    `openForRead` in class `Tap<JobConf,RecordReader,OutputCollector>`
  Parameters:
    `flowProcess` - of type FlowProcess
    `input` - of type Input
  Returns:
    TupleEntryIterator
  Throws:
    `IOException` - when the resource cannot be opened

### openForWrite

    public TupleEntryCollector openForWrite(FlowProcess<JobConf> flowProcess,
                                            OutputCollector output)
                                     throws IOException

  Description copied from class: `Tap`
  Method openForWrite opens the resource represented by this Tap instance for writing.

  This method is used internally and does not honor the `SinkMode` setting. If SinkMode is `SinkMode.REPLACE`, this call may fail. See `Tap.openForWrite(cascading.flow.FlowProcess)`.

  `output` value may be null, if so, sub-classes must inquire with the underlying `Scheme` via `Scheme.sinkConfInit(cascading.flow.FlowProcess, Tap, Object)` to get the proper output type and instantiate it before calling `super.openForWrite()`.

  Specified by:
    `openForWrite` in class `Tap<JobConf,RecordReader,OutputCollector>`
  Parameters:
    `flowProcess` - of type FlowProcess
    `output` - of type Output
  Returns:
    TupleEntryCollector
  Throws:
    `IOException` - when the resource cannot be opened

### createResource

    public boolean createResource(JobConf conf)
                           throws IOException

  Description copied from class: `Tap`
  Method createResource creates the underlying resource.

  Specified by:
    `createResource` in class `Tap<JobConf,RecordReader,OutputCollector>`
  Parameters:
    `conf` - of type Config
  Returns:
    boolean
  Throws:
    `IOException` - when there is an error making directories

### deleteResource

    public boolean deleteResource(JobConf conf)
                           throws IOException

  Description copied from class: `Tap`
  Method deleteResource deletes the resource represented by this instance.

  Specified by:
    `deleteResource` in class `Tap<JobConf,RecordReader,OutputCollector>`
  Parameters:
    `conf` - of type Config
  Returns:
    boolean
  Throws:
    `IOException` - when the resource cannot be deleted

### resourceExists

    public boolean resourceExists(JobConf conf)
                           throws IOException

  Description copied from class: `Tap`
  Method resourceExists returns true if the path represented by this instance exists.

  Specified by:
    `resourceExists` in class `Tap<JobConf,RecordReader,OutputCollector>`
  Parameters:
    `conf` - of type Config
  Returns:
    true if the underlying resource already exists
  Throws:
    `IOException` - when the status cannot be determined

### isDirectory

    public boolean isDirectory(JobConf conf)
                        throws IOException

  Description copied from interface: `FileType`
  Method isDirectory returns true if the underlying resource represents a directory or folder instead of an individual file.

  Specified by:
    `isDirectory` in interface `FileType<JobConf>`
  Parameters:
    `conf` - of JobConf
  Returns:
    boolean
  Throws:
    `IOException`

### getSize

    public long getSize(JobConf conf)
                 throws IOException

  Description copied from interface: `FileType`
  Method getSize returns the size of the file referenced by this tap.

  Specified by:
    `getSize` in interface `FileType<JobConf>`
  Parameters:
    `conf` - of type Config
  Returns:
    The size of the file reference by this tap.
  Throws:
    `IOException`

### getBlockSize

    public long getBlockSize(JobConf conf)
                      throws IOException

  Method getBlockSize returns the `blocksize` specified by the underlying file system for this resource.

  Parameters:
    `conf` - of JobConf
  Returns:
    long
  Throws:
    `IOException` - when

### getReplication

    public int getReplication(JobConf conf)
                       throws IOException

  Method getReplication returns the `replication` specified by the underlying file system for this resource.

  Parameters:
    `conf` - of JobConf
  Returns:
    int
  Throws:
    `IOException` - when

### getChildIdentifiers

    public String[] getChildIdentifiers(JobConf conf)
                                 throws IOException

  Description copied from interface: `FileType`
  Method getChildIdentifiers returns an array of child identifiers if this resource is a directory.

  This method will skip Hadoop log directories (`_log`).

  Specified by:
    `getChildIdentifiers` in interface `FileType<JobConf>`
  Parameters:
    `conf` - of JobConf
  Returns:
    String[]
  Throws:
    `IOException`

### getModifiedTime

    public long getModifiedTime(JobConf conf)
                         throws IOException

  Description copied from class: `Tap`
  Method getModifiedTime returns the date this resource was last modified.

  Specified by:
    `getModifiedTime` in class `Tap<JobConf,RecordReader,OutputCollector>`
  Parameters:
    `conf` - of type Config
  Returns:
    The date this resource was last modified.
  Throws:
    `IOException`

### getTempPath

    public static Path getTempPath(JobConf conf)

### makeTemporaryPathDirString

    protected String makeTemporaryPathDirString(String name)

Overview | Package | Class | Tree | Deprecated | Index | Help
PREV CLASS NEXT CLASS | FRAMES NO FRAMES
SUMMARY: NESTED | FIELD | CONSTR | METHOD | DETAIL: FIELD | CONSTR | METHOD

Copyright © 2007-2013 Concurrent, Inc. All Rights Reserved.