cascading.flow.planner

## Class BaseFlowStep<Config>

- java.lang.Object
- - cascading.flow.planner.BaseFlowStep<Config>

- All Implemented Interfaces:
    FlowStep<Config>, ProcessModel, ProcessLogger, Serializable
  Direct Known Subclasses:
    Hadoop2TezFlowStep, HadoopFlowStep, LocalFlowStep, ProcessFlowStep

      public abstract class BaseFlowStep<Config>
      extends Object
      implements FlowStep<Config>, ProcessLogger, Serializable

  Class FlowStep is an internal representation of a given Job to be executed on a remote cluster. During planning, pipe assemblies are broken down into "steps" and encapsulated in this class.

  FlowSteps are submitted in order of dependency. If two or more steps do not share the same dependencies and all can be scheduled simultaneously, the `getSubmitPriority()` value determines the order in which all steps will be submitted for execution. The default submit priority is 5.

  This class is for internal use, there are no stable public methods.
  See Also:
    Serialized Form

- - ### Field Summary

    Fields
    Modifier and Type | Field and Description
    --- | ---
    `protected FlowStepStats` | `flowStepStats`
    `protected Map<Tap,Set<String>>` | `sinks` Field sink
    `protected Map<Tap,Set<String>>` | `sources` Field sources
    `protected Tap` | `tempSink` Field tempSink

    - ### Fields inherited from interface cascading.flow.FlowStep

      `CASCADING_FLOW_STEP_ID`

    - ### Fields inherited from interface cascading.util.ProcessLogger

      `NULL`

  - ### Constructor Summary

    Constructors
    Modifier | Constructor and Description
    --- | ---
    `protected` | `BaseFlowStep(ElementGraph elementStepGraph, FlowNodeGraph flowNodeGraph)`
    `protected` | `BaseFlowStep(ElementGraph elementStepGraph, FlowNodeGraph flowNodeGraph, Map<String,String> flowStepDescriptor)`
    `protected` | `BaseFlowStep(String name, int ordinal)`
    `protected` | `BaseFlowStep(String name, int ordinal, FlowNodeGraph flowNodeGraph, Map<String,String> flowStepDescriptor)`
    `protected` | `BaseFlowStep(String name, int ordinal, Map<String,String> flowStepDescriptor)`

  - ### Method Summary

    Methods
    Modifier and Type | Method and Description
    --- | ---
    `protected void` | `addAllGroups()`
    `void` | `addGroup(Group group)`
    `void` | `addGroups(Collection<Group> groups)`
    `void` | `addListener(FlowStepListener flowStepListener)` Method addListener registers the given `FlowStepListener` with this instance.
    `void` | `addProcessAnnotation(Enum annotation)`
    `void` | `addProcessAnnotation(String key, String value)`
    `void` | `addSink(String name, Tap sink)`
    `protected static void` | `addSinks(BaseFlowStep flowStep, ElementGraph elementGraph, Set<Tap> sinks)`
    `void` | `addSource(String name, Tap source)`
    `protected static void` | `addSources(BaseFlowStep flowStep, ElementGraph elementGraph, Set<Tap> sources)`
    `void` | `clean()`
    `abstract void` | `clean(Config config)`
    `protected Throwable` | `commitSinks()`
    `protected void` | `configure()`
    `boolean` | `containsPipeNamed(String pipeName)` Returns true if this FlowStep contains a pipe/branch with the given name.
    `protected ClientState` | `createClientState(FlowProcess flowProcess)`
    `protected abstract FlowStepJob` | `createFlowStepJob(ClientState clientState, FlowProcess<Config> flowProcess, Config initializedStepConfig)`
    `abstract Config` | `createInitializedConfig(FlowProcess<Config> flowProcess, Config parentConfig)` Public for testing.
    `boolean` | `equals(Object object)`
    `protected void` | `fireOnCompleted()`
    `protected void` | `fireOnRunning()`
    `protected void` | `fireOnStarting()`
    `protected void` | `fireOnStopping()`
    `protected void` | `fireOnThrowable(Throwable throwable)`
    `Set<Tap>` | `getAllAccumulatedSources()`
    `Collection<Operation>` | `getAllOperations()`
    `protected EnumMultiMap` | `getAnnotations()`
    `Config` | `getConfig()` Method getConfig returns the current initialized configuration.
    `Map<Object,Object>` | `getConfigAsProperties()` Method getConfigAsProperties converts the internal configuration object into a `Map` of key value pairs.
    `FlowStepJob<Config>` | `getCreateFlowStepJob(FlowProcess<Config> flowProcess, Config parentConfig)`
    `ElementGraph` | `getElementGraph()`
    `Flow<Config>` | `getFlow()`
    `String` | `getFlowID()`
    `String` | `getFlowName()` Method getParentFlowName returns the parentFlowName of this FlowStep object.
    `FlowNodeGraph` | `getFlowNodeGraph()`
    `Map<String,String>` | `getFlowStepDescriptor()` Returns an immutable map of properties giving more details about the FlowStep object.
    `FlowStepJob<Config>` | `getFlowStepJob()`
    `FlowStepStats` | `getFlowStepStats()`
    `Group` | `getGroup()`
    `Collection<Group>` | `getGroups()`
    `String` | `getID()` Method getId returns the id of this FlowStep object.
    `String` | `getName()` Method getName returns the name of this FlowStep object.
    `FlowElement` | `getNextFlowElement(Scope scope)`
    `Scope` | `getNextScope(FlowElement flowElement)` Method getNextScope returns the next Scope instance in the graph.
    `protected String` | `getNodeDisplayName(FlowNode flowNode, int idLength)`
    `int` | `getNumFlowNodes()`
    `int` | `getOrdinal()`
    `Set<Scope>` | `getPreviousScopes(FlowElement flowElement)` Method getPreviousScopes returns the previous Scope instances.
    `Map<String,String>` | `getProcessAnnotations()` Method getProcessAnnotations returns an immutable Map of platform specific annotations or meta-data that describe the current model type.
    `Tap` | `getSink()`
    `Set<FlowElement>` | `getSinkElements()`
    `Set<String>` | `getSinkName(Tap sink)`
    `Set<Tap>` | `getSinkTaps()`
    `Tap` | `getSinkWith(String identifier)`
    `Set<FlowElement>` | `getSourceElements()`
    `Set<String>` | `getSourceName(Tap source)`
    `Set<Tap>` | `getSourceTaps()`
    `Tap` | `getSourceWith(String identifier)`
    `String` | `getStepDisplayName()` Method getStepDisplayName returns the stepDisplayName of this FlowStep object.
    `protected String` | `getStepDisplayName(int idLength)`
    `int` | `getSubmitPriority()` Method getSubmitPriority returns the submitPriority of this FlowStep object.
    `Tap` | `getTrap(String name)`
    `Map<String,Tap>` | `getTrapMap()`
    `Set<Tap>` | `getTraps()`
    `int` | `hashCode()`
    `boolean` | `hasListeners()` Method hasListeners returns true if `FlowStepListener` instances have been registered.
    `protected void` | `initConfFromNodeConfigDef(ElementGraph nodeElementGraph, ConfigDef.Setter setter)`
    `protected void` | `initConfFromStepConfigDef(ConfigDef.Setter setter)`
    `boolean` | `isDebugEnabled()`
    `boolean` | `isInfoEnabled()`
    `void` | `logDebug(String message, Object... arguments)`
    `void` | `logError(String message, Object... arguments)`
    `void` | `logError(String message, Throwable throwable)`
    `void` | `logInfo(String message, Object... arguments)`
    `void` | `logWarn(String message)`
    `void` | `logWarn(String message, Object... arguments)`
    `void` | `logWarn(String message, Throwable throwable)`
    `protected Throwable` | `prepareResources()`
    `boolean` | `removeListener(FlowStepListener flowStepListener)` Method removeListener removes the given flowStepListener from this instance.
    `protected Throwable` | `rollbackSinks()`
    `protected void` | `setConfig(Config flowStepConf)` Set the initialized flowStepConf Config instance
    `void` | `setFlow(Flow<Config> flow)`
    `protected void` | `setFlowName(String flowName)`
    `protected void` | `setFlowStepDescriptor(Map<String,String> flowStepDescriptor)`
    `void` | `setFlowStepStats(FlowStepStats flowStepStats)`
    `void` | `setName(String name)`
    `void` | `setOrdinal(int ordinal)`
    `void` | `setSubmitPriority(int submitPriority)` Method setSubmitPriority sets the submitPriority of this FlowStep object.
    `String` | `toString()`

    - ### Methods inherited from class java.lang.Object

      `clone, finalize, getClass, notify, notifyAll, wait, wait, wait`

- - ### Field Detail

    - #### sources

          protected final Map<Tap,Set<String>> sources

      Field sources

    - #### sinks

          protected final Map<Tap,Set<String>> sinks

      Field sink

    - #### tempSink

          protected Tap tempSink

      Field tempSink

    - #### flowStepStats

          protected transient FlowStepStats flowStepStats

  - ### Constructor Detail

    - #### BaseFlowStep

          protected BaseFlowStep(String name,
                      int ordinal)

    - #### BaseFlowStep

          protected BaseFlowStep(String name,
                      int ordinal,
                      Map<String,String> flowStepDescriptor)

    - #### BaseFlowStep

          protected BaseFlowStep(String name,
                      int ordinal,
                      FlowNodeGraph flowNodeGraph,
                      Map<String,String> flowStepDescriptor)

    - #### BaseFlowStep

          protected BaseFlowStep(ElementGraph elementStepGraph,
                      FlowNodeGraph flowNodeGraph)

    - #### BaseFlowStep

          protected BaseFlowStep(ElementGraph elementStepGraph,
                      FlowNodeGraph flowNodeGraph,
                      Map<String,String> flowStepDescriptor)

  - ### Method Detail

    - #### configure

          protected void configure()

    - #### addAllGroups

          protected void addAllGroups()

    - #### getID

          public String getID()

      Description copied from interface: `FlowStep`
      Method getId returns the id of this FlowStep object.
      Specified by:
        `getID` in interface `FlowStep<Config>`
      Specified by:
        `getID` in interface `ProcessModel`
      Returns:
        the id (type int) of this FlowStep object.

    - #### setOrdinal

          public void setOrdinal(int ordinal)

    - #### getOrdinal

          public int getOrdinal()

      Specified by:
        `getOrdinal` in interface `FlowStep<Config>`
      Specified by:
        `getOrdinal` in interface `ProcessModel`

    - #### getName

          public String getName()

      Description copied from interface: `FlowStep`
      Method getName returns the name of this FlowStep object.
      Specified by:
        `getName` in interface `FlowStep<Config>`
      Specified by:
        `getName` in interface `ProcessModel`
      Returns:
        the name (type String) of this FlowStep object.

    - #### setName

          public void setName(String name)

    - #### getFlowStepDescriptor

          public Map<String,String> getFlowStepDescriptor()

      Description copied from interface: `FlowStep`
      Returns an immutable map of properties giving more details about the FlowStep object.

      FlowStep descriptions provide meta-data to monitoring systems describing the workload a given FlowStep represents. For known description types, see `FlowStepDescriptors`.
      Specified by:
        `getFlowStepDescriptor` in interface `FlowStep<Config>`
      Returns:
        Map

    - #### setFlowStepDescriptor

          protected void setFlowStepDescriptor(Map<String,String> flowStepDescriptor)

    - #### getProcessAnnotations

          public Map<String,String> getProcessAnnotations()

      Description copied from interface: `ProcessModel`
      Method getProcessAnnotations returns an immutable Map of platform specific annotations or meta-data that describe the current model type.

      Use `ProcessModel.addProcessAnnotation(Enum)` or `ProcessModel.addProcessAnnotation(String, String)` to insert annotations or meta-data.
      Specified by:
        `getProcessAnnotations` in interface `ProcessModel`
      Returns:
        an immutable Map

    - #### addProcessAnnotation

          public void addProcessAnnotation(Enum annotation)

      Specified by:
        `addProcessAnnotation` in interface `ProcessModel`

    - #### addProcessAnnotation

          public void addProcessAnnotation(String key,
                                  String value)

      Specified by:
        `addProcessAnnotation` in interface `ProcessModel`

    - #### setFlow

          public void setFlow(Flow<Config> flow)

    - #### getFlow

          public Flow<Config> getFlow()

      Specified by:
        `getFlow` in interface `FlowStep<Config>`

    - #### getFlowID

          public String getFlowID()

      Specified by:
        `getFlowID` in interface `FlowStep<Config>`

    - #### getFlowName

          public String getFlowName()

      Description copied from interface: `FlowStep`
      Method getParentFlowName returns the parentFlowName of this FlowStep object.
      Specified by:
        `getFlowName` in interface `FlowStep<Config>`
      Returns:
        the parentFlowName (type Flow) of this FlowStep object.

    - #### setFlowName

          protected void setFlowName(String flowName)

    - #### getConfig

          public Config getConfig()

      Description copied from interface: `FlowStep`
      Method getConfig returns the current initialized configuration.

      The returned configuration is mutable and may be changed prior to this step being started or submitted.
      Specified by:
        `getConfig` in interface `FlowStep<Config>`
      Returns:
        the current initialized configuration

    - #### getConfigAsProperties

          public Map<Object,Object> getConfigAsProperties()

      Description copied from interface: `FlowStep`
      Method getConfigAsProperties converts the internal configuration object into a `Map` of key value pairs.
      Specified by:
        `getConfigAsProperties` in interface `FlowStep<Config>`
      Returns:
        a Map of key/value pairs, may return an empty collection if unsupported

    - #### setConfig

          protected void setConfig(Config flowStepConf)

      Set the initialized flowStepConf Config instance
      Parameters:
        `flowStepConf` - of type Config

    - #### getStepDisplayName

          public String getStepDisplayName()

      Description copied from interface: `FlowStep`
      Method getStepDisplayName returns the stepDisplayName of this FlowStep object.
      Specified by:
        `getStepDisplayName` in interface `FlowStep<Config>`
      Returns:
        the stepName (type String) of this FlowStep object.

    - #### getStepDisplayName

          protected String getStepDisplayName(int idLength)

    - #### getNodeDisplayName

          protected String getNodeDisplayName(FlowNode flowNode,
                                  int idLength)

    - #### getSubmitPriority

          public int getSubmitPriority()

      Description copied from interface: `FlowStep`
      Method getSubmitPriority returns the submitPriority of this FlowStep object.

      10 is lowest, 1 is the highest, 5 is the default.
      Specified by:
        `getSubmitPriority` in interface `FlowStep<Config>`
      Specified by:
        `getSubmitPriority` in interface `ProcessModel`
      Returns:
        the submitPriority (type int) of this FlowStep object.

    - #### setSubmitPriority

          public void setSubmitPriority(int submitPriority)

      Description copied from interface: `FlowStep`
      Method setSubmitPriority sets the submitPriority of this FlowStep object.

      10 is lowest, 1 is the highest, 5 is the default.
      Specified by:
        `setSubmitPriority` in interface `FlowStep<Config>`
      Parameters:
        `submitPriority` - the submitPriority of this FlowStep object.

    - #### setFlowStepStats

          public void setFlowStepStats(FlowStepStats flowStepStats)

      Specified by:
        `setFlowStepStats` in interface `FlowStep<Config>`

    - #### getFlowStepStats

          public FlowStepStats getFlowStepStats()

      Specified by:
        `getFlowStepStats` in interface `FlowStep<Config>`

    - #### getElementGraph

          public ElementGraph getElementGraph()

      Specified by:
        `getElementGraph` in interface `ProcessModel`

    - #### getAnnotations

          protected EnumMultiMap getAnnotations()

    - #### getFlowNodeGraph

          public FlowNodeGraph getFlowNodeGraph()

      Specified by:
        `getFlowNodeGraph` in interface `FlowStep<Config>`

    - #### getNumFlowNodes

          public int getNumFlowNodes()

      Specified by:
        `getNumFlowNodes` in interface `FlowStep<Config>`

    - #### getSourceElements

          public Set<FlowElement> getSourceElements()

      Specified by:
        `getSourceElements` in interface `ProcessModel`

    - #### getSinkElements

          public Set<FlowElement> getSinkElements()

      Specified by:
        `getSinkElements` in interface `ProcessModel`

    - #### getGroup

          public Group getGroup()

      Specified by:
        `getGroup` in interface `FlowStep<Config>`

    - #### getGroups

          public Collection<Group> getGroups()

      Specified by:
        `getGroups` in interface `ProcessModel`

    - #### addGroups

          public void addGroups(Collection<Group> groups)

    - #### addGroup

          public void addGroup(Group group)

    - #### getAllAccumulatedSources

          public Set<Tap> getAllAccumulatedSources()

    - #### addSource

          public void addSource(String name,
                       Tap source)

    - #### addSink

          public void addSink(String name,
                     Tap sink)

    - #### getSourceTaps

          public Set<Tap> getSourceTaps()

      Specified by:
        `getSourceTaps` in interface `ProcessModel`

    - #### getSinkTaps

          public Set<Tap> getSinkTaps()

      Specified by:
        `getSinkTaps` in interface `ProcessModel`

    - #### getSink

          public Tap getSink()

      Specified by:
        `getSink` in interface `FlowStep<Config>`

    - #### getSourceName

          public Set<String> getSourceName(Tap source)

      Specified by:
        `getSourceName` in interface `FlowStep<Config>`

    - #### getSinkName

          public Set<String> getSinkName(Tap sink)

      Specified by:
        `getSinkName` in interface `FlowStep<Config>`

    - #### getSourceWith

          public Tap getSourceWith(String identifier)

      Specified by:
        `getSourceWith` in interface `FlowStep<Config>`

    - #### getSinkWith

          public Tap getSinkWith(String identifier)

      Specified by:
        `getSinkWith` in interface `FlowStep<Config>`

    - #### getTrapMap

          public Map<String,Tap> getTrapMap()

      Specified by:
        `getTrapMap` in interface `ProcessModel`

    - #### getTraps

          public Set<Tap> getTraps()

      Specified by:
        `getTraps` in interface `FlowStep<Config>`

    - #### getTrap

          public Tap getTrap(String name)

      Specified by:
        `getTrap` in interface `FlowStep<Config>`

    - #### prepareResources

          protected Throwable prepareResources()

    - #### commitSinks

          protected Throwable commitSinks()

    - #### rollbackSinks

          protected Throwable rollbackSinks()

    - #### createInitializedConfig

          public abstract Config createInitializedConfig(FlowProcess<Config> flowProcess,
                                       Config parentConfig)

      Public for testing.
      Parameters:
        `flowProcess` -
        `parentConfig` -
      Returns:

    - #### getPreviousScopes

          public Set<Scope> getPreviousScopes(FlowElement flowElement)

      Method getPreviousScopes returns the previous Scope instances. If the flowElement is a Group (specifically a CoGroup), there will be more than one instance.
      Parameters:
        `flowElement` - of type FlowElement
      Returns:
        Set

    - #### getNextScope

          public Scope getNextScope(FlowElement flowElement)

      Method getNextScope returns the next Scope instance in the graph. There will always only be one next.
      Parameters:
        `flowElement` - of type FlowElement
      Returns:
        Scope

    - #### getNextFlowElement

          public FlowElement getNextFlowElement(Scope scope)

    - #### getAllOperations

          public Collection<Operation> getAllOperations()

    - #### containsPipeNamed

          public boolean containsPipeNamed(String pipeName)

      Description copied from interface: `FlowStep`
      Returns true if this FlowStep contains a pipe/branch with the given name.
      Specified by:
        `containsPipeNamed` in interface `FlowStep<Config>`
      Parameters:
        `pipeName` - the name of the Pipe
      Returns:
        a boolean

    - #### clean

          public void clean()

    - #### clean

          public abstract void clean(Config config)

    - #### hasListeners

          public boolean hasListeners()

      Description copied from interface: `FlowStep`
      Method hasListeners returns true if `FlowStepListener` instances have been registered.
      Specified by:
        `hasListeners` in interface `FlowStep<Config>`
      Returns:
        boolean

    - #### addListener

          public void addListener(FlowStepListener flowStepListener)

      Description copied from interface: `FlowStep`
      Method addListener registers the given `FlowStepListener` with this instance.
      Specified by:
        `addListener` in interface `FlowStep<Config>`
      Parameters:
        `flowStepListener` - of type flowStepListener

    - #### removeListener

          public boolean removeListener(FlowStepListener flowStepListener)

      Description copied from interface: `FlowStep`
      Method removeListener removes the given flowStepListener from this instance.
      Specified by:
        `removeListener` in interface `FlowStep<Config>`
      Parameters:
        `flowStepListener` - of type FlowStepListener
      Returns:
        true if the listener was removed

    - #### fireOnCompleted

          protected void fireOnCompleted()

    - #### fireOnThrowable

          protected void fireOnThrowable(Throwable throwable)

    - #### fireOnStopping

          protected void fireOnStopping()

    - #### fireOnStarting

          protected void fireOnStarting()

    - #### fireOnRunning

          protected void fireOnRunning()

    - #### createClientState

          protected ClientState createClientState(FlowProcess flowProcess)

    - #### getFlowStepJob

          public FlowStepJob<Config> getFlowStepJob()

    - #### getCreateFlowStepJob

          public FlowStepJob<Config> getCreateFlowStepJob(FlowProcess<Config> flowProcess,
                                                 Config parentConfig)

    - #### createFlowStepJob

          protected abstract FlowStepJob createFlowStepJob(ClientState clientState,
                                      FlowProcess<Config> flowProcess,
                                      Config initializedStepConfig)

    - #### initConfFromNodeConfigDef

          protected void initConfFromNodeConfigDef(ElementGraph nodeElementGraph,
                                       ConfigDef.Setter setter)

    - #### initConfFromStepConfigDef

          protected void initConfFromStepConfigDef(ConfigDef.Setter setter)

    - #### addSources

          protected static void addSources(BaseFlowStep flowStep,
                        ElementGraph elementGraph,
                        Set<Tap> sources)

    - #### addSinks

          protected static void addSinks(BaseFlowStep flowStep,
                      ElementGraph elementGraph,
                      Set<Tap> sinks)

    - #### equals

          public boolean equals(Object object)

      Overrides:
        `equals` in class `Object`

    - #### hashCode

          public int hashCode()

      Overrides:
        `hashCode` in class `Object`

    - #### toString

          public String toString()

      Overrides:
        `toString` in class `Object`

    - #### isInfoEnabled

          public final boolean isInfoEnabled()

      Specified by:
        `isInfoEnabled` in interface `ProcessLogger`

    - #### isDebugEnabled

          public final boolean isDebugEnabled()

      Specified by:
        `isDebugEnabled` in interface `ProcessLogger`

    - #### logDebug

          public void logDebug(String message,
                      Object... arguments)

      Specified by:
        `logDebug` in interface `ProcessLogger`

    - #### logInfo

          public void logInfo(String message,
                     Object... arguments)

      Specified by:
        `logInfo` in interface `ProcessLogger`

    - #### logWarn

          public void logWarn(String message)

      Specified by:
        `logWarn` in interface `ProcessLogger`

    - #### logWarn

          public void logWarn(String message,
                     Throwable throwable)

      Specified by:
        `logWarn` in interface `ProcessLogger`

    - #### logWarn

          public void logWarn(String message,
                     Object... arguments)

      Specified by:
        `logWarn` in interface `ProcessLogger`

    - #### logError

          public void logError(String message,
                      Object... arguments)

      Specified by:
        `logError` in interface `ProcessLogger`

    - #### logError

          public void logError(String message,
                      Throwable throwable)

      Specified by:
        `logError` in interface `ProcessLogger`

Copyright © 2007-2015 Concurrent, Inc. All Rights Reserved.