tf::DataPipeline class | Taskflow QuickStart

#
template<typename... Ps>
tf::DataPipeline class

class to create a data-parallel pipeline scheduling framework

Template parameters
Ps | data pipe types

Similar to tf::Pipeline, a tf::DataPipeline is a composable graph object for users to create a data-parallel pipeline scheduling framework using a module task in a taskflow. The only difference is that tf::DataPipeline provides a data abstraction for users to quickly express dataflow in a pipeline. The following example creates a data-parallel pipeline of three stages that generate dataflow from `void` to `int`, `std::string`, and `void`.

    #include <taskflow/taskflow.hpp>
    #include <taskflow/algorithm/data_pipeline.hpp>

    int main() {

      // data flow => void -> int -> std::string -> void
      tf::Taskflow taskflow("pipeline");
      tf::Executor executor;

      const size_t num_lines = 4;

      tf::DataPipeline pl(num_lines,
        tf::make_data_pipe<void, int>(tf::PipeType::SERIAL, [&](tf::Pipeflow& pf) -> int{
          if(pf.token() == 5) {
            pf.stop();
            return 0;
          }
          else {
            return pf.token();
          }
        }),
        tf::make_data_pipe<int, std::string>(tf::PipeType::SERIAL, [](int& input) {
          return std::to_string(input + 100);
        }),
        tf::make_data_pipe<std::string, void>(tf::PipeType::SERIAL, [](std::string& input) {
          std::cout << input << std::endl;
        })
      );

      // build the pipeline graph using composition
      taskflow.composed_of(pl).name("pipeline");

      // dump the pipeline graph structure (with composition)
      taskflow.dump(std::cout);

      // run the pipeline
      executor.run(taskflow).wait();

      return 0;
    }

The pipeline schedules five tokens over four parallel lines in a circular fashion, as depicted below:

    o -> o -> o
    |    |    |
    v    v    v
    o -> o -> o
    |    |    |
    v    v    v
    o -> o -> o
    |    |    |
    v    v    v
    o -> o -> o

## Public types

using data_t = unique_variant_t<std::variant<std::conditional_t<std::is_void_v<typename Ps::output_t>, std::monostate, std::decay_t<typename Ps::output_t>>...>>
  internal storage type for each data token (default std::variant)

## Constructors, destructors, conversion operators

DataPipeline(size_t num_lines, Ps && ... ps)
  constructs a data-parallel pipeline object
DataPipeline(size_t num_lines, std::tuple<Ps...>&& ps)
  constructs a data-parallel pipeline object

## Public functions

auto num_lines() const -> size_t noexcept
  queries the number of parallel lines
auto num_pipes() const -> size_t constexpr noexcept
  queries the number of pipes
void reset()
  resets the pipeline
auto num_tokens() const -> size_t noexcept
  queries the number of generated tokens in the pipeline
auto graph() -> Graph&
  obtains the graph object associated with the pipeline construct

## Function documentation

###
template<typename... Ps>
tf::DataPipeline<Ps>::DataPipeline(size_t num_lines, Ps && ... ps)

constructs a data-parallel pipeline object

Parameters
num_lines | the number of parallel lines
ps | a list of pipes

Constructs a data-parallel pipeline of up to `num_lines` parallel lines to schedule tokens through the given linear chain of pipes. The first pipe must define a serial direction (tf::PipeType::SERIAL) or an exception will be thrown.

###
template<typename... Ps>
tf::DataPipeline<Ps>::DataPipeline(size_t num_lines, std::tuple<Ps...>&& ps)

constructs a data-parallel pipeline object

Parameters
num_lines | the number of parallel lines
ps | a tuple of pipes

Constructs a data-parallel pipeline of up to `num_lines` parallel lines to schedule tokens through the given linear chain of pipes stored in a std::tuple. The first pipe must define a serial direction (tf::PipeType::SERIAL) or an exception will be thrown.

###
template<typename... Ps>
size_t tf::DataPipeline<Ps>::num_lines() const noexcept

queries the number of parallel lines

The function returns the number of parallel lines given by the user upon the construction of the pipeline. The number of lines represents the maximum parallelism this pipeline can achieve.

###
template<typename... Ps>
size_t tf::DataPipeline<Ps>::num_pipes() const constexpr noexcept

queries the number of pipes

The Function returns the number of pipes given by the user upon the construction of the pipeline.

###
template<typename... Ps>
void tf::DataPipeline<Ps>::reset()

resets the pipeline

Resetting the pipeline to the initial state. After resetting a pipeline, its token identifier will start from zero as if the pipeline was just constructed.

###
template<typename... Ps>
size_t tf::DataPipeline<Ps>::num_tokens() const noexcept

queries the number of generated tokens in the pipeline

The number represents the total scheduling tokens that has been generated by the pipeline so far.

###
template<typename... Ps>
Graph& tf::DataPipeline<Ps>::graph()

obtains the graph object associated with the pipeline construct

This method is primarily used as an opaque data structure for creating a module task of this pipeline.

Tab / T to search, Esc to close
…

Search for symbols, directories, files, pages or modules. You can omit any prefix from the symbol or file path; adding a `:` or `/` suffix lists all members of given symbol or directory.

Use ↓ / ↑ to navigate through the list, Enter to go. Tab autocompletes common prefix, you can copy a link to the result using ⌘ L while ⌘ M produces a Markdown link.

Sorry, nothing was found.