Thrill  0.1
 All Classes Namespaces Files Functions Variables Typedefs Enumerations Enumerator Friends Macros Groups Pages
Source Nodes
+ Collaboration diagram for Source Nodes:

Functions

template<typename ValueType >
auto ConcatToDIA (Context &ctx, const std::vector< ValueType > &in_vector)
 ConcatToDIA is a Source-DOp, which takes a vector of data on all workers, and CONCATENATES them into a DIA. More...
 
template<typename ValueType >
auto EqualToDIA (Context &ctx, const std::vector< ValueType > &in_vector)
 EqualToDIA is a Source-DOp, which takes a vector of data EQUAL on all workers, and returns the data in a DIA. More...
 
template<typename ValueType >
auto EqualToDIA (Context &ctx, std::vector< ValueType > &&in_vector)
 EqualToDIA is an Source-DOp, which takes a vector of data EQUAL on all workers, and returns the data in a DIA. More...
 
template<typename GenerateFunction >
auto Generate (Context &ctx, size_t size, const GenerateFunction &generate_function)
 Generate is a Source-DOp, which creates a DIA of given size using a generator function. More...
 
static auto Generate (Context &ctx, size_t size)
 Generate is a Source-DOp, which creates a DIA of given size containing the size_t indexes [0,size). More...
 
template<typename ValueType >
DIA< ValueType > ReadBinary (Context &ctx, const std::vector< std::string > &filepath, uint64_t size_limit=ReadBinaryNode< ValueType >::no_size_limit_)
 ReadBinary is a DOp, which reads a file written by WriteBinary from the file system and creates a DIA. More...
 
template<typename ValueType >
DIA< ValueType > ReadBinary (Context &ctx, const std::string &filepath, uint64_t size_limit=ReadBinaryNode< ValueType >::no_size_limit_)
 ReadBinary is a DOp, which reads a file written by WriteBinary from the file system and creates a DIA. More...
 
DIA< std::string > ReadLines (Context &ctx, const std::string &filepath)
 ReadLines is a DOp, which reads a file from the file system and creates an ordered DIA according to a given read function. More...
 
DIA< std::string > ReadLines (Context &ctx, const std::vector< std::string > &filepaths)
 ReadLines is a DOp, which reads a file from the file system and creates an ordered DIA according to a given read function. More...
 
template<typename ValueType >
auto Distribute (Context &ctx, const std::vector< ValueType > &in_vector, size_t source_id=0)
 Distribute is a Source DOp, which scatters the vector data from the source_id to all workers, partitioning equally, and returning the data in a DIA. More...
 
template<typename ValueType >
auto Distribute (Context &ctx, std::vector< ValueType > &&in_vector, size_t source_id=0)
 Distribute is a Source DOp, which scatters the vector data from the source_id to all workers, partitioning equally, and returning the data in a DIA. More...
 

Detailed Description

Function Documentation

auto thrill::api::ConcatToDIA ( Context &  ctx,
const std::vector< ValueType > &  in_vector 
)

ConcatToDIA is a Source-DOp, which takes a vector of data on all workers, and CONCATENATES them into a DIA.

Use Distribute to actually distribute data from a single worker, ConcatToDIA is a wrapper if the data is already distributed.

Parameters
ctxReference to the Context object
in_vectorVector to concatenate into a DIA, the contents is COPIED into the DIANode.

Definition at line 74 of file concat_to_dia.hpp.

auto thrill::api::Distribute ( Context &  ctx,
const std::vector< ValueType > &  in_vector,
size_t  source_id = 0 
)

Distribute is a Source DOp, which scatters the vector data from the source_id to all workers, partitioning equally, and returning the data in a DIA.

Definition at line 105 of file distribute.hpp.

auto thrill::api::Distribute ( Context &  ctx,
std::vector< ValueType > &&  in_vector,
size_t  source_id = 0 
)

Distribute is a Source DOp, which scatters the vector data from the source_id to all workers, partitioning equally, and returning the data in a DIA.

Definition at line 125 of file distribute.hpp.

auto thrill::api::EqualToDIA ( Context &  ctx,
const std::vector< ValueType > &  in_vector 
)

EqualToDIA is a Source-DOp, which takes a vector of data EQUAL on all workers, and returns the data in a DIA.

Use Distribute to actually distribute data from a single worker, EqualToDIA is a wrapper if the data is already distributed.

Parameters
ctxReference to the Context object
in_vectorVector to convert to a DIA, the contents is COPIED into the DIANode.

Definition at line 77 of file equal_to_dia.hpp.

Referenced by ConstructRLBWT().

auto thrill::api::EqualToDIA ( Context &  ctx,
std::vector< ValueType > &&  in_vector 
)

EqualToDIA is an Source-DOp, which takes a vector of data EQUAL on all workers, and returns the data in a DIA.

Use Distribute to actually distribute data from a single worker, EqualToDIA is a wrapper if the data is already distributed.

Parameters
ctxReference to the Context object
in_vectorVector to convert to a DIA, the contents is MOVED into the DIANode.

Definition at line 99 of file equal_to_dia.hpp.

auto thrill::api::Generate ( Context &  ctx,
size_t  size,
const GenerateFunction &  generate_function 
)

Generate is a Source-DOp, which creates a DIA of given size using a generator function.

The generator function called for each index in the range of [0,size) and must output exactly one item.

Parameters
ctxReference to the Context object
sizeSize of the output DIA
generate_functionGenerator function, which maps size_t from [0,size) to elements. Input type has to be size_t.
Examples:
examples/tutorial/k-means_step1.cpp, examples/tutorial/k-means_step2.cpp, examples/tutorial/k-means_step3.cpp, examples/tutorial/k-means_step4.cpp, examples/tutorial/k-means_step5.cpp, and examples/tutorial/k-means_step6.cpp.

Definition at line 85 of file generate.hpp.

References gen_data::value.

Referenced by examples::suffix_sorting::ConstructLCP(), CountTrianglesGenerated(), thrill::api::Generate(), GenerateInput(), GeneratePoints(), GenerateTestData(), main(), examples::page_rank::PageRank(), examples::page_rank::PageRankJoin(), Parallel(), Process(), RunHashWordCountGenerated(), RunKMeansGenerated(), RunPageRankGenerated(), RunPageRankJoinGenerated(), RunSelect(), RunStochasticGradGenerated(), and RunWordCountGenerated().

static auto thrill::api::Generate ( Context &  ctx,
size_t  size 
)
inlinestatic

Generate is a Source-DOp, which creates a DIA of given size containing the size_t indexes [0,size).

Parameters
ctxReference to the Context object
sizeSize of the output DIA

Definition at line 122 of file generate.hpp.

References thrill::api::Generate().

DIA<ValueType> thrill::api::ReadBinary ( Context &  ctx,
const std::vector< std::string > &  filepath,
uint64_t  size_limit = ReadBinaryNode<ValueType>::no_size_limit_ 
)

ReadBinary is a DOp, which reads a file written by WriteBinary from the file system and creates a DIA.

Parameters
ctxReference to the context object
filepathPath of the file in the file system
size_limitOptional limit to the total file size (e.g. for testing algorithms on prefixes)

Definition at line 349 of file read_binary.hpp.

DIA<ValueType> thrill::api::ReadBinary ( Context &  ctx,
const std::string &  filepath,
uint64_t  size_limit = ReadBinaryNode<ValueType>::no_size_limit_ 
)

ReadBinary is a DOp, which reads a file written by WriteBinary from the file system and creates a DIA.

Parameters
ctxReference to the context object
filepathPath of the file in the file system
size_limitOptional limit to the total file size (e.g. for testing algorithms on prefixes)

Definition at line 383 of file read_binary.hpp.

DIA<std::string> thrill::api::ReadLines ( Context &  ctx,
const std::string &  filepath 
)

ReadLines is a DOp, which reads a file from the file system and creates an ordered DIA according to a given read function.

Parameters
ctxReference to the context object
filepathPath of the file in the file system
Examples:
examples/tutorial/k-means_step5.cpp, and examples/tutorial/k-means_step6.cpp.

Definition at line 452 of file read_lines.hpp.

Referenced by CountTrianglesPerLine(), JoinTPCH4(), LoadBFSGraph(), LoadPoints(), main(), Percentiles(), ReadInputFile(), RunHashWordCount(), RunJoinPageRankEdgePerLine(), RunKMeansFile(), RunPageRankEdgePerLine(), RunStochasticGradFile(), RunWordCount(), and WordCount().

DIA<std::string> thrill::api::ReadLines ( Context &  ctx,
const std::vector< std::string > &  filepaths 
)

ReadLines is a DOp, which reads a file from the file system and creates an ordered DIA according to a given read function.

Parameters
ctxReference to the context object
filepathsPath of the file in the file system

Definition at line 467 of file read_lines.hpp.