Thrill  0.1
Context Class Reference

Detailed Description

The Context of a job is a unique instance per worker which holds references to all underlying parts of Thrill.

The context is able to give references to the stream multiplexer, the netgroup" \ref common::Stats "stats" and \ref common::StatsGraph "stats graph". Threads share the stream multiplexer and the net group via the context object.

Examples:
examples/tutorial/k-means_step1.cpp, examples/tutorial/k-means_step2.cpp, examples/tutorial/k-means_step3.cpp, examples/tutorial/k-means_step4.cpp, examples/tutorial/k-means_step5.cpp, and examples/tutorial/k-means_step6.cpp.

Definition at line 221 of file context.hpp.

+ Collaboration diagram for Context:

#include <context.hpp>

Public Member Functions

 Context (HostContext &host_context, size_t local_worker_id)
 
common::Range CalculateLocalRange (size_t global_size) const
 
common::Range CalculateLocalRangeOnHost (size_t global_size) const
 
bool consume () const
 return value of consume flag. More...
 
void enable_consume (bool consume=true)
 Sets consume-mode flag such that DIA contents may be consumed during PushData(). More...
 
void Launch (const std::function< void(Context &)> &job_startpoint)
 method used to launch a job's main procedure. it wraps it in log output. More...
 
const MemoryConfigmem_config () const
 host-global memory config More...
 
mem::Managermem_manager ()
 returns the host-global memory manager More...
 
net::Managernet_manager ()
 
size_t next_dia_id ()
 Returns next_dia_id_ to generate DIA::id_ serial. More...
 
template<typename Type >
void PrintCollectiveMeanStdev (const char *text, const Type &local)
 
Data Subsystem
data::File GetFile (size_t dia_id)
 Returns a new File object containing a sequence of local Blocks. More...
 
data::File GetFile (DIABase *dia)
 Returns a new File object containing a sequence of local Blocks. More...
 
data::FilePtr GetFilePtr (size_t dia_id)
 
data::FilePtr GetFilePtr (DIABase *dia)
 
data::CatStreamPtr GetNewCatStream (size_t dia_id)
 
data::CatStreamPtr GetNewCatStream (DIABase *dia)
 
data::MixStreamPtr GetNewMixStream (size_t dia_id)
 
data::MixStreamPtr GetNewMixStream (DIABase *dia)
 
template<typename Stream >
tlx::CountingPtr< Stream > GetNewStream (size_t dia_id)
 
data::BlockPoolblock_pool ()
 the block manager keeps all data blocks moving through the system. More...
 

Public Attributes

Shared Objects
std::default_random_engine rng_
 a random generator More...
 
Network Subsystem
net::FlowControlChannelnet
 
Logging System
common::JsonLogger base_logger_
 base logger exclusive for this worker More...
 
common::JsonLogger logger_
 

Private Attributes

data::BlockPoolblock_pool_
 data block pool More...
 
bool consume_ = false
 flag to set which enables selective consumption of DIA contents! More...
 
net::FlowControlChannelManagerflow_manager_
 net::FlowControlChannelManager instance that is shared among workers More...
 
size_t last_dia_id_ = 0
 the number of valid DIA ids. 0 is reserved for invalid. More...
 
size_t local_host_id_
 id among all local hosts (in test program runs) More...
 
size_t local_worker_id_
 number of this host context, 0..p-1, within this host More...
 
const MemoryConfigmem_config_
 memory configuration in HostContext More...
 
size_t mem_limit_
 memory limit of this worker Context for local data structures More...
 
mem::Managermem_manager_
 host-global memory manager More...
 
data::Multiplexermultiplexer_
 data::Multiplexer instance that is shared among workers More...
 
net::Managernet_manager_
 net::Manager instance that is shared among workers More...
 
size_t workers_per_host_
 number of workers hosted per host More...
 

System Information

size_t num_hosts () const
 Returns the total number of hosts. More...
 
size_t workers_per_host () const
 Returns the number of workers that is hosted on each host. More...
 
size_t my_rank () const
 Global rank of this worker among all other workers in the system. More...
 
size_t mem_limit () const
 memory limit of this worker Context for local data structures More...
 
size_t num_workers () const
 Global number of workers in the system. More...
 
size_t host_rank () const
 
size_t local_worker_id () const
 
size_t local_host_id () const
 id among all local hosts (in test program runs) More...
 
std::ostream & operator<< (std::ostream &os, const Context &ctx)
 Outputs the context as [host id]:[local worker id] to an std::ostream. More...
 

Constructor & Destructor Documentation

◆ Context()

Context ( HostContext host_context,
size_t  local_worker_id 
)

Member Function Documentation

◆ block_pool()

◆ CalculateLocalRange()

common::Range CalculateLocalRange ( size_t  global_size) const
inline

given a global range [0,global_size) and p PEs to split the range, calculate the [local_begin,local_end) index range assigned to the PE i. Takes the information from the Context.

Definition at line 339 of file context.hpp.

References thrill::common::CalculateLocalRange().

Referenced by ReadLinesNode::InputLineIteratorCompressed::InputLineIteratorCompressed(), ReadLinesNode::InputLineIteratorUncompressed::InputLineIteratorUncompressed(), EqualToDIANode< ValueType >::PushData(), GenerateNode< ValueType, GenerateFunction >::PushData(), and ReadBinaryNode< ValueType >::ReadBinaryNode().

◆ CalculateLocalRangeOnHost()

◆ consume()

bool consume ( ) const
inline

return value of consume flag.

Definition at line 378 of file context.hpp.

Referenced by DIANode< StackInput >::RunPushData().

◆ enable_consume()

void enable_consume ( bool  consume = true)
inline

Sets consume-mode flag such that DIA contents may be consumed during PushData().

When in consume mode the DIA contents is destroyed online when it is transmitted to the next operation. This enables reusing the space of the consume operations. This enabled processing more data with less space. However, by default this mode is DISABLED, because it requires deliberate insertion of .Keep() calls.

Definition at line 388 of file context.hpp.

Referenced by JoinTPCH4(), main(), RunHashWordCount(), RunHashWordCountGenerated(), RunJoinPageRankEdgePerLine(), RunPageRankEdgePerLine(), RunPageRankGenerated(), RunPageRankJoinGenerated(), RunWordCount(), and RunWordCountGenerated().

◆ GetFile() [1/2]

◆ GetFile() [2/2]

data::File GetFile ( DIABase dia)

Returns a new File object containing a sequence of local Blocks.

Definition at line 1196 of file context.cpp.

References DIABase::dia_id(), and Context::GetFile().

◆ GetFilePtr() [1/2]

◆ GetFilePtr() [2/2]

data::FilePtr GetFilePtr ( DIABase dia)

Returns a new File, wrapped in a CountingPtr, containing a sequence of local Blocks.

Definition at line 1205 of file context.cpp.

References DIABase::dia_id(), and Context::GetFilePtr().

◆ GetNewCatStream() [1/2]

data::CatStreamPtr GetNewCatStream ( size_t  dia_id)

◆ GetNewCatStream() [2/2]

data::CatStreamPtr GetNewCatStream ( DIABase dia)

Returns a reference to a new CatStream. This method alters the state of the context and must be called on all Workers to ensure correct communication coordination.

Definition at line 1213 of file context.cpp.

References DIABase::dia_id(), and Context::GetNewCatStream().

◆ GetNewMixStream() [1/2]

data::MixStreamPtr GetNewMixStream ( size_t  dia_id)

Returns a reference to a new MixStream. This method alters the state of the context and must be called on all Workers to ensure correct communication coordination.

Definition at line 1217 of file context.cpp.

References Multiplexer::GetNewMixStream(), Context::local_worker_id_, and Context::multiplexer_.

Referenced by thrill::api::Context::GetNewStream< data::MixStream >(), Context::GetNewMixStream(), and SortNode< ValueType, CompareFunction, SortAlgorithm, Stable >::MainOp().

◆ GetNewMixStream() [2/2]

data::MixStreamPtr GetNewMixStream ( DIABase dia)

Returns a reference to a new MixStream. This method alters the state of the context and must be called on all Workers to ensure correct communication coordination.

Definition at line 1221 of file context.cpp.

References DIABase::dia_id(), and Context::GetNewMixStream().

◆ GetNewStream()

tlx::CountingPtr<Stream> GetNewStream ( size_t  dia_id)

Returns a reference to a new CatStream or MixStream, selectable via template parameter.

◆ host_rank()

size_t host_rank ( ) const
inline

Returns id of this host in the cluser A host is a machine in the cluster that hosts multiple workers

Definition at line 257 of file context.hpp.

Referenced by thrill::api::RunBackendLoopback().

◆ Launch()

◆ local_host_id()

size_t local_host_id ( ) const
inline

id among all local hosts (in test program runs)

Definition at line 268 of file context.hpp.

◆ local_worker_id()

size_t local_worker_id ( ) const
inline

Returns the local id ot this worker on the host A worker is locally identified by this id

Definition at line 263 of file context.hpp.

Referenced by Context::Context(), SortNode< ValueType, CompareFunction, SortAlgorithm, Stable >::MainOp(), ReadBinaryNode< ValueType >::VfsFileBlockSource::NextBlock(), and WriteBinaryNode< ValueType >::OpenNextFile().

◆ mem_config()

const MemoryConfig& mem_config ( ) const
inline

host-global memory config

Definition at line 329 of file context.hpp.

Referenced by Context::Launch().

◆ mem_limit()

size_t mem_limit ( ) const
inline

memory limit of this worker Context for local data structures

Definition at line 248 of file context.hpp.

◆ mem_manager()

mem::Manager& mem_manager ( )
inline

returns the host-global memory manager

Definition at line 332 of file context.hpp.

◆ my_rank()

size_t my_rank ( ) const
inline

Global rank of this worker among all other workers in the system.

Definition at line 243 of file context.hpp.

Referenced by CountTrianglesGenerated(), examples::suffix_sorting::DC3Recursive(), examples::suffix_sorting::DC7Recursive(), doubleSweepDiameter(), DistributeNode< ValueType >::Execute(), WriteLinesOneNode< ValueType >::Execute(), SampleNode< ValueType >::Execute(), ConcatNode< ValueType >::Execute(), thrill::api::FindStages(), GroupToIndexNode< ValueType, KeyExtractor, GroupFunction >::GroupToIndexNode(), JoinTPCH4(), Context::Launch(), main(), MergeNode< ValueType, Comparator, kNumInputs >::MainOp(), SortNode< ValueType, CompareFunction, SortAlgorithm, Stable >::MainOp(), CollapseNode< ValueType >::OnPreOpFile(), WriteBinaryNode< ValueType >::OpenNextFile(), Parallel(), Percentiles(), examples::select::PickPivots(), WriteLinesNode< ValueType >::PreOp(), MergeNode< ValueType, Comparator, kNumInputs >::Stats::Print(), OverlapWindowNode< ValueType, Input, WindowFunction, PartialWindowFunction >::PushData(), SortNode< ValueType, CompareFunction, SortAlgorithm, Stable >::PushData(), DisjointWindowNode< ValueType, Input, WindowFunction, PartialWindowFunction >::PushData(), RunHashWordCount(), RunJoinPageRankEdgePerLine(), RunKMeansFile(), RunKMeansGenerated(), RunPageRankEdgePerLine(), RunPageRankGenerated(), RunPageRankJoinGenerated(), RunStochasticGradFile(), RunStochasticGradGenerated(), RunWordCount(), examples::select::Select(), and WriteLinesNode< ValueType >::WriteLinesNode().

◆ net_manager()

◆ next_dia_id()

size_t next_dia_id ( )
inline

Returns next_dia_id_ to generate DIA::id_ serial.

Definition at line 391 of file context.hpp.

◆ num_hosts()

◆ num_workers()

◆ PrintCollectiveMeanStdev()

◆ workers_per_host()

size_t workers_per_host ( ) const
inline

Returns the number of workers that is hosted on each host.

Definition at line 238 of file context.hpp.

Referenced by Context::Context().

Friends And Related Function Documentation

◆ operator<<

std::ostream& operator<< ( std::ostream &  os,
const Context ctx 
)
friend

Outputs the context as [host id]:[local worker id] to an std::ostream.

Definition at line 272 of file context.hpp.

Referenced by thrill::api::Context::GetNewStream< data::MixStream >().

Member Data Documentation

◆ base_logger_

common::JsonLogger base_logger_

base logger exclusive for this worker

Definition at line 457 of file context.hpp.

Referenced by Context::Context().

◆ block_pool_

data::BlockPool& block_pool_
private

data block pool

Definition at line 419 of file context.hpp.

Referenced by Context::GetFilePtr().

◆ consume_

bool consume_ = false
private

flag to set which enables selective consumption of DIA contents!

Definition at line 425 of file context.hpp.

◆ flow_manager_

net::FlowControlChannelManager& flow_manager_
private

net::FlowControlChannelManager instance that is shared among workers

Definition at line 416 of file context.hpp.

◆ last_dia_id_

size_t last_dia_id_ = 0
private

the number of valid DIA ids. 0 is reserved for invalid.

Definition at line 428 of file context.hpp.

◆ local_host_id_

size_t local_host_id_
private

id among all local hosts (in test program runs)

Definition at line 395 of file context.hpp.

Referenced by Context::Launch().

◆ local_worker_id_

size_t local_worker_id_
private

number of this host context, 0..p-1, within this host

Definition at line 398 of file context.hpp.

Referenced by Context::Context(), Context::GetFilePtr(), Context::GetNewCatStream(), Context::GetNewMixStream(), and Context::Launch().

◆ logger_

Initial value:
{
&base_logger_, "host_rank", host_rank(), "worker_rank", my_rank()
}

public member which delivers key:value pairs as JSON log lines. this logger is local to this Context which is exclusive for one worker thread.

Definition at line 462 of file context.hpp.

Referenced by Context::Launch().

◆ mem_config_

const MemoryConfig& mem_config_
private

memory configuration in HostContext

Definition at line 407 of file context.hpp.

◆ mem_limit_

size_t mem_limit_
private

memory limit of this worker Context for local data structures

Definition at line 404 of file context.hpp.

◆ mem_manager_

mem::Manager& mem_manager_
private

host-global memory manager

Definition at line 410 of file context.hpp.

◆ multiplexer_

data::Multiplexer& multiplexer_
private

data::Multiplexer instance that is shared among workers

Definition at line 422 of file context.hpp.

Referenced by Context::GetNewCatStream(), and Context::GetNewMixStream().

◆ net

Initial value:

public member which exposes all network primitives from FlowControlChannel for DOp implementations. Use it as context_.net.Method().

Definition at line 446 of file context.hpp.

Referenced by CountTrianglesGenerated(), HyperLogLogNode< p, ValueType >::Execute(), SizeNode< ValueType >::Execute(), AllReduceNode< ValueType, ReduceFunction >::Execute(), RebalanceNode< ValueType >::Execute(), WriteLinesOneNode< ValueType >::Execute(), SampleNode< ValueType >::Execute(), PrefixSumNode< ValueType, SumFunction, Inclusive >::Execute(), ZipWithIndexNode< ValueType, ZipFunction >::Execute(), OverlapWindowNode< ValueType, Input, WindowFunction, PartialWindowFunction >::Execute(), ConcatNode< ValueType >::Execute(), DisjointWindowNode< ValueType, Input, WindowFunction, PartialWindowFunction >::Execute(), DuplicateDetection::FindNonDuplicates(), MergeNode< ValueType, Comparator, kNumInputs >::GetGlobalRanks(), JoinTPCH4(), Context::Launch(), main(), ZipNode< ValueType, ZipFunction, Pad, UnequalCheck, NoRebalance, kNumInputs >::MainOp(), MergeNode< ValueType, Comparator, kNumInputs >::MainOp(), SortNode< ValueType, CompareFunction, SortAlgorithm, Stable >::MainOp(), Percentiles(), examples::select::PickPivots(), MergeNode< ValueType, Comparator, kNumInputs >::Stats::Print(), RunHashWordCount(), RunKMeansFile(), RunKMeansGenerated(), RunWordCount(), examples::select::Select(), and MergeNode< ValueType, Comparator, kNumInputs >::SelectPivots().

◆ net_manager_

net::Manager& net_manager_
private

net::Manager instance that is shared among workers

Definition at line 413 of file context.hpp.

Referenced by Context::Launch().

◆ rng_

std::default_random_engine rng_

◆ workers_per_host_

size_t workers_per_host_
private

number of workers hosted per host

Definition at line 401 of file context.hpp.


The documentation for this class was generated from the following files: