Thrill  0.1
 All Classes Namespaces Files Functions Variables Typedefs Enumerations Enumerator Friends Macros Groups Pages
Context Class Reference

Detailed Description

The Context of a job is a unique instance per worker which holds references to all underlying parts of Thrill.

The context is able to give references to the stream multiplexer, the netgroup" \ref common::Stats "stats" and \ref common::StatsGraph "stats graph". Threads share the stream multiplexer and the net group via the context object.

Definition at line 210 of file context.hpp.

+ Collaboration diagram for Context:

#include <context.hpp>

Public Member Functions

 Context (HostContext &host_context, size_t local_worker_id)
 
common::Range CalculateLocalRange (size_t global_size) const
 
common::Range CalculateLocalRangeOnHost (size_t global_size) const
 
bool consume () const
 return value of consume flag. More...
 
void enable_consume (bool consume=true)
 Sets consume-mode flag such that DIA contents may be consumed during PushData(). More...
 
void Launch (const std::function< void(Context &)> &job_startpoint)
 method used to launch a job's main procedure. it wraps it in log output. More...
 
const MemoryConfigmem_config () const
 host-global memory config More...
 
mem::Managermem_manager ()
 returns the host-global memory manager More...
 
net::Managernet_manager ()
 
size_t next_dia_id ()
 Returns next_dia_id_ to generate DIA::id_ serial. More...
 
template<typename Type >
void PrintCollectiveMeanStdev (const char *text, const Type &local)
 
Data Subsystem
data::File GetFile (size_t dia_id)
 Returns a new File object containing a sequence of local Blocks. More...
 
data::File GetFile (DIABase *dia)
 Returns a new File object containing a sequence of local Blocks. More...
 
data::FilePtr GetFilePtr (size_t dia_id)
 
data::FilePtr GetFilePtr (DIABase *dia)
 
data::CatStreamPtr GetNewCatStream (size_t dia_id)
 
data::CatStreamPtr GetNewCatStream (DIABase *dia)
 
data::MixStreamPtr GetNewMixStream (size_t dia_id)
 
data::MixStreamPtr GetNewMixStream (DIABase *dia)
 
template<typename Stream >
tlx::CountingPtr< Stream > GetNewStream (size_t dia_id)
 
data::BlockPoolblock_pool ()
 the block manager keeps all data blocks moving through the system. More...
 

Public Attributes

Network Subsystem
net::FlowControlChannelnet
 
Logging System
common::JsonLogger base_logger_
 base logger exclusive for this worker More...
 
common::JsonLogger logger_
 

Private Attributes

data::BlockPoolblock_pool_
 data block pool More...
 
bool consume_ = false
 flag to set which enables selective consumption of DIA contents! More...
 
net::FlowControlChannelManagerflow_manager_
 net::FlowControlChannelManager instance that is shared among workers More...
 
size_t last_dia_id_ = 0
 the number of valid DIA ids. 0 is reserved for invalid. More...
 
size_t local_host_id_
 id among all local hosts (in test program runs) More...
 
size_t local_worker_id_
 number of this host context, 0..p-1, within this host More...
 
const MemoryConfigmem_config_
 memory configuration in HostContext More...
 
size_t mem_limit_
 memory limit of this worker Context for local data structures More...
 
mem::Managermem_manager_
 host-global memory manager More...
 
data::Multiplexermultiplexer_
 data::Multiplexer instance that is shared among workers More...
 
net::Managernet_manager_
 net::Manager instance that is shared among workers More...
 
size_t workers_per_host_
 number of workers hosted per host More...
 

System Information

size_t num_hosts () const
 Returns the total number of hosts. More...
 
size_t workers_per_host () const
 Returns the number of workers that is hosted on each host. More...
 
size_t my_rank () const
 Global rank of this worker among all other workers in the system. More...
 
size_t mem_limit () const
 memory limit of this worker Context for local data structures More...
 
size_t num_workers () const
 Global number of workers in the system. More...
 
size_t host_rank () const
 
size_t local_worker_id () const
 
size_t local_host_id () const
 id among all local hosts (in test program runs) More...
 
std::ostream & operator<< (std::ostream &os, const Context &ctx)
 Outputs the context as [host id]:[local worker id] to an std::ostream. More...
 

Constructor & Destructor Documentation

Context ( HostContext host_context,
size_t  local_worker_id 
)
inline

Definition at line 213 of file context.hpp.

References Context::workers_per_host().

Member Function Documentation

common::Range CalculateLocalRange ( size_t  global_size) const
inline
bool consume ( ) const
inline

return value of consume flag.

Definition at line 380 of file context.hpp.

References Context::consume_.

Referenced by Context::enable_consume(), DIA< ValueType_, Stack_ >::Keep(), and DIANode< StackInput >::RunPushData().

void enable_consume ( bool  consume = true)
inline

Sets consume-mode flag such that DIA contents may be consumed during PushData().

When in consume mode the DIA contents is destroyed online when it is transmitted to the next operation. This enables reusing the space of the consume operations. This enabled processing more data with less space. However, by default this mode is DISABLED, because it requires deliberate insertion of .Keep() calls.

Definition at line 390 of file context.hpp.

References Context::consume(), and Context::consume_.

Referenced by JoinTPCH4(), main(), RunHashWordCount(), RunHashWordCountGenerated(), RunJoinPageRankEdgePerLine(), RunPageRankEdgePerLine(), RunPageRankGenerated(), RunPageRankJoinGenerated(), RunWordCount(), and RunWordCountGenerated().

data::File GetFile ( DIABase dia)

Returns a new File object containing a sequence of local Blocks.

Definition at line 1023 of file context.cpp.

References Context::GetFile(), and DIABase::id().

data::FilePtr GetFilePtr ( size_t  dia_id)

Returns a new File, wrapped in a CountingPtr, containing a sequence of local Blocks.

Definition at line 1027 of file context.cpp.

References Context::block_pool_, and Context::local_worker_id_.

Referenced by JoinNode< ValueType, FirstDIA, SecondDIA, KeyExtractor1, KeyExtractor2, JoinFunction, HashFunction, UseLocationDetection >::AddEqualKeysToVec(), and Context::GetFilePtr().

data::FilePtr GetFilePtr ( DIABase dia)

Returns a new File, wrapped in a CountingPtr, containing a sequence of local Blocks.

Definition at line 1032 of file context.cpp.

References Context::GetFilePtr(), and DIABase::id().

data::CatStreamPtr GetNewCatStream ( size_t  dia_id)
data::CatStreamPtr GetNewCatStream ( DIABase dia)

Returns a reference to a new CatStream. This method alters the state of the context and must be called on all Workers to ensure correct communication coordination.

Definition at line 1040 of file context.cpp.

References Context::GetNewCatStream(), and DIABase::id().

data::MixStreamPtr GetNewMixStream ( size_t  dia_id)

Returns a reference to a new MixStream. This method alters the state of the context and must be called on all Workers to ensure correct communication coordination.

Definition at line 1044 of file context.cpp.

References Multiplexer::GetNewMixStream(), Context::local_worker_id_, and Context::multiplexer_.

Referenced by Context::GetNewMixStream(), and SortNode< ValueType, CompareFunction, SortAlgorithm >::MainOp().

data::MixStreamPtr GetNewMixStream ( DIABase dia)

Returns a reference to a new MixStream. This method alters the state of the context and must be called on all Workers to ensure correct communication coordination.

Definition at line 1048 of file context.cpp.

References Context::GetNewMixStream(), and DIABase::id().

tlx::CountingPtr<Stream> GetNewStream ( size_t  dia_id)

Returns a reference to a new CatStream or MixStream, selectable via template parameter.

size_t host_rank ( ) const
inline

Returns id of this host in the cluser A host is a machine in the cluster that hosts multiple workers

Definition at line 259 of file context.hpp.

References Manager::my_host_rank(), and Context::net_manager_.

Referenced by Context::my_rank().

size_t local_host_id ( ) const
inline

id among all local hosts (in test program runs)

Definition at line 270 of file context.hpp.

References Context::local_host_id_.

size_t local_worker_id ( ) const
inline
const MemoryConfig& mem_config ( ) const
inline

host-global memory config

Definition at line 331 of file context.hpp.

References Context::mem_config_.

Referenced by Context::Launch().

size_t mem_limit ( ) const
inline

memory limit of this worker Context for local data structures

Definition at line 250 of file context.hpp.

References Context::mem_limit_.

mem::Manager& mem_manager ( )
inline

returns the host-global memory manager

Definition at line 334 of file context.hpp.

References Context::mem_manager_.

Referenced by DIABase::mem_manager().

size_t next_dia_id ( )
inline
size_t workers_per_host ( ) const
inline

Returns the number of workers that is hosted on each host.

Definition at line 240 of file context.hpp.

References Context::workers_per_host_.

Referenced by Context::CalculateLocalRangeOnHost(), Context::Context(), Context::my_rank(), and Context::num_workers().

Friends And Related Function Documentation

std::ostream& operator<< ( std::ostream &  os,
const Context ctx 
)
friend

Outputs the context as [host id]:[local worker id] to an std::ostream.

Definition at line 274 of file context.hpp.

Member Data Documentation

common::JsonLogger base_logger_

base logger exclusive for this worker

Definition at line 450 of file context.hpp.

data::BlockPool& block_pool_
private

data block pool

Definition at line 421 of file context.hpp.

Referenced by Context::block_pool(), Context::GetFile(), and Context::GetFilePtr().

bool consume_ = false
private

flag to set which enables selective consumption of DIA contents!

Definition at line 427 of file context.hpp.

Referenced by Context::consume(), and Context::enable_consume().

net::FlowControlChannelManager& flow_manager_
private

net::FlowControlChannelManager instance that is shared among workers

Definition at line 418 of file context.hpp.

size_t last_dia_id_ = 0
private

the number of valid DIA ids. 0 is reserved for invalid.

Definition at line 430 of file context.hpp.

Referenced by Context::next_dia_id().

size_t local_host_id_
private

id among all local hosts (in test program runs)

Definition at line 397 of file context.hpp.

Referenced by Context::Launch(), and Context::local_host_id().

size_t local_worker_id_
private

number of this host context, 0..p-1, within this host

Definition at line 400 of file context.hpp.

Referenced by Context::GetFile(), Context::GetFilePtr(), Context::GetNewCatStream(), Context::GetNewMixStream(), Context::Launch(), and Context::local_worker_id().

Initial value:
{
&base_logger_, "host_rank", host_rank(), "worker_rank", my_rank()
}

public member which delivers key:value pairs as JSON log lines. this logger is local to this Context which is exclusive for one worker thread.

Definition at line 455 of file context.hpp.

Referenced by DIA< ValueType_, Stack_ >::Filter(), DIA< ValueType_, Stack_ >::FlatMap(), Context::Launch(), and DIA< ValueType_, Stack_ >::Map().

const MemoryConfig& mem_config_
private

memory configuration in HostContext

Definition at line 409 of file context.hpp.

Referenced by Context::mem_config().

size_t mem_limit_
private

memory limit of this worker Context for local data structures

Definition at line 406 of file context.hpp.

Referenced by Context::mem_limit().

mem::Manager& mem_manager_
private

host-global memory manager

Definition at line 412 of file context.hpp.

Referenced by Context::mem_manager().

data::Multiplexer& multiplexer_
private

data::Multiplexer instance that is shared among workers

Definition at line 424 of file context.hpp.

Referenced by Context::GetNewCatStream(), Context::GetNewMixStream(), and Context::Launch().

net::Manager& net_manager_
private

net::Manager instance that is shared among workers

Definition at line 415 of file context.hpp.

Referenced by Context::host_rank(), Context::Launch(), Context::net_manager(), and Context::num_hosts().

size_t workers_per_host_
private

number of workers hosted per host

Definition at line 403 of file context.hpp.

Referenced by Context::workers_per_host().


The documentation for this class was generated from the following files: