Detailed Description

The HostContext contains all data structures shared among workers on the same host.

It is used to construct and destroy them. For testing multiple instances are run in the same process.

Definition at line 90 of file context.hpp.

Collaboration diagram for HostContext:

#include <context.hpp>

Public Member Functions
	HostContext (size_t local_host_id, const MemoryConfig &mem_config, std::unique_ptr< net::DispatcherThread > dispatcher, std::array< net::GroupPtr, net::Manager::kGroupCount > &&groups, size_t workers_per_host)
	constructor from existing net Groups. Used by the construction methods. More...

	~HostContext ()
	destructor More...

data::BlockPool &	block_pool ()
	the block manager keeps all data blocks moving through the system. More...

data::Multiplexer &	data_multiplexer ()
	data multiplexer transmits large amounts of data asynchronously. More...

net::FlowControlChannelManager &	flow_manager ()
	the flow control group is used for collective communication. More...

size_t	host_rank () const

size_t	local_host_id () const
	Returns local_host_id_. More...

std::string	MakeHostLogPath (size_t host_rank)
	create host log More...

MemoryConfig &	mem_config ()
	host-global memory config More...

mem::Manager &	mem_manager ()
	host-global memory manager More...

net::Manager &	net_manager ()
	net manager constructs communication groups to other hosts. More...

size_t	worker_mem_limit () const
	memory limit of each worker Context for local data structures More...

size_t	workers_per_host () const
	number of workers per host (all have the same). More...

Static Public Member Functions
static std::vector< std::unique_ptr< HostContext > >	ConstructLoopback (size_t num_hosts, size_t workers_per_host)
	Construct a number of mock hosts running in this process. More...

Public Attributes
Logging System
common::JsonLogger	base_logger_
	base logger exclusive for this host context More...

common::JsonLogger	logger_

std::unique_ptr< common::ProfileThread >	profiler_
	thread for scheduling profiling methods for statistical output More...

Private Attributes
data::BlockPool	block_pool_
	data block pool More...

common::ProfileTaskRegistration	block_pool_profiler_
	register BlockPool's profiling method More...

data::Multiplexer	data_multiplexer_
	data multiplexer transmits large amounts of data asynchronously. More...

std::unique_ptr< net::DispatcherThread >	dispatcher_
	main host network dispatcher thread backend More...

net::FlowControlChannelManager	flow_manager_
	the flow control group is used for collective communication. More...

size_t	local_host_id_
	id among all local hosts (in test program runs) More...

MemoryConfig	mem_config_
	memory configuration More...

mem::Manager	mem_manager_ { nullptr, "HostContext" }
	host-global memory manager for internal memory only More...

net::Manager	net_manager_
	net manager constructs communication groups to other hosts. More...

common::ProfileTaskRegistration	net_manager_profiler_
	register net_manager_'s profiling method More...

size_t	workers_per_host_
	number of workers per host (all have the same). More...

Constructor & Destructor Documentation

◆ HostContext()

HostContext	(	size_t	local_host_id,
		const MemoryConfig &	mem_config,
		std::unique_ptr< net::DispatcherThread >	dispatcher,
		std::array< net::GroupPtr, net::Manager::kGroupCount > &&	groups,
		size_t	workers_per_host
	)

constructor from existing net Groups. Used by the construction methods.

Definition at line 1123 of file context.cpp.

References MemoryConfig::enable_proc_profiler_, thrill::common::LogCmdlineParams(), HostContext::logger_, HostContext::mem_config_, HostContext::profiler_, thrill::common::StartLinuxProcStatsProfiler(), and thrill::mem::StartMemProfiler().

◆ ~HostContext()

~HostContext ( )

destructor

Definition at line 1149 of file context.cpp.

References HostContext::dispatcher_.

Member Function Documentation

◆ block_pool()

data::BlockPool& block_pool ( )

inline

the block manager keeps all data blocks moving through the system.

Definition at line 139 of file context.hpp.

◆ ConstructLoopback()

std::vector< std::unique_ptr< HostContext > > ConstructLoopback	(	size_t	num_hosts,
		size_t	workers_per_host
	)

static

Construct a number of mock hosts running in this process.

Definition at line 308 of file context.cpp.

References HostContext::mem_config(), MemoryConfig::setup(), MemoryConfig::verbose_, and HostContext::workers_per_host().

◆ data_multiplexer()

data::Multiplexer& data_multiplexer ( )

inline

data multiplexer transmits large amounts of data asynchronously.

Definition at line 142 of file context.hpp.

◆ flow_manager()

net::FlowControlChannelManager& flow_manager ( )

inline

the flow control group is used for collective communication.

Definition at line 136 of file context.hpp.

◆ host_rank()

size_t host_rank ( ) const

inline

Returns id of this host in the cluser. A host is a machine in the cluster that hosts multiple workers

Definition at line 133 of file context.hpp.

◆ local_host_id()

size_t local_host_id ( ) const

inline

Returns local_host_id_.

Definition at line 112 of file context.hpp.

◆ MakeHostLogPath()

std::string MakeHostLogPath ( size_t host_rank )

create host log

Definition at line 1154 of file context.cpp.

References HostContext::mem_config(), and thrill::mem::to_string().

◆ mem_config()

MemoryConfig& mem_config ( )

inline

host-global memory config

Definition at line 123 of file context.hpp.

Referenced by HostContext::ConstructLoopback(), HostContext::MakeHostLogPath(), thrill::api::RunBackendLoopback(), thrill::api::RunLocalSameThread(), and thrill::api::RunLocalTests().

◆ mem_manager()

mem::Manager& mem_manager ( )

inline

host-global memory manager

Definition at line 126 of file context.hpp.

◆ net_manager()

net::Manager& net_manager ( )

inline

net manager constructs communication groups to other hosts.

Definition at line 129 of file context.hpp.

◆ worker_mem_limit()

size_t worker_mem_limit ( ) const

inline

memory limit of each worker Context for local data structures

Definition at line 118 of file context.hpp.

◆ workers_per_host()

size_t workers_per_host ( ) const

inline

number of workers per host (all have the same).

Definition at line 115 of file context.hpp.

Referenced by HostContext::ConstructLoopback(), and thrill::api::RunBackendLoopback().

Member Data Documentation

◆ base_logger_

common::JsonLogger base_logger_

base logger exclusive for this host context

Definition at line 153 of file context.hpp.

◆ block_pool_

data::BlockPool block_pool_

private

Initial value:

{
        mem_config_.ram_block_pool_soft_, mem_config_.ram_block_pool_hard_,
        &logger_, &mem_manager_, workers_per_host_
    }

data block pool

Definition at line 194 of file context.hpp.

◆ block_pool_profiler_

common::ProfileTaskRegistration block_pool_profiler_

private

Initial value:

{
        std::chrono::milliseconds(500), *profiler_, &block_pool_
    }

register BlockPool's profiling method

Definition at line 201 of file context.hpp.

◆ data_multiplexer_

data::Multiplexer data_multiplexer_

private

Initial value:

{
        mem_manager_, block_pool_,
        *dispatcher_, net_manager_.GetDataGroup(), workers_per_host_
    }

data multiplexer transmits large amounts of data asynchronously.

Definition at line 207 of file context.hpp.

◆ dispatcher_

std::unique_ptr<net::DispatcherThread> dispatcher_

private

main host network dispatcher thread backend

Definition at line 176 of file context.hpp.

Referenced by HostContext::~HostContext().

◆ flow_manager_

net::FlowControlChannelManager flow_manager_

private

Initial value:

{
        net_manager_.GetFlowGroup(), workers_per_host_
    }

the flow control group is used for collective communication.

Definition at line 189 of file context.hpp.

◆ local_host_id_

size_t local_host_id_

private

id among all local hosts (in test program runs)

Definition at line 167 of file context.hpp.

◆ logger_

common::JsonLogger logger_

public member which delivers key:value pairs as JSON log lines. this logger is local to this Context which is exclusive for one worker thread.

Definition at line 158 of file context.hpp.

Referenced by HostContext::HostContext().

◆ mem_config_

MemoryConfig mem_config_

private

memory configuration

Definition at line 146 of file context.hpp.

Referenced by HostContext::HostContext().

◆ mem_manager_

mem::Manager mem_manager_ { nullptr, "HostContext" }

private

host-global memory manager for internal memory only

Definition at line 173 of file context.hpp.

◆ net_manager_

net::Manager net_manager_

private

net manager constructs communication groups to other hosts.

Definition at line 179 of file context.hpp.

◆ net_manager_profiler_

common::ProfileTaskRegistration net_manager_profiler_

private

Initial value:

{
        std::chrono::milliseconds(500), *profiler_, &net_manager_
    }

register net_manager_'s profiling method

Definition at line 183 of file context.hpp.

◆ profiler_

std::unique_ptr<common::ProfileThread> profiler_

thread for scheduling profiling methods for statistical output

Definition at line 161 of file context.hpp.

Referenced by HostContext::HostContext().

◆ workers_per_host_

size_t workers_per_host_

private

number of workers per host (all have the same).

Definition at line 170 of file context.hpp.

The documentation for this class was generated from the following files:

thrill/api/context.hpp
thrill/api/context.cpp

Detailed Description

Public Member Functions

Static Public Member Functions

Public Attributes

Private Attributes

Constructor & Destructor Documentation

◆ HostContext()

◆ ~HostContext()

Member Function Documentation

◆ block_pool()

◆ ConstructLoopback()

◆ data_multiplexer()

◆ flow_manager()

◆ host_rank()

◆ local_host_id()

◆ MakeHostLogPath()

◆ mem_config()

◆ mem_manager()

◆ net_manager()

◆ worker_mem_limit()

◆ workers_per_host()

Member Data Documentation

◆ base_logger_

◆ block_pool_

◆ block_pool_profiler_

◆ data_multiplexer_

◆ dispatcher_

◆ flow_manager_

◆ local_host_id_

◆ logger_

◆ mem_config_

◆ mem_manager_

◆ net_manager_

◆ net_manager_profiler_

◆ profiler_

◆ workers_per_host_