Detailed Description

template<typename ValueType, typename FirstDIA, typename SecondDIA, typename KeyExtractor1, typename KeyExtractor2, typename JoinFunction, typename HashFunction, bool UseLocationDetection>
class thrill::api::JoinNode< ValueType, FirstDIA, SecondDIA, KeyExtractor1, KeyExtractor2, JoinFunction, HashFunction, UseLocationDetection >

Performs an inner join between two DIAs.

The key from each DIA element is hereby extracted with a key extractor function. All pairs of elements with equal keys from both DIAs are then joined with the join function.

Template Parameters

KeyExtractor1	Type of the key_extractor1 function. This is a function ValueType to the key type.
KeyExtractor2	Type of the key_extractor2 function. This is a function from SecondDIA::ValueType to the key type.
JoinFunction	Type of the join_function. This is a function from ValueType and SecondDIA::ValueType to the type of the output DIA.

Parameters

SecondDIA	Other DIA joined with this DIA.
key_extractor1	Key extractor for first DIA
key_extractor2	Key extractor for second DIA
join_function	Join function applied to all equal key pairs

Definition at line 61 of file inner_join.hpp.

Inheritance diagram for JoinNode< ValueType, FirstDIA, SecondDIA, KeyExtractor1, KeyExtractor2, JoinFunction, HashFunction, UseLocationDetection >:

Collaboration diagram for JoinNode< ValueType, FirstDIA, SecondDIA, KeyExtractor1, KeyExtractor2, JoinFunction, HashFunction, UseLocationDetection >:

#include <inner_join.hpp>

Classes
class	HashCount
	hash counter used by LocationDetection More...

Public Member Functions
	JoinNode (const FirstDIA &parent1, const SecondDIA &parent2, const KeyExtractor1 &key_extractor1, const KeyExtractor2 &key_extractor2, const JoinFunction &join_function, const HashFunction &hash_function)
	Constructor for a JoinNode. More...

void	Dispose () final
	Virtual clear method. Triggers actual disposing in sub-classes. More...

void	Execute () final
	Virtual execution method. Triggers actual computation in sub-classes. More...

template<typename ElementType , typename CompareFunction >
auto	MakePuller (std::deque< data::File > &files, std::vector< data::File::Reader > &seq, CompareFunction compare_function, bool consume)

void	PushData (bool consume) final
	Virtual method for pushing data. Triggers actual pushing in sub-classes. More...

Public Member Functions inherited from DOpNode< ValueType >
	DOpNode (Context &ctx, const char *label, const std::initializer_list< size_t > &parent_ids, const std::initializer_list< DIABasePtr > &parents)
	Constructor for a DOpNode, which sets references to the parent nodes. More...

	DOpNode (Context &ctx, const char *label, std::vector< size_t > &&parent_ids, std::vector< DIABasePtr > &&parents)
	Constructor for a DOpNode, which sets references to the parent nodes. More...

Public Member Functions inherited from DIANode< ValueType >
	DIANode (Context &ctx, const char *label, const std::initializer_list< size_t > &parent_ids, const std::initializer_list< DIABasePtr > &parents)
	Constructor for a DIANode, which sets references to the parent nodes. More...

	DIANode (Context &ctx, const char *label, std::vector< size_t > &&parent_ids, std::vector< DIABasePtr > &&parents)
	Constructor for a DIANode, which sets references to the parent nodes. More...

virtual void	AddChild (DIABase *node, const Callback &callback=Callback(), size_t parent_index=0)
	Enables children to push their "folded" function chains to their parent. More...

std::vector< DIABase * >	children () const override
	Returns the children of this DIABase. More...

void	PushFile (data::File &file, bool consume) const

void	PushItem (const ValueType &item) const
	Method for derived classes to Push a single item to all children. More...

void	RemoveAllChildren () override

void	RemoveChild (DIABase *node) override

void	RunPushData () override

Public Member Functions inherited from DIABase
	DIABase (Context &ctx, const char *label, const std::initializer_list< size_t > &parent_ids, const std::initializer_list< DIABasePtr > &parents)
	The constructor for a DIABase. More...

	DIABase (Context &ctx, const char *label, std::vector< size_t > &&parent_ids, std::vector< DIABasePtr > &&parents)
	The constructor for a DIABase. More...

	DIABase (const DIABase &)=delete
	non-copyable: delete copy-constructor More...

	DIABase (DIABase &&)=default
	move-constructor: default More...

virtual	~DIABase ()
	Virtual destructor for a DIABase. More...

virtual size_t	consume_counter () const
	Returns consume_counter_. More...

Context &	context ()
	Returns the api::Context of this DIABase. More...

virtual void	DecConsumeCounter (size_t counter)

const size_t &	dia_id () const
	return unique id of DIANode subclass as stored by StatsNode More...

virtual bool	ForwardDataOnly () const

virtual void	IncConsumeCounter (size_t counter)

const char *	label () const
	return label() of DIANode subclass as stored by StatsNode More...

mem::Manager &	mem_manager ()
	Return the Context's memory manager. More...

DIABase &	operator= (const DIABase &)=delete
	non-copyable: delete assignment operator More...

DIABase &	operator= (DIABase &&)=default
	move-assignment operator: default More...

std::vector< size_t >	parent_ids () const
	Returns the parents of this DIABase. More...

const std::vector< DIABasePtr > &	parents () const
	Returns the parents of this DIABase. More...

void	RemoveParent (DIABase *p)
	Remove a parent. More...

virtual bool	RequireParentPushData (size_t) const

void	RunScope ()

void	set_mem_limit (const DIAMemUse &mem_limit)

void	set_state (const DIAState &state)

virtual void	SetConsumeCounter (size_t counter)

DIAState	state () const

virtual bool	OnPreOpFile (const data::File &, size_t)

Public Member Functions inherited from ReferenceCounter
	ReferenceCounter () noexcept
	new objects have zero reference count More...

	ReferenceCounter (const ReferenceCounter &) noexcept
	coping still creates a new object with zero reference count More...

	~ReferenceCounter ()

bool	dec_reference () const noexcept
	Call whenever resetting (i.e. More...

void	inc_reference () const noexcept
	Call whenever setting a pointer to the object. More...

ReferenceCounter &	operator= (const ReferenceCounter &) noexcept
	assignment operator, leaves pointers unchanged More...

size_t	reference_count () const noexcept
	Return the number of references to this object (for debugging) More...

bool	unique () const noexcept
	Test if the ReferenceCounter is referenced by only one CountingPtr. More...

Private Types
using	InputTypeFirst = typename FirstDIA::ValueType

using	InputTypeSecond = typename SecondDIA::ValueType

using	Key = typename common::FunctionTraits< KeyExtractor1 >::result_type
	Key type of join. must be equal to the other key extractor. More...

using	Super = DOpNode< ValueType >

Private Member Functions
template<typename KeyExtractor , typename MergeTree >
std::pair< bool, bool >	AddEqualKeysToFile (MergeTree &puller, const KeyExtractor &key_extractor, data::File::Writer &writer, const Key &key)
	Adds all elements from merge tree to a data::File, potentially to external memory, afterwards sets the first_element pointer to the first element with a different key. More...

template<typename ItemType , typename KeyExtractor , typename MergeTree >
std::pair< bool, bool >	AddEqualKeysToVec (std::vector< ItemType > &vec, MergeTree &puller, const KeyExtractor &key_extractor, data::FilePtr &file_ptr)
	Adds all elements from merge tree to a vector, afterwards sets the first_element pointer to the first element with a different key. More...

DIAMemUse	ExecuteMemUse () final
	Amount of RAM used by Execute() More...

void	JoinAllElements (const std::vector< InputTypeFirst > &vec1, bool external1, const std::vector< InputTypeSecond > &vec2, bool external2)
	Joins all elements in cartesian product of both vectors. More...

template<typename ItemType >
size_t	JoinCapacity ()

void	MainOp ()
	Receive elements from other workers, create pre-sorted files. More...

template<typename ItemType , typename CompareFunction >
void	MergeFiles (std::deque< data::File > &files, CompareFunction compare_function)
	Merge files when there are too many for the merge tree to handle. More...

void	PreOp1 (const InputTypeFirst &input)

void	PreOp2 (const InputTypeSecond &input)

DIAMemUse	PreOpMemUse () final
	Amount of RAM used by PreOp after StartPreOp() More...

DIAMemUse	PushDataMemUse () final
	Amount of RAM used by PushData() More...

template<typename ItemType , typename KeyExtractor >
void	ReceiveItems (size_t capacity, data::MixStream::MixReader &reader, std::deque< data::File > &files, const KeyExtractor &key_extractor)
	Recieve all elements from a stream and write them to files sorted by key. More...

template<typename ItemType , typename KeyExtractor >
void	SortAndWriteToFile (std::vector< ItemType > &vec, std::deque< data::File > &files, const KeyExtractor &key_extractor)
	Sorts all elements in a vector and writes them to a file. More...

void	StartPreOp (size_t parent_index) final
	Virtual method for preparing start of PushData. More...

void	StopPreOp (size_t parent_index) final
	Virtual method for preparing end of PushData. More...

Private Attributes
std::deque< data::File >	files1_
	files for sorted datasets More...

std::deque< data::File >	files2_

HashFunction	hash_function_

data::MixStreamPtr	hash_stream1_ { context_.GetNewMixStream(this) }
	data streams for inter-worker communication of DIA elements More...

data::MixStreamPtr	hash_stream2_ { context_.GetNewMixStream(this) }

data::MixStream::Writers	hash_writers1_ { hash_stream1_->GetWriters() }

data::MixStream::Writers	hash_writers2_ { hash_stream2_->GetWriters() }

data::FilePtr	join_file1_

data::FilePtr	join_file2_

JoinFunction	join_function_

KeyExtractor1	key_extractor1_
	user-defined functions More...

KeyExtractor2	key_extractor2_

core::LocationDetection< HashCount >	location_detection_ { context_, Super::dia_id() }

bool	location_detection_initialized_ = false

data::File	pre_file1_ { context_.GetFile(this) }
	location detection and associated files More...

data::File	pre_file2_ { context_.GetFile(this) }

data::File::Writer	pre_writer1_

data::File::Writer	pre_writer2_

Static Private Attributes
static constexpr bool	debug = false

Additional Inherited Members
Public Types inherited from DOpNode< ValueType >
using	Super = DIANode< ValueType >

Public Types inherited from DIANode< ValueType >
using	Callback = tlx::delegate< void(const ValueType &)>

Public Types inherited from DIABase
using	DIABasePtr = tlx::CountingPtr< DIABase >

Public Attributes inherited from DIABase
common::JsonLogger	logger_

Static Public Attributes inherited from DIABase
static constexpr size_t	kNeverConsume = static_cast<size_t>(-1)
	Never full consume. More...

Protected Attributes inherited from DIANode< ValueType >
std::vector< Child >	children_
	Callback functions from the child nodes. More...

Protected Attributes inherited from DIABase
Context &	context_
	associated Context More...

const size_t	dia_id_
	DIA serial id. More...

const char *const	label_
	DOp node static label. More...

DIAState	state_ = DIAState::NEW
	State of the DIANode. State is NEW on creation. More...

std::vector< DIABasePtr >	parents_
	Parents of this DIABase. More...

DIAMemUse	mem_limit_ = 0

size_t	consume_counter_ = 1

Member Typedef Documentation

◆ InputTypeFirst

using InputTypeFirst = typename FirstDIA::ValueType

private

Definition at line 69 of file inner_join.hpp.

◆ InputTypeSecond

using InputTypeSecond = typename SecondDIA::ValueType

private

Definition at line 70 of file inner_join.hpp.

◆ Key

using Key = typename common::FunctionTraits<KeyExtractor1>::result_type

private

Key type of join. must be equal to the other key extractor.

Definition at line 73 of file inner_join.hpp.

◆ Super

using Super = DOpNode<ValueType>

private

Definition at line 66 of file inner_join.hpp.

Constructor & Destructor Documentation

◆ JoinNode()

JoinNode	(	const FirstDIA &	parent1,
		const SecondDIA &	parent2,
		const KeyExtractor1 &	key_extractor1,
		const KeyExtractor2 &	key_extractor2,
		const JoinFunction &	join_function,
		const HashFunction &	hash_function
	)

inline

Constructor for a JoinNode.

Definition at line 133 of file inner_join.hpp.

References JoinNode< ValueType, FirstDIA, SecondDIA, KeyExtractor1, KeyExtractor2, JoinFunction, HashFunction, UseLocationDetection >::hash_function_, JoinNode< ValueType, FirstDIA, SecondDIA, KeyExtractor1, KeyExtractor2, JoinFunction, HashFunction, UseLocationDetection >::join_function_, JoinNode< ValueType, FirstDIA, SecondDIA, KeyExtractor1, KeyExtractor2, JoinFunction, HashFunction, UseLocationDetection >::key_extractor1_, JoinNode< ValueType, FirstDIA, SecondDIA, KeyExtractor1, KeyExtractor2, JoinFunction, HashFunction, UseLocationDetection >::key_extractor2_, JoinNode< ValueType, FirstDIA, SecondDIA, KeyExtractor1, KeyExtractor2, JoinFunction, HashFunction, UseLocationDetection >::PreOp1(), and JoinNode< ValueType, FirstDIA, SecondDIA, KeyExtractor1, KeyExtractor2, JoinFunction, HashFunction, UseLocationDetection >::PreOp2().

Member Function Documentation

◆ AddEqualKeysToFile()

std::pair<bool, bool> AddEqualKeysToFile	(	MergeTree &	puller,
		const KeyExtractor &	key_extractor,
		data::File::Writer &	writer,
		const Key &	key
	)

inlineprivate

Adds all elements from merge tree to a data::File, potentially to external memory, afterwards sets the first_element pointer to the first element with a different key.

Parameters

puller	Input merge tree
key_extractor	Key extractor function
writer	File writer
key	target key

Returns: Pair of bools, first bool indicates whether the merge tree is emptied, second bool indicates whether external memory was needed (always true, when this method was called).

Definition at line 435 of file inner_join.hpp.

References BlockWriter< BlockSink >::Put().

Referenced by JoinNode< ValueType, FirstDIA, SecondDIA, KeyExtractor1, KeyExtractor2, JoinFunction, HashFunction, UseLocationDetection >::AddEqualKeysToVec().

◆ AddEqualKeysToVec()

std::pair<bool, bool> AddEqualKeysToVec	(	std::vector< ItemType > &	vec,
		MergeTree &	puller,
		const KeyExtractor &	key_extractor,
		data::FilePtr &	file_ptr
	)

inlineprivate

Adds all elements from merge tree to a vector, afterwards sets the first_element pointer to the first element with a different key.

Parameters

vec	target vector
puller	Input merge tree
key_extractor	Key extractor function
file_ptr	Pointer to a data::File

Returns: Pair of bools, first bool indicates whether the merge tree is emptied, second bool indicates whether external memory was needed.

vec is very large when this happens swap with empty vector to free the memory

Definition at line 380 of file inner_join.hpp.

References JoinNode< ValueType, FirstDIA, SecondDIA, KeyExtractor1, KeyExtractor2, JoinFunction, HashFunction, UseLocationDetection >::AddEqualKeysToFile(), DIABase::context_, Context::GetFilePtr(), thrill::mem::memory_exceeded, BlockWriter< BlockSink >::Put(), and tlx::vector_free().

Referenced by JoinNode< ValueType, FirstDIA, SecondDIA, KeyExtractor1, KeyExtractor2, JoinFunction, HashFunction, UseLocationDetection >::PushData().

◆ Dispose()

void Dispose ( )

inlinefinalvirtual

Virtual clear method. Triggers actual disposing in sub-classes.

Reimplemented from DIABase.

Definition at line 289 of file inner_join.hpp.

References JoinNode< ValueType, FirstDIA, SecondDIA, KeyExtractor1, KeyExtractor2, JoinFunction, HashFunction, UseLocationDetection >::files1_, and JoinNode< ValueType, FirstDIA, SecondDIA, KeyExtractor1, KeyExtractor2, JoinFunction, HashFunction, UseLocationDetection >::files2_.

◆ Execute()

void Execute ( )

inlinefinalvirtual

Virtual execution method. Triggers actual computation in sub-classes.

Implements DIABase.

Definition at line 159 of file inner_join.hpp.

◆ ExecuteMemUse()

DIAMemUse ExecuteMemUse ( )

inlinefinalprivatevirtual

Amount of RAM used by Execute()

Reimplemented from DIABase.

Definition at line 483 of file inner_join.hpp.

References DIAMemUse::Max().

◆ JoinAllElements()

void JoinAllElements	(	const std::vector< InputTypeFirst > &	vec1,
		bool	external1,
		const std::vector< InputTypeSecond > &	vec2,
		bool	external2
	)

inlineprivate

Joins all elements in cartesian product of both vectors.

Uses files when one of the data sets is too large to fit in memory. (indicated by 'external' bools)

file 2 needs to be read multiple times

non-consuming reader, need to clear now

Definition at line 569 of file inner_join.hpp.

References BlockReader< BlockSource >::HasNext(), JoinNode< ValueType, FirstDIA, SecondDIA, KeyExtractor1, KeyExtractor2, JoinFunction, HashFunction, UseLocationDetection >::join_function_, JoinNode< ValueType, FirstDIA, SecondDIA, KeyExtractor1, KeyExtractor2, JoinFunction, HashFunction, UseLocationDetection >::key_extractor1_, JoinNode< ValueType, FirstDIA, SecondDIA, KeyExtractor1, KeyExtractor2, JoinFunction, HashFunction, UseLocationDetection >::key_extractor2_, LOG1, thrill::mem::memory_exceeded, and DIANode< ValueType >::PushItem().

Referenced by JoinNode< ValueType, FirstDIA, SecondDIA, KeyExtractor1, KeyExtractor2, JoinFunction, HashFunction, UseLocationDetection >::PushData().

◆ JoinCapacity()

size_t JoinCapacity ( )

inlineprivate

Definition at line 360 of file inner_join.hpp.

References DIABase::mem_limit_.

◆ MainOp()

void MainOp ( )

inlineprivate

Receive elements from other workers, create pre-sorted files.

Definition at line 343 of file inner_join.hpp.

References JoinNode< ValueType, FirstDIA, SecondDIA, KeyExtractor1, KeyExtractor2, JoinFunction, HashFunction, UseLocationDetection >::files1_, JoinNode< ValueType, FirstDIA, SecondDIA, KeyExtractor1, KeyExtractor2, JoinFunction, HashFunction, UseLocationDetection >::files2_, JoinNode< ValueType, FirstDIA, SecondDIA, KeyExtractor1, KeyExtractor2, JoinFunction, HashFunction, UseLocationDetection >::hash_stream1_, JoinNode< ValueType, FirstDIA, SecondDIA, KeyExtractor1, KeyExtractor2, JoinFunction, HashFunction, UseLocationDetection >::hash_stream2_, JoinNode< ValueType, FirstDIA, SecondDIA, KeyExtractor1, KeyExtractor2, JoinFunction, HashFunction, UseLocationDetection >::key_extractor1_, JoinNode< ValueType, FirstDIA, SecondDIA, KeyExtractor1, KeyExtractor2, JoinFunction, HashFunction, UseLocationDetection >::key_extractor2_, and DIABase::mem_limit_.

Referenced by JoinNode< ValueType, FirstDIA, SecondDIA, KeyExtractor1, KeyExtractor2, JoinFunction, HashFunction, UseLocationDetection >::Execute().

◆ MakePuller()

auto MakePuller	(	std::deque< data::File > &	files,
		std::vector< data::File::Reader > &	seq,
		CompareFunction	compare_function,
		bool	consume
	)

inline

Definition at line 196 of file inner_join.hpp.

References Context::block_pool(), DIABase::context_, BlockPool::MaxMergeDegreePrefetch(), and thrill::data::StartPrefetch().

◆ MergeFiles()

void MergeFiles	(	std::deque< data::File > &	files,
		CompareFunction	compare_function
	)

inlineprivate

Merge files when there are too many for the merge tree to handle.

Definition at line 519 of file inner_join.hpp.

References Context::block_pool(), DIABase::context_, Context::GetFile(), BlockPool::MaxMergeDegreePrefetch(), sLOG1, and thrill::data::StartPrefetch().

◆ PreOp1()

void PreOp1 ( const InputTypeFirst & input )

inlineprivate

Definition at line 320 of file inner_join.hpp.

References DIABase::context_, JoinNode< ValueType, FirstDIA, SecondDIA, KeyExtractor1, KeyExtractor2, JoinFunction, HashFunction, UseLocationDetection >::HashCount::hash, JoinNode< ValueType, FirstDIA, SecondDIA, KeyExtractor1, KeyExtractor2, JoinFunction, HashFunction, UseLocationDetection >::hash_function_, JoinNode< ValueType, FirstDIA, SecondDIA, KeyExtractor1, KeyExtractor2, JoinFunction, HashFunction, UseLocationDetection >::hash_writers1_, JoinNode< ValueType, FirstDIA, SecondDIA, KeyExtractor1, KeyExtractor2, JoinFunction, HashFunction, UseLocationDetection >::key_extractor1_, JoinNode< ValueType, FirstDIA, SecondDIA, KeyExtractor1, KeyExtractor2, JoinFunction, HashFunction, UseLocationDetection >::location_detection_, Context::num_workers(), and BlockWriter< BlockSink >::Put().

Referenced by JoinNode< ValueType, FirstDIA, SecondDIA, KeyExtractor1, KeyExtractor2, JoinFunction, HashFunction, UseLocationDetection >::JoinNode().

◆ PreOp2()

void PreOp2 ( const InputTypeSecond & input )

inlineprivate

Definition at line 331 of file inner_join.hpp.

References DIABase::context_, JoinNode< ValueType, FirstDIA, SecondDIA, KeyExtractor1, KeyExtractor2, JoinFunction, HashFunction, UseLocationDetection >::HashCount::hash, JoinNode< ValueType, FirstDIA, SecondDIA, KeyExtractor1, KeyExtractor2, JoinFunction, HashFunction, UseLocationDetection >::hash_function_, JoinNode< ValueType, FirstDIA, SecondDIA, KeyExtractor1, KeyExtractor2, JoinFunction, HashFunction, UseLocationDetection >::hash_writers2_, JoinNode< ValueType, FirstDIA, SecondDIA, KeyExtractor1, KeyExtractor2, JoinFunction, HashFunction, UseLocationDetection >::key_extractor2_, JoinNode< ValueType, FirstDIA, SecondDIA, KeyExtractor1, KeyExtractor2, JoinFunction, HashFunction, UseLocationDetection >::location_detection_, Context::num_workers(), and BlockWriter< BlockSink >::Put().

Referenced by JoinNode< ValueType, FirstDIA, SecondDIA, KeyExtractor1, KeyExtractor2, JoinFunction, HashFunction, UseLocationDetection >::JoinNode().

◆ PreOpMemUse()

DIAMemUse PreOpMemUse ( )

inlinefinalprivatevirtual

Amount of RAM used by PreOp after StartPreOp()

Reimplemented from DIABase.

Definition at line 451 of file inner_join.hpp.

References DIAMemUse::Max().

◆ PushData()

void PushData ( bool consume )

inlinefinalvirtual

Virtual method for pushing data. Triggers actual pushing in sub-classes.

Merge files when there are too many for the merge tree.

cache for elements with equal keys, cartesian product of both caches are joined with the join_function

find elements with equal key

Implements DIABase.

Definition at line 213 of file inner_join.hpp.

◆ PushDataMemUse()

DIAMemUse PushDataMemUse ( )

inlinefinalprivatevirtual

Amount of RAM used by PushData()

Reimplemented from DIABase.

Definition at line 487 of file inner_join.hpp.

References DIAMemUse::Max().

◆ ReceiveItems()

void ReceiveItems	(	size_t	capacity,
		data::MixStream::MixReader &	reader,
		std::deque< data::File > &	files,
		const KeyExtractor &	key_extractor
	)

inlineprivate

Recieve all elements from a stream and write them to files sorted by key.

Definition at line 495 of file inner_join.hpp.

References MixBlockQueueReader::HasNext(), and JoinNode< ValueType, FirstDIA, SecondDIA, KeyExtractor1, KeyExtractor2, JoinFunction, HashFunction, UseLocationDetection >::SortAndWriteToFile().

◆ SortAndWriteToFile()

void SortAndWriteToFile	(	std::vector< ItemType > &	vec,
		std::deque< data::File > &	files,
		const KeyExtractor &	key_extractor
	)

inlineprivate

Sorts all elements in a vector and writes them to a file.

Definition at line 649 of file inner_join.hpp.

References BlockPool::AdviseFree(), Context::block_pool(), DIABase::context_, and Context::GetFile().

Referenced by JoinNode< ValueType, FirstDIA, SecondDIA, KeyExtractor1, KeyExtractor2, JoinFunction, HashFunction, UseLocationDetection >::ReceiveItems().

◆ StartPreOp()

void StartPreOp ( size_t )

inlinefinalprivatevirtual

Virtual method for preparing start of PushData.

Reimplemented from DIABase.

Definition at line 455 of file inner_join.hpp.

References File::GetWriter(), JoinNode< ValueType, FirstDIA, SecondDIA, KeyExtractor1, KeyExtractor2, JoinFunction, HashFunction, UseLocationDetection >::location_detection_, LOG, DIABase::mem_limit_, DIABase::parent_ids(), JoinNode< ValueType, FirstDIA, SecondDIA, KeyExtractor1, KeyExtractor2, JoinFunction, HashFunction, UseLocationDetection >::pre_file1_, and JoinNode< ValueType, FirstDIA, SecondDIA, KeyExtractor1, KeyExtractor2, JoinFunction, HashFunction, UseLocationDetection >::pre_file2_.

◆ StopPreOp()

void StopPreOp ( size_t )

inlinefinalprivatevirtual

Virtual method for preparing end of PushData.

Reimplemented from DIABase.

Definition at line 472 of file inner_join.hpp.

References BlockWriter< BlockSink >::Close(), and LOG.

Member Data Documentation

◆ debug

constexpr bool debug = false

staticprivate

Definition at line 64 of file inner_join.hpp.

◆ files1_

std::deque<data::File> files1_

private

files for sorted datasets

Definition at line 296 of file inner_join.hpp.

Referenced by JoinNode< ValueType, FirstDIA, SecondDIA, KeyExtractor1, KeyExtractor2, JoinFunction, HashFunction, UseLocationDetection >::Dispose(), JoinNode< ValueType, FirstDIA, SecondDIA, KeyExtractor1, KeyExtractor2, JoinFunction, HashFunction, UseLocationDetection >::MainOp(), and JoinNode< ValueType, FirstDIA, SecondDIA, KeyExtractor1, KeyExtractor2, JoinFunction, HashFunction, UseLocationDetection >::PushData().

◆ files2_

std::deque<data::File> files2_

private

Definition at line 297 of file inner_join.hpp.

Referenced by JoinNode< ValueType, FirstDIA, SecondDIA, KeyExtractor1, KeyExtractor2, JoinFunction, HashFunction, UseLocationDetection >::Dispose(), JoinNode< ValueType, FirstDIA, SecondDIA, KeyExtractor1, KeyExtractor2, JoinFunction, HashFunction, UseLocationDetection >::MainOp(), and JoinNode< ValueType, FirstDIA, SecondDIA, KeyExtractor1, KeyExtractor2, JoinFunction, HashFunction, UseLocationDetection >::PushData().

◆ hash_function_

HashFunction hash_function_

private

Definition at line 303 of file inner_join.hpp.

Referenced by JoinNode< ValueType, FirstDIA, SecondDIA, KeyExtractor1, KeyExtractor2, JoinFunction, HashFunction, UseLocationDetection >::Execute(), JoinNode< ValueType, FirstDIA, SecondDIA, KeyExtractor1, KeyExtractor2, JoinFunction, HashFunction, UseLocationDetection >::JoinNode(), JoinNode< ValueType, FirstDIA, SecondDIA, KeyExtractor1, KeyExtractor2, JoinFunction, HashFunction, UseLocationDetection >::PreOp1(), and JoinNode< ValueType, FirstDIA, SecondDIA, KeyExtractor1, KeyExtractor2, JoinFunction, HashFunction, UseLocationDetection >::PreOp2().

◆ hash_stream1_

data::MixStreamPtr hash_stream1_ { context_.GetNewMixStream(this) }

private

data streams for inter-worker communication of DIA elements

Definition at line 306 of file inner_join.hpp.

Referenced by JoinNode< ValueType, FirstDIA, SecondDIA, KeyExtractor1, KeyExtractor2, JoinFunction, HashFunction, UseLocationDetection >::MainOp().

◆ hash_stream2_

data::MixStreamPtr hash_stream2_ { context_.GetNewMixStream(this) }

private

Definition at line 308 of file inner_join.hpp.

Referenced by JoinNode< ValueType, FirstDIA, SecondDIA, KeyExtractor1, KeyExtractor2, JoinFunction, HashFunction, UseLocationDetection >::MainOp().

◆ hash_writers1_

data::MixStream::Writers hash_writers1_ { hash_stream1_->GetWriters() }

private

Definition at line 307 of file inner_join.hpp.

Referenced by JoinNode< ValueType, FirstDIA, SecondDIA, KeyExtractor1, KeyExtractor2, JoinFunction, HashFunction, UseLocationDetection >::Execute(), and JoinNode< ValueType, FirstDIA, SecondDIA, KeyExtractor1, KeyExtractor2, JoinFunction, HashFunction, UseLocationDetection >::PreOp1().

◆ hash_writers2_

data::MixStream::Writers hash_writers2_ { hash_stream2_->GetWriters() }

private

Definition at line 309 of file inner_join.hpp.

Referenced by JoinNode< ValueType, FirstDIA, SecondDIA, KeyExtractor1, KeyExtractor2, JoinFunction, HashFunction, UseLocationDetection >::Execute(), and JoinNode< ValueType, FirstDIA, SecondDIA, KeyExtractor1, KeyExtractor2, JoinFunction, HashFunction, UseLocationDetection >::PreOp2().

◆ join_file1_

data::FilePtr join_file1_

private

Definition at line 561 of file inner_join.hpp.

Referenced by JoinNode< ValueType, FirstDIA, SecondDIA, KeyExtractor1, KeyExtractor2, JoinFunction, HashFunction, UseLocationDetection >::PushData().

◆ join_file2_

data::FilePtr join_file2_

private

Definition at line 562 of file inner_join.hpp.

Referenced by JoinNode< ValueType, FirstDIA, SecondDIA, KeyExtractor1, KeyExtractor2, JoinFunction, HashFunction, UseLocationDetection >::PushData().

◆ join_function_

JoinFunction join_function_

private

Definition at line 302 of file inner_join.hpp.

Referenced by JoinNode< ValueType, FirstDIA, SecondDIA, KeyExtractor1, KeyExtractor2, JoinFunction, HashFunction, UseLocationDetection >::JoinAllElements(), and JoinNode< ValueType, FirstDIA, SecondDIA, KeyExtractor1, KeyExtractor2, JoinFunction, HashFunction, UseLocationDetection >::JoinNode().

◆ key_extractor1_

KeyExtractor1 key_extractor1_

private

user-defined functions

Definition at line 300 of file inner_join.hpp.

Referenced by JoinNode< ValueType, FirstDIA, SecondDIA, KeyExtractor1, KeyExtractor2, JoinFunction, HashFunction, UseLocationDetection >::Execute(), JoinNode< ValueType, FirstDIA, SecondDIA, KeyExtractor1, KeyExtractor2, JoinFunction, HashFunction, UseLocationDetection >::JoinAllElements(), JoinNode< ValueType, FirstDIA, SecondDIA, KeyExtractor1, KeyExtractor2, JoinFunction, HashFunction, UseLocationDetection >::JoinNode(), JoinNode< ValueType, FirstDIA, SecondDIA, KeyExtractor1, KeyExtractor2, JoinFunction, HashFunction, UseLocationDetection >::MainOp(), JoinNode< ValueType, FirstDIA, SecondDIA, KeyExtractor1, KeyExtractor2, JoinFunction, HashFunction, UseLocationDetection >::PreOp1(), and JoinNode< ValueType, FirstDIA, SecondDIA, KeyExtractor1, KeyExtractor2, JoinFunction, HashFunction, UseLocationDetection >::PushData().

◆ key_extractor2_

KeyExtractor2 key_extractor2_

private

Definition at line 301 of file inner_join.hpp.

Referenced by JoinNode< ValueType, FirstDIA, SecondDIA, KeyExtractor1, KeyExtractor2, JoinFunction, HashFunction, UseLocationDetection >::Execute(), JoinNode< ValueType, FirstDIA, SecondDIA, KeyExtractor1, KeyExtractor2, JoinFunction, HashFunction, UseLocationDetection >::JoinAllElements(), JoinNode< ValueType, FirstDIA, SecondDIA, KeyExtractor1, KeyExtractor2, JoinFunction, HashFunction, UseLocationDetection >::JoinNode(), JoinNode< ValueType, FirstDIA, SecondDIA, KeyExtractor1, KeyExtractor2, JoinFunction, HashFunction, UseLocationDetection >::MainOp(), JoinNode< ValueType, FirstDIA, SecondDIA, KeyExtractor1, KeyExtractor2, JoinFunction, HashFunction, UseLocationDetection >::PreOp2(), and JoinNode< ValueType, FirstDIA, SecondDIA, KeyExtractor1, KeyExtractor2, JoinFunction, HashFunction, UseLocationDetection >::PushData().

◆ location_detection_

core::LocationDetection<HashCount> location_detection_ { context_, Super::dia_id() }

private

Definition at line 317 of file inner_join.hpp.

Referenced by JoinNode< ValueType, FirstDIA, SecondDIA, KeyExtractor1, KeyExtractor2, JoinFunction, HashFunction, UseLocationDetection >::Execute(), JoinNode< ValueType, FirstDIA, SecondDIA, KeyExtractor1, KeyExtractor2, JoinFunction, HashFunction, UseLocationDetection >::PreOp1(), JoinNode< ValueType, FirstDIA, SecondDIA, KeyExtractor1, KeyExtractor2, JoinFunction, HashFunction, UseLocationDetection >::PreOp2(), and JoinNode< ValueType, FirstDIA, SecondDIA, KeyExtractor1, KeyExtractor2, JoinFunction, HashFunction, UseLocationDetection >::StartPreOp().

◆ location_detection_initialized_

bool location_detection_initialized_ = false

private

Definition at line 318 of file inner_join.hpp.

◆ pre_file1_

data::File pre_file1_ { context_.GetFile(this) }

private

location detection and associated files

Definition at line 312 of file inner_join.hpp.

Referenced by JoinNode< ValueType, FirstDIA, SecondDIA, KeyExtractor1, KeyExtractor2, JoinFunction, HashFunction, UseLocationDetection >::Execute(), and JoinNode< ValueType, FirstDIA, SecondDIA, KeyExtractor1, KeyExtractor2, JoinFunction, HashFunction, UseLocationDetection >::StartPreOp().

◆ pre_file2_

data::File pre_file2_ { context_.GetFile(this) }

private

Definition at line 314 of file inner_join.hpp.

Referenced by JoinNode< ValueType, FirstDIA, SecondDIA, KeyExtractor1, KeyExtractor2, JoinFunction, HashFunction, UseLocationDetection >::Execute(), and JoinNode< ValueType, FirstDIA, SecondDIA, KeyExtractor1, KeyExtractor2, JoinFunction, HashFunction, UseLocationDetection >::StartPreOp().

◆ pre_writer1_

data::File::Writer pre_writer1_

private

Definition at line 313 of file inner_join.hpp.

◆ pre_writer2_

data::File::Writer pre_writer2_

private

Definition at line 315 of file inner_join.hpp.

The documentation for this class was generated from the following file:

thrill/api/inner_join.hpp

Detailed Description

Classes

Public Member Functions

Private Types

Private Member Functions

Private Attributes

Static Private Attributes

Additional Inherited Members

Member Typedef Documentation

◆ InputTypeFirst

◆ InputTypeSecond

◆ Key

◆ Super

Constructor & Destructor Documentation

◆ JoinNode()

Member Function Documentation

◆ AddEqualKeysToFile()

◆ AddEqualKeysToVec()

◆ Dispose()

◆ Execute()

◆ ExecuteMemUse()

◆ JoinAllElements()

◆ JoinCapacity()

◆ MainOp()

◆ MakePuller()

◆ MergeFiles()

◆ PreOp1()

◆ PreOp2()

◆ PreOpMemUse()

◆ PushData()

◆ PushDataMemUse()

◆ ReceiveItems()

◆ SortAndWriteToFile()

◆ StartPreOp()

◆ StopPreOp()

Member Data Documentation

◆ debug

◆ files1_

◆ files2_

◆ hash_function_

◆ hash_stream1_

◆ hash_stream2_

◆ hash_writers1_

◆ hash_writers2_

◆ join_file1_

◆ join_file2_

◆ join_function_

◆ key_extractor1_

◆ key_extractor2_

◆ location_detection_

◆ location_detection_initialized_

◆ pre_file1_

◆ pre_file2_

◆ pre_writer1_

◆ pre_writer2_