Thrill  0.1
 All Classes Namespaces Files Functions Variables Typedefs Enumerations Enumerator Friends Macros Groups Pages
ReadBinaryNode< ValueType > Class Template Referencefinal

Detailed Description

template<typename ValueType>
class thrill::api::ReadBinaryNode< ValueType >

A DIANode which performs a line-based Read operation.

Read reads a file from the file system and emits it as a DIA.

Definition at line 44 of file read_binary.hpp.

+ Inheritance diagram for ReadBinaryNode< ValueType >:
+ Collaboration diagram for ReadBinaryNode< ValueType >:

#include <read_binary.hpp>

Classes

struct  FileInfo
 structure to store info on what to read from files More...
 
class  VfsFileBlockSource
 

Public Types

using Super = SourceNode< ValueType >
 
using VfsFileBlockReader = data::BlockReader< VfsFileBlockSource >
 
- Public Types inherited from SourceNode< ValueType >
using Super = DIANode< ValueType >
 
- Public Types inherited from DIANode< ValueType >
using Callback = tlx::delegate< void(const ValueType &)>
 
- Public Types inherited from DIABase
using DIABasePtr = tlx::CountingPtr< DIABase >
 

Public Member Functions

 ReadBinaryNode (Context &ctx, const std::vector< std::string > &globlist, uint64_t size_limit, bool local_storage)
 
 ReadBinaryNode (Context &ctx, const std::string &glob, uint64_t size_limit, bool local_storage)
 
- Public Member Functions inherited from SourceNode< ValueType >
 SourceNode (Context &ctx, const char *label)
 
- Public Member Functions inherited from DIANode< ValueType >
 DIANode (Context &ctx, const char *label, const std::initializer_list< size_t > &parent_ids, const std::initializer_list< DIABasePtr > &parents)
 Constructor for a DIANode, which sets references to the parent nodes. More...
 
 DIANode (Context &ctx, const char *label, std::vector< size_t > &&parent_ids, std::vector< DIABasePtr > &&parents)
 Constructor for a DIANode, which sets references to the parent nodes. More...
 
virtual void AddChild (DIABase *node, const Callback &callback=Callback(), size_t parent_index=0)
 Enables children to push their "folded" function chains to their parent. More...
 
std::vector< DIABase * > children () const override
 Returns the children of this DIABase. More...
 
void PushFile (data::File &file, bool consume) const
 
void PushItem (const ValueType &item) const
 Method for derived classes to Push a single item to all children. More...
 
void RemoveAllChildren () override
 
void RemoveChild (DIABase *node) override
 
void RunPushData () override
 
- Public Member Functions inherited from DIABase
 DIABase (Context &ctx, const char *label, const std::initializer_list< size_t > &parent_ids, const std::initializer_list< DIABasePtr > &parents)
 The constructor for a DIABase. More...
 
 DIABase (Context &ctx, const char *label, std::vector< size_t > &&parent_ids, std::vector< DIABasePtr > &&parents)
 The constructor for a DIABase. More...
 
 DIABase (const DIABase &)=delete
 non-copyable: delete copy-constructor More...
 
 DIABase (DIABase &&)=default
 move-constructor: default More...
 
virtual ~DIABase ()
 Virtual destructor for a DIABase. More...
 
virtual size_t consume_counter () const
 Returns consume_counter_. More...
 
Contextcontext ()
 Returns the api::Context of this DIABase. More...
 
virtual void DecConsumeCounter (size_t counter)
 
virtual bool ForwardDataOnly () const
 
const size_t & id () const
 return unique id() of DIANode subclass as stored by StatsNode More...
 
virtual void IncConsumeCounter (size_t counter)
 
const char * label () const
 return label() of DIANode subclass as stored by StatsNode More...
 
mem::Managermem_manager ()
 Return the Context's memory manager. More...
 
DIABaseoperator= (const DIABase &)=delete
 non-copyable: delete assignment operator More...
 
DIABaseoperator= (DIABase &&)=default
 move-assignment operator: default More...
 
std::vector< size_t > parent_ids () const
 Returns the parents of this DIABase. More...
 
const std::vector< DIABasePtr > & parents () const
 Returns the parents of this DIABase. More...
 
void RemoveParent (DIABase *p)
 Remove a parent. More...
 
virtual bool RequireParentPushData (size_t) const
 
void RunScope ()
 
void set_mem_limit (const DIAMemUse &mem_limit)
 
void set_state (const DIAState &state)
 
virtual void SetConsumeCounter (size_t counter)
 
DIAState state () const
 
virtual DIAMemUse PreOpMemUse ()
 Amount of RAM used by PreOp after StartPreOp() More...
 
virtual void StartPreOp (size_t)
 Virtual method for preparing start of PushData. More...
 
virtual bool OnPreOpFile (const data::File &, size_t)
 
virtual void StopPreOp (size_t)
 Virtual method for preparing end of PushData. More...
 
virtual DIAMemUse ExecuteMemUse ()
 Amount of RAM used by Execute() More...
 
virtual void Execute ()=0
 Virtual execution method. Triggers actual computation in sub-classes. More...
 
virtual DIAMemUse PushDataMemUse ()
 Amount of RAM used by PushData() More...
 
virtual void PushData (bool consume)=0
 Virtual method for pushing data. Triggers actual pushing in sub-classes. More...
 
virtual void Dispose ()
 Virtual clear method. Triggers actual disposing in sub-classes. More...
 
- Public Member Functions inherited from ReferenceCounter
 ReferenceCounter () noexcept
 new objects have zero reference count More...
 
 ReferenceCounter (const ReferenceCounter &) noexcept
 coping still creates a new object with zero reference count More...
 
 ~ReferenceCounter ()
 
bool dec_reference () const noexcept
 Call whenever resetting (i.e. More...
 
void inc_reference () const noexcept
 Call whenever setting a pointer to the object. More...
 
ReferenceCounteroperator= (const ReferenceCounter &) noexcept
 assignment operator, leaves pointers unchanged More...
 
size_t reference_count () const noexcept
 Return the number of references to this object (for debugging) More...
 
bool unique () const noexcept
 Test if the ReferenceCounter is referenced by only one CountingPtr. More...
 

Public Attributes

data::File ext_file_ { context_.GetFile(this) }
 
 local_storage
 
 size_limit
 
size_t stats_total_bytes = 0
 
size_t stats_total_reads = 0
 
bool use_ext_file_ = false
 File containing Blocks mapped directly to a io fileimpl. More...
 
- Public Attributes inherited from DIABase
common::JsonLogger logger_
 

Static Public Attributes

static constexpr size_t fixed_size_
 fixed size of ValueType or zero. More...
 
static constexpr bool is_fixed_size_
 flag whether ValueType is fixed size More...
 
static constexpr uint64_t no_size_limit_
 sentinel to disable size limit More...
 
- Static Public Attributes inherited from DIABase
static constexpr size_t kNeverConsume = static_cast<size_t>(-1)
 Never full consume. More...
 

Static Private Attributes

static constexpr bool debug = false
 
static constexpr bool debug_no_extfile = false
 for testing old method of pushing items instead of PushFile(). More...
 

Additional Inherited Members

- Protected Attributes inherited from DIANode< ValueType >
std::vector< Childchildren_
 Callback functions from the child nodes. More...
 
- Protected Attributes inherited from DIABase
Contextcontext_
 associated Context More...
 
const size_t id_
 DIA serial id. More...
 
const char *const label_
 DOp node static label. More...
 
DIAState state_ = DIAState::NEW
 State of the DIANode. State is NEW on creation. More...
 
std::vector< DIABasePtrparents_
 Parents of this DIABase. More...
 
DIAMemUse mem_limit_ = 0
 
size_t consume_counter_ = 1
 

Member Typedef Documentation

using Super = SourceNode<ValueType>

Definition at line 55 of file read_binary.hpp.

Constructor & Destructor Documentation

ReadBinaryNode ( Context ctx,
const std::string &  glob,
uint64_t  size_limit,
bool  local_storage 
)
inline

Definition at line 223 of file read_binary.hpp.

Member Data Documentation

constexpr bool debug = false
staticprivate

Definition at line 46 of file read_binary.hpp.

constexpr bool debug_no_extfile = false
staticprivate

for testing old method of pushing items instead of PushFile().

Definition at line 49 of file read_binary.hpp.

Referenced by ReadBinaryNode< ValueType >::ReadBinaryNode().

data::File ext_file_ { context_.GetFile(this) }

Definition at line 267 of file read_binary.hpp.

Referenced by ReadBinaryNode< ValueType >::ReadBinaryNode().

constexpr size_t fixed_size_
static
Initial value:
=
data::Serialization<VfsFileBlockReader, ValueType>::fixed_size

fixed size of ValueType or zero.

Definition at line 65 of file read_binary.hpp.

Referenced by ReadBinaryNode< ValueType >::ReadBinaryNode().

constexpr bool is_fixed_size_
static
Initial value:
=
data::Serialization<VfsFileBlockReader, ValueType>::is_fixed_size

flag whether ValueType is fixed size

Definition at line 61 of file read_binary.hpp.

Referenced by ReadBinaryNode< ValueType >::ReadBinaryNode().

local_storage

Definition at line 226 of file read_binary.hpp.

constexpr uint64_t no_size_limit_
static
Initial value:

sentinel to disable size limit

Definition at line 78 of file read_binary.hpp.

Referenced by ReadBinaryNode< ValueType >::ReadBinaryNode().

size_limit

Definition at line 225 of file read_binary.hpp.

size_t stats_total_bytes = 0

Definition at line 269 of file read_binary.hpp.

size_t stats_total_reads = 0

Definition at line 270 of file read_binary.hpp.

bool use_ext_file_ = false

File containing Blocks mapped directly to a io fileimpl.

Definition at line 266 of file read_binary.hpp.

Referenced by ReadBinaryNode< ValueType >::ReadBinaryNode().


The documentation for this class was generated from the following file: