Overview and basic concepts | The data sets involved in some modern applications are too large to fit in the main memory of even the most powerful computers and must therefore reside on disk |
Compiling and installing TPIE | You will need Git, CMake, Boost and a C++ compiler in order to use TPIE on any system |
Repository overview | |
File list | Dummy Doxygen destination linking to the autogenerated file list |
Authors | The TPIE development team consists of the following people: |
Reading and writing file streams | The central class in TPIE for reading from files and writing to files is the file_stream class |
Internal sorting | Internal memory sorting is implemented in parallel_sort.h as a parallel quick sort with a sequential partition step |
External sorting | The recommended way to sort large data set (which will typically be stored in a tpie::file_stream in TPIE) is to use one of the tpie::sort functions defined in sort.h |
Priority queue | TPIE provides a very efficient external memory priority queue based off Sanders, Fast Priority Queues for Cached Memory |
Memory manager | TPIE provides various external memory algorithms and data structures, most prominently an external memory priority queue (tpie::ami::priority_queue) and external memory sorting (tpie::sort) |
Progress reporting | TPIE has a couple classes for reporting progress to the user |
Fraction database | When using fractional progress reporting, TPIE is able to measure and remember the proportion of execution time spent between the fractions |
External memory queue implementation | Let us see how to implement an external memory (EM) queue using TPIE's files and streams |
Pipelining | |
Pipelining Nodes | |
Serialization streams | |
Compressed streams | The TPIE stream header has a bit that indicates whether compression is used for the stream |