Most elegant way to implement Pipe and Filter Pattern

10,782

Solution 1

There is a draft in the upcoming C++14 standard that covers this area:

C++ Pipelines - ISO/IEC JTC1 SC22 WG21 N3534 = 2013-03-15

And here is an implementation for it:

code.google.com/p/google-concurrency-library/source/browse/include/pipeline.h

Solution 2

What you are describing is some sort of streaming architecture or pipeline architecture. Standard C++ doesn't have anything this specific, but it provides you, as the library author, the necessary primitives in the language and standard library in order to build out such an architecture. Simply create classes and interfaces as per usual object-oriented programming and then instantiate them and connect them in a pipeline as per your needs.

For example you may have a Source interface and a Sink interface - and a Filter abstract class that implements both Source and Sink, as well as a Pipe class that implements both Source and Sink and just passes the data straight through. This is just one of many ways to name and organize such a framework.

Share:
10,782
benjist
Author by

benjist

Updated on June 04, 2022

Comments

  • benjist
    benjist almost 2 years

    I want to create a Pipe and Filter based data handler that should manipulate incoming data sets like so, but not neccessarily limited to:

    source pipe(could be a data set from a db) <-sink-source-> filter(add an additional field) <-sink-source-> filter(manipulate some more data / remove ie nullify data set)

    I have an idea how such architecture will look like in C/C++. But given all the goodies that come with C++11's functional aspects, I hope this task can be done in an elegant way so that it is able to:

    • easily scale and use it in a multithreaded environment (e.g. by performing filter tasks as lambda funcions, thus probably avoiding at least some thread handling)
    • easily add and remove filters at runtime
    • handle input streams lazily
  • benjist
    benjist almost 11 years
    You say "standard C++ doesn't have anything this specific", which is correct. However, I already know how a common implementation would look like in C++ (like in your example). But was interested in something
  • Andrew Tomazos
    Andrew Tomazos almost 11 years
    @benjist: The generality of my answer matches the generality of your question. Your question would do better if you asked something more specific. "What is the most elegant way to implement design pattern X?" is overly broad and subjective.
  • benjist
    benjist almost 11 years
    Sorry, my previous comment was shortened. You're correct that the question is a bit general. But my question still is if there isn't a more elegant way towards the three requirements/wishes that I stated now with C++11. For an example, wouldn't it be possible to have a vector of filters and use std::transform to apply these filters on the data? Couldn't it make parallel tasks easier? Etc.
  • Andrew Tomazos
    Andrew Tomazos almost 11 years
    If you are interested in parallel execution of this form then take a look at the Intel Threading Building Blocks library.
  • benjist
    benjist almost 11 years
    Nice. Thanks a lot. That looks promising indeed.
  • malat
    malat almost 9 years
    @AndrewTomazos could you give more details on Source, Sink and Filter API ? Esp. the type of data that can be passed from one to the other ?