Pipeline Using Multiple Queues (Ref : MSDN)
The prosed solution here uses a bit different approach than above and use a single queue instead of multiple queues as shown below.
Pipeline Using a Single Queue
To practically implement a solution like above the common approach would be to maintain a state for each queue message and assign a state for each workers(filters) to filter the messages accordingly before execution. However recently when I was trying out Firebase Realtime Database I found Firebase Queue which is a queue implementation on top of the Firebase Realtime Database and it turned out that it comes with a out of the box solution for maintaining a state (their term is specs) for queue messages as described above. So I worked on a PoC to evaluate this and in my personal opinion it’s really cool. But this is not enough to make a dicission so I was looking at various benefits we can achieve here with the usage of Firebase Realtime Database.
- Using firebase realtime behaviour we can easily show the progress of the running tasks on a front-end
- Since it’s single queue it’s less hazzle to implement and maintain
- Uses push instead of polling to notify workers about new tasks
- Implementation has been made super easy with their library
- Nicely support to be run in a clustered environment distributing workload among nodes
- In scenarios where REST APIs facilitate time consuming processing heavy backend functions
- In scenarios where triggered background tasks run for streaming data