Processing Multi-Stage Tasks using Firebase Queue

This is an implementation of Pipes and Filter Pattern which is a commonly used Cloud Design Pattern. The most common approach to implement this pattern is via multiple queues as shown in the following figure.

ic709558

Pipeline Using Multiple Queues (Ref : MSDN)

The prosed solution here uses a bit different approach than above and use a single queue instead of multiple queues as shown below.

Data Flow Diagram

Pipeline Using a Single Queue

To practically implement a solution like above the common approach would be to maintain a state for each queue message and assign a state for each workers(filters) to filter the messages accordingly before execution. However recently when I was trying out Firebase Realtime Database I found Firebase Queue which is a queue implementation on top of the Firebase Realtime Database and it turned out that it comes with a out of the box solution for maintaining a state (their term is specs) for queue messages as described above. So I worked on a PoC to evaluate this and in my personal opinion it’s really cool. But this is not enough to make a dicission so I was looking at various benefits we can achieve here with the usage of Firebase Realtime Database.

Benefits

  • Using firebase realtime behaviour we can easily show the progress of the running tasks on a front-end
  • Since it’s single queue it’s less hazzle to implement and maintain
  • Uses push instead of polling to notify workers about new tasks
  • Implementation has been made super easy with their library
  • Nicely support to be run in a clustered environment distributing workload among nodes

Use Cases

  • In scenarios where REST APIs facilitate time consuming processing heavy back­end functions
  • In scenarios where triggered background tasks run for streaming data
Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s