Skip to main content

How to query data from different microservices

Introduction

Our application is divided into logical pieces or subsystem which has its own responsibility or data stores. When comes to development we can develop and deploy individually, and we can realize the fastest improvement in the development of logical pieces. But it has its Cons in aggregation the data for reporting from different microservice or subsystems. Let us discuss deep into this.

Overview



Here Different microservices have different data stores and have their own responsibility it is processed via API Gateway to the client.we may have straight forward request against the microservice from the client for a particular module.

Problem Statement

When it comes to reporting we must produce the communication between the microservices in order to bring the report.we should not intercommunicate between microservice/datastore which is against the architecture principle. where it will bring the scalability issue. let us see that

Anti-Pattern – Intercommunication between datastores

Scenarios

1.       Materialized view
2.       Stored procedure
3.       Trigger

Description

Microservices datastore has its bounded context when we intercommunicate between the datastores we tend to create the tightly coupled datastore which is very hard to scale and it won't support the modern databases like NoSql or it won't support the heterogeneous datastores. This is considered an anti-pattern in microservices.

Cons

1.       Tight coupling between datastores
2.       Again become like monolithic, means you cant able to deploy services individually.
3.       Hard to scale
4.       Not supported for heterogeneous datastores

Anti-Pattern – Intercommunication between Microservices

Scenarios

1.       Joining REST API calls in between microservices to produce aggregated data

Description

Each Microservices has its own context and handled by a different small team and when it comes to intercommunication, the dependency between teams is created and it is very hard to think about continuous delivery or deployment. Over a period of time, it will become the chatty application and it will produce problems in maintenance and performance.

Cons

1.       Interdependency between services will become a problem to continuous delivery
2.       Very hard to maintain

Microservices-Pattern – Orchestrator pattern


Scenarios

1.       Composition of multiple requests into a single request
2.       Reporting services usually collect data from different data service

Description

When comes to reporting we need to collect data from different service using different calls /request and compose them into a single request. As it involves different calls this service will not be responsive and business logic will become very clumsy or complex for a maintenance point of view. Usually this type of pattern will be recommended for composing straight forward request from different calls between microservices.

Cons

1.       Not responsive due to Network latency of huge calls
2.       Not Resilient(If one of the calls fails in composition it is very hard to figure out)
3.       Hard to maintain
4.       Hard to troubleshoot



Microservices-Pattern – Aggregator pattern

Scenarios

1.       Straightforward aggregate call without composition/orchestration
2.       Serverless function act as orchestrator from data receives to the topic
3.       The responsive solution where client request can be responded without any network delay
4.       Resilient approach
5.       High data availability

Description

We can use the service broker mechanism and produce the topics for the data we need aggregation. Here the listener service will act as orchestrator and composition will be happened through the topics and stored in a persistent database. Aggregator microservice will expose its own RESTFul endpoint from where the client can make a request via proxy and it will be responded without any network delay or failure. I will be recommending this approach for quick response and high throughput.

Cons

1.       Complex design
2.       Careful consideration of topic and service design is required

Conclusion

Though the aggregator pattern is the most resilient approach to the query from different microservices. In simple words, the aggregator data store will act as the data warehouse and aggregator microservice will act as SSRS in our naive programming. From this article, I have provided an overview of querying data from different microservices.

References

·        https://microservices.io/
https://www.youtube.com/watch?v=fZkMxA_TKS4
Labels:

Comments

Post a Comment

Popular posts from this blog

How to resolve ASP.NET core web API 2 mins timeout issue

Introduction We are in the new world of microservices and cross-platform applications which will be supported for multiple platforms and multiple heterogeneous teams can work on the same application. I like ASP.NET Core by the way its groomed to support modern architecture and adhere to the software principles. I am a big fan of dot net and now I become the craziest fan after seeing the sophisticated facility by dot net core to support infrastructure level where we can easily perform vertical and horizontal scaling. It very important design aspect is to keep things simple and short and by the way, RESTFul applications are build and it is a powerful mantra for REST-based application and frameworks. Some times we need to overrule some principles and order to handle some situations. I would like to share my situation of handling HTTP long polling to resolve the ASP.Net core 2 mins issue. What is HTTP Long polling? In the RESTFul term, when a client asks for a query from the serv
6 comments
Read more

How to Resolve ASP.NET Core Key Protection Ring Problem in AWS Lambda

Introduction When it comes to server less web application design using asp.net core razor pages, we definitely need to consider a factor of data protection key management and its lifetime in asp.net core. I developed a site using AWS toolkit of ASP.NET Core Razor Pages. The main advantage of ASP.NET Core is cross-platform from where we can deploy our application in MAC, Linux or windows. I deployed my site initially in IIS Server from which I got the results as expected .but later period I decided to host my site in AWS Lambda in order to meet our client requirement. Strangely, I got unexpected behavior from my site. I just refer the cloud information Lambda Log to identify or pinpoint the case, I got the error Information like “Error Unprotecting the session cookie” from the log. In this article, I tried to explain the root cause of the problem and its solution to overcome such kind of issue. Data Protection in ASP.NET Core This is feature in ASP.NET Core which acts as repl
4 comments
Read more

Which linq method performs better: Where(expression).FirstorDefault() vs .FirstOrDefault(expression)

 Introduction When it comes to LINQ, we always have multiple options to execute the query for the same scenario. Choosing correct one is always challenging aspect and debatable one. In one of our previous articles   Any Vs Count  , we have done performance testing about best LINQ methods over .NET types. In this article, I would like to share about  Where(expression).FirstorDefault() vs .FirstOrDefault(expression) Approaches Performance testing for  Where(expression).FirstorDefault() vs .FirstOrDefault(expression) is very interesting IEnumerable<T> or ICollcetion<T>  .FirstOrDefault(expression) is better than  Where(expression).FirstorDefault() Public API To check the performance, I need some amount of data which should already available. So I decided to choose this  public api . Thanks to publicapis Public API Models Entry class using System ; using System.Collections.Generic ; using System.Text ;   namespace AnyVsCount { public class Entry { pub
Post a Comment
Read more