Edition 6 - How to consolidate information from databases in a micro-services architecture?

Not every enterprise started with microservices; many moved to it gradually when the need arose. I was fortunate enough to contribute to this movement across a few companies. The run-time separation of concerns is relatively easy when you compare it to the data complexities arising from it.

The transition from monoliths creates fragmented information and security concerns that many underestimate. But with the right consolidation strategy, these obstacles can be overcome.

In this edition of LeadReads, you'll get an expert-guided playbook for consolidating data across databases in a microservices architecture.

Information consolidation: Monolith to Microservices.

The movement to a micro-services architecture is not one without consequence. There are quite a few pros - and one of the cons is that your data is no longer consolidated.

Single-tenant systems are another example where the data isn’t consolidated.
Multi-tenant systems suffer from a different problem - data isolation. All of this requires post-processing of data.

So, , the first step in this is setting up security and access control. Once that is out of the way, we move on to;

Data sources - key identifiers, standardization, and extraction

• Identify all the databases to pull data from.

  1. Focus on key identifiers that hold meaning across databases, eg. the country_id key used to tag data from the same country across services.

• Create a weighted data dictionary.

• Standardise data across storage mechanisms within the microservices ecosystem

• Identify data streaming and batch use cases.

• Identify data extraction tools and schedules.

Data security, access, and governance

• Implement access control from the start. Consider compliance requirements, organizational structure, and regulations across countries and for multiple tenants.

• Prioritize role-based privileges over user-level access to prevent unnecessary data access and focus on privacy and governance.

• Protect your data vigorously. Use robust encryption at rest, in transit, and during usage.

• I can’t stress how important this is. Ensure that your team knows the Audit Requirements when building your system. Don’t use the audit report like a checkmark.

• Embrace GDPR’s data protection by design and default.

• Have a clear, unambiguous plan for responding to data breaches.

DATA TALES

Amazon is a prime example of a company that has successfully adopted microservices. They dismantled their monolithic applications and refactored them into microservices-based architectures. For instance, Amazon uses a specialized microservice to process shipping orders. Link

Top Microservices Trends

1. Container Orchestration with Kubernetes

2. AIOps for Smarter Operations

3. Serverless Architecture

4. Service Meshes

WHAT I’M READING

I’m currently reading "Only the Paranoid Survive" by former Intel CEO Andrew Grove, which offers an invaluable playbook on how to lead with urgency and harness change - essential principles for any disruptive startup.

HAPPENINGS AT WEDNESDAY

1. Wednesday’s website has an updated look (it’s pretty cool if you ask me), plus we’re uploading new articles.

2. The Wednesday Show has ramped up season 2 and the last episode is out. Listen to it here.

3. Wednesday Solutions is live on GoodFirms and DesignRush. Feel free to check us out.

Hi, I’m Mohammed Ali Chherawalla (Mac), Co-founder & CTO at Wednesday Solutions, a specialized engineering services company with a focus on Gen AI, Data, and Application Modernization. 10% of India's Unicorns are our customers. I make it a point to read every message from my subscribers, so don't hesitate to share your thoughts with me.