The Essence of Modularity: Navigating Modular Monoliths and Microservices

Software Architecture

Modularity is paramount in software design, irrespective of deployment. This guide explores architectural styles, from simple modular monoliths to microservices, detailing their trade-offs for robust development.

The Essence of Modularity: Navigating Modular Monoliths and Microservices

Modularity is a fundamental concept in software design and creation. Regardless of whether your chosen architectural style involves a single unit of deployment (a Monolith) or many units of deployment (Microservices/Services), modularity remains a crucial quality that should be considered independently of the number of deployable units.

The goal is to divide systems into logical, functional modules that are as independent as possible. Ideally, each module would be self-contained, requiring no knowledge of other modules to deliver its functionality. While this ideal is often unattainable in practice, it serves as a guiding principle to strive for high cohesion and low/loose coupling.

What is a Module?

In this context, a module is a segment of software—be it a single object, class, function, or a collaborative set of them—responsible for delivering a specific functionality or a closely related set of functionalities. These functionalities should be defined from the user's perspective, whether human or machine. Every module requires an input, a public API, which allows external entities to utilize its capabilities.

Why Modularity Matters

Why is modularity so important, and what benefits does it offer?

  • Organization: Well-defined modules provide clarity about a system's purpose. This simplifies the division of work and responsibilities among individuals or teams, enabling parallel, independent development.
  • Comprehension: It's significantly easier to understand a few small, distinct modules than to untangle a large, interconnected codebase.
  • Resource Utilization: Modules can have vastly different resource requirements. For instance, a highly available, user-facing module might need substantial resources, while a background task module might run infrequently with minimal exposure. Modularity allows for tailored resource allocation, leading to more optimal utilization.
  • Reusability: By assigning clear responsibilities and public interfaces to modules, their functionalities can be easily reused across the system or in other projects.
  • Testability: Verifying the functionality of a small to medium-sized module is simpler than testing a large, complex system with numerous unclear dependencies. Well-defined module boundaries often reduce the need for extensive inter-module interaction testing, as adherence to established contracts is primarily verified.

When to Modularize?

If a system is tiny, developed by a single person, and inherently represents a single logical module, the benefits of modularity might be negligible. However, as systems grow, modularity's advantages become increasingly apparent. As soon as distinct modules begin to emerge within a system, it is advisable to proceed with modularization.

To illustrate, consider a system named "Curious Notes to the Interesting Quotes," where users can add notes to famous quotes. A possible modular design could include:

  • users: Responsible for user creation, account management, authorization, and authentication.
  • quotes: Manages quotes, primarily by privileged users.
  • notes: Allows users to add, edit, delete, and like notes associated with quotes.

Module Dependencies:

  • users: No dependencies.
  • quotes: Depends on users to verify if a user is authorized to manage quotes.
  • notes: Depends on users to verify user authorization for note management, and on quotes to confirm a quote's existence.

This logical division should largely remain independent of the chosen physical architecture. These three modules could form a modular monolith, organized as separate folders or fully isolated, independently versioned packages. Alternatively, they could be implemented as three (micro)services communicating over a network, either synchronously or asynchronously. The physical division into one or many deployable units should be a secondary consideration in system design. The primary drivers should be domain understanding, functional requirements, existing concepts, and their interdependencies. Only after addressing these factors should non-functional, performance, and resource utilization considerations influence implementation details.

Software Development in the Real World

The modularity requirements described above are ideal for situations with complete knowledge of the software and its functional requirements. Unfortunately, real-world development often involves partial and incomplete knowledge, primarily due to two reasons:

  1. Limited Client Access/Information: Often, direct access to the client driving requirements is limited, or the client/business/product owner lacks time. Yet, development must begin due to time constraints and deadlines, leaving developers with only a subset of feature details.
  2. Dynamic or Undefined Domains: Sometimes, the system models a constantly changing environment, like in a startup, or involves conscious experimentation where domain details are unclear and still being explored.

The level of ambiguity and the number of unknowns are critical factors in defining modules and selecting an implementation strategy (modular monolith vs. (micro)services). The first rule in such scenarios is to gather all necessary information to make sound design decisions and minimize unknowns. If this isn't possible, or information is scarce, the decision-making strategy must adapt. The more ambiguity, dynamism, and uncertainty about the final system shape, the more one should favor a strategy that minimizes the cost of redesigning and rearranging modules. In highly ambiguous situations, starting with the simplest modular monolith strategy is often best. Its structure is inexpensive to change, and migration to more sophisticated architectures can occur as the domain stabilizes and becomes more defined. Pursuing microservices when module numbers and responsibilities are constantly changing makes little sense; a simple modular monolith is more appropriate. Furthermore, avoid the "stability illusion"—systems constantly evolve, a factor to remember when deciding on module isolation and granularity.

Implementations of Modularity

Many practical strategies exist for implementing modularity. A good design, characterized by effective module separation, should not depend on the physical architecture (single or multiple deployment units). Understanding the domain, functional requirements, natural concepts, boundaries, and especially the dependencies between modules is crucial, as dependencies often determine a system's success or failure. Only once these aspects are clear and stable should the physical runtime structure be evaluated. This involves answering questions such as:

  • Should there be one or multiple runtime deployment units (applications)?
  • If one, what level of complexity is needed for the modular monolith structure, and what constraints (if any) should be imposed?
  • If multiple, what constraints (if any) should be imposed on the (micro)services—e.g., tech stack, allowed network communication types (synchronous/asynchronous), timing of communication (foreground/background), and handling of distributed transactions?

Let's explore different strategies based on these factors.

Simple Modular Monolith: Modules as Folders

This is the most straightforward approach to modularization. Here, folders are treated as separate modules without independent versioning or additional boundaries. However, a contract or convention must be established for inter-module communication.

Its greatest advantage—simplicity—can also be its biggest drawback. Extra precautions are needed to ensure module boundaries are respected, preventing the modular monolith (modulith) from degrading into a complex, hard-to-understand, test, and change monolith (often termed a "big ball of mud"). To mitigate this, establish and adhere to conventions:

  • Create a _contracts folder for common interfaces, models, and application events.
  • Each module can have a dedicated folder/file defining its public interface; all other parts are considered private and should not be directly accessed.
  • If a shared database is used, implement separate schemas for each module or a consistent table naming convention. This ensures each module owns and is responsible for its data.

Using the _contracts folder approach with the "Curious Notes to the Interesting Quotes" system (featuring users, quotes, and notes modules), one might define interfaces like:

interface UsersClient {
  boolean canAddQuote(UUID userId);
  boolean canEditQuote(UUID userId, UUID ownerId, UUID quoteId);
  boolean canDeleteQuote(UUID userId, UUID ownerId, UUID quoteId);
  boolean canManageNote(UUID userId, UUID ownerId, UUID noteId);
}

interface QuotesClient {
  boolean quoteExists(UUID quoteId);    
}

record QuoteCreatedEvent(UUID quoteId, UUID ownerId) { }

record QuoteDeletedEvent(UUID quoteId, UUID ownerId) { }

The quotes module, which depends on the users module, would only interact via the UsersClient interface. The users module implements this interface, which is then injected into the quotes module at runtime. This way, the quotes module remains unaware of the users module's implementation details, resulting in loose coupling. Similarly, the notes module would communicate with quotes solely through the QuotesClient interface. Additionally, application events (e.g., QuoteCreatedEvent) can be published to other modules. For example, the users module might send notifications upon new quote creation, or the notes module might delete all associated notes when a quote is deleted.

To reiterate, this is the simplest and most agile modularization implementation. For solo developers, single teams, or when domain knowledge/module separation is uncertain, this strategy is highly suitable. Adhering to a few basic rules facilitates easy migration to more elaborate structures later. Its main drawbacks are:

  • Ease of inadvertently violating module boundaries and established conventions.
  • Inability to deploy different modules independently from various Git branches, as there's no independent versioning and everything is built from a single source code repository (though this isn't an issue if deployments always occur from a single master branch after merges).

These drawbacks might be acceptable in specific scenarios. A team might be disciplined enough to respect module boundaries (and tools exist to help enforce this). Furthermore, if the modular monolith can be fully run and tested locally, independent module deployments might not be necessary. If these conditions hold, embrace this strategy and enjoy the often-underestimated benefits of simplicity!

Modular Monolith with Independently Deployable Modules

This strategy represents a more rigorous version of the modular monolith architecture. It still involves a single unit of deployment, but each module is an independently versioned package (e.g., Java jar, NPM package, Go binary, C# assembly). This additional isolation brings several consequences:

  • Independent Versioning and Deployment: Each module is versioned and can be deployed independently, assuming a private artifact repository (like Nexus) is used for package storage.
  • Transparent Dependencies: If a module depends on others, these dependencies must be explicitly added, making them more transparent.
  • Enforced Boundaries: Increased isolation makes it harder to cross module boundaries, thereby maintaining architectural integrity.
  • Further Decoupling with Databases: Modules can use separate databases for even greater decoupling (also achievable in the simple modular monolith approach).
  • Separate Git Repositories: For maximum insulation, each module can reside in its own Git repository.

Regarding independent versioning, a common practice (e.g., in Java) is using SNAPSHOT versions. This allows modules with the same version number but changed source code to be uploaded to a private artifact repository. For instance, module A can be uploaded from feature-A Git branch, and the modular monolith built and deployed with this updated A, while other modules remain unchanged. This approach requires a central application/main module that depends on and assembles all domain modules into a single executable. Using SNAPSHOT versioning negates the need to bump the version of a changed module in the application/main module. Since modules are in an artifact repository, different versions can be pushed from separate Git branches without affecting other team members working on different modules on other branches. Placing each module in a separate Git repository further enhances isolation and facilitates parallel work.

For increased decoupling, using a distinct database for each module is an option. Does this provide more independence than separate database schemas or naming conventions within a shared database? If a rule dictates that transactions must be confined to a single module and cannot span across modules, then having separate databases offers minimal additional gain. However, with multiple databases, cross-database transactions are technically impossible, enforcing this boundary. If a transaction must span multiple modules, it often indicates a design flaw. Yet, sometimes distributed transactions are a necessary evil, requiring patterns like the Transactional Outbox Pattern for reliable message delivery and/or the Saga Pattern for managing transactions across multiple independent databases. Fortunately, the latter can often be avoided through smart background data reconciliation. In this scenario, data might be duplicated across modules (and their databases) to allow them to function independently. Given the complexity of distributed transactions, it's highly recommended to design modules to avoid them. At worst, data synchronization between modules can occur via application events or simple function calls—after all, it's still a monolith, a single deployment and execution unit. Data copying and synchronization between modules is simpler, as it can be performed in the background, independently of normal request processing, with eventual success guaranteed through retries.

This approach is significantly more scalable than a simple modular monolith. With isolated, independently versioned modules, each possessing its own database (or adhering to a "no cross-module transactions" rule), and potentially residing in separate Git repositories, many individuals and teams can collaborate on a single modular monolith in parallel, experiencing minimal conflicts and synchronization needs. The primary requirements for alignment are fundamental aspects of the tech stack: programming language, build system, perhaps a framework (multiple can be used), and core libraries.

Constrained Microservices (Services/Microliths)

This approach offers a powerful way to mitigate many of the complexities of traditional microservices by adhering to a simple rule: When processing any external network request (synchronous or asynchronous), a service cannot make any network calls to other services (synchronous or asynchronous).

Why is this constraint so effective?

  • Forces Better Design: It compels developers to design services to be self-sufficient, possessing all data required for their functionality, often simplifying and improving system design.
  • Increased Reliability: Services become more reliable because this constraint eliminates their dependency on other services for handling external requests.
  • Eliminates Distributed Transactions: By definition, distributed transactions become impossible, removing a major source of complexity.
  • Easier Comprehension and Debugging: Such systems are significantly easier to understand and debug compared to a pure microservices architecture. Since no network requests are made during external request processing, the only potential point of failure is the service itself. Background network requests are still allowed, but the constraint ensures that downstream services won't make further network calls when handling those requests, simplifying failure tracing.
  • Fewer Services: This approach naturally leads to fewer services overall, as services are incentivized to keep all necessary functionality and data directly available. This results in simpler infrastructure and fewer moving parts to analyze, understand, observe, and maintain.

An interesting hybrid can emerge: a few larger services, each effectively a modular monolith. This allows reaping the benefits of both strategies: having a few independently scalable deployment units without the need for complex infrastructure, advanced observability tools, or the network indeterminism inherent in pure, unconstrained microservices.

Modular Monolith with Helper Services

In this strategy, a modulith (modular monolith) core serves as the default, where most functionality is initially added, emphasizing good module structure. In rare instances where a specific module has distinct resource or technology requirements, or differs significantly from the modulith core for other reasons, it is extracted into a separate, independently deployed and run service. The rules for these helper services largely mirror those of microliths: network communication is generally not allowed when serving external requests. This maintains system simplicity while allowing specialized, single-purpose services to address unique needs that wouldn't fit well within the core monolith.

Microservices: Modules as Applications

This is the most complex implementation of modularization. Here, each module is essentially a separate application, independently deployed and executed. Consequently, the number of deployment units often equals the number of modules.

This level of separation comes with the highest cost: complexity.

  • Infrastructure Complexity: With one, two, or three deployment units, building and deploying applications to a VPS or PaaS with simple scripts is feasible. However, microservices involve numerous separate applications, typically communicating via remote network calls. This necessitates complex infrastructure setup (e.g., Kubernetes, which requires significant expertise) or reliance on managed services.
  • Observability Challenges: Monitoring (logs, metrics, alerts) a system with many independent applications is far more challenging. One must contend with events published to message brokers/queue systems that can fail anytime, and synchronous HTTP calls that can also fail unpredictably. This plunges development into the complex world of network indeterminism and truly distributed systems, a complexity often underestimated.

Despite the challenges, microservices can be justified and offer significant benefits when managed effectively:

  • Technology Diversity: Each module, being a separate application, can theoretically use different programming languages or technologies tailored to its specific needs (though practical multi-tech stacks come with their own trade-offs).
  • Optimized Resource Utilization: Different modules can have varying resource requirements—some needing high availability and load handling, others having modest needs. As each module is a separate application, distinct resources and a dynamic number of replicas can be assigned, leading to highly optimized resource utilization. In contrast, scaling a modular monolith for one module's increased resource need often means scaling the entire monolith (though approaches like "Monolith++", discussed in resources, address this).
  • Scalability for Large Teams: While a "Modular Monolith with Independently Deployable Modules" can scale to a few teams, microservices are better suited for tens or hundreds of teams, enabling more independent streams of work. While similar parallelism can be achieved with a modular monolith through discipline, microservices often offer this by default with the right implementation. The optimal choice depends on specific organizational growth and project needs.

In summary, this architectural style should be adopted only when absolutely necessary, as its inherent complexity can significantly slow development if implemented without valid reasons. However, in specific scenarios, such solutions are indeed indispensable.

Parallel Work and Independent Deployments

For teams, maximizing independent work streams is crucial for parallelization. The more modular a system, the easier it is to divide work and responsibilities. Loosely coupled, independent modules allow more individuals/teams to modify code concurrently without conflicts or interruptions. Two key dimensions are:

  1. Introducing Changes to the Codebase: All presented approaches are relatively equal here. The bottleneck for parallel work, regardless of a simple modular monolith or complex microservices, is the quality of system design—specifically, its degree of modularity. Ten highly coupled microservices will impede parallel feature development more than a simple modular monolith with fifty loosely coupled modules.
  2. Testing Concurrent Changes: This depends heavily on environment details and available strategies. Microservices/separate services, by definition, offer this capability almost inherently, assuming correct implementation. When working on a separate service, only that service needs deployment and testing, without affecting other services' changes—provided services are reasonably independent. This is also achievable with a modular monolith: if modules are isolated as independently versioned packages in an artifact repository, the monolith can be deployed with an updated module without affecting others. For simple modular monoliths with multiple teams, separate dev/stage environments per team are an option. Many other strategies exist for independent deployments with a modular monolith, though they require setup effort. Crucially, due to infrastructure simplicity, a modular monolith can often run and be reliably tested end-to-end locally, potentially eliminating the need for shared dev/stage environments and allowing direct deployment to production.

What about the Frontend?

So far, the focus has been on backend systems or systems without a strict frontend/backend deployment distinction, such as traditional server-side rendered applications. With old-school server-side rendering or technologies like HTMX, views are defined within the same modules as the backend, effectively solving the modularity problem by eliminating the frontend/backend deployment distinction. In this approach, frontend/views are scoped to their respective modules.

However, pragmatically, what about today's dominant JavaScript-focused Single Page Applications (SPAs)?

For SPAs, similar modularity rules apply. One can start with a simple modular monolith, organizing modules as separate folders. A _contracts folder/module can define inter-module communication and shared state. Limiting global styles (a concern often addressed by Tailwind CSS principles) and components, and relying mostly on module-scoped styles and components—sharing only the most generic ones—is beneficial. Universal rules of coupling and cohesion also apply: more independent, self-contained modules enable more independent work.

With the simple modular monolith strategy, independent deployment of different modules isn't natively supported. If "modules as folders" aren't isolated enough, code splitting or dynamic imports can create multiple independent entry points for the application. As a last resort, one can implement a separate SPA for selected routes (multiple SPAs approach) or utilize Micro Frontends. Both offer complete physical isolation of modules, akin to the microliths/microservices concepts for the backend.

Closing Thoughts

This discussion has delved deeply into modularity: its importance, consequences, and various implementations. Specifically, we compared single-unit deployments (the (modular)monolith) with multiple-unit deployments (the (micro)services). We also illuminated the details and trade-offs of different modularity implementations, reminding that "there is no free lunch"—every choice has consequences. To reiterate, we explored strategies from simplest to most complex:

  • Simple Modular Monolith
  • Modular Monolith with Isolated and Independently Deployable Modules
  • Modular Monolith with Helper Services
  • Constrained Microservices (Microliths)
  • Microservices

It is crucial to thoughtfully consider which strategy aligns with particular requirements, always erring on the side of simplicity, as transitioning to a more complex approach is always possible.

Ultimately, design and architecture are forms of art; there are no absolute, final solutions, only trade-offs. Weigh them wisely!

Related Resources

Related videos on my YouTube channel

Notes and Resources