DOCSEARCHbyAlgolia
Submit Docs

A-Nomad System Architecture Documentation

website·Indexed 31 days ago

https://architecture.a-nomad.com/

A guide to distributed system architecture and cloud-native design. Covers NoSQL databases including Redis and MongoDB, logging frameworks such as ELK (Elasticsearch, Logstash, Kibana) and Exceptionless, database locking mechanisms, and the evolution of system architecture from single servers to containerized microservices using Docker.

Tokens
13.2K
Snippets
64
Records
84
Agent score
50%

What's inside A-Nomad Architecture

  1. Overview of the ELK Stack

    ELK is a classic distributed logging framework consisting of three open-source components:

    • Elasticsearch: A distributed search engine used for collecting, analyzing, and storing data. It features a RESTful API, automatic sharding, and replica mechanisms.
    • Logstash: A tool for collecting, analyzing, and filtering logs. It typically operates on a client-server architecture where clients collect logs and the server filters/modifies them before sending them to Elasticsearch.
    • Kibana: A web interface for visualizing and searching the log data stored in Elasticsearch.

  2. Ensure consistency in SOA and MQ architectures

    In Service-Oriented Architecture (SOA) or Message Queue (MQ) based systems, traditional atomic transactions cannot be guaranteed across service boundaries. Implement a transaction compensation mechanism to ensure eventual consistency of data.

  3. Deploy a Redis Master-Slave architecture with Docker Compose

    Redis Master-Slave architecture enables read-write splitting and data backup. The master node handles writes, while slave nodes are typically read-only. Slaves are configured using the slaveof parameter to bind to the master.
    version: '3.7'

services:
  redis-master:
    image: redis:alpine
    container_name: redis-master
    command: redis-server --requirepass master_password
    ports: 
      - "6379:6379"
    restart: always

  redis-slave1:
    image: redis:alpine
    container_name: redis-slave1
    command: redis-server --slaveof redis-master 6379 --masterauth master_password --port 6380 --requirepass slave_password
    ports: 
      - "6380:6380"
    restart: always
    depends_on:
      - redis-master

  redis-slave2:
    image: redis:alpine
    container_name: redis-slave2
    command: redis-server --slaveof redis-master 6379 --masterauth master_password --port 6381 --requirepass slave_password
    ports: 
      - "6381:6381"
    restart: always
    depends_on:
      - redis-master

  4. Understand the purpose and use cases of NoSQL databases

    NoSQL (Not Only SQL) is designed to complement rather than replace relational databases. It is specifically optimized for scenarios where relational databases are inefficient, particularly for ultra-large-scale data storage (e.g., trillions of bits of data collected daily by platforms like Google or Facebook). NoSQL is ideal for data that does not require a fixed schema and needs to scale horizontally without complex overhead, such as user profiles, social networks, geolocation data, user-generated content, and operation logs.

  5. Perform advanced data processing with the MongoDB Aggregation Pipeline

    The aggregate() method processes data through a pipeline of stages. Each stage transforms the data and passes the result to the next. Common pipeline stages include:

    • $match: Filters documents (similar to find query).
    • $project: Reshapes documents (renaming, adding, or removing fields).
    • $sort: Sorts documents.
    • $skip / $limit: Controls the number of documents passed to the next stage.
    • $group: Groups documents by a key (_id) and applies accumulator expressions.
    • $unwind: Deconstructs an array field into multiple documents.

    Note: aggregate() cannot be used with find(), sort(), skip(), or limit() methods; you must use their corresponding pipeline stages instead.

    // Example: Filter adults, project only name and age
db.students.aggregate([
    {$match: {age: {$gte: 18}}},
    {$project: {_id: 0, name: 1, age: 1}}
])

  6. Install Redis using Docker

    The recommended way to deploy a Redis server is using Docker for efficiency and simplicity. The following command runs a Redis instance based on the Alpine Linux image with a required password.
    docker run \
-d \
--name redis \
-p 6379:6379 \
redis:alpine \
--requirepass "password"

  7. Implement pessimistic locking in SQL Server with EF or Raw SQL

    Pessimistic locking assumes conflicts will occur and locks the data upon retrieval. In SQL Server, this is achieved using table hints. To implement a pessimistic lock, you must execute the query and the update within the same transaction. Use the with (xlock, ROWLOCK) hint: xlock specifies an exclusive lock (blocking others until the transaction ends), and ROWLOCK ensures only the specific row is locked rather than the entire table.
    // Using EF with Raw SQL for pessimistic locking
using (var tx = ctx.Database.BeginTransaction())
{
    // The query must be executed (e.g., via .Single()) to apply the lock
    var record = ctx.Database.SqlQuery<Girl>("select * from T_Girls with (xlock, ROWLOCK) where id=1").Single();
    
    ctx.Database.ExecuteSqlCommand("Update T_Girls set BF={0} where id=1", bf);
    tx.Commit();
}

  8. Implement optimistic locking using rowversion in EF Core

    Optimistic locking is recommended for 'read-heavy, write-light' systems. It allows multiple users to read data and only checks for conflicts during the update. In SQL Server, this is implemented using a rowversion column (or timestamp).

    1. Add a rowversion column to the database table.
    2. In the EF entity class, define the property as byte[].
    3. Configure the property using Fluent API: Property(e => e.RowVersion).IsRowVersion();.
    4. Wrap ctx.SaveChanges() in a try-catch block to handle DbUpdateConcurrencyException, which is thrown if the rowversion has changed since the data was read.
    // Entity Configuration
modelBuilder.Entity<Girl>().Property(e => e.RowVersion).IsRowVersion();

// Usage
var g = ctx.Girls.First();
g.BF = "New Name";
try
{
    ctx.SaveChanges();
}
catch (DbUpdateConcurrencyException)
{
    // Handle the conflict: the record was modified by another user
    Console.WriteLine("Update failed due to concurrency conflict");
}

  9. Implement data sharding and high availability with Redis Cluster

    Redis Cluster is the recommended solution for data sharding and high availability. Unlike Sentinel, where every node holds all data, Redis Cluster distributes data across multiple master nodes using a hash slot system (16,384 slots). A key's slot is determined by CRC16(key) % 16384.

    Key constraints and behaviors:

    • Requires at least 3 master nodes to initialize the cluster.
    • If the number of surviving master nodes falls below half of the total master nodes, the cluster becomes unavailable.
    • Each master has associated slaves for failover; if a master fails, a slave is automatically promoted.

  10. Perform batch operations in Redis

    To reduce network requests and improve performance, use batch operations. Most Redis operations support array types for bulk processing. Additionally, the IBatch interface allows multiple commands to be queued and submitted to the server in a single call.
    IBatch batch = db.CreateBatch();
db.StringSet("abc", "123");
db.GeoAdd("ShopsGeo1", new GeoEntry(116.34039, 39.94218, "1"));
batch.Execute(); // All operations between CreateBatch and Execute are submitted at once

  11. Implement high availability with Redis Sentinel

    Redis Sentinel provides high availability by automatically monitoring master and slave instances, notifying administrators of failures, and performing automatic failover. When a master node fails, Sentinel promotes a slave to be the new master and updates other slaves to replicate from the new master. It also acts as a configuration center, providing application clients with the current address of the master node.

    Deployment requires at least three Sentinel nodes for stability.

    port 26379

# sentinel monitor <master-name> <ip> <port> <quorum>
# Example: monitor 'redis-service' at 'redis-master:6379' with a quorum of 2
sentinel monitor redis-service redis-master 6379 2

# Time in milliseconds before a master is considered unreachable
sentinel down-after-milliseconds redis-service 5000

# Timeout for the failover process
sentinel failover-timeout redis-service 60000

sentinel parallel-syncs redis-service 1

# Password for the master node
sentinel auth-pass redis-service master_password

  12. Integrate .NET Core and NLog with ELK

    To send logs from a .NET Core application to ELK, use NLog with a Network target pointing to the Logstash TCP port (default 5000). The NLog layout must match the grok pattern defined in the Logstash filter.
    <targets>
    <!-- Console output for local debugging -->
    <target xsi:type="Console" name="console"
            layout="${date} | ${level:uppercase=true} | ${logger} | ${message} | ${replace-newlines:${exception:format=toString}}" />
    
    <!-- Network output to Logstash -->
    <target xsi:type="Network" name="logstash" keepConnection="false"
            address="tcp://192.168.0.211:5000"
            layout="${level:uppercase=true}#${logger}#${callsite:includeSourcePath=true} ${callsite-linenumber}#${message}#${replace-newlines:${event-properties:item=EventId}}#${replace-newlines:replacement=|:${exception:format=ToString}}"/>
</targets>

<rules>
    <logger name="Microsoft.*" maxlevel="Info" final="true"/>
    <logger name="*" minlevel="Trace" writeTo="console"/>
    <logger name="*" minlevel="Info" writeTo="logstash"/>
</rules>