ShardingCore Documentation

ShardingCore is a library designed for database sharding (splitting data across multiple databases or tables) with minimal learning curve for EF Core users. It provides extensions for IQueryable to support sharding while maintaining compatibility with standard EF Core operations.

Key Features

• Custom Sharding: Supports custom database sharding (routing to different data sources) and table sharding (splitting a single entity into multiple physical tables).
• Flexible Table Sharding: Uses a pattern of [BaseName] + [TailPrefix] + [Tail] (e.g., user_0, user_1 or user202101, user202102).
• Advanced Query Support: Supports join, group by, max, count, min, avg, and sum across sharded tables.
• Performance Optimizations: Includes high-performance pagination with low-memory streaming for large page jumps.
• Ecosystem Compatibility: Supports extensions like EFCore.BulkExtensions for batch operations.
• Read/Write Splitting: Built-in support for separating read and write operations.

Limitations

• Connection Consumption: Performing join operations between sharded tables without index-based filtering can result in Cartesian products, potentially leading to connection exhaustion. Future updates aim to address this via configuration.

ShardingCore supports a primary-replica (one master, multiple slaves) architecture via AddReadWriteSeparation.

Strategies

• ReadStrategyEnum.Loop: Rotates through read connections.
• ReadStrategyEnum.Random: Selects a read connection randomly.

Consistency and Connection Strategies

To prevent data inconsistency (e.g., during pagination where a count and list are fetched separately), use ReadConnStringGetStrategyEnum:

• LatestEveryTime: Fetches a new connection every time (may cause data gaps in pagination).
• LatestFirstTime: Fetches a connection once per DbContext unit (ensures consistency within that context).

Manual Connection Switching

You can manually force a DbContext to use a specific connection type:

• _virtualDbContext.ReadWriteSeparationReadOnly(): Switch to read-only databases.
• _virtualDbContext.ReadWriteSeparationWriteOnly(): Switch to the write connection (useful to resolve read-write delay issues).
• _virtualDbContext.ReadWriteSeparation = false: Disables read-write separation for that context, defaulting to the write connection.

To ensure the sharding logic works correctly, verify the following requirements in your project:

Entity Requirements:

• Table Sharding: Entities intended for table sharding must inherit from IShardingTable and contain a ShardingKey.
• Database Sharding: Entities intended for database sharding must inherit from IShardingDataSource and contain a ShardingDataSourceKey.
• Routing: Ensure the entity object has implemented a virtual route.

Application Setup Requirements:

• Assembly Loading: The framework uses AppDomain.CurrentDomain.GetAssemblies(). To prevent issues where required assemblies are not loaded, ensure all necessary DLLs are loaded at the API layer.
• Routing Configuration: Ensure the virtual route has been added to your Startup configuration.
• Initialization: Ensure bootstrapper.start() has been called in your Startup configuration.

If you override certain services in your ShardingDbContext or DefaultDbContext, the ShardingCore framework may fail to function correctly. If you need to use these services, you must manually implement the extensions provided by the framework.

Specifically, the framework relies on replacing the following services:

1. ShardingDbContext overrides:

• IDbSetSource (replaced with ShardingDbSetSource)
• IQueryCompiler (replaced with ShardingQueryCompiler)
• IDbContextTransactionManager (replaced with ShardingRelationalTransactionManager<TShardingDbContext>)
• IRelationalTransactionFactory (replaced with ShardingRelationalTransactionFactory<TShardingDbContext>)

2. DefaultDbContext overrides:

• IModelCacheKeyFactory (replaced with ShardingModelCacheKeyFactory)
• IModelCustomizer (replaced with ShardingModelCustomizer<TShardingDbContext>)

Additionally, the framework uses AppDomain.CurrentDomain.GetAssemblies() to locate components. To prevent issues where required assemblies are not loaded, ensure that all necessary DLLs are loaded at the API layer.

// Example of the service replacement pattern used by the framework
return optionsBuilder.UseShardingWrapMark()
                .ReplaceService<IDbSetSource, ShardingDbSetSource>()
                .ReplaceService<IQueryCompiler, ShardingQueryCompiler>()
                .ReplaceService<IDbContextTransactionManager, ShardingRelationalTransactionManager<TShardingDbContext>>()
                .ReplaceService<IRelationalTransactionFactory, ShardingRelationalTransactionFactory<TShardingDbContext>>();

To enable sharding capabilities, your DbContext must inherit from AbstractShardingDbContext.

If you are implementing table sharding (splitting tables), you must also implement the IShardingTableDbContext interface. If you are only implementing database sharding (splitting databases), implementing the interface is optional.

When implementing IShardingTableDbContext, you must provide a RouteTail property of type IRouteTail.

public class MyDbContext : AbstractShardingDbContext, IShardingTableDbContext
{
    public MyDbContext(DbContextOptions<MyDbContext> options) : base(options)
    {
    }

    protected override void OnModelCreating(ModelBuilder modelBuilder)
    {
        base.OnModelCreating(modelBuilder);
        modelBuilder.Entity<Order>(entity =>
        {
            entity.HasKey(o => o.Id);
            entity.Property(o => o.Id).IsRequired().IsUnicode(false).HasMaxLength(50);
            entity.Property(o=>o.Payer).IsRequired().IsUnicode(false).HasMaxLength(50);
            entity.Property(o => o.Area).IsRequired().IsUnicode(false).HasMaxLength(50);
            entity.Property(o => o.OrderStatus).HasConversion<int>();
            entity.ToTable(nameof(Order));
        });
    }

    /// <summary>
    /// empty impl if use sharding table
    /// </summary>
    public IRouteTail RouteTail { get; set; }
}

Register ShardingCore in your IServiceCollection using AddShardingDbContext.

When configuring the data source via AddDefaultDataSource, provide your connection string. Important: Do not modify the internal parameters of the UseXXX delegates (like UseSqlServer); instead, use the provided connStr or connection parameters to ensure the sharding logic correctly applies the connection to the routed shards.

public void ConfigureServices(IServiceCollection services)
{
    // Add sharding configuration
    services.AddShardingDbContext<MyDbContext>()
        .UseRouteConfig(op =>
        {
            op.AddShardingTableRoute<OrderVirtualTableRoute>();
        }).UseConfig(op =>
        {
            op.UseShardingQuery((connStr, builder) =>
            {
                // connStr is delegate input param
                builder.UseSqlServer(connStr);
            });
            op.UseShardingTransaction((connection, builder) =>
            {
                // connection is delegate input param
                builder.UseSqlServer(connection);
            });
            // use your database connection string
            op.AddDefaultDataSource(Guid.NewGuid().ToString("n"),
                "Data Source=localhost;Initial Catalog=EFCoreShardingTableDB;Integrated Security=True;");
        }).AddShardingCore();
}

public void Configure(IApplicationBuilder app, IWebHostEnvironment env)
{
    if (env.IsDevelopment())
    {
        app.UseDeveloperExceptionPage();
    }
    // Optional: enable check for missing tables and auto-creation on startup
    app.ApplicationServices.UseAutoTryCompensateTable();
    // other configure....
}