Shoko Server Documentation

In Shoko Server, a FilterExpression is a data transformation method that takes zero or more arguments and returns a result. These expressions are structured as a Binary Expression Tree.

Key constraints for designing expressions:

• Argument Limit: An expression should not have more than 5 arguments of any single type. If an expression requires more, it should be redesigned (e.g., instead of one large function, use nested logic like And(HasTag("comedy"), HasTag("action"))).
• Complexity: Expressions should be kept as simple as possible. For example, NAND logic should be expressed as Not(And()) rather than a dedicated operator, unless the complexity exceeds 2 or 3 operations.
• Supported Argument Types: Arguments must be one of the following types (which can be mapped to other CLR types like enums):
  • String
  • Double (integers are coerced to Double)
  • DateTime

// Example of a nested FilterExpression structure:
And(Or(HasTag('comedy'), HasTag('action')), Not(HasTag('18 restricted')))

The JobWatchdog runs as a background task and polls executing jobs every 15 seconds. If a job runs longer than WatchdogTimeoutSeconds (default 90s), it is flagged as a potential deadlock.

Watchdog Behavior:

1. First Detection: Logs an LogError containing the job type, elapsed time, and the call stack captured at the last IJobFactory.Execute entry point.
2. Subsequent Polls: Logs a LogWarning heartbeat indicating the job is still stuck.

How to prevent false positives: If a job is designed to run longer than the timeout, decorate it with the [LongRunning] attribute. This exempts the job from the watchdog's deadlock detection.

Capturing Call Stacks: Because .NET Core cannot suspend threads to get managed stack traces, the system relies on JobFactory.Execute<T> to capture the call stack when a worker job is invoked. This stack is stored in a SubExecutionTracker and included in the first-detection error log.

[LongRunning]
[LimitConcurrency(2)]
public class HashFileJob : IQueueJob { … }

To optimize performance, the "series with multiple releases" list uses two progressively expensive pre-filters before running the full grouping pipeline. These filters are designed to be never under-inclusive (they may produce false positives, but never false negatives).

1. GetAnimeIDsWithMultipleFilesPerEpisode(): Scans CrossRef_File_Episode for any anime ID where at least one episode has more than one distinct file (excluding orphaned references).
2. MightHaveMultipleCandidates(videos, animeId): A heuristic check on StoredReleaseInfo that evaluates group keys, source, codec/resolution/bit-depth, version collisions, and whether the same episode is covered by files in different (ManagedFolderID, ParentDirectory) partitions.

Important: Files in different folders are treated as hard separators. Even if all other signals match, the grouper will split them into separate candidates.

Shoko uses episode coverage to determine if files belong to the same release or represent parallel releases.

Patching vs. Parallel Releases

• Patching (Same Candidate): If files cover different episodes (e.g., ep 1 is v1, ep 3 is v2), they are considered part of the same release/patch series and land in the same candidate.
• Parallel Releases (Two Candidates): If a complete set of episodes is covered by one version (v1) and then a complete set is covered by another version (v2), this is a complete collision. Shoko splits these into two separate candidates (e.g., Candidate A for v1, Candidate B for v2).

Quality-Tier Splitting

When a complete collision occurs between two sets of files that have the same version number, Shoko performs a quality-tier split based on (IsCorrupted, IsChaptered). This allows the comparison service to rank a "clean" batch higher than a "corrupt" batch and mark the latter as redundant.

After initial grouping, if a bucket contains multiple files covering the same episode, the system applies the following logic in order to decide whether to keep them together or split them into separate candidates:

1. No collision: Each episode is covered by exactly one file. Action: Keep together.
2. Partial collision: Some episodes are covered by multiple files, but at least one episode is only covered by a single file (e.g., a release where only a few episodes were patched to v2). Action: Keep together.
3. All episodes collide, different versions: Every episode in the group has a file on both sides, and they have different StoredReleaseInfo.Version values. Action: Split into one candidate per version.
4. All episodes collide, same version, different quality tier: Every episode has a file on both sides, they share the same version, but IsCorrupted or IsChaptered flags differ. Action: Split into one candidate per quality tier.
5. All episodes collide, same version, same quality tier: The files are ambiguous (e.g., a combined-episode file vs a single-episode file). Action: Keep together.

Shoko Server uses three primary services to manage video files and organize them into logical releases:

1. VideoReleaseGroupingService: Takes a flat list of VideoLocal_Place objects and groups them into VideoReleaseCandidate buckets. Each bucket represents files estimated to belong to the same release (e.g., all episodes of a specific Blu-ray encode).
2. ReleaseComparisonService: Ranks these candidates using a configurable tie-breaker system. It identifies a 'primary' release and flags redundant 'secondary' releases that are safe to delete.
3. ReleaseAutoManagementService: Triggered at the end of the import pipeline (within FinalizeReleaseSearchJob). If AllowDeletion is enabled, it automatically removes files belonging to redundant candidates.

Grouping is deterministic (same inputs produce same buckets) but is a heuristic based on provider metadata and MediaInfo stream data.