Fraud Detection Demo

repository·master·Indexed 43 days ago

https://github.com/afedulov/fraud-detection-demo

A demonstration system for real-time transaction monitoring and rule-based fraud detection. It features a Java Spring Boot backend, a React frontend, and a Flink job for dynamic rule evaluation. The system integrates Kafka for data streaming, H2 for in-memory storage, and uses Docker Compose to orchestrate Zookeeper, Kafka, and Schema Registry.

Tokens
6.9K
Snippets
28
Records
33
Agent score
55%

What's inside Fraud Detection Demo

  1. Format rule updates for the Fraud Detection Demo

    master

    When submitting rules via netcat, use the following comma-separated format:

    rule_id, (rule_state), (aggregation keys), (unique keys), (aggregateFieldName field), (aggregation function), (limit operator), (limit), (window size in minutes)

    CSV Examples:

    • 1,(active),(paymentType),,(paymentAmount),(SUM),(>),(50),(20)
    • 1,(delete),(paymentType),,(paymentAmount),(SUM),(>),(50),(20)
    • 2,(active),(payeeId),,(paymentAmount),(SUM),(>),(10),(20)
    • 2,(pause),(payeeId),,(paymentAmount),(SUM),(>),(10),(20)
    • 1,(active),(paymentType),,(COUNT_FLINK),(SUM),(>),(50),(20) (using special function COUNT_FLINK)

    JSON Format:

    Alternatively, rules can be submitted using this JSON structure:

    { "ruleId": 1, "ruleState": "ACTIVE", "groupingKeyNames": ["paymentType"], "unique": [], "aggregateFieldName": "paymentAmount", "aggregatorFunctionType": "SUM","limitOperatorType": "GREATER","limit": 500, "windowMinutes": 20}
    1,(active),(paymentType),,(paymentAmount),(SUM),(>),(50),(20)
  2. Run the Dynamic Fraud Detection Demo locally with netcat

    master

    To execute the fraud detection demo locally using netcat as the data source, follow these steps:

    1. Start a netcat listener on port 9999:
      nc -lk 9999
    2. Run the main method of the class com.ververica.field.dynamicrules.Main in your IDE or via CLI.
    3. Send rule updates or control commands to the netcat listener in the required format.
    nc -lk 9999
  3. Define the Alert and Rule relationship

    master

    The dashboard visualizes the flow from Transactions $\rightarrow$ Rules $\rightarrow$ Alerts.

    • Rules: Fetched from /api/rules. Each rule is associated with a specific transaction.
    • Alerts: Received via WebSocket. An alert is linked to a rule via the ruleId property.
    • Visual Logic:
      • Lines connect Transactions to Rules.
      • Lines connect Rules to Alerts.
      • If a rule is currently triggering an alert, the connection line color changes to red (#dc3545); otherwise, it remains orange (#ff7f50).
  4. Configure the Kafka infrastructure via Docker Compose

    master

    The project uses a docker-compose.yaml file to orchestrate a fraud detection data pipeline consisting of Zookeeper, Kafka, Schema Registry, and a Schema Registry UI.

    To start the entire stack, run:

    docker-compose up

    Service Overview

    • Zookeeper (zoo1): Manages Kafka cluster state. Accessible on port 2181.
    • Kafka (kafka1): The message broker. It is configured with two listeners:
      • LISTENER_DOCKER_INTERNAL: Used for communication within the Docker network (port 19092).
      • LISTENER_DOCKER_EXTERNAL: Used for external access from the host machine (port 9092).
    • Schema Registry (schema-registry): Manages Avro schemas. Accessible on port 18081.
    • Schema Registry UI (schema-registry-ui): A web interface to view schemas. Accessible on port 8001 (mapped from container port 8000).
  5. Run the backend service via Main.java

    master

    The backend service is a Spring Boot application. You can start the application by executing the main method in the com.ververica.demo.backend.Main class. This initializes the Spring context, enables auto-configuration, and performs a component scan across the com.ververica package to wire up the necessary beans for the fraud detection demo.

    package com.ververica.demo.backend;
    
    import org.springframework.boot.SpringApplication;
    import org.springframework.boot.autoconfigure.EnableAutoConfiguration;
    import org.springframework.boot.autoconfigure.SpringBootApplication;
    import org.springframework.context.annotation.ComponentScan;
    
    @SpringBootApplication
    @EnableAutoConfiguration
    @ComponentScan("com.ververica")
    public class Main {
      public static void main(String[] args) {
        SpringApplication.run(Main.class, args);
      }
    }
  6. Adjust transaction generation speed via API

    master

    The UI includes a slider to control the speed of the transaction generator. Changing the slider value updates a local storage key generatorSpeed and triggers a fetch request to the backend to adjust the simulation speed.

    Note that the value sent to the API is transformed by getFakeValue(), which scales the slider input (0-30) into a higher magnitude value used by the generator.

  7. Configure Schema Registry and UI

    master

    The Schema Registry and its UI are used to manage data schemas for Kafka messages.

    Schema Registry (schema-registry)

    • Port: 18081
    • Bootstrap Servers: Connects to Kafka via PLAINTEXT://kafka1:19092.
    • Listeners: Listens on http://0.0.0.0:18081 (external) and http://schema-registry:28081 (internal).

    Schema Registry UI (schema-registry-ui)

    • Port: 8001 (Host) / 8000 (Container)
    • Connection: Configured to point to the registry via SCHEMAREGISTRY_URL: http://schema-registry:28081/.
    schema-registry:
        image: confluentinc/cp-schema-registry:5.2.1
        hostname: schema-registry
        ports:
        - "18081:18081"
        environment:
          SCHEMA_REGISTRY_KAFKASTORE_BOOTSTRAP_SERVERS: PLAINTEXT://kafka1:19092
          SCHEMA_REGISTRY_HOST_NAME: schema-registry
          SCHEMA_REGISTRY_LISTENERS: http://0.0.0.0:18081,http://schema-registry:28081
    
    schema-registry-ui:
        image: landoop/schema-registry-ui:0.9.4
        hostname: schema-registry-ui
        ports:
        - "8001:8000"
        environment:
          SCHEMAREGISTRY_URL: http://schema-registry:28081/
          PROXY: "true"
  8. Kafka connection settings and listeners

    master

    The Kafka service (kafka1) is configured to support both internal Docker networking and external host access.

    • External Access: Connect to Kafka from your host machine using 127.0.0.1:9092 (or the value of ${DOCKER_HOST_IP}).
    • Internal Access: Other services within the Docker network should use kafka1:19092.

    Key environment variables used for this configuration:

    • KAFKA_ADVERTISED_LISTENERS: Defines the connection strings for internal and external listeners.
    • KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: Maps listener names to PLAINTEXT.
    • KAFKA_INTER_BROKER_LISTENER_NAME: Set to LISTENER_DOCKER_INTERNAL for inter-broker communication.
    kafka1:
        image: confluentinc/cp-kafka:5.0.0
        hostname: kafka1
        ports:
        - "9092:9092"
        environment:
          KAFKA_ADVERTISED_LISTENERS: LISTENER_DOCKER_INTERNAL://kafka1:19092,LISTENER_DOCKER_EXTERNAL://${DOCKER_HOST_IP:-127.0.0.1}:9092
          KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: LISTENER_DOCKER_INTERNAL:PLAINTEXT,LISTENER_DOCKER_EXTERNAL:PLAINTEXT
          KAFKA_INTER_BROKER_LISTENER_NAME: LISTENER_DOCKER_INTERNAL
          KAFKA_ZOOKEEPER_CONNECT: "zoo1:2181"
          KAFKA_BROKER_ID: 1
  9. Use Control Commands for rule management

    master

    You can send special control commands to the system using JSON format to manage the state of the fraud detection engine. The ruleState must be set to "CONTROL".

    Available controlType values:

    • DELETE_RULES_ALL: Deletes all rules.
    • EXPORT_RULES_CURRENT: Exports the current rules.
    • CLEAR_STATE_ALL: Clears all internal state.

    Examples:

    {"ruleState": "CONTROL", "controlType":"DELETE_RULES_ALL"}
    {"ruleState": "CONTROL", "controlType":"EXPORT_RULES_CURRENT"}
    {"ruleState": "CONTROL", "controlType":"CLEAR_STATE_ALL"}
    {"ruleState": "CONTROL", "controlType":"DELETE_RULES_ALL"}