Scavenger Documentation

In Scavenger, a codebase is a representation of the code being scanned at the time of collection.

New Codebase Triggers

Scavenger treats the code as a new codebase and performs a new upload if:

1. The content of the code changes.
2. The scanning settings (such as packages) are modified.

Data Preservation and Garbage Collection

• Method Identification: Each method is identified by a unique signature (a function signal).
• Call Information: Call information is recorded based on the method in each codebase. If a method's signature remains unchanged between codebase uploads, its call information is preserved.
• Garbage Collection: When a new codebase is uploaded, a garbage collector runs at regular intervals to remove methods from the old codebase that are no longer present in the new one, while inserting new methods.

Scavenger operates through several interconnected components:

• Scavenger Agent: Collects the codebase and regularly sends host application invocation data to collectors. There are versions for Java and a (BETA) Python agent.
• Scavenger Collector: Receives data from the agent and stores it in a database (supporting MySQL, Vitess, or H2). It also handles garbage cleanup.
• Scavenger API: Provides APIs for exploring and querying invocation data.
• Scavenger Frontend: Provides the web-based user interface for interacting with the system.

Keep the following technical limitations in mind when using the Python agent:

1. Reference-based Instrumentation: The agent works by replacing the reference of the current module/class. It will not work if a function is called without using a reference (e.g., if a framework uses a Decorator to save only the function reference without the module context). Example of what currently fails:

   @app.route('/')
   def hello_world():
       return 'Hello World!'

2. No pyc Support: Direct instrumentation for .pyc files is not supported; the agent requires access to the original Python source code.
3. Manual Shutdown: As noted, agent.shutdown() must be manually invoked.

If you are using the Scavenger Agent, you need a configuration file to define its runtime settings. Scavenger provides a built-in generator to simplify this process:

1. In the Scavenger UI, click the Show Scavenger configuration file generator button.
2. In the Create scavenger configuration file modal, configure your desired settings.
3. Click the Download button to save the configuration file for use with your agent.

If you prefer to build the Scavenger Agent manually rather than downloading a pre-built JAR, follow these steps:

1. Clone the Scavenger repository from Git.
2. Use the Gradle wrapper to assemble the Java agent specifically.

Note: You must have the Scavenger project built before running the agent-specific assembly command.

./gradlew assemble -p scavenger-agent-java

Before using Scavenger, you must install both the Collector and the API.

By default, this guide assumes the following network configuration:

• Collector: port 8080
• API: port 8081

Once a snapshot is created, you can analyze the invocation state through the Scavenger UI. The analysis view provides:

• Snapshot Recreation: Ability to recreate a snapshot based on the most recent invocation state currently collected.
• Call Treemap: A visual graph of calls within a selected package.
  • Area Size: Represents the number of methods (larger area = more methods).
  • Color: Represents the call percentage (black color = lower percentage of calls).
• Package-level Status: Displays the number of methods and the invocation rate per package.
• Code Integration: Links to open the specific class directly in IntelliJ or GitHub.

To debug the Scavenger Agent, you must configure a Remote JVM Debug session in your IDE (such as IntelliJ IDEA) and run the agent with specific JVM options to allow the debugger to attach via a socket.

1. Prepare the Debug Configuration

In your IDE, create a new Run/Debug Configuration of type Remote JVM Debug. Use the following settings:

• Host: localhost
• Port: 5005
• JVM option: -agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=*:5005

2. Start Debugging

1. Execute the agent startup script: ./run-agent.sh.
2. Run the Remote JVM Debug configuration you created in step 1.
3. Begin debugging using your IDE's standard tools (breakpoints, stepping, etc.).

# Step 1: Run the agent
./run-agent.sh

# Step 2: Use these JVM options in your configuration
-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=*:5005

To begin managing your data, connect to the Scavenger API at http://{API host}:8081/scavenger and follow these steps:

1. Click the Create workspace button in the bottom left corner.
2. In the modal that appears, enter a name for your workspace.
3. Click Create to finalize the workspace setup.

The API can be obtained via the GitHub releases page or built manually.

To build the API manually, run:

./gradlew assemble -p scavenger-api

To start the API, you must specify the Collector's URL using the scavenger.collector-server-url property:

java -Dscavenger.collector-server-url=http://localhost:8080 -jar scavenger-api-boot.jar