AI Diplomacy

The Diplomacy Game Analysis report provides a post-game breakdown of strategic performance, model reliability, and behavioral patterns. It is used to evaluate how different LLM agents (Power Models) perform in a Diplomacy game environment.

Key sections of the report include:

• Game Narrative: A chronological summary of the game's progression.
• Summary: Quantitative metrics of strategic actions (e.g., Betrayals, Collaborations, Brilliant Strategies, Strategic Blunders).
• Score Distribution: A breakdown of the quality scores assigned to analyzed moments.
• Power Models: A mapping of game powers (e.g., AUSTRIA, ENGLAND) to the specific LLM models used.
• Invalid Moves by Model: A count of illegal or impossible moves attempted by each model.
• Lies Analysis: A breakdown of deceptive behavior, distinguishing between intentional and unintentional lies.
• Notable Lies: Detailed logs of specific instances where a model's promise to another player contradicted its actual orders, including the negotiation_summary from its internal diary.

The AI agents in this project are built around several key capabilities:

• Stateful Agents: Uses DiplomacyAgent to manage dynamic goals, relationship tracking (Enemy, Unfriendly, Neutral, Friendly, Ally), and personality-driven system prompts.
• Rich Negotiations: Supports multi-round private and global message exchanges with relationship-aware strategies and message history tracking.
• Strategic Order Generation: Combines BFS pathfinding for movement analysis with context-aware order selection. It supports multiple LLM providers including OpenAI, Claude, Gemini, DeepSeek, and OpenRouter.
• Advanced Game Analysis: Includes betrayal detection (comparing orders vs. negotiations), phase summaries, and strategic planning phases.
• Memory Management: Uses a dual-layer system consisting of a Private Diary (structured, phase-prefixed entries) and Yearly Consolidation (automatic summarization to prevent context overflow).

The game analysis output provides a detailed breakdown of strategic moments in a Diplomacy game. Each analyzed moment (e.g., S1901M, F1901M) typically includes:

• Score: A qualitative assessment of the strategic quality (e.g., 7.0/10).
• Powers Involved: The specific players and the LLM models driving them (e.g., RUSSIA (claude-opus-4-20250514)).
• Agreement: A summary of the diplomatic pacts or understandings reached between players.
• Action Taken: The actual moves executed by the players in that turn.
• Impact: The strategic consequence of those moves (e.g., securing supply centers or establishing dominance).
• Diary Context: Detailed, player-specific logs containing:
  • Negotiation Summary: A narrative of diplomatic exchanges.
  • Intent: The player's planned moves and strategic goals.
  • Relationships: A mapping of player attitudes toward others (e.g., {'AUSTRIA': 'Enemy', 'ITALY': 'Friendly'}).

When implementing or debugging success rate logic, you must account for two distinct game data formats. Failure to distinguish between them will result in 0% success rates for models playing in older game formats.

New Game Format

• Result Keying: Results are keyed by power name.
• Success Indicator: A string value of "success" indicates a successful order.

Old Game Format

• Result Keying: Results are keyed by unit location (e.g., "A PAR" extracted from an order string like "A PAR - PIC").
• Success Indicator: Success is indicated by an empty list [], an empty string "", or None.

# Logic for parsing Old Format successes
unit_loc = None
if ' - ' in order_str or ' S ' in order_str or ' C ' in order_str or ' H' in order_str:
    parts = order_str.strip().split(' ')
    if len(parts) >= 2 and parts[0] in ['A', 'F']:
        unit_loc = f"{parts[0]} {parts[1]}"

if unit_loc and unit_loc in results_dict:
    result_value = results_dict[unit_loc]
    if isinstance(result_value, list) and len(result_value) == 0:
        model_phase_data[model]['order_successes'][order_type] += 1
    elif isinstance(result_value, str) and result_value == "":
        model_phase_data[model]['order_successes'][order_type] += 1
    elif result_value is None:
        model_phase_data[model]['order_successes'][order_type] += 1