Storage Configuration¶

The storage configuration defines how agent interactions, messages, and tool usage are logged. It's defined at the root level of the manifest.

Basic Structure¶

storage:
  # List of storage providers
  providers:
    - type: "sql"  # SQL database (default)
      url: "sqlite:///history.db"
      pool_size: 5
      auto_migration: true
      agents: ["planner", "executor"]  # Only log these agents
      log_messages: true
      log_conversations: true
      log_tool_calls: true
      log_commands: true
      log_context: true

  # Global agent filtering
  agents: ["planner", "executor", "analyzer"]  # Global agent filter
  filter_mode: "and"  # How to combine filters: "and" | "override"

  # Default provider for history queries
  default_provider: "sql"

  # Global logging flags (apply to all providers)
  log_messages: true      # Log all messages
  log_conversations: true # Log conversation metadata
  log_tool_calls: true   # Log tool executions
  log_commands: true     # Log command executions
  log_context: true      # Log context additions

Agent Filtering¶

You can filter which agents get logged at both global and provider levels:

storage:
  # Global filter - affects all providers
  agents: ["planner", "executor", "analyzer"]

  # How filters are combined:
  filter_mode: "and"     # Both filters must allow the agent
  # filter_mode: "override" # Provider filter overrides global if set

  providers:
    - type: "sql"
      url: "sqlite:///history.db"
      agents: ["planner", "executor"]  # Provider-specific filter

With filter_mode: "and":

An agent must be allowed by both global AND provider filters
If either filter is None, only the other filter applies
If both are None, all agents are logged

With filter_mode: "override":

Provider filter takes precedence if set
Falls back to global filter if provider filter is None
If both are None, all agents are logged

Available Providers¶

SQL Storage (Default)¶

Uses SQLModel for database storage. Supports SQLite, PostgreSQL, etc.

storage:
  providers:
    - type: "sql"
      url: "sqlite:///history.db"  # Database URL
      pool_size: 5  # Connection pool size
      auto_migration: true  # Automatically add missing columns
      agents: ["planner", "executor"]  # Agent filter
      # Logging flags (override global)
      log_messages: true
      log_conversations: true
      log_tool_calls: true
      log_commands: true
      log_context: true

Text Log Templates¶

The text log provider supports customizable Jinja2 templates. You can either use predefined formats or provide your own template file.

storage:
  providers:
    - type: "text_file"
      path: "logs/chat.log"
      # Use predefined template:
      format: "chronological"  # or "conversations"
      # Or use custom template:
      template: "templates/custom.j2"

Available Templates¶

Two predefined formats are available:

chronological: All events in chronological order
conversations: Grouped by conversation with commands at the end

Custom Templates¶

You can create custom templates with access to these variables:

Entry Types and Fields¶

Each entry has a type field and type-specific data:

# Message entry
{
    "type": "message",
    "timestamp": datetime,
    "conversation_id": str,
    "content": str,
    "role": str,
    "name": str | None,
    "model": str | None,
    "cost_info": {
        "token_usage": {
            "total": int,
            "prompt": int,
            "completion": int
        },
        "total_cost": float
    },
    "response_time": float | None,
    "forwarded_from": list[str] | None
}

# Conversation start entry
{
    "type": "conversation_start",
    "timestamp": datetime,
    "conversation_id": str,
    "agent_name": str
}

# Tool call entry
{
    "type": "tool_call",
    "timestamp": datetime,
    "conversation_id": str,
    "message_id": str,
    "tool_name": str,
    "args": dict[str, Any],
    "result": Any
}

# Command entry
{
    "type": "command",
    "timestamp": datetime,
    "agent_name": str,
    "session_id": str,
    "command": str,
    "context_type": str | None,
    "metadata": dict[str, Any] | None
}

Refer to the source for the details, an in-depth explanation will follow.

File Storage¶

Stores data in structured files (JSON, YAML, etc.).

storage:
  providers:
    - type: "file"
      path: "data/history.json"  # Storage file path
      format: "auto"  # "auto" | "json" | "yaml" | "toml" | "ini"
      encoding: "utf-8"  # File encoding
      agents: ["planner"]  # Agent filter
      # Logging flags
      log_messages: true
      log_conversations: true
      log_tool_calls: true
      log_commands: true
      log_context: true

Memory Storage¶

In-memory storage for testing.

storage:
  providers:
    - type: "memory"  # No additional configuration needed
      agents: null  # Log all agents
      # Logging flags
      log_messages: true
      log_conversations: true
      log_tool_calls: true
      log_commands: true
      log_context: true

Multiple Providers¶

You can use multiple providers simultaneously, each with their own filtering:

storage:
  # Global settings
  agents: ["planner", "executor", "analyzer"]
  filter_mode: "and"
  log_messages: true
  log_conversations: true
  log_commands: true
  log_context: true
  default_provider: "sql"

  providers:
    - type: "sql"  # Primary storage in database
      url: "sqlite:///history.db"
      agents: ["planner", "executor"]  # Only execution pipeline

    - type: "text_file"  # Analysis logs
      path: "logs/analysis.log"
      agents: ["analyzer"]  # Only analysis pipeline
      format: "chronological"

    - type: "file"  # Complete history
      path: "data/history.json"
      agents: null  # Log everything

Provider Selection¶

When loading history, providers are selected in this order:

Explicitly specified preferred provider
Default provider if configured in default_provider
First capable provider in the list
Raises error if no capable provider found

Logging Flags¶

All logging flags can be set both globally and per provider:

log_messages: Log all messages exchanged
log_conversations: Log conversation metadata
log_tool_calls: Log tool executions and results
log_commands: Log command executions
log_context: Log context additions and changes

Provider flags are combined with global flags using AND logic:

storage:
  log_messages: true     # Global setting
  providers:
    - type: "sql"
      log_messages: true # Must also be true here to log messages

Notes¶

Individual provider settings are combined with global settings
SQL provider is recommended for production use
Memory provider is useful for testing
Text logs are good for debugging and monitoring
File storage is suitable for simple deployments
Agent filtering allows splitting logs across different providers