Skip to content

UPathTools

UPathTools is a Python library that extends UPath and fsspec with powerful async capabilities, improved type safety, and a collection of specialized filesystem implementations.

Key Features

🚀 Async-First Design

  • AsyncUPath: Fully async path operations with aread_bytes(), awrite_text(), aiterdir(), etc.
  • Async filesystem operations: Batch file reading with configurable parallelism
  • Automatic fallback: Thread-pool execution for filesystems without native async support

🎯 Type Safety

  • Generic filesystem base classes with typed info dictionaries / UPath subclasses
  • Type-safe path classes specific to each filesystem
  • Proper overloads for detail/non-detail variants of operations

📦 Specialized Filesystems

  • Virtual filesystems: Browse Python AST, module structures, SQLite databases
  • Developer tools: CLI command output, Git repositories, package metadata
  • Aggregation: Filesystem wrappers to compose multiple filesystems into one
  • Remote: GitHub Gists / Issues / Wiki, Notion, Linear and multiple other remote filesystems

🛠️ Utility Functions

  • Async batch operations: read_folder(), list_files(), read_folder_as_text()
  • Path helpers: to_upath(), upath_to_fs(), type conversions
  • Tree visualization: Rich directory tree rendering

Quick Start

Basic Usage

from upathtools import AsyncUPath
import asyncio

async def main():
    # Read a file asynchronously
    path = AsyncUPath("https://example.com/data.json")
    content = await path.aread_text()

    # Iterate directory contents
    async for item in path.aiterdir():
        print(item)

asyncio.run(main())

Batch File Operations

from upathtools import read_folder
import asyncio

async def main():
    # Read all Python files in parallel
    files = await read_folder(
        "src/",
        pattern="**/*.py",
        load_parallel=True,
        chunk_size=50
    )

    for filepath, content in files.items():
        print(f"{filepath}: {len(content)} bytes")

asyncio.run(main())

Type-Safe Filesystem Access

from upathtools.filesystems import ModuleFileSystem, ModulePath

# Browse a Python module as a filesystem
fs = ModuleFileSystem(module_name="upathtools")
path: ModulePath = fs.get_upath("async_upath.py")

# Typed info dictionaries
info = path.info()
# info has type-safe fields specific to ModuleFileSystem

Generic base classes with full typing

UPathTools provides:

from upathtools.filesystems.base import BaseAsyncFileSystem

class MyFileSystem[MyPath, MyInfoDict](BaseAsyncFileSystem):
    """Fully typed filesystem with custom info structure."""

    upath_cls = MyPath

    async def _ls(self, path: str, detail: bool = True) -> list[MyInfoDict]:
        # Return typed info dictionaries
        ...

Installation

uv add upathtools[httpx]

# All optional dependencies
uv add upathtools[all]

Extras

Name Dependencies Description
appwrite appwrite Appwrite integration
beam beam-client>=0.2.187; sys_platform != 'win32' or python_version < '3.14' Beam integration
daytona daytona; python_version < '3.14' Daytona integration
e2b e2b-code-interpreter E2B integration
gitlab python-gitlab>=7.0.0 GitLab integration
httpx httpx, yarl, brotli HTTPX based filesystem
mcp fastmcp, mcp Virtual MCP Tools & MCP Resources filesystem
microsandbox microsandbox Microsandbox integration
modal modal Modal integration
notion-client notion-client Notion integration
openapi3 openapi3 OpenAPI integration
sqlite sqlalchemy[aiosqlite] SQLite integration
tree-sitter tree-sitter>=0.25.2, tree-sitter-python>=0.25.0, tree-sitter-c>=0.24.1, tree-sitter-javascript>=0.25.0 Tree-sitter based filesystem
vercel vercel Vercel integration

Package Overview

📂 upathtools
┣━━ 📂 configs
┃   ┣━━ 🐍 __init__.py (1.2 kB)
┃   ┣━━ 🐍 base.py (6.6 kB)
┃   ┣━━ 🐍 custom_fs_configs.py (9.5 kB)
┃   ┣━━ 🐍 file_based_fs_configs.py (5.5 kB)
┃   ┣━━ 🐍 fsspec_fs_configs.py (23.0 kB)
┃   ┣━━ 🐍 remote_fs_configs.py (8.6 kB)
┃   ┗━━ 🐍 sandbox_fs_configs.py (9.4 kB)
┣━━ 📂 filesystems
┃   ┣━━ 📂 base
┃   ┃   ┣━━ 🐍 __init__.py (977 Bytes)
┃   ┃   ┣━━ 🐍 basefilesystem.py (13.2 kB)
┃   ┃   ┣━━ 🐍 baseupath.py (15.9 kB)
┃   ┃   ┣━━ 🐍 file_objects.py (10.2 kB)
┃   ┃   ┣━━ 🐍 filefilesystem.py (10.3 kB)
┃   ┃   ┗━━ 🐍 wrapper.py (14.0 kB)
┃   ┣━━ 📂 file_filesystems
┃   ┃   ┣━━ 🐍 __init__.py (1.4 kB)
┃   ┃   ┣━━ 🐍 jsonschema_fs.py (30.0 kB)
┃   ┃   ┣━━ 🐍 markdown_fs.py (11.6 kB)
┃   ┃   ┣━━ 🐍 openapi_fs.py (37.5 kB)
┃   ┃   ┣━━ 🐍 sqlite_fs.py (16.9 kB)
┃   ┃   ┗━━ 🐍 treesitter_fs.py (18.9 kB)
┃   ┣━━ 📂 fsspec_filesystems
┃   ┃   ┣━━ 🐍 __init__.py (2.4 kB)
┃   ┃   ┣━━ 🐍 azure.py (456 Bytes)
┃   ┃   ┣━━ 🐍 gcs.py (430 Bytes)
┃   ┃   ┣━━ 🐍 hdfs.py (439 Bytes)
┃   ┃   ┣━━ 🐍 hf.py (486 Bytes)
┃   ┃   ┣━━ 🐍 s3.py (389 Bytes)
┃   ┃   ┣━━ 🐍 sftp.py (424 Bytes)
┃   ┃   ┣━━ 🐍 smb.py (416 Bytes)
┃   ┃   ┗━━ 🐍 webdav.py (430 Bytes)
┃   ┣━━ 📂 remote_filesystems
┃   ┃   ┣━━ 🐍 __init__.py (532 Bytes)
┃   ┃   ┣━━ 🐍 appwrite_fs.py (31.1 kB)
┃   ┃   ┣━━ 🐍 gist_fs.py (29.0 kB)
┃   ┃   ┣━━ 🐍 gitlab_fs.py (12.7 kB)
┃   ┃   ┣━━ 🐍 issue_fs.py (17.5 kB)
┃   ┃   ┣━━ 🐍 linear_fs.py (47.7 kB)
┃   ┃   ┣━━ 🐍 mcp_fs.py (12.1 kB)
┃   ┃   ┣━━ 🐍 mcp_tools_fs.py (15.6 kB)
┃   ┃   ┣━━ 🐍 notion_fs.py (18.8 kB)
┃   ┃   ┗━━ 🐍 wiki_fs.py (20.4 kB)
┃   ┣━━ 📂 sandbox_filesystems
┃   ┃   ┣━━ 🐍 __init__.py (1.3 kB)
┃   ┃   ┣━━ 🐍 beam_fs.py (15.3 kB)
┃   ┃   ┣━━ 🐍 daytona_fs.py (15.9 kB)
┃   ┃   ┣━━ 🐍 e2b_fs.py (14.2 kB)
┃   ┃   ┣━━ 🐍 microsandbox_fs.py (10.9 kB)
┃   ┃   ┣━━ 🐍 modal_fs.py (17.4 kB)
┃   ┃   ┣━━ 🐍 srt_fs.py (12.3 kB)
┃   ┃   ┗━━ 🐍 vercel_fs.py (9.8 kB)
┃   ┣━━ 🐍 __init__.py (3.4 kB)
┃   ┣━━ 🐍 async_local.py (7.8 kB)
┃   ┣━━ 🐍 auto_expand_fs.py (14.0 kB)
┃   ┣━━ 🐍 basemodel_fs.py (19.7 kB)
┃   ┣━━ 🐍 basemodel_instance_fs.py (18.6 kB)
┃   ┣━━ 🐍 delegating_fs.py (19.9 kB)
┃   ┣━━ 🐍 distribution_fs.py (6.9 kB)
┃   ┣━━ 🐍 flat_union_fs.py (20.0 kB)
┃   ┣━━ 🐍 httpx_fs.py (34.5 kB)
┃   ┣━━ 🐍 module_fs.py (9.5 kB)
┃   ┣━━ 🐍 overlay.py (12.0 kB)
┃   ┣━━ 🐍 package_fs.py (7.2 kB)
┃   ┣━━ 🐍 skills_fs.py (4.3 kB)
┃   ┗━━ 🐍 union_fs.py (9.9 kB)
┣━━ 📂 filetree
┃   ┣━━ 🐍 __init__.py (510 Bytes)
┃   ┣━━ 🐍 data.py (2.1 kB)
┃   ┣━━ 🐍 file_icons.py (18.9 kB)
┃   ┗━━ 🐍 filetree.py (13.2 kB)
┣━━ 🐍 __init__.py (3.9 kB)
┣━━ 🐍 async_ops.py (12.4 kB)
┣━━ 🐍 async_upath.py (8.2 kB)
┣━━ 🐍 common_types.py (292 Bytes)
┣━━ 🐍 helpers.py (8.9 kB)
┣━━ 🐍 log.py (582 Bytes)
┣━━ 📄 py.typed (0 Bytes)
┗━━ 🐍 pydantic_type.py (4.4 kB)

Dependencies

Name Summary Marker
tree-sitter Python bindings to the Tree-sitter parsing library extra == "tree-sitter"
httpx The next generation HTTP client. extra == "httpx"
python-dotenv Read key-value pairs from a .env file and set them as environment variables
appwrite Appwrite is an open-source self-hosted backend server that abstract and simplify complex and repetitive development tasks behind a very simple REST API extra == "appwrite"
universal_pathlib pathlib api extended to use fsspec backends
tree-sitter-python Python grammar for tree-sitter extra == "tree-sitter"
python-gitlab The python wrapper for the GitLab REST and GraphQL APIs. extra == "gitlab"
openapi3 Client and Validator of OpenAPI 3 Specifications extra == "openapi3"
brotli Python bindings for the Brotli compression library extra == "httpx"
fastmcp The fast, Pythonic way to build MCP servers and clients. extra == "mcp"
daytona Python SDK for Daytona python_full_version < "3.14" and extra == "daytona"
tree-sitter-javascript JavaScript grammar for tree-sitter extra == "tree-sitter"
notion-client Python client for the official Notion API extra == "notion-client"
morefs A collection of self-contained fsspec-based filesystems
SQLAlchemy Database Abstraction Library extra == "sqlite"
e2b-code-interpreter E2B Code Interpreter - Stateful code execution extra == "e2b"
microsandbox Microsandbox Python SDK extra == "microsandbox"
tree-sitter-c C grammar for tree-sitter extra == "tree-sitter"
modal Python client library for Modal extra == "modal"
vercel-sandbox Python SDK for Vercel Sandbox extra == "vercel"
beam-client (python_full_version < "3.14" and extra == "beam") or (sys_platform != "win32" and extra == "beam")
yarl Yet another URL library extra == "httpx"
fsspec File-system specification
mcp Model Context Protocol SDK extra == "mcp"

License

MIT License - see LICENSE for details.

Basic Usage

AsyncUPath - Async File Operations

from upathtools import AsyncUPath
import asyncio

async def main():
    # Read a file asynchronously
    path = AsyncUPath("https://example.com/data.json")
    content = await path.aread_text()

    # Write asynchronously
    output = AsyncUPath("/tmp/output.txt")
    await output.awrite_text("Hello, async!")

    # Iterate directory
    async for item in AsyncUPath("/tmp").aiterdir():
        if await item.ais_file():
            print(item)

asyncio.run(main())

Batch File Operations

from upathtools import read_folder, list_files
import asyncio

async def main():
    # Read multiple files in parallel
    files = await read_folder(
        "src/",
        pattern="**/*.py",
        load_parallel=True
    )

    for filepath, content in files.items():
        print(f"{filepath}: {len(content)} bytes")

    # Just list files
    py_files = await list_files(
        "src/",
        pattern="**/*.py",
        recursive=True
    )

asyncio.run(main())

Type-Safe Filesystems

from upathtools.filesystems import ModuleFileSystem

# Browse Python modules as filesystems
fs = ModuleFileSystem(module_name="upathtools")

# List module contents
files = fs.ls("/", detail=True)

# Read file with type safety
path = fs.get_upath("async_upath.py")
code = path.read_text()

Common Patterns

Convert Paths

from upathtools.helpers import to_upath
from pathlib import Path

# Convert any path type to UPath
path = to_upath("/tmp/file.txt")
path = to_upath(Path("/tmp/file.txt"))

# Get async wrapper
async_path = to_upath("/tmp/file.txt", as_async=True)

Read Single File Async

from upathtools import read_path

async def main():
    # Text mode
    text = await read_path("file.txt", mode="rt")

    # Binary mode
    data = await read_path("image.png", mode="rb")

Write Files

from upathtools.helpers import write_file

# Handles parent directory creation automatically
write_file("content", "/deep/nested/path/file.txt")
write_file(b"binary", "/path/to/file.bin")

Copy Files

from upathtools.helpers import copy

# Copy between filesystems
copy("s3://bucket/source.txt", "/tmp/local.txt")

# Copy to directory (preserves name)
copy("/tmp/file.txt", "/backup/")

Find Files

from upathtools.helpers import multi_glob

# Find with multiple patterns
files = multi_glob(
    directory="src/",
    keep_globs=["**/*.py", "**/*.md"],
    drop_globs=["**/test_*.py", "**/__pycache__/**"]
)

Combine Multiple Files

from upathtools import read_folder_as_text

async def main():
    # Create single document from multiple files
    combined = await read_folder_as_text(
        "docs/",
        pattern="**/*.md",
        recursive=True
    )
    print(combined)

Cheat Sheet

Async Operations

Operation Code
Read text await path.aread_text()
Read bytes await path.aread_bytes()
Write text await path.awrite_text("data")
Write bytes await path.awrite_bytes(b"data")
Check exists await path.aexists()
Is file await path.ais_file()
Is dir await path.ais_dir()
Create dir await path.amkdir(parents=True)
Remove file await path.aunlink()
Iterate dir async for item in path.aiterdir()
Glob async for match in path.aglob("*.py")

Batch Operations

Operation Function
Read multiple files read_folder(path, pattern="**/*")
List files list_files(path, pattern="**/*")
Combine files read_folder_as_text(path, pattern="**/*")
Read single read_path(path, mode="rt")

Helpers

Operation Function
Convert to UPath to_upath(path)
Get filesystem upath_to_fs(path)
Copy file copy(src, dst)
Write file write_file(content, path)
Multi-glob multi_glob(keep_globs=[...])
Clean dir clean_directory(path)