# Networked File Transfer Research

Research into efficient file transfer between agents in a networked owl messaging system.

**Date:** 2026-04-04
**Status:** Research / Proposal
**Context:** Current architecture uses length-prefixed TCP (16 MB max), UTF-8 only, SQLite spool fallback, filesystem inbox fallback, zero external dependencies.

---

## Current Constraints

- **16 MB hard limit** on message size (protocol layer)
- **UTF-8 only** — binary data must be encoded
- **Message atomicity** — entire message read into memory
- **Spool TTL** — 30 minutes before messages expire
- **Zero-dependency** — portability on Windows is a hard constraint

---

## Option 1: Inline Base64 in Messages

Encode the file as base64 and send it as a regular message with a metadata header.

```
__FILE__:report.csv
__SIZE__:34218
__HASH__:sha256:a1b2c3...
SGVsbG8gV29ybGQ...
```

| Pros | Cons |
|------|------|
| Zero new infrastructure — works with existing protocol | 33% size overhead from base64 |
| Spool/inbox fallback works unchanged | 16 MB limit means ~11 MB real file max |
| Atomic delivery — file arrives or it doesn't | Entire file in memory at once |
| Works through every transport layer (TCP, spool, inbox) | Not viable for large files |
| Trivial to implement | Receiver must know to decode vs treat as text |

**Best for:** Config files, small data, logs, code snippets — anything under ~10 MB.

---

## Option 2: Chunked Transfer Protocol

Add a transfer ID and chunk headers to the protocol. Sender splits the file, receiver reassembles.

```
__XFER__:xfer-001  chunk:1/5  file:data.bin  hash:sha256:...
<base64 chunk>
```

Each chunk is a normal owl message. Receiver buffers chunks by transfer ID, assembles when complete.

| Pros | Cons |
|------|------|
| No file size limit (chunks fit under 16 MB each) | Ordering and reassembly logic needed |
| Each chunk uses existing delivery pipeline | Partial transfers need cleanup/timeout |
| Resumable — can re-request missing chunks | Multiple messages for one file = more traffic |
| Backpressure natural (one message at a time) | Spool TTL (30 min) could expire early chunks |
| Still zero new dependencies | Receiver needs temp staging directory |

**Best for:** Medium files (10 MB - 1 GB). Good balance of complexity vs capability.

---

## Option 3: Out-of-Band with Shared Storage Reference

Don't send the file through owl at all. Write it to a shared location and send a reference message.

```
__FILE_REF__:owl-xfer/xfer-001/model-weights.bin
__SIZE__:2147483648
__HASH__:sha256:...
```

For **local** agents: shared filesystem path (e.g., `{SPT_HOME}/owlery/.xfer/`). <!-- path updated Phase 18.2 -->
For **networked** agents: presigned URL (S3, R2, etc.), or a temporary HTTP server.

| Pros | Cons |
|------|------|
| Zero overhead — file stays on disk/object store, only a pointer moves | Requires shared storage both sides can reach |
| No size limit whatsoever | Introduces external dependency (for network case) |
| No encoding overhead | Two-phase: send ref, then receiver must fetch |
| Streaming-friendly — receiver can stream from source | Cleanup responsibility (who deletes after fetch?) |
| Works with existing message protocol as-is | Local-only variant doesn't help networked agents |

**Best for:** Large files (100 MB+), binary blobs, model weights. The only viable option past ~1 GB.

---

## Option 4: Binary Protocol Extension

Extend the length-prefixed protocol with a message type byte. Type `0x00` = text (current), `0x01` = binary file transfer with header.

```
[4-byte length][0x01][header-length][JSON header][raw binary bytes]
```

Header: `{"file":"data.bin","size":34218,"hash":"sha256:..."}`

| Pros | Cons |
|------|------|
| Zero encoding overhead — raw bytes on the wire | Breaks UTF-8 assumption throughout codebase |
| Most bandwidth-efficient option | Spool (SQLite TEXT column) can't store binary |
| Clean protocol evolution | Inbox (.msg files) need binary-safe I/O |
| Can combine with chunking for large files | Every consumer must handle both message types |
| Single TCP round-trip for files under 16 MB | Biggest code change surface area |

**Best for:** Performance-critical transfers where base64 overhead matters. High implementation cost.

---

## Recommendation: Layered Approach

### Phase 1 — Inline base64 (Option 1)

Start here. A few dozen lines of code, covers 90% of agent file sharing (configs, code, logs, small datasets). Add a `--file` flag to `owl send`.

### Phase 2 — Shared storage references (Option 3)

Add when going networked. For internet-connected agents, a presigned URL pattern over S3/R2 is the cleanest. The owl message is just the reference; the heavy lifting is a simple HTTP GET. Avoids pushing large binary data through the message spool entirely.

### Skip: Binary protocol extension (Option 4)

The UTF-8 invariant is load-bearing across the entire stack (protocol, spool, inbox, message parsing). Breaking it for marginal throughput gains isn't worth the complexity unless profiling shows base64 is a real bottleneck.

### Defer: Chunked transfer (Option 2)

Middle-ground if large file support is needed but shared storage can't be added. Most complex to get right (ordering, timeouts, partial cleanup) for a benefit that Option 3 handles more cleanly.