vaskes/viexp
1
0
Fork 0
Code Issues 10 Pull Requests Actions Packages Projects 1 Releases Wiki Activity

docs: bootstrap vault, roadmap and research

Browse Source
This commit is contained in:
Vasily Guzov
2026-10-17 21:14:19 +03:00
parent 749afadcd4
commit 8fa0632d69
34 changed files with 1508 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/research/competitor-landscape.md
+76
View File
@@ -0,0 +1,76 @@
---
type: research
status: analyzed
confidence: high
created: 2026-10-17
question: Which terminal file managers exist in 2026, what does each do best, and which ideas should viexp adopt or avoid?
tags: [research, file-manager, yazi, nnn, lf, vifm, broot]
---
# Competitor landscape (October 2026)
## Feature matrix
| Manager | Lang | TUI | Image preview | Extensions | Config | Async | State (last release) |
|---|---|---|---|---|---|---|---|
| **yazi** | Rust | ratatui | native kitty/iterm2/sixel/ueberzug++/chafa | Lua (mlua, vendored) + `ya pack` | TOML ×3 | tokio, fully | Very active, v26.9.1 (2026-09) |
| **nnn** | C | ncurses | via FIFO plugins | any executable; `NNN_PIPE`/`NNN_FIFO` protocol | none (flags+env) | no (tiny by design) | Active, v5.3 (2026-08) |
| **lf** | Go | own | sixel native; else scripts | shell cmds, `push`, remote socket API | lfrc DSL | async I/O | Active, r42 (2026-07) |
| **joshuto** | Rust | tui-rs/termion | external scripts | config only | 5 TOMLs | partial | Semi-stalled, v0.9.9 (2025-05) |
| **felix** | Rust | termion | chafa + bat | none | YAML | partial | Dormant since 2025-04 |
| **hunter** | Rust | own | halfblock/sixel/kitty | mime actions | conf+keymap | heavily threaded | Dead (2020/2022) |
| **vifm** | C | ncurses | community scripts only | vim-like `:commands` + Lua | vifmrc | no | Active, v0.14.4 (2026-05) |
| **broot** | Rust | crossterm+termimad | kitty graphics | verbs, skins | TOML | interruptible search | Very active, v1.61 (2026-10) |
| **mc** | C | own | none | extfs VFS helpers | menus | background ops | Active, 4.8.33 |
## What the winners do right
- **yazi** (~43k stars): ~25-crate workspace, everything on tokio; micro/macro worker pools with
priorities (mime/previews preempt copying); **paged preloading** — only visible rows get work;
three cancellation mechanisms (tokio abort, atomic tickets for CPU work, instruction-count
checks for Lua); two-pass image pipeline (downscale+cache → fit to terminal); DDS pub/sub
between instances without a daemon. v25.4 doubled startup speed via config parser rewrite.
- **nnn**: 150 KiB binary, zero config; plugins = executables with env-var contract
(`$1` hovered file, `$2` cwd, `$3` picker) and a tiny write-back protocol via `NNN_PIPE`;
`NNN_FIFO` streams hovered paths so external scripts render live previews.
- **lf**: external `previewer(file, w, h, x, y)` + `cleaner` contract with exit-code-controlled
caching; `push` simulates keystrokes making mappings composable; server model over Unix socket.
- **broot**: every keystroke interrupts the running search; fuzzy+regex+content combined with
boolean operators; collapsed tree respecting .gitignore.
- **felix**: built-in trash + undo/redo + zoxide `z` jump — small features with huge daily value.
## Ideas worth stealing (ranked)
1. yazi's paged task scheduling + prioritized workers + cancellation tickets (perceived speed)
2. Protocol auto-detection adapter layer behind one trait (kitty/iterm2/sixel/ueberzug++/chafa)
3. lf's previewer/cleaner argv contract — cheapest path to infinite formats
4. nnn's FIFO hover event stream for external live previews
5. yazi's TOML tables with `prepend_*`/`append_*` merge semantics for rules
6. TOML for config + (later) Lua for behavior + pinned plugin packages
7. broot's interruptible, composable search
8. lf's `push` key-simulation primitive
9. felix's trash + undo + `z` jump
10. Performance as a marketable feature (both nnn and yazi publish benchmarks)
## Anti-patterns to avoid
| Trap | Victim | Lesson |
|------|--------|--------|
| Full ranger parity alone | joshuto | 5 config files, janky command line, 14-month release gap → audience left for yazi. Ship narrow and polished. |
| Nightly compiler + heavy C deps | hunter | GStreamer + nightly = unbuildable for users. Stable Rust, spawned binaries for media. |
| No native graphics path | vifm, nnn, mc | Preview zoo of mutually incompatible scripts. Build the adapter layer early. |
| Bespoke config DSL | lf | A whole parser users must learn; standard TOML + scripting runtime beats it. |
| Doctrinal non-features | lf (no tabs) | Minimal defaults and power features are not contradictory (yazi proves it). |
| Monolith | mc | Thin core + clean contracts. |
| API churn without pinning | yazi (mild) | Plugin version pinning story from day one. |
| Silence without governance | felix, hunter | Release cadence = perceived life. |
## Outcome
Shaped ADR-001 (ratatui), ADR-002 (extension phases), ADR-004 (protocol chain), and the
roadmap order (core narrow first, previews second, plugins third, Lua last).
## Sources
- yazi: github.com/sxyazi/yazi, https://yazi-rs.github.io/blog/why-is-yazi-fast/, releases v25–v26
- nnn: github.com/jarun/nnn + plugins/README.md
- lf: github.com/gokcehan/lf + doc.md
- joshuto, felix, hunter, vifm, broot, mc: repos + release APIs (dates verified 2026-10-17)
docs/research/plugin-architecture.md
+69
View File
@@ -0,0 +1,69 @@
---
type: research
status: analyzed
confidence: high
created: 2026-10-17
question: How should viexp's file-type modules and plugins be designed so Linux users can disable, replace and extend them easily?
tags: [research, plugins, architecture, lua, wasm]
---
# Extension architecture research
## Mechanism comparison
| Mechanism | Safety | Ease for users | Host complexity | Hot reload | Proof |
|---|---|---|---|---|---|
| External process contract (argv/env/stdout) | low (= shell) | **highest** — any language | very low | free | nnn plugins, lf previewer, ranger scope.sh |
| Declarative rules (TOML / rifle / mailcap) | high (data, not code) | high — one-liners | low | free | rifle.conf, yazi `[plugin]` tables |
| In-process Lua (mlua) | medium — sandboxes leak (2025 CVE); Luau is the hardened dialect | medium | medium-high | yes | yazi (mlua 0.12, vendored Lua 5.5), wezterm, neovim |
| Rhai | high | low (niche language) | low-medium | yes | niche |
| JS (deno_core/V8) | medium-high | medium | high (~30 MB) | yes | Deno |
| WASM (wasmtime/extism) | **highest** | lowest (compile a component) | high (WIT churn) | yes | zellij |
| Host-language plugins (Python) | lowest (monkeypatching) | high for pythonistas | medium, fragile | poor | ranger |
## Case-study lessons
- **ranger**: the script pipeline (scope.sh + rifle) carried ~90% of real customization; the
Python layer exists because previews/commands need *state*. Three disjoint mechanisms = confusion.
- **nnn**: cleanest "process as API": `$1`=file, `$2`=cwd, `$3`=picker; context in env
(`NNN_FIFO` hover stream); control back via `NNN_PIPE` with tiny opcodes (`1c/etc` = cd ctx 1).
- **lf**: minimum viable preview contract = `(path, w, h, x, y)` + `cleaner` hook + exit-code
caching semantics. Document SIGPIPE/exit codes or users suffer.
- **yazi**: built-ins are preset plugins routed through the same `[plugin]` TOML tables users
edit (`prepend_*`/`append_*` merge) — one dispatch model for built-in and user code. mlua
with vendored Lua so users install nothing. Still BETA → in-process scripting is the most
expensive layer to stabilize.
- **neovim**: Lua won because the host API is rich, stable and uniform — half-exposed APIs are
worse than external scripts.
- **zellij**: WASM gives real sandboxing but locks architecture (single-threaded synchronous
plugins; WASIp2 migration "a significant undertaking"). Not worth it for v1.
- **kakoune**: `disabled_hooks` regex = cheap runtime kill-switch worth copying.
## Cargo feature flags = compile-time layer, not the primary switch
Features (`pdf`, `video`, … default-on) trim binary size and C deps, but feature unification
breaks packagers, combinations explode, and removing a default feature is a breaking change.
**Runtime `enabled = false` in modules config must remain the real disable mechanism**; every
`#[cfg]`-gated module should degrade to a TOML rule pointing at an external tool.
## Recommended phasing (adopted)
- **v0.1 — declarative core**: every file-type module = rule in a TOML registry mapping
mime/glob → previewer/opener, with per-module `enabled` switch and `prepend_/append_` merge.
Built-ins already dispatch through this table → replacing/reordering needs no recompile.
- **v0.2 — external process protocol**: freeze the script contract —
`argv = (path, width, height, x, y)`, env context, FIFO/socket write-back, `cleaner` hook.
Ship bash + Python reference plugins. This is where 90% of Linux-user extensibility lives.
- **v1.0+ — embedded Lua** (mlua, vendored): only for what scripts cannot do — stateful,
in-process previews/fetchers — registered through the same tables. Evaluate Luau if
sandboxing becomes a requirement. WASM explicitly rejected for v1.
## Outcome
Adopted as [[decisions/ADR-002-extension-architecture|ADR-002]] and [[decisions/ADR-003-file-type-modules|ADR-003]].
## Sources
- yazi docs (plugins overview, configuration), yazi Cargo.tomls (mlua 0.12 lua55 vendored)
- nnn plugins README; lf doc.md; rifle(1); RFC 1524 (mailcap)
- zellij plugin docs + discussion #4139; wezterm config docs; neovim lua.txt; kakoune FAQ
- mlua/rhai/gluon repos; deno blog; Lua sandbox-bypass CVE coverage (2025)
- Cargo book: features.md
docs/research/ranger-deep-dive.md
+77
View File
@@ -0,0 +1,77 @@
---
type: research
status: analyzed
confidence: high
created: 2026-10-17
question: What exactly makes ranger great, what makes it slow and fragile, and what must a Rust successor keep or fix?
tags: [research, ranger, file-manager]
---
# Ranger deep dive
## Research question
Ranger is the most feature-complete terminal file manager but famously slow in specific
scenarios. We need a precise keep/fix list before writing a single line of viexp.
## Findings
### Project state (Oct 2026)
Python + curses, Miller-columns UI. 17.4k stars, ~900 open issues, last tag v1.9.4 (Nov 2024),
commits still trickling. Maintainers openly state they lack manpower (issue #1717).
### Architecture
- Config = 4 files with **different merge semantics**: `rc.conf` merges, `rifle.conf` replaces,
`scope.sh` replaces (~700-line bash), `commands.py` merges (Python subclassing).
- Preview pipeline = external `scope.sh`; communicates via exit codes:
`0` = stdout preview, `1` = none, `2` = fixed-height text, `5` = image written to cache dir.
- Plugins = Python monkeypatching with full interpreter access ("Plugins have full access to
your computer!"). Ecosystem: ranger-devicons, ranger-zoxide, ranger-fzf-filter, etc.
- Image protocols: w3m, ueberzug, urxvt, kitty, sixel, iterm2, terminology.
### Why it is slow (root causes)
| Cause | Evidence |
|-------|----------|
| Synchronous UI loop — directory listing + `stat` of every file run inline | issue #1173: 100k files = multi-second uninterruptible hang (open since 2018) |
| No async I/O anywhere; threading refactor stalled | issue #1025 literally labeled `wants-async` |
| `statvfs` on **every status-bar refresh** → 4–10 s j/k lag on sshfs/NFS/CryFS/s3fs | issue #1025; vifm stays fast in identical setups |
| Python interpreter startup ~1 s cold | issue #1717 |
| Every preview spawns scope.sh + a chain of external binaries | architecture |
### Pain points (user-verified)
1. ueberzug unmaintained and X11-only; ueberzugpp fixes exist but are bolted on
(ranger #3188, #3158, #2995; ueberzugpp #122 fractional-scaling bug on Sway/Hyprland).
2. Network mounts unusable without manual `setlocal` hacks or `F` (freeze_files).
3. `:flat` broken past ~30k files (#379).
4. No built-in trash — ArchWiki documents `map DD shell gio trash` as the workaround.
5. Config fragmentation: four files, four formats, four merge behaviors.
### What users love (must keep)
Vim keys, Miller columns, `:bulkrename` via `$EDITOR` (the killer feature), rifle-style
ordered opener menu (`r`), preview breadth (20+ types), tabs/bookmarks/marks, `:filter`,
`--choosefile/--choosedir` + cd-on-exit, per-path options (`setlocal`).
## Keep / Fix table
| Keep | Fix |
|------|-----|
| Miller 3-pane layout, vim key vocabulary | Native binary startup (no interpreter) |
| `:bulkrename` via `$EDITOR` | Async dir loading / stat / preview — UI never blocks |
| Rifle-style opener rules + chooser menu | No `statvfs` per frame; poll with timeout+backoff |
| Exit-code preview contract (scope.sh idea) | First-class trash (`gio trash`-compatible) + undo |
| Preview breadth with caching | Native kitty/sixel/ueberzug++ graphics, Wayland-safe |
| `setlocal` per-path options | One config format (TOML) with documented merge semantics |
| `--choosefile/--choosedir`, cd-on-exit | Virtual scrolling for 100k+ entry dirs |
| `F` freeze escape hatch | Automatic freeze on network/fuse mounts |
## Outcome
Direct input to [[decisions/ADR-002-extension-architecture|ADR-002]] (extension model),
ADR-004 (preview protocols) and the roadmap's async requirements (Sprint 8).
## Sources
- https://github.com/ranger/ranger — README, wiki, examples/plugins
- https://wiki.archlinux.org/title/Ranger
- Issues: #1717 (startup), #1173 (100k files), #1025 (network lag), #379 (`:flat`),
#3188 / #3158 / #2995 (ueberzug family)
- https://github.com/seebye/ueberzug (unmaintained notice), https://github.com/jstkdng/ueberzugpp
- Community: r/commandline "Is yazi overhyped?", r/linux FM threads, x-cmd six-FM comparison,
https://mqdir.com/blog/file-management/nnn-vs-yazi
docs/research/rust-preview-stack.md
+75
View File
@@ -0,0 +1,75 @@
---
type: research
status: analyzed
confidence: high
created: 2026-10-17
question: Which Rust crates should render previews (images, PDF, DJVU, EPUB, markdown, media) in a terminal, with what licenses and fallbacks?
tags: [research, rust, crates, preview, ratatui]
---
# Rust preview stack (October 2026)
Versions verified against crates.io metadata on 2026-10-17.
## Recommended stack
| Concern | Crate | Why | Notes / license |
|---|---|---|---|
| TUI | **ratatui 0.30** + **crossterm 0.29** | De-facto standard; 0.30 split into `ratatui-core`/`-widgets` facade; yazi & felix use it | MIT; MSRV 1.88 |
| Images in TUI | **ratatui-image 11.1** | kitty, sixel, iTerm2, halfblocks + `Picker` protocol/font detection; lives in the ratatui org | MIT |
| Image decode | **image 0.25** | yazi's choice; jpeg/png/gif/webp/bmp/tiff (+avif feature) | MIT/Apache |
| Resize | **fast_image_resize 6.1** (SIMD) or `image::thumbnail` (triangle filter, yazi-style) | hot loop | MIT/Apache |
| Sixel encode | **quantette 0.6** (+ moxcms, palette) | yazi's actual sixel pipeline | MIT |
| PDF | **pdfium-render 0.9** + pdfium binary | only mature page→image in-process option | pdfium is **Apache-2.0** (b redistribute, ~5 MB); spawn `pdftoppm` as fallback |
| DJVU | **djvu-rs 0.43** (pure Rust, render→image) | first usable pure-Rust DjVu decoder | **6 months old** — keep `ddjvu` fallback |
| EPUB | hand-rolled: **zip** + quick-xml/html-to-text | the `epub` crate is **GPL-3.0 — viral** if statically linked | avoid the trap |
| Markdown | **comrak 0.56** → styled ratatui lines, or **termimad** (ready-made renderer) | termimad = fastest integration (broot uses it) | MIT |
| Syntax highlight | **syntect 5.3** | yazi's choice; regex Sublime syntaxes, no C deps | tree-sitter = heavier, per-language grammars |
| Media metadata | **spawn `ffprobe -print_format json`** | no linking pain; what yazi does | ffmpeg-next ties you to system FFmpeg — avoid |
| Video thumbnail | **spawn `ffmpeg -ss … -vframes 1 -vf scale…`** (+ `attached_pic` cover-art fallback) | yazi's proven recipe | — |
| Audio (optional in-process) | **symphonia 0.6** | pure-Rust metadata/album art | MPL-2.0 |
| MIME | **infer 0.22** (magic bytes) + **xdg-mime 0.4** (extension DB) | fast sniffing + correct `text/*` | `new_mime_guess` unmaintained — avoid |
| FS events | **notify 8.2** | yazi's watcher | CC0 |
| Fuzzy | **nucleo 0.5** | Helix's matcher, library-first | MPL-2.0 |
| Config | **serde + toml 1.1** | yazi's set | solid |
| Errors/logs | **anyhow + thiserror 2.x + tracing** | boring and correct | — |
| Async | **tokio 1.x** with results marshalled to a single-threaded UI actor | yazi's exact architecture; ecosystem is tokio-first | start threads-only, adopt tokio in Sprint 8 |
## Image protocol fallback chain (first match wins)
1. **kitty graphics** (unicode-placeholder mode) — kitty ≥0.28, Ghostty ← *our primary target*
2. **iTerm2 inline** — WezTerm, iTerm2, Warp, VSCode terminal
3. **sixel** — foot, Konsole 22.04+, Windows Terminal ≥1.22, WezTerm, rio
4. **ueberzug++ subprocess** — X11 + Wayland (wlroots compositors: sway/hyprland/niri/wayfire);
the only option for alacritty; not GNOME/KDE compositors
5. **chafa / halfblocks** — works everywhere incl. SSH and tmux
6. text: filename + size + mime
Terminal facts (Oct 2026): kitty has **no sixel by design**; alacritty rejected both protocols;
tmux needs `allow-passthrough on` and never passes kitty-old. The daily terminal here is
**kitty without a multiplexer** → tier 1 covers the primary user; tiers 3–5 cover everyone else.
## Thumbnail caching (yazi-proven)
Decode once at preview size (max ~600×900, triangle filter) → persist rendered pages/frames to
a disk cache dir → keep decoded previews in an in-process LRU → preload neighbors in background.
Re-encoding for a protocol is cheap; re-decoding is not.
## Risk notes
- GPL traps: `epub` crate (static link), ueberzug++ (fine — spawned), pdftoppm (fine — spawned).
- pdfium must be shipped as a binary (~5 MB/platform) or static-linked via pdfium-render `static`.
- djvu-rs is young; config-selectable `ddjvu` fallback required.
- ratatui 0.30 subcrate split may cause ecosystem churn in 2026 — pin carefully.
## Outcome
Feeds ADR-001, ADR-004 and Sprints 5–8 crate choices.
## Sources
- https://github.com/sxyazi/yazi (adapter/Cargo.tomls, preset plugins pdf.lua/video.lua)
- https://yazi-rs.github.io/docs/image-preview, /blog/why-is-yazi-fast/
- crates.io version metadata (ratatui, ratatui-image, image, fast_image_resize, quantette,
pdfium-render, djvu-rs, epub, comrak, termimad, syntect, symphonia, notify, nucleo, infer, xdg-mime)
- https://github.com/ajrcarey/pdfium-render; pdfium LICENSE (Apache-2.0)
- https://www.arewesixelyet.com; https://sw.kovidgoyal.net/kitty/graphics-protocol
- https://github.com/jstkdng/ueberzugpp; https://github.com/matyushkin/djvu-rs