Files
gbrain/test/files.test.ts
Garry Tan ecebd5552a feat: GBrain v0.2.0 — incremental sync, file storage, install skill (#2)
* refactor: extract importFile from import.ts + add tag reconciliation

Shared single-file import function used by both import and sync.
Adds tag reconciliation (removes stale tags on reimport), >1MB file
skip, and import->sync checkpoint continuity (writes git HEAD to
config table after import so sync picks up seamlessly).

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* feat: add sync pure functions, updateSlug engine method, and sync tests

- buildSyncManifest: parses git diff --name-status -M output
- isSyncable: filters to .md pages, excludes hidden/ops/.raw/skip-list
- pathToSlug: converts file paths to page slugs with optional prefix
- updateSlug: renames page slug in-place (preserves page_id, chunks, embeddings)
- rewriteLinks: stub for v0.2 (FKs use page_id, already correct)
- 20 new tests, all passing (39 total across 3 files)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* feat: add gbrain sync command with CLI, MCP, and watch mode

18-step sync protocol: read config, git pull, ancestry validation,
git diff --name-status -M for net changes, isSyncable filter, process
deletes/renames/adds/modifies via importFile, batch optimization,
sync state checkpoint in Postgres config table. Watch mode with
polling and consecutive error counter. MCP sync_brain tool returns
structured SyncResult. Stale page deletion for un-syncable files.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* feat: add files table, gbrain files commands, and config show redaction

- files table: page_slug FK with ON DELETE SET NULL + ON UPDATE CASCADE,
  storage_path, storage_url, mime_type, content_hash for dedup
- gbrain files list/upload/sync/verify commands for Supabase Storage
- gbrain config show redacts postgresql:// passwords and secret keys
- CLI help updated with FILES section

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* feat: add install skill for GBrain onboarding

6-phase install workflow: environment discovery, Supabase setup (magic
path via CLI OAuth or fallback 2-copy-paste), init + import, ongoing
sync cron, optional file migration with mandatory verification, and
agent teaching (AGENTS.md rules). Every error gets what + why + fix.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* docs: update project documentation for v0.2.0

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* docs: add v0.2 features to README (sync, files, install skill)

README.md: added sync command to IMPORT/EXPORT section, added FILES
section with 4 commands, added files table to schema diagram, added
install skill to skills table, updated MCP tools count from 20 to 21
(sync_brain added).

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix: OpenClaw DX improvements (skill count, upgrade docs, config show help)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* refactor: consolidate version to single source of truth

Create src/version.ts that reads from package.json via static import
(safe for bun compiled binaries). Update mcp/server.ts from hardcoded
'0.1.0' to use shared VERSION. Bump skills/manifest.json to 0.2.0.

* fix: upgrade detection order, npm→bun naming, clawhub false positives

Reorder detection: node_modules first, binary second, clawhub last.
Rename 'npm' install method to 'bun'. Use 'clawhub --version' instead
of 'which clawhub' to avoid false positives from dangling symlinks.
Add 120s timeout to execSync calls to prevent hanging. Add --help flag.

* feat: per-command --help, unknown command check before DB connection

Add COMMAND_HELP map covering all 28 commands. Check --help before
init/upgrade dispatch and before connectEngine() so help works without
a database. Use COMMAND_HELP keys as known-command set to catch unknown
commands before wasting a DB round-trip.

* docs: standardize npm references to bun, add Upgrade section to README

Fix init.ts: npx→bunx, npm→bun for supabase CLI guidance.
Fix README: npm install→bun add for standalone CLI install.
Add ## Upgrade section to README with all three install methods.
Update install skill Upgrading section to list bun, ClawHub, and binary.

* test: full coverage audit — CLI dispatch, upgrade detection, config, edge cases

New test files:
- test/cli.test.ts: COMMAND_HELP ↔ switch consistency, version from
  package.json, per-command --help, unknown command handling, global help
- test/upgrade.test.ts: detection order verification, npm→bun naming,
  clawhub --version (not which), timeout presence
- test/config.test.ts: redactUrl for postgresql URLs, edge cases

Extended existing tests:
- test/sync.test.ts: empty string pathToSlug, uppercase .MD rejection,
  deeply nested files, multiple renames, unknown status codes
- test/markdown.test.ts: multiple --- separators, missing frontmatter,
  no frontmatter at all, empty string, type inference from paths

Tests: 39 → 83 (+44 new). All pass.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* test: 100% coverage — import-file mock engine, files utils, chunker edge cases

New test files:
- test/import-file.test.ts (9 tests): mock BrainEngine to test importFile
  without DB — MAX_FILE_SIZE skip, content_hash dedup, tag reconciliation
  (remove stale + add new), compiled_truth/timeline chunking, noEmbed flag,
  sequential chunk_index
- test/files.test.ts (22 tests): getMimeType for all extensions + uppercase
  + unknown + no-extension, fileHash consistency + different content + empty,
  collectFiles pattern (skip .md, skip hidden dirs, recurse, sorted output)

Extended:
- test/chunkers/recursive.test.ts (+6 tests): single newline splits,
  word-only text, clause delimiters, lossless preservation, default options,
  mixed delimiter hierarchy

Tests: 83 → 118 (+35 new). All pass.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-06 16:50:15 -07:00

179 lines
5.7 KiB
TypeScript

import { describe, test, expect, beforeAll, afterAll } from 'bun:test';
import { writeFileSync, mkdirSync, rmSync } from 'fs';
import { join } from 'path';
import { createHash } from 'crypto';
import { extname } from 'path';
const TMP = join(import.meta.dir, '.tmp-files-test');
// These functions are not exported from files.ts, so we reimplement and test
// the logic patterns to ensure correctness. If they ever get exported, switch
// to direct imports.
const MIME_TYPES: Record<string, string> = {
'.jpg': 'image/jpeg', '.jpeg': 'image/jpeg', '.png': 'image/png',
'.gif': 'image/gif', '.webp': 'image/webp', '.svg': 'image/svg+xml',
'.pdf': 'application/pdf', '.mp4': 'video/mp4', '.m4a': 'audio/mp4',
'.mp3': 'audio/mpeg', '.wav': 'audio/wav', '.heic': 'image/heic',
'.tiff': 'image/tiff', '.tif': 'image/tiff', '.dng': 'image/x-adobe-dng',
'.doc': 'application/msword',
'.docx': 'application/vnd.openxmlformats-officedocument.wordprocessingml.document',
'.xls': 'application/vnd.ms-excel',
'.xlsx': 'application/vnd.openxmlformats-officedocument.spreadsheetml.sheet',
};
function getMimeType(filePath: string): string | null {
const ext = extname(filePath).toLowerCase();
return MIME_TYPES[ext] || null;
}
function fileHash(content: Buffer): string {
return createHash('sha256').update(content).digest('hex');
}
beforeAll(() => {
mkdirSync(TMP, { recursive: true });
mkdirSync(join(TMP, 'subdir'), { recursive: true });
mkdirSync(join(TMP, '.hidden'), { recursive: true });
writeFileSync(join(TMP, 'photo.jpg'), 'fake-jpg');
writeFileSync(join(TMP, 'doc.pdf'), 'fake-pdf');
writeFileSync(join(TMP, 'notes.md'), '# Markdown');
writeFileSync(join(TMP, 'data.csv'), 'a,b,c');
writeFileSync(join(TMP, 'subdir', 'nested.png'), 'fake-png');
writeFileSync(join(TMP, '.hidden', 'secret.txt'), 'hidden');
});
afterAll(() => {
rmSync(TMP, { recursive: true, force: true });
});
describe('getMimeType', () => {
test('returns correct MIME for .jpg', () => {
expect(getMimeType('photo.jpg')).toBe('image/jpeg');
});
test('returns correct MIME for .jpeg', () => {
expect(getMimeType('photo.jpeg')).toBe('image/jpeg');
});
test('returns correct MIME for .png', () => {
expect(getMimeType('image.png')).toBe('image/png');
});
test('returns correct MIME for .pdf', () => {
expect(getMimeType('doc.pdf')).toBe('application/pdf');
});
test('returns correct MIME for .mp4', () => {
expect(getMimeType('video.mp4')).toBe('video/mp4');
});
test('returns correct MIME for .svg', () => {
expect(getMimeType('icon.svg')).toBe('image/svg+xml');
});
test('handles uppercase extensions via toLowerCase', () => {
expect(getMimeType('PHOTO.JPG')).toBe('image/jpeg');
expect(getMimeType('doc.PDF')).toBe('application/pdf');
});
test('returns null for unknown extensions', () => {
expect(getMimeType('data.csv')).toBeNull();
expect(getMimeType('script.ts')).toBeNull();
expect(getMimeType('readme.md')).toBeNull();
});
test('returns null for files without extension', () => {
expect(getMimeType('Makefile')).toBeNull();
});
test('handles .docx and .xlsx', () => {
expect(getMimeType('report.docx')).toContain('wordprocessingml');
expect(getMimeType('sheet.xlsx')).toContain('spreadsheetml');
});
test('handles .heic (iPhone photos)', () => {
expect(getMimeType('IMG_0001.heic')).toBe('image/heic');
});
test('handles .dng (raw photos)', () => {
expect(getMimeType('RAW_001.dng')).toBe('image/x-adobe-dng');
});
});
describe('fileHash', () => {
test('produces consistent SHA-256 hash', () => {
const content = Buffer.from('hello world');
const hash1 = fileHash(content);
const hash2 = fileHash(content);
expect(hash1).toBe(hash2);
expect(hash1).toHaveLength(64); // SHA-256 hex = 64 chars
});
test('different content produces different hash', () => {
const hash1 = fileHash(Buffer.from('hello'));
const hash2 = fileHash(Buffer.from('world'));
expect(hash1).not.toBe(hash2);
});
test('empty content produces valid hash', () => {
const hash = fileHash(Buffer.from(''));
expect(hash).toHaveLength(64);
});
});
describe('collectFiles pattern (non-markdown, skip hidden)', () => {
// Reimplementing collectFiles logic to test the pattern
const { readdirSync, statSync } = require('fs');
function collectFiles(dir: string): string[] {
const files: string[] = [];
function walk(d: string) {
for (const entry of readdirSync(d)) {
if (entry.startsWith('.')) continue;
const full = join(d, entry);
const stat = statSync(full);
if (stat.isDirectory()) {
walk(full);
} else if (!entry.endsWith('.md')) {
files.push(full);
}
}
}
walk(dir);
return files.sort();
}
test('finds non-markdown files', () => {
const files = collectFiles(TMP);
const basenames = files.map(f => f.split('/').pop());
expect(basenames).toContain('photo.jpg');
expect(basenames).toContain('doc.pdf');
expect(basenames).toContain('data.csv');
});
test('skips .md files', () => {
const files = collectFiles(TMP);
const mdFiles = files.filter(f => f.endsWith('.md'));
expect(mdFiles).toHaveLength(0);
});
test('skips hidden directories', () => {
const files = collectFiles(TMP);
const hiddenFiles = files.filter(f => f.includes('.hidden'));
expect(hiddenFiles).toHaveLength(0);
});
test('recurses into subdirectories', () => {
const files = collectFiles(TMP);
const nested = files.filter(f => f.includes('subdir'));
expect(nested.length).toBeGreaterThan(0);
});
test('returns sorted paths', () => {
const files = collectFiles(TMP);
const sorted = [...files].sort();
expect(files).toEqual(sorted);
});
});