nand2tetris-zed/DEVELOPMENT.md

# Nand2Tetris Zed Extension - Development Guide

This guide provides detailed information for developers working on or contributing to the Nand2Tetris Zed extension.

## Overview

The extension provides comprehensive language support for all nand2tetris course languages through custom Tree-sitter grammars and Zed language configurations.

## Architecture

### Extension Structure
```
nand2tetris-zed/
├── extension.toml              # Main extension configuration
├── languages/                  # Zed language configurations
│   ├── hdl/                   # Hardware Description Language
│   ├── jack/                  # Jack programming language
│   ├── hack-assembly/         # Hack Assembly language
│   ├── vm/                    # Virtual Machine language
│   ├── test-script/           # Test Script language
│   └── compare-output/        # Compare/Output file format
├── grammars/                   # Tree-sitter grammar sources
│   ├── hdl/                   # External grammar (quantonganh)
│   ├── jack/                  # External grammar (nverno)
│   ├── hack-assembly/         # Custom grammar
│   ├── vm/                    # Custom grammar
│   ├── test-script/           # Custom grammar
│   └── compare-output/        # Custom grammar
└── examples/                   # Test files for validation
```

### Language Support Levels

1. **HDL & Jack**: Use external, mature Tree-sitter grammars
2. **Assembly, VM, Test Script, Compare/Output**: Custom grammars built specifically for this extension

## Grammar Development

### Prerequisites

- **Rust** (installed via rustup)
- **Node.js** and npm
- **tree-sitter-cli**: `npm install -g tree-sitter-cli`

### Grammar Structure

Each custom grammar follows this structure:

```
grammars/[language]/
├── grammar.js                 # Grammar definition
├── tree-sitter.json          # Tree-sitter configuration
├── package.json               # npm package configuration
├── Cargo.toml                 # Rust package configuration
├── binding.gyp                # Node.js binding configuration
├── src/                       # Generated parser code
│   ├── parser.c              # Generated C parser
│   ├── grammar.json          # Generated grammar metadata
│   └── node-types.json       # Generated node type definitions
├── bindings/                  # Language bindings
│   └── rust/                 # Rust bindings
│       ├── lib.rs            # Library interface
│       └── build.rs          # Build script
└── queries/                  # Tree-sitter queries
    └── highlights.scm        # Syntax highlighting queries
```

### Creating a New Grammar

1. **Create directory structure**:
   ```bash
   mkdir -p grammars/my-language/{bindings/rust,queries}
   cd grammars/my-language
   ```

2. **Create `grammar.js`**:
   ```javascript
   module.exports = grammar({
     name: 'my_language',
     
     rules: {
       source_file: $ => repeat($._item),
       
       _item: $ => choice(
         $.comment,
         // Add your language constructs here
       ),
       
       comment: $ => token(seq('//', /.*/)),
     },
     
     extras: $ => [
       /\s/,
       $.comment
     ]
   });
   ```

3. **Create configuration files**:
   - `tree-sitter.json` (see existing examples)
   - `package.json` (Node.js package)
   - `Cargo.toml` (Rust package)
   - `binding.gyp` (Node.js bindings)

4. **Generate parser**:
   ```bash
   tree-sitter generate
   ```

5. **Create Rust bindings**:
   - `bindings/rust/lib.rs`
   - `bindings/rust/build.rs`

6. **Create highlighting queries**:
   - `queries/highlights.scm`

7. **Test the grammar**:
   ```bash
   tree-sitter test
   tree-sitter parse test-file.ext
   ```

### Grammar Rules Best Practices

#### Token Naming
- Use snake_case for rule names
- Prefix internal rules with `_`
- Use semantic names that describe the construct

#### Rule Structure
```javascript
// Good: Clear semantic meaning
instruction: $ => choice(
  $.a_instruction,
  $.c_instruction
),

// Bad: Generic naming
thing: $ => choice(
  $.type1,
  $.type2
),
```

#### Comments and Whitespace
Always handle comments and whitespace properly:
```javascript
extras: $ => [
  /\s/,        // Whitespace
  $.comment    // Comments
],
```

#### String Tokens
Use `token()` for multi-character operators:
```javascript
// Good
if_goto: $ => token('if-goto'),

// Bad (can cause parsing issues)
if_goto: $ => 'if-goto',
```

## Language Configuration

Each language needs a `config.toml` file in `languages/[language]/`:

```toml
name = "Language Name"
grammar = "grammar_name"
scope = "source.language_name"
path_suffixes = ["ext"]
line_comments = ["// "]
block_comments = [["/*", "*/"]]  # Optional
tab_size = 4
hard_tabs = false
```

### Highlighting Queries

Create `.scm` files that map grammar nodes to semantic tokens:

```scheme
; Comments
(comment) @comment

; Keywords
"function" @keyword.function
"return" @keyword.control

; Identifiers
(identifier) @variable
(function_name) @function

; Literals
(number) @constant.numeric
(string) @string
```

### Query Development Tips

1. **Check node types**: Use `tree-sitter parse --debug` to see actual node structure
2. **Test queries**: Use `tree-sitter highlight` to test highlighting
3. **Use semantic tokens**: Follow TextMate/LSP token conventions
4. **Prioritize specificity**: More specific queries override general ones

## Testing

### Grammar Testing
```bash
cd grammars/[language]
tree-sitter test                    # Run test suite
tree-sitter parse example.ext       # Parse specific file
tree-sitter highlight example.ext   # Test highlighting
```

### Integration Testing
1. Install extension as dev extension in Zed
2. Open test files and verify:
   - Syntax highlighting works
   - Bracket matching functions
   - Code outline appears
   - Indentation behaves correctly

### Example Files
Create comprehensive example files in `examples/` that cover:
- All language constructs
- Edge cases
- Common patterns
- Error conditions

## Common Issues and Solutions

### Grammar Generation Errors

**Empty string rules**:
```
Error: The rule contains an empty string
```
Solution: Remove empty alternatives or use `optional()`

**Invalid node types in queries**:
```
Query error: Invalid node type 'foo'
```
Solution: Check `src/node-types.json` for actual node names

### Highlighting Issues

**Tokens not highlighting**:
1. Verify node exists in `node-types.json`
2. Check query syntax
3. Ensure grammar generates expected nodes

**Conflicting highlights**:
- More specific queries take precedence
- Use `#match?` predicates for conditional highlighting

### Zed Integration Issues

**Extension not loading**:
1. Check `extension.toml` syntax
2. Verify all referenced grammars exist
3. Check Zed logs for specific errors

**Grammar compilation fails**:
1. Ensure Rust is installed via rustup
2. Check that all required files are present
3. Verify tree-sitter grammar generates successfully

## Contributing Guidelines

### Code Style
- Follow existing patterns in grammar definitions
- Use clear, semantic naming
- Add comprehensive comments
- Test thoroughly before submitting

### Documentation
- Update README.md for user-facing changes
- Update this guide for development changes
- Include example files for new languages
- Document any breaking changes

### Pull Request Process
1. Create feature branch
2. Implement changes with tests
3. Update documentation
4. Test in Zed environment
5. Submit PR with clear description

### Commit Messages
Use conventional commits:
```
feat: add support for new language construct
fix: resolve highlighting issue with comments
docs: update installation instructions
test: add comprehensive test cases
```

## Release Process

1. **Version Bump**: Update `extension.toml` version
2. **Test**: Verify all grammars compile and work in Zed
3. **Documentation**: Update README.md and CHANGELOG.md
4. **Tag**: Create git tag for version
5. **Publish**: Submit to Zed extension registry (when available)

## Performance Considerations

### Grammar Optimization
- Avoid excessive backtracking in rules
- Use `token()` for multi-character sequences
- Minimize conflicts between rules
- Profile with `tree-sitter parse --time`

### Query Optimization
- Use specific node patterns over generic ones
- Avoid complex predicates when possible
- Test query performance on large files

## Debugging

### Grammar Debugging
```bash
tree-sitter generate --debug
tree-sitter parse --debug-graph file.ext
tree-sitter test --debug
```

### Query Debugging
```bash
tree-sitter query queries/highlights.scm file.ext
tree-sitter highlight file.ext
```

### Zed Debugging
- Check `zed: open log` for extension errors
- Use `--foreground` flag for verbose logging
- Test with minimal example files

## Resources

- [Tree-sitter Documentation](https://tree-sitter.github.io/tree-sitter/)
- [Zed Extension Guide](https://zed.dev/docs/extensions)
- [Nand2Tetris Course Materials](https://www.nand2tetris.org)
- [TextMate Grammar Guide](https://macromates.com/manual/en/language_grammars)

## Support

For development questions or issues:
1. Check existing GitHub issues
2. Review this development guide
3. Test with minimal reproducible examples
4. Provide detailed error messages and logs

---

This extension represents a complete implementation of nand2tetris language support, demonstrating how to create comprehensive Tree-sitter grammars and integrate them into modern editors.