279 lines
8.7 KiB
Markdown
279 lines
8.7 KiB
Markdown
# bpe_framework
|
|
## Byte Pair Encoding Framework
|
|
Large Language Model for Agentic AI
|
|
|
|
Fully internationalized framework for Agentic AI research
|
|
|
|
Requires:
|
|
1. nlohman/json (https://github.com/nlohmann/json
|
|
2. Internationalzation library for Unicode by Frederick Roubert (https://github.com/unicode-org/icu)
|
|
3. OpenNMT Tokenizer by Thuc Pham (https://github.com/OpenNMT/Tokenize)
|
|
4. Eigen header files (https://github.com/PX4/eigen)
|
|
|
|
Build: cmake -DCMAKE_POLICY_VERSION_MINIMUM=3.5 -DEIGEN_LOC=<eigen3 folder> ..
|
|
|
|
#### The test_bpe application is a comprehensive test program that validates the functionality of the BPE tokenizer implementation in the LM Framework. Here's how it works:
|
|
1. Initialization:
|
|
Creates an instance of BPETokenizer
|
|
Defines a training corpus with sample English text
|
|
|
|
2. Training Process:
|
|
Calls tokenizer.train(corpus, 500) to train the tokenizer
|
|
The training process:
|
|
Initializes with byte-level vocabulary (0-255)
|
|
Analyzes word frequencies in the corpus
|
|
Iteratively merges the most frequent character pairs
|
|
Builds a vocabulary of 500 tokens (as specified)
|
|
|
|
3. Encoding Test:
|
|
Encodes the test string "the quick brown fox"
|
|
The encoding process:
|
|
Splits text into words
|
|
Converts each character to its initial token ID
|
|
Applies learned BPE merges to combine tokens
|
|
Returns a sequence of integer token IDs
|
|
|
|
4. Decoding Test:
|
|
Decodes the token IDs back to text
|
|
The decoding process:
|
|
Converts each token ID back to its string representation
|
|
Concatenates the strings to reconstruct the original text
|
|
|
|
5. Serialization Test
|
|
Saves the trained tokenizer to "bpe_model.txt"
|
|
The serialization process:
|
|
Writes vocabulary size and token-ID mappings
|
|
Records all learned merge rules
|
|
|
|
6. Deserialization Test
|
|
Loads the tokenizer from "bpe_model.txt"
|
|
Verifies the loaded tokenizer has the same vocabulary size
|
|
Confirms the tokenizer can perform encoding/decoding
|
|
|
|
Expected Output
|
|
text
|
|
|
|
Training tokenizer...
|
|
Vocabulary size: 500
|
|
Original: the quick brown fox
|
|
Tokens: [list of token IDs]
|
|
Decoded: the quick brown fox
|
|
Successfully loaded tokenizer
|
|
Loaded vocabulary size: 500
|
|
|
|
Key Validations
|
|
Training Completes without errors
|
|
Encoding/Decoding Round-Trip preserves the original text
|
|
Serialization/Deserialization maintains tokenizer state
|
|
Vocabulary Size matches the specified target (500)
|
|
Token IDs are consistent between sessions
|
|
|
|
## test_unicode.cpp
|
|
|
|
### Lower-level Unicode-specific tests:
|
|
|
|
Unicode normalization functions
|
|
|
|
Character boundary detection
|
|
|
|
Grapheme cluster handling
|
|
|
|
Encoding conversion utilities
|
|
|
|
Validation of Unicode compliance
|
|
|
|
|
|
## BPE Tokenizer Performance Test Suite
|
|
|
|
### Overview
|
|
|
|
The performance test application is a comprehensive benchmarking tool designed to evaluate the efficiency and scalability of the Byte Pair Encoding (BPE) tokenizer implementation. The test suite measures critical performance metrics including training time, memory usage, encoding/decoding speed, and serialization performance across various configurations.
|
|
|
|
## Key Features
|
|
|
|
### 1. Corpus Generation
|
|
- Automatically generates realistic test corpora using common AI/ML terminology
|
|
- Configurable sentence count and word range parameters
|
|
- Creates diverse text samples that mimic real-world language patterns
|
|
|
|
### 2. Multi-Dimensional Testing
|
|
- Tests multiple corpus sizes (100, 1000, 5000 sentences)
|
|
- Evaluates different vocabulary sizes (500, 1000, 2000 tokens)
|
|
- Measures performance across various workload scenarios
|
|
|
|
### 3. Comprehensive Performance Metrics
|
|
- **Training Time**: Measures how long it takes to build the BPE vocabulary from a corpus
|
|
- **Memory Usage**: Tracks peak memory consumption during training (Linux-specific)
|
|
- **Encoding Speed**: Calculates processing time per token during text encoding
|
|
- **Round-Trip Verification**: Ensures encoding/decoding preserves original content
|
|
- **Serialization Performance**: Measures model save/load operations
|
|
|
|
### 4. Validation Checks
|
|
- Verifies encoding/decoding consistency
|
|
- Detects potential data corruption issues
|
|
- Validates vocabulary construction
|
|
|
|
## Usage Scenarios
|
|
|
|
This performance test is ideal for:
|
|
- Benchmarking different BPE implementations
|
|
- Evaluating hardware suitability for language processing tasks
|
|
- Identifying performance bottlenecks in tokenization pipelines
|
|
- Testing scalability of tokenizer implementations
|
|
- Comparing optimization techniques
|
|
|
|
## Technical Implementation
|
|
|
|
The test suite utilizes:
|
|
- High-resolution timing with `<chrono>` for precise measurements
|
|
- Linux-specific memory tracking via `/proc/self/status`
|
|
- Randomized corpus generation with configurable parameters
|
|
- Exception handling for robust testing
|
|
- Automatic cleanup of temporary files
|
|
|
|
## Output Metrics
|
|
|
|
The application provides detailed performance reports including:
|
|
- Training duration in milliseconds
|
|
- Peak memory usage in megabytes
|
|
- Encoding speed in microseconds per token
|
|
- Serialization/deserialization times
|
|
- Vocabulary size validation
|
|
- Round-trip integrity verification
|
|
|
|
This test framework serves as an essential tool for developers and researchers working with BPE tokenizers, providing quantitative data to guide optimization efforts and implementation choices.
|
|
|
|
## Technical Summary: BPE Framework
|
|
### Overview
|
|
|
|
The BPE Framework is a C++-based neural network framework designed for building and training language models with Byte Pair Encoding (BPE) tokenization. It implements a complete deep learning stack with automatic differentiation, optimization, and model serialization capabilities.
|
|
Core Components
|
|
#### 1. Tensor Operations with Autograd
|
|
|
|
Header-only Tensor class with Eigen backend for efficient linear algebra
|
|
|
|
Automatic differentiation with backward propagation
|
|
|
|
Comprehensive operator support: element-wise operations, matrix multiplication, reductions
|
|
|
|
Activation functions: ReLU, GELU, Softmax, Sigmoid with gradient support
|
|
|
|
Memory-efficient implementation with shape-aware operations
|
|
|
|
#### 2. BPE Tokenizer
|
|
|
|
PIMPL pattern implementation for API stability
|
|
|
|
Efficient vocabulary management with merge operations
|
|
|
|
Encoding/decoding support for text processing
|
|
|
|
Non-copyable design (uses unique_ptr) for proper resource management
|
|
|
|
#### 3. Neural Network Architecture
|
|
|
|
Transformer-based language model implementation
|
|
|
|
Configurable dimensions: embedding size, hidden layers, attention heads
|
|
|
|
Parameter management with named parameters for serialization
|
|
|
|
Training/inference modes support
|
|
|
|
#### 4. Training Infrastructure
|
|
|
|
Adam optimizer with configurable hyperparameters
|
|
|
|
Gradient accumulation and moment estimation
|
|
|
|
Batch processing with sequence padding
|
|
|
|
Loss computation (cross-entropy) with masking support
|
|
|
|
#### 5. Model Serialization
|
|
|
|
Binary format with versioning and magic number validation
|
|
|
|
Parameter-by-name storage and retrieval
|
|
|
|
Shape preservation and data integrity checks
|
|
|
|
Error handling for file operations and format validation
|
|
|
|
### Key Technical Features
|
|
#### Memory Management
|
|
|
|
Eigen integration for optimized matrix operations
|
|
|
|
Shape-aware memory allocation preventing unnecessary copies
|
|
|
|
RAII principles for resource management
|
|
|
|
#### Performance Considerations
|
|
|
|
Header-only design for Tensor class enabling compiler optimizations
|
|
|
|
Batch processing for efficient training
|
|
|
|
In-place operations where possible to reduce memory overhead
|
|
|
|
#### Extensibility
|
|
|
|
Modular architecture allowing component replacement
|
|
|
|
Clear interfaces between tokenizer, model, and training components
|
|
|
|
Parameter naming convention supporting complex architectures
|
|
|
|
#### Architecture Patterns
|
|
|
|
PIMPL Idiom: Used in tokenizer for stable ABI
|
|
|
|
RAII: Comprehensive resource management throughout
|
|
|
|
Builder Pattern: Model configuration through constructor parameters
|
|
|
|
Strategy Pattern: Optimizer implementation allowing algorithm changes
|
|
|
|
#### Current Capabilities
|
|
|
|
* Automatic differentiation with reverse-mode autograd
|
|
|
|
* BPE tokenization with vocabulary learning
|
|
|
|
* Transformer language model training
|
|
|
|
* Adam optimization with moment estimation
|
|
|
|
* Model serialization/deserialization
|
|
|
|
* Configurable network architectures
|
|
|
|
* Batch processing with padding
|
|
|
|
### Technical Stack
|
|
|
|
C++17 with standard library components
|
|
|
|
Eigen for linear algebra operations
|
|
|
|
CMake for build system management
|
|
|
|
Header-only design for core components
|
|
|
|
#### Usage Example
|
|
|
|
// Initialize components
|
|
BPETokenizer tokenizer(corpus);
|
|
LanguageModel model(tokenizer.vocab_size(), 512, 2048, 8);
|
|
LanguageModelTrainer trainer(tokenizer, 512, 2048, 8);
|
|
|
|
// Train model
|
|
trainer.train(training_corpus, 10, 32, 256);
|
|
|
|
trainer.save_model("language_model.bin");
|
|
|
|
|
|
Based on the research of Timothy O'Neil, Frederick Warren, et. al.
|
|
|