Mitchins/retnet-literary-explicitness-classifier

RetNet Explicitness Classifier

A high-performance RetNet model for classifying text content by explicitness level, designed for large-scale content moderation and filtering applications.

🚀 Model Overview

Attribute	Value
Model Type	RetNet (Linear Attention)
Parameters	45,029,943
Task	7-class text classification
Performance	74.4% accuracy, 63.9% macro F1
Speed	1,574 paragraphs/second
Training Time	4.9 hours

📊 Performance Comparison

Model	Parameters	Accuracy	Macro F1	Speed	Architecture
DeBERTa-v3-small	~44M	82.3%*	75.8%*	~500 p/s	O(n²) attention
RetNet	45M	74.4%	63.9%	1,574 p/s	O(n) linear

*Results on different data splits. RetNet offers 3x speed advantage with competitive performance.

🏷️ Classification Labels

The model classifies text into 7 categories of explicitness:

1. NON-EXPLICIT - Safe, general audience content
2. SUGGESTIVE - Mild romantic or suggestive themes
3. SEXUAL-REFERENCE - References to sexual topics without explicit detail
4. EXPLICIT-SEXUAL - Graphic sexual content
5. EXPLICIT-OFFENSIVE - Strong profanity and offensive language
6. EXPLICIT-VIOLENT - Graphic violence and disturbing content
7. EXPLICIT-DISCLAIMER - Content warnings and disclaimers

🚀 Quick Start

Installation

# Install dependencies
pip install torch transformers safetensors

Basic Usage

from test_model import RetNetExplicitnessClassifier

# Initialize classifier
classifier = RetNetExplicitnessClassifier()

# Classify single text
result = classifier.classify("Your text here...")
print(f"Category: {result['predicted_class']}")
print(f"Confidence: {result['confidence']:.3f}")

# Batch classification for better performance
texts = ["Text 1", "Text 2", "Text 3"]
results = classifier.classify_batch(texts)

Test the Model

python test_model.py

📁 Model Files

retnet-explicitness-classifier/
├── README.md                    # This file
├── config.json                  # Model configuration  
├── model.py                     # RetNet architecture code
├── model.safetensors            # Trained model weights (SafeTensors format)
├── model_metadata.json          # Model metadata
├── retnet_training_results.json # Training metrics
└── test_model.py               # Test script and API

🏗️ Architecture Details

RetNet Advantages

• Linear O(n) attention vs traditional O(n²) transformers
• 3x faster inference - ideal for high-throughput applications
• Memory efficient for long sequences
• Parallel training with recurrent inference capabilities

Model Configuration

{
  "model_dim": 512,
  "num_layers": 6,
  "num_heads": 8,
  "max_length": 512,
  "vocab_size": 50257
}

📈 Training Details

Dataset

• Total samples: 119,023 paragraphs
• Training: 101,771 samples (85.5%)
• Validation: 11,304 samples (9.5%)
• Holdout: 5,948 samples (5.0%)
• Data source: Literary content with GPT-4 annotations

Training Configuration

• Epochs: 5
• Batch size: 32
• Learning rate: 1e-4
• Loss function: Focal Loss (γ=2.0) for class imbalance
• Optimizer: AdamW with cosine scheduling
• Hardware: Apple Silicon (MPS)
• Duration: 4.9 hours

Performance Metrics (Holdout Set)

Class	Precision	Recall	F1-Score	Support
EXPLICIT-DISCLAIMER	1.00	0.93	0.96	57
EXPLICIT-OFFENSIVE	0.70	0.76	0.73	1,208
EXPLICIT-SEXUAL	0.85	0.91	0.88	1,540
EXPLICIT-VIOLENT	0.58	0.25	0.35	73
NON-EXPLICIT	0.75	0.83	0.79	2,074
SEXUAL-REFERENCE	0.61	0.37	0.46	598
SUGGESTIVE	0.38	0.26	0.30	398
Macro Average	0.70	0.61	0.64	5,948

⚡ Performance Benchmarks

Speed Comparison

• RetNet: 1,574 paragraphs/second
• Book processing: ~8-15 books/second (assuming 100-200 paragraphs/book)
• Million book processing: ~19-31 hours
• Memory usage: Optimized for batch processing

Use Cases

✅ Ideal for:

• Large-scale content filtering (millions of documents)
• Real-time content moderation
• High-throughput publishing pipelines
• Content recommendation systems

⚠️ Consider alternatives for:

• Maximum accuracy requirements (use DeBERTa)
• Small-scale applications where speed isn't critical
• Academic research requiring state-of-the-art performance

🔧 Technical Implementation

RetNet Architecture

class ProductionRetNet(nn.Module):
    def __init__(self, vocab_size=50257, dim=512, num_layers=6, 
                 num_heads=8, num_classes=7, max_length=512):
        # FastRetentionMechanism with linear attention
        # Rotary positional encoding
        # Pre-layer normalization
        # Classification head with dropout

Key Features

• Rotary positional encoding for better position awareness
• Fast retention mechanism replacing traditional attention
• Layer normalization for stable training
• Focal loss to handle class imbalance
• Gradient clipping for training stability

🚀 Production Deployment

Docker Example

FROM python:3.9-slim

COPY retnet-explicitness-classifier/ /app/
WORKDIR /app

RUN pip install torch transformers

EXPOSE 8000
CMD ["python", "-m", "uvicorn", "api:app", "--host", "0.0.0.0"]

API Endpoint Example

from fastapi import FastAPI
from test_model import RetNetExplicitnessClassifier

app = FastAPI()
classifier = RetNetExplicitnessClassifier()

@app.post("/classify")
async def classify_text(text: str):
    return classifier.classify(text)

📚 Citation

If you use this model in your research, please cite:

@misc{retnet_explicitness_2024,
  title={RetNet for Explicitness Classification: Linear Attention for High-Throughput Content Moderation},
  author={Claude Code Assistant},
  year={2024},
  note={Production-scale RetNet implementation for 7-class explicitness classification}
}

📄 License

This model is released for research and educational purposes. Please ensure compliance with content moderation guidelines and applicable laws when using for production applications.

🔗 Related Work

• RetNet: Retentive Network: A Successor to Transformer for Large Language Models
• DeBERTa: Decoding-enhanced BERT with Disentangled Attention
• Focal Loss for Dense Object Detection

---

Model Version: 1.0
Last Updated: August 2024
Framework: PyTorch 2.0+
Minimum Python: 3.8+