Spaces:
Sleeping
Sleeping
prabhjkaur
commited on
Commit
Β·
d8eceeb
1
Parent(s):
b6cf318
Added streamlit files
Browse files- README.md +354 -14
- Recording_system.py +1169 -0
- analysis_system.py +868 -0
- main_app.py +576 -0
- packages.txt +2 -0
- requirements.txt +43 -3
- runtime.txt +1 -0
- scoring_dashboard.py +745 -0
- yolov8n-cls.pt +3 -0
- yolov8n.pt +3 -0
README.md
CHANGED
|
@@ -1,20 +1,360 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
---
|
| 2 |
-
|
| 3 |
-
|
| 4 |
-
|
| 5 |
-
|
| 6 |
-
|
| 7 |
-
|
| 8 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 9 |
- streamlit
|
| 10 |
-
|
| 11 |
-
|
| 12 |
-
license: apache-2.0
|
| 13 |
---
|
| 14 |
|
| 15 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 16 |
|
| 17 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 18 |
|
| 19 |
-
|
| 20 |
-
forums](https://discuss.streamlit.io).
|
|
|
|
| 1 |
+
# AI Interview System - Modular Architecture
|
| 2 |
+
|
| 3 |
+
## π Project Structure
|
| 4 |
+
|
| 5 |
+
```
|
| 6 |
+
ai_interview_system/
|
| 7 |
+
β
|
| 8 |
+
βββ main_app.py # Main integration file (run this)
|
| 9 |
+
βββ recording_system.py # Module 1: Recording & Violation Detection
|
| 10 |
+
βββ analysis_system.py # Module 2: Multi-Modal Analysis
|
| 11 |
+
βββ scoring_dashboard.py # Module 3: Scoring & Dashboard
|
| 12 |
+
βββ README.md # This file
|
| 13 |
+
```
|
| 14 |
+
|
| 15 |
+
## π― Module Overview
|
| 16 |
+
|
| 17 |
+
### **Module 1: `recording_system.py`**
|
| 18 |
+
**Real-Time Interview Recording and Violation Detection System**
|
| 19 |
+
|
| 20 |
+
**Responsibilities:**
|
| 21 |
+
- Video and audio recording
|
| 22 |
+
- Real-time violation detection (multiple people, looking away, no face, cheating items)
|
| 23 |
+
- Eye contact tracking
|
| 24 |
+
- Blink detection
|
| 25 |
+
- Head pose estimation
|
| 26 |
+
- Lighting analysis
|
| 27 |
+
- Audio transcription
|
| 28 |
+
|
| 29 |
+
**Key Class:** `RecordingSystem`
|
| 30 |
+
|
| 31 |
+
**Main Method:** `record_interview(question_data, duration, ui_callbacks)`
|
| 32 |
+
|
| 33 |
+
---
|
| 34 |
+
|
| 35 |
+
### **Module 2: `analysis_system.py`**
|
| 36 |
+
**Multi-Modal Analysis System**
|
| 37 |
+
|
| 38 |
+
**Responsibilities:**
|
| 39 |
+
- Facial emotion analysis (DeepFace)
|
| 40 |
+
- Audio quality assessment (fluency, accuracy, WPM)
|
| 41 |
+
- Visual outfit analysis (YOLO)
|
| 42 |
+
- Semantic similarity scoring
|
| 43 |
+
- Emotion aggregation and fusion
|
| 44 |
+
|
| 45 |
+
**Key Class:** `AnalysisSystem`
|
| 46 |
+
|
| 47 |
+
**Main Method:** `analyze_recording(recording_data, question_data, duration)`
|
| 48 |
+
|
| 49 |
+
---
|
| 50 |
+
|
| 51 |
+
### **Module 3: `scoring_dashboard.py`**
|
| 52 |
+
**Scoring, Hiring Decision, and Results Dashboard**
|
| 53 |
+
|
| 54 |
+
**Responsibilities:**
|
| 55 |
+
- Calculate hiring decision based on metrics
|
| 56 |
+
- Display immediate question results
|
| 57 |
+
- Render performance overview dashboard
|
| 58 |
+
- Question-by-question detailed analysis
|
| 59 |
+
- CSV export functionality
|
| 60 |
+
|
| 61 |
+
**Key Class:** `ScoringDashboard`
|
| 62 |
+
|
| 63 |
+
**Main Methods:**
|
| 64 |
+
- `decide_hire(result)`
|
| 65 |
+
- `render_dashboard(results)`
|
| 66 |
+
|
| 67 |
+
---
|
| 68 |
+
|
| 69 |
+
### **Integration File: `main_app.py`**
|
| 70 |
+
**Main Application Entry Point**
|
| 71 |
+
|
| 72 |
+
**Responsibilities:**
|
| 73 |
+
- Load all AI models (once, with caching)
|
| 74 |
+
- Initialize all three systems
|
| 75 |
+
- Handle Streamlit UI and routing
|
| 76 |
+
- Manage session state
|
| 77 |
+
- Coordinate data flow between modules
|
| 78 |
+
|
| 79 |
+
---
|
| 80 |
+
|
| 81 |
+
## π How Modules Communicate
|
| 82 |
+
|
| 83 |
+
### **Loose Coupling Design**
|
| 84 |
+
|
| 85 |
+
Each module is **completely independent** and communicates through **standardized dictionaries**:
|
| 86 |
+
|
| 87 |
+
```python
|
| 88 |
+
# Module 1 Output β Module 2 Input
|
| 89 |
+
recording_data = {
|
| 90 |
+
'video_path': str,
|
| 91 |
+
'audio_path': str,
|
| 92 |
+
'frames': list,
|
| 93 |
+
'transcript': str,
|
| 94 |
+
'eye_contact_pct': float,
|
| 95 |
+
'blink_count': int,
|
| 96 |
+
'face_box': tuple,
|
| 97 |
+
'violation_detected': bool,
|
| 98 |
+
'violation_reason': str,
|
| 99 |
+
'violations': list
|
| 100 |
+
}
|
| 101 |
+
|
| 102 |
+
# Module 2 Output β Module 3 Input
|
| 103 |
+
analysis_results = {
|
| 104 |
+
'fused_emotions': dict,
|
| 105 |
+
'emotion_scores': dict,
|
| 106 |
+
'accuracy': float,
|
| 107 |
+
'fluency': float,
|
| 108 |
+
'wpm': float,
|
| 109 |
+
'outfit': str,
|
| 110 |
+
'has_valid_data': bool
|
| 111 |
+
}
|
| 112 |
+
|
| 113 |
+
# Module 3 Output
|
| 114 |
+
final_result = {
|
| 115 |
+
'hire_decision': str,
|
| 116 |
+
'hire_reasons': list,
|
| 117 |
+
... (all previous data merged)
|
| 118 |
+
}
|
| 119 |
+
```
|
| 120 |
+
|
| 121 |
+
---
|
| 122 |
+
|
| 123 |
+
## β
Benefits of This Architecture
|
| 124 |
+
|
| 125 |
+
### **1. Independent Development**
|
| 126 |
+
- Modify `recording_system.py` without touching analysis logic
|
| 127 |
+
- Update `analysis_system.py` algorithms without affecting UI
|
| 128 |
+
- Change `scoring_dashboard.py` visualizations without breaking recording
|
| 129 |
+
|
| 130 |
+
### **2. Easy Testing**
|
| 131 |
+
```python
|
| 132 |
+
# Test Module 1 independently
|
| 133 |
+
recording_system = RecordingSystem(models)
|
| 134 |
+
result = recording_system.record_interview(question, 20, callbacks)
|
| 135 |
+
|
| 136 |
+
# Test Module 2 independently
|
| 137 |
+
analysis_system = AnalysisSystem(models)
|
| 138 |
+
analysis = analysis_system.analyze_recording(recording_data, question)
|
| 139 |
+
|
| 140 |
+
# Test Module 3 independently
|
| 141 |
+
dashboard = ScoringDashboard()
|
| 142 |
+
decision, reasons = dashboard.decide_hire(merged_result)
|
| 143 |
+
```
|
| 144 |
+
|
| 145 |
+
### **3. Easy Extension**
|
| 146 |
+
Want to add a new feature? Just modify one module:
|
| 147 |
+
|
| 148 |
+
- **New violation rule** β Edit `recording_system.py`
|
| 149 |
+
- **New emotion detection** β Edit `analysis_system.py`
|
| 150 |
+
- **New chart/metric** β Edit `scoring_dashboard.py`
|
| 151 |
+
|
| 152 |
+
### **4. Reusability**
|
| 153 |
+
Each module can be imported and used in other projects:
|
| 154 |
+
|
| 155 |
+
```python
|
| 156 |
+
# Use only the recording system in another app
|
| 157 |
+
from recording_system import RecordingSystem
|
| 158 |
+
recorder = RecordingSystem(models)
|
| 159 |
+
```
|
| 160 |
+
|
| 161 |
+
---
|
| 162 |
+
|
| 163 |
+
## π How to Run
|
| 164 |
+
|
| 165 |
+
### **1. Install Dependencies**
|
| 166 |
+
```bash
|
| 167 |
+
pip install streamlit opencv-python numpy pandas deepface mediapipe ultralytics sentence-transformers speechrecognition pyaudio
|
| 168 |
+
```
|
| 169 |
+
|
| 170 |
+
### **2. Run the Application**
|
| 171 |
+
```bash
|
| 172 |
+
streamlit run main_app.py
|
| 173 |
+
```
|
| 174 |
+
|
| 175 |
+
### **3. Project Structure**
|
| 176 |
+
Make sure all 4 files are in the same directory:
|
| 177 |
+
```
|
| 178 |
+
your_folder/
|
| 179 |
+
βββ main_app.py
|
| 180 |
+
βββ recording_system.py
|
| 181 |
+
βββ analysis_system.py
|
| 182 |
+
βββ scoring_dashboard.py
|
| 183 |
+
```
|
| 184 |
+
|
| 185 |
+
---
|
| 186 |
+
|
| 187 |
+
## π§ Customization Guide
|
| 188 |
+
|
| 189 |
+
### **Change Violation Rules**
|
| 190 |
+
Edit `recording_system.py`:
|
| 191 |
+
```python
|
| 192 |
+
# In record_interview() method, adjust thresholds:
|
| 193 |
+
if elapsed > 3.0: # Change from 2.0 to 3.0 seconds
|
| 194 |
+
self.violation_detected = True
|
| 195 |
+
```
|
| 196 |
+
|
| 197 |
+
### **Change Analysis Algorithms**
|
| 198 |
+
Edit `analysis_system.py`:
|
| 199 |
+
```python
|
| 200 |
+
# In evaluate_english_fluency(), adjust weights:
|
| 201 |
+
combined = (0.4 * alpha_ratio) + (0.3 * len_score) + ...
|
| 202 |
+
```
|
| 203 |
+
|
| 204 |
+
### **Change Scoring Logic**
|
| 205 |
+
Edit `scoring_dashboard.py`:
|
| 206 |
+
```python
|
| 207 |
+
# In decide_hire(), adjust thresholds:
|
| 208 |
+
if pos >= 6: # More strict (was 5)
|
| 209 |
+
decision = "β
Hire"
|
| 210 |
+
```
|
| 211 |
+
|
| 212 |
+
### **Change UI/Dashboard**
|
| 213 |
+
Edit `scoring_dashboard.py` or `main_app.py`:
|
| 214 |
+
```python
|
| 215 |
+
# Add new charts, change colors, modify layout
|
| 216 |
+
```
|
| 217 |
+
|
| 218 |
+
---
|
| 219 |
+
|
| 220 |
+
## π¨ Module Interfaces (API)
|
| 221 |
+
|
| 222 |
+
### **RecordingSystem API**
|
| 223 |
+
```python
|
| 224 |
+
class RecordingSystem:
|
| 225 |
+
def __init__(self, models_dict)
|
| 226 |
+
def record_interview(self, question_data, duration, ui_callbacks) -> dict
|
| 227 |
+
def detect_cheating_items(self, detected_objects) -> list
|
| 228 |
+
def calculate_eye_gaze(self, face_landmarks, frame_shape) -> bool
|
| 229 |
+
def estimate_head_pose(self, face_landmarks, frame_shape) -> tuple
|
| 230 |
+
```
|
| 231 |
+
|
| 232 |
+
### **AnalysisSystem API**
|
| 233 |
+
```python
|
| 234 |
+
class AnalysisSystem:
|
| 235 |
+
def __init__(self, models_dict)
|
| 236 |
+
def analyze_recording(self, recording_data, question_data, duration) -> dict
|
| 237 |
+
def analyze_frame_emotion(self, frame_bgr) -> dict
|
| 238 |
+
def evaluate_answer_accuracy(self, answer, question, ideal) -> float
|
| 239 |
+
def evaluate_english_fluency(self, text) -> float
|
| 240 |
+
def analyze_outfit(self, frame, face_box) -> tuple
|
| 241 |
+
```
|
| 242 |
+
|
| 243 |
+
### **ScoringDashboard API**
|
| 244 |
+
```python
|
| 245 |
+
class ScoringDashboard:
|
| 246 |
+
def __init__(self)
|
| 247 |
+
def decide_hire(self, result) -> tuple
|
| 248 |
+
def render_dashboard(self, results) -> None
|
| 249 |
+
def display_immediate_results(self, result) -> None
|
| 250 |
+
def export_results_csv(self, results) -> str
|
| 251 |
+
```
|
| 252 |
+
|
| 253 |
---
|
| 254 |
+
|
| 255 |
+
## π¦ Dependencies by Module
|
| 256 |
+
|
| 257 |
+
### **Module 1 (recording_system.py)**
|
| 258 |
+
- cv2 (opencv-python)
|
| 259 |
+
- numpy
|
| 260 |
+
- mediapipe
|
| 261 |
+
- ultralytics
|
| 262 |
+
- speech_recognition
|
| 263 |
+
|
| 264 |
+
### **Module 2 (analysis_system.py)**
|
| 265 |
+
- cv2 (opencv-python)
|
| 266 |
+
- numpy
|
| 267 |
+
- pandas
|
| 268 |
+
- deepface
|
| 269 |
+
- sentence-transformers
|
| 270 |
+
- ultralytics
|
| 271 |
+
|
| 272 |
+
### **Module 3 (scoring_dashboard.py)**
|
| 273 |
+
- streamlit
|
| 274 |
+
- numpy
|
| 275 |
+
- pandas
|
| 276 |
+
|
| 277 |
+
### **Main App (main_app.py)**
|
| 278 |
- streamlit
|
| 279 |
+
- All dependencies from modules 1-3
|
| 280 |
+
|
|
|
|
| 281 |
---
|
| 282 |
|
| 283 |
+
## π‘οΈ Error Handling
|
| 284 |
+
|
| 285 |
+
Each module handles its own errors:
|
| 286 |
+
|
| 287 |
+
- **Module 1**: Returns `{'error': 'message'}` if camera fails
|
| 288 |
+
- **Module 2**: Returns default values (0.0) if analysis fails
|
| 289 |
+
- **Module 3**: Handles missing data gracefully in UI
|
| 290 |
|
| 291 |
+
The main app checks for errors and displays appropriate messages.
|
| 292 |
+
|
| 293 |
+
---
|
| 294 |
+
|
| 295 |
+
## π Data Flow Diagram
|
| 296 |
+
|
| 297 |
+
```
|
| 298 |
+
βββββββββββββββββββ
|
| 299 |
+
β main_app.py β
|
| 300 |
+
β (Orchestrator) β
|
| 301 |
+
ββββββββββ¬βββββββββ
|
| 302 |
+
β
|
| 303 |
+
ββββΊ 1. Load Models (cached)
|
| 304 |
+
β
|
| 305 |
+
ββββΊ 2. RecordingSystem.record_interview()
|
| 306 |
+
β β
|
| 307 |
+
β ββββΊ Returns: recording_data
|
| 308 |
+
β
|
| 309 |
+
ββββΊ 3. AnalysisSystem.analyze_recording(recording_data)
|
| 310 |
+
β β
|
| 311 |
+
β ββββΊ Returns: analysis_results
|
| 312 |
+
β
|
| 313 |
+
ββββΊ 4. Merge recording_data + analysis_results
|
| 314 |
+
β
|
| 315 |
+
ββββΊ 5. ScoringDashboard.decide_hire(merged_result)
|
| 316 |
+
β
|
| 317 |
+
ββββΊ Returns: (decision, reasons)
|
| 318 |
+
```
|
| 319 |
+
|
| 320 |
+
---
|
| 321 |
+
|
| 322 |
+
## π‘ Best Practices
|
| 323 |
+
|
| 324 |
+
1. **Never modify dictionary keys** between modules - this breaks compatibility
|
| 325 |
+
2. **Always provide default values** in case of missing data
|
| 326 |
+
3. **Use type hints** when adding new methods
|
| 327 |
+
4. **Test each module independently** before integration
|
| 328 |
+
5. **Keep UI logic in main_app.py** or scoring_dashboard.py only
|
| 329 |
+
|
| 330 |
+
---
|
| 331 |
+
|
| 332 |
+
## π Version History
|
| 333 |
+
|
| 334 |
+
- **v2.0**: Modular architecture with 3 independent systems
|
| 335 |
+
- **v1.0**: Monolithic single-file application
|
| 336 |
+
|
| 337 |
+
---
|
| 338 |
+
|
| 339 |
+
## π€ Contributing
|
| 340 |
+
|
| 341 |
+
When adding features:
|
| 342 |
+
|
| 343 |
+
1. Identify which module it belongs to
|
| 344 |
+
2. Add method to that module only
|
| 345 |
+
3. Update the module's docstrings
|
| 346 |
+
4. Test independently before integration
|
| 347 |
+
5. Update this README if adding new APIs
|
| 348 |
+
|
| 349 |
+
---
|
| 350 |
+
|
| 351 |
+
## π§ Support
|
| 352 |
+
|
| 353 |
+
For questions about:
|
| 354 |
+
- **Recording issues** β Check `recording_system.py`
|
| 355 |
+
- **Analysis issues** β Check `analysis_system.py`
|
| 356 |
+
- **UI/Dashboard issues** β Check `scoring_dashboard.py` or `main_app.py`
|
| 357 |
+
|
| 358 |
+
---
|
| 359 |
|
| 360 |
+
**Built with β€οΈ using Modular Design Principles**
|
|
|
Recording_system.py
ADDED
|
@@ -0,0 +1,1169 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
|
| 2 |
+
"""
|
| 3 |
+
Real-Time Interview Recording and Violation Detection System
|
| 4 |
+
UPDATED VERSION:
|
| 5 |
+
- Fixed cv2.FONT_HERSHEY_BOLD error (use FONT_HERSHEY_SIMPLEX)
|
| 6 |
+
- Captures violation images
|
| 7 |
+
- Continues to next question after violation
|
| 8 |
+
- Stores violation metadata for display in results
|
| 9 |
+
"""
|
| 10 |
+
|
| 11 |
+
import cv2
|
| 12 |
+
import numpy as np
|
| 13 |
+
import threading
|
| 14 |
+
import time
|
| 15 |
+
import tempfile
|
| 16 |
+
import os
|
| 17 |
+
import speech_recognition as sr
|
| 18 |
+
import warnings
|
| 19 |
+
from collections import deque
|
| 20 |
+
|
| 21 |
+
warnings.filterwarnings('ignore')
|
| 22 |
+
os.environ['TF_CPP_MIN_LOG_LEVEL'] = '3'
|
| 23 |
+
|
| 24 |
+
class RecordingSystem:
|
| 25 |
+
"""Handles video/audio recording with real-time violation detection"""
|
| 26 |
+
|
| 27 |
+
def __init__(self, models_dict):
|
| 28 |
+
"""
|
| 29 |
+
Initialize recording system with loaded models
|
| 30 |
+
|
| 31 |
+
Args:
|
| 32 |
+
models_dict: Dictionary containing pre-loaded AI models
|
| 33 |
+
"""
|
| 34 |
+
self.models = models_dict
|
| 35 |
+
self.violation_detected = False
|
| 36 |
+
self.violation_reason = ""
|
| 37 |
+
|
| 38 |
+
# Frame boundaries (for sitting position: left, right, top only)
|
| 39 |
+
self.frame_margin = 50 # pixels from edge
|
| 40 |
+
|
| 41 |
+
# Position adjustment tracking
|
| 42 |
+
self.position_adjusted = False
|
| 43 |
+
self.baseline_environment = None # Store initial environment scan
|
| 44 |
+
|
| 45 |
+
# Violation storage directory
|
| 46 |
+
self.violation_images_dir = tempfile.mkdtemp(prefix="violations_")
|
| 47 |
+
|
| 48 |
+
# Initialize pose detection if available
|
| 49 |
+
try:
|
| 50 |
+
import mediapipe as mp
|
| 51 |
+
self.mp_pose = mp.solutions.pose
|
| 52 |
+
self.pose_detector = self.mp_pose.Pose(
|
| 53 |
+
static_image_mode=False,
|
| 54 |
+
model_complexity=1,
|
| 55 |
+
smooth_landmarks=True,
|
| 56 |
+
min_detection_confidence=0.5,
|
| 57 |
+
min_tracking_confidence=0.5
|
| 58 |
+
)
|
| 59 |
+
self.pose_available = True
|
| 60 |
+
except:
|
| 61 |
+
self.pose_detector = None
|
| 62 |
+
self.pose_available = False
|
| 63 |
+
|
| 64 |
+
def save_violation_image(self, frame, question_number, violation_reason):
|
| 65 |
+
"""
|
| 66 |
+
Save an image of the violation for later display
|
| 67 |
+
FIXED: Changed cv2.FONT_HERSHEY_BOLD to cv2.FONT_HERSHEY_SIMPLEX
|
| 68 |
+
|
| 69 |
+
Args:
|
| 70 |
+
frame: BGR image frame showing the violation
|
| 71 |
+
question_number: Current question number
|
| 72 |
+
violation_reason: Description of the violation
|
| 73 |
+
|
| 74 |
+
Returns:
|
| 75 |
+
Path to saved violation image
|
| 76 |
+
"""
|
| 77 |
+
try:
|
| 78 |
+
# Create filename with timestamp
|
| 79 |
+
timestamp = int(time.time() * 1000)
|
| 80 |
+
filename = f"violation_q{question_number}_{timestamp}.jpg"
|
| 81 |
+
filepath = os.path.join(self.violation_images_dir, filename)
|
| 82 |
+
|
| 83 |
+
# Add violation text overlay to image
|
| 84 |
+
overlay_frame = frame.copy()
|
| 85 |
+
h, w = overlay_frame.shape[:2]
|
| 86 |
+
|
| 87 |
+
# Add semi-transparent red overlay
|
| 88 |
+
red_overlay = overlay_frame.copy()
|
| 89 |
+
cv2.rectangle(red_overlay, (0, 0), (w, h), (0, 0, 255), -1)
|
| 90 |
+
overlay_frame = cv2.addWeighted(overlay_frame, 0.7, red_overlay, 0.3, 0)
|
| 91 |
+
|
| 92 |
+
# Add thick red border
|
| 93 |
+
cv2.rectangle(overlay_frame, (0, 0), (w-1, h-1), (0, 0, 255), 10)
|
| 94 |
+
|
| 95 |
+
# Add violation text with background - FIXED FONT
|
| 96 |
+
text = "VIOLATION DETECTED"
|
| 97 |
+
cv2.rectangle(overlay_frame, (0, 0), (w, 80), (0, 0, 0), -1)
|
| 98 |
+
cv2.putText(overlay_frame, text, (w//2 - 200, 50),
|
| 99 |
+
cv2.FONT_HERSHEY_SIMPLEX, 1.2, (0, 0, 255), 3) # FIXED: Was FONT_HERSHEY_BOLD
|
| 100 |
+
|
| 101 |
+
# Add violation reason at bottom
|
| 102 |
+
cv2.rectangle(overlay_frame, (0, h-100), (w, h), (0, 0, 0), -1)
|
| 103 |
+
|
| 104 |
+
# Split long violation text into multiple lines
|
| 105 |
+
words = violation_reason.split()
|
| 106 |
+
lines = []
|
| 107 |
+
current_line = ""
|
| 108 |
+
for word in words:
|
| 109 |
+
test_line = current_line + " " + word if current_line else word
|
| 110 |
+
if len(test_line) > 50:
|
| 111 |
+
lines.append(current_line)
|
| 112 |
+
current_line = word
|
| 113 |
+
else:
|
| 114 |
+
current_line = test_line
|
| 115 |
+
if current_line:
|
| 116 |
+
lines.append(current_line)
|
| 117 |
+
|
| 118 |
+
# Draw violation reason lines
|
| 119 |
+
y_offset = h - 90
|
| 120 |
+
for line in lines[:2]: # Max 2 lines
|
| 121 |
+
cv2.putText(overlay_frame, line, (10, y_offset),
|
| 122 |
+
cv2.FONT_HERSHEY_SIMPLEX, 0.6, (255, 255, 255), 2)
|
| 123 |
+
y_offset += 30
|
| 124 |
+
|
| 125 |
+
# Save image
|
| 126 |
+
cv2.imwrite(filepath, overlay_frame)
|
| 127 |
+
return filepath
|
| 128 |
+
|
| 129 |
+
except Exception as e:
|
| 130 |
+
print(f"Error saving violation image: {e}")
|
| 131 |
+
return None
|
| 132 |
+
|
| 133 |
+
def scan_environment(self, frame):
|
| 134 |
+
"""
|
| 135 |
+
Scan and catalog the environment before test starts
|
| 136 |
+
"""
|
| 137 |
+
if self.models['yolo'] is None:
|
| 138 |
+
return {'objects': [], 'positions': []}
|
| 139 |
+
|
| 140 |
+
try:
|
| 141 |
+
results = self.models['yolo'].predict(frame, conf=0.25, verbose=False)
|
| 142 |
+
|
| 143 |
+
environment_data = {
|
| 144 |
+
'objects': [],
|
| 145 |
+
'positions': [],
|
| 146 |
+
'person_position': None
|
| 147 |
+
}
|
| 148 |
+
|
| 149 |
+
if results and len(results) > 0:
|
| 150 |
+
names = self.models['yolo'].names
|
| 151 |
+
boxes = results[0].boxes
|
| 152 |
+
|
| 153 |
+
for box in boxes:
|
| 154 |
+
cls_id = int(box.cls[0])
|
| 155 |
+
obj_name = names[cls_id]
|
| 156 |
+
x1, y1, x2, y2 = box.xyxy[0].cpu().numpy()
|
| 157 |
+
|
| 158 |
+
environment_data['objects'].append(obj_name)
|
| 159 |
+
environment_data['positions'].append({
|
| 160 |
+
'name': obj_name,
|
| 161 |
+
'bbox': (int(x1), int(y1), int(x2), int(y2)),
|
| 162 |
+
'center': (int((x1+x2)/2), int((y1+y2)/2))
|
| 163 |
+
})
|
| 164 |
+
|
| 165 |
+
if obj_name == 'person':
|
| 166 |
+
environment_data['person_position'] = (int((x1+x2)/2), int((y1+y2)/2))
|
| 167 |
+
|
| 168 |
+
return environment_data
|
| 169 |
+
|
| 170 |
+
except Exception as e:
|
| 171 |
+
return {'objects': [], 'positions': []}
|
| 172 |
+
|
| 173 |
+
def detect_new_objects(self, frame):
|
| 174 |
+
"""
|
| 175 |
+
Detect NEW objects that weren't in baseline environment
|
| 176 |
+
"""
|
| 177 |
+
if self.models['yolo'] is None or self.baseline_environment is None:
|
| 178 |
+
return False, []
|
| 179 |
+
|
| 180 |
+
try:
|
| 181 |
+
results = self.models['yolo'].predict(frame, conf=0.25, verbose=False)
|
| 182 |
+
|
| 183 |
+
if results and len(results) > 0:
|
| 184 |
+
names = self.models['yolo'].names
|
| 185 |
+
boxes = results[0].boxes
|
| 186 |
+
|
| 187 |
+
current_objects = []
|
| 188 |
+
for box in boxes:
|
| 189 |
+
cls_id = int(box.cls[0])
|
| 190 |
+
obj_name = names[cls_id]
|
| 191 |
+
x1, y1, x2, y2 = box.xyxy[0].cpu().numpy()
|
| 192 |
+
current_center = (int((x1+x2)/2), int((y1+y2)/2))
|
| 193 |
+
|
| 194 |
+
current_objects.append({
|
| 195 |
+
'name': obj_name,
|
| 196 |
+
'center': current_center,
|
| 197 |
+
'bbox': (int(x1), int(y1), int(x2), int(y2))
|
| 198 |
+
})
|
| 199 |
+
|
| 200 |
+
baseline_objects = self.baseline_environment['positions']
|
| 201 |
+
new_items = []
|
| 202 |
+
|
| 203 |
+
for curr_obj in current_objects:
|
| 204 |
+
if curr_obj['name'] == 'person':
|
| 205 |
+
continue
|
| 206 |
+
|
| 207 |
+
is_baseline = False
|
| 208 |
+
for base_obj in baseline_objects:
|
| 209 |
+
if curr_obj['name'] == base_obj['name']:
|
| 210 |
+
dist = np.sqrt(
|
| 211 |
+
(curr_obj['center'][0] - base_obj['center'][0])**2 +
|
| 212 |
+
(curr_obj['center'][1] - base_obj['center'][1])**2
|
| 213 |
+
)
|
| 214 |
+
if dist < 100:
|
| 215 |
+
is_baseline = True
|
| 216 |
+
break
|
| 217 |
+
|
| 218 |
+
if not is_baseline:
|
| 219 |
+
new_items.append(curr_obj['name'])
|
| 220 |
+
|
| 221 |
+
if new_items:
|
| 222 |
+
return True, list(set(new_items))
|
| 223 |
+
|
| 224 |
+
return False, []
|
| 225 |
+
|
| 226 |
+
except Exception as e:
|
| 227 |
+
return False, []
|
| 228 |
+
|
| 229 |
+
def detect_suspicious_movements(self, frame):
|
| 230 |
+
"""Detect suspicious hand movements"""
|
| 231 |
+
if self.models['hands'] is None:
|
| 232 |
+
return False, ""
|
| 233 |
+
|
| 234 |
+
rgb_frame = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB)
|
| 235 |
+
h, w = frame.shape[:2]
|
| 236 |
+
|
| 237 |
+
try:
|
| 238 |
+
hand_results = self.models['hands'].process(rgb_frame)
|
| 239 |
+
|
| 240 |
+
if hand_results.multi_hand_landmarks:
|
| 241 |
+
for hand_landmarks in hand_results.multi_hand_landmarks:
|
| 242 |
+
wrist = hand_landmarks.landmark[0]
|
| 243 |
+
index_tip = hand_landmarks.landmark[8]
|
| 244 |
+
|
| 245 |
+
wrist_y = wrist.y * h
|
| 246 |
+
tip_y = index_tip.y * h
|
| 247 |
+
|
| 248 |
+
if wrist_y > h * 0.75:
|
| 249 |
+
return True, "Hand movement below desk level detected"
|
| 250 |
+
|
| 251 |
+
if wrist_y < h * 0.15:
|
| 252 |
+
return True, "Suspicious hand movement at top of frame"
|
| 253 |
+
|
| 254 |
+
except Exception as e:
|
| 255 |
+
pass
|
| 256 |
+
|
| 257 |
+
return False, ""
|
| 258 |
+
|
| 259 |
+
def calculate_eye_gaze(self, face_landmarks, frame_shape):
|
| 260 |
+
"""Calculate if eyes are looking at camera"""
|
| 261 |
+
h, w = frame_shape[:2]
|
| 262 |
+
|
| 263 |
+
left_eye_indices = [468, 469, 470, 471, 472]
|
| 264 |
+
right_eye_indices = [473, 474, 475, 476, 477]
|
| 265 |
+
left_eye_center = [33, 133, 157, 158, 159, 160, 161, 163, 144, 145, 153, 154, 155]
|
| 266 |
+
right_eye_center = [362, 263, 387, 386, 385, 384, 398, 382, 381, 380, 373, 374, 390]
|
| 267 |
+
|
| 268 |
+
landmarks = face_landmarks.landmark
|
| 269 |
+
|
| 270 |
+
left_iris_x = np.mean([landmarks[i].x for i in left_eye_indices if i < len(landmarks)])
|
| 271 |
+
left_eye_x = np.mean([landmarks[i].x for i in left_eye_center if i < len(landmarks)])
|
| 272 |
+
|
| 273 |
+
right_iris_x = np.mean([landmarks[i].x for i in right_eye_indices if i < len(landmarks)])
|
| 274 |
+
right_eye_x = np.mean([landmarks[i].x for i in right_eye_center if i < len(landmarks)])
|
| 275 |
+
|
| 276 |
+
left_gaze_ratio = (left_iris_x - left_eye_x) if left_iris_x and left_eye_x else 0
|
| 277 |
+
right_gaze_ratio = (right_iris_x - right_eye_x) if right_iris_x and right_eye_x else 0
|
| 278 |
+
|
| 279 |
+
avg_gaze = (left_gaze_ratio + right_gaze_ratio) / 2
|
| 280 |
+
|
| 281 |
+
return abs(avg_gaze) < 0.02
|
| 282 |
+
|
| 283 |
+
def estimate_head_pose(self, face_landmarks, frame_shape):
|
| 284 |
+
"""Estimate head pose angles"""
|
| 285 |
+
h, w = frame_shape[:2]
|
| 286 |
+
landmarks_3d = np.array([(lm.x * w, lm.y * h, lm.z) for lm in face_landmarks.landmark])
|
| 287 |
+
|
| 288 |
+
required_indices = [1, 33, 263, 61, 291]
|
| 289 |
+
image_points = np.array([landmarks_3d[i] for i in required_indices], dtype="double")
|
| 290 |
+
|
| 291 |
+
model_points = np.array([
|
| 292 |
+
(0.0, 0.0, 0.0), (-30.0, -125.0, -30.0),
|
| 293 |
+
(30.0, -125.0, -30.0), (-60.0, -70.0, -60.0),
|
| 294 |
+
(60.0, -70.0, -60.0)
|
| 295 |
+
])
|
| 296 |
+
|
| 297 |
+
focal_length = w
|
| 298 |
+
center = (w / 2, h / 2)
|
| 299 |
+
camera_matrix = np.array([
|
| 300 |
+
[focal_length, 0, center[0]],
|
| 301 |
+
[0, focal_length, center[1]],
|
| 302 |
+
[0, 0, 1]
|
| 303 |
+
], dtype="double")
|
| 304 |
+
dist_coeffs = np.zeros((4, 1))
|
| 305 |
+
|
| 306 |
+
success, rotation_vector, _ = cv2.solvePnP(
|
| 307 |
+
model_points, image_points, camera_matrix,
|
| 308 |
+
dist_coeffs, flags=cv2.SOLVEPNP_ITERATIVE
|
| 309 |
+
)
|
| 310 |
+
|
| 311 |
+
if success:
|
| 312 |
+
rmat, _ = cv2.Rodrigues(rotation_vector)
|
| 313 |
+
pose_mat = cv2.hconcat((rmat, rotation_vector))
|
| 314 |
+
_, _, _, _, _, _, euler = cv2.decomposeProjectionMatrix(pose_mat)
|
| 315 |
+
yaw, pitch, roll = [float(a) for a in euler]
|
| 316 |
+
return yaw, pitch, roll
|
| 317 |
+
|
| 318 |
+
return 0, 0, 0
|
| 319 |
+
|
| 320 |
+
def detect_blink(self, face_landmarks):
|
| 321 |
+
"""Detect if eye is blinking"""
|
| 322 |
+
upper_lid = face_landmarks.landmark[159]
|
| 323 |
+
lower_lid = face_landmarks.landmark[145]
|
| 324 |
+
eye_openness = abs(upper_lid.y - lower_lid.y)
|
| 325 |
+
return eye_openness < 0.01
|
| 326 |
+
|
| 327 |
+
def analyze_lighting(self, frame):
|
| 328 |
+
"""Analyze lighting conditions"""
|
| 329 |
+
gray = cv2.cvtColor(frame, cv2.COLOR_BGR2GRAY)
|
| 330 |
+
mean_brightness = np.mean(gray)
|
| 331 |
+
std_brightness = np.std(gray)
|
| 332 |
+
|
| 333 |
+
if mean_brightness < 60:
|
| 334 |
+
return "Too Dark", mean_brightness
|
| 335 |
+
elif mean_brightness > 200:
|
| 336 |
+
return "Too Bright", mean_brightness
|
| 337 |
+
elif std_brightness < 25:
|
| 338 |
+
return "Low Contrast", mean_brightness
|
| 339 |
+
else:
|
| 340 |
+
return "Good", mean_brightness
|
| 341 |
+
|
| 342 |
+
|
| 343 |
+
def check_frame_boundaries(self, frame, face_box):
|
| 344 |
+
"""Check if person is within frame boundaries"""
|
| 345 |
+
if face_box is None:
|
| 346 |
+
return False, "No face detected", "NO_FACE"
|
| 347 |
+
|
| 348 |
+
h, w = frame.shape[:2]
|
| 349 |
+
margin = self.frame_margin
|
| 350 |
+
x, y, fw, fh = face_box
|
| 351 |
+
|
| 352 |
+
face_center_x = x + fw // 2
|
| 353 |
+
face_top = y
|
| 354 |
+
face_left = x
|
| 355 |
+
face_right = x + fw
|
| 356 |
+
|
| 357 |
+
if face_left < margin:
|
| 358 |
+
return False, "Person too close to LEFT edge", "LEFT_VIOLATION"
|
| 359 |
+
|
| 360 |
+
if face_right > (w - margin):
|
| 361 |
+
return False, "Person too close to RIGHT edge", "RIGHT_VIOLATION"
|
| 362 |
+
|
| 363 |
+
if face_top < margin:
|
| 364 |
+
return False, "Person too close to TOP edge", "TOP_VIOLATION"
|
| 365 |
+
|
| 366 |
+
return True, "Within boundaries", "OK"
|
| 367 |
+
|
| 368 |
+
def detect_person_outside_frame(self, frame):
|
| 369 |
+
"""Detect if any person/living being is outside boundaries"""
|
| 370 |
+
if self.models['yolo'] is None:
|
| 371 |
+
return False, "", ""
|
| 372 |
+
|
| 373 |
+
h, w = frame.shape[:2]
|
| 374 |
+
margin = self.frame_margin
|
| 375 |
+
|
| 376 |
+
try:
|
| 377 |
+
results = self.models['yolo'].predict(frame, conf=0.4, verbose=False)
|
| 378 |
+
|
| 379 |
+
if results and len(results) > 0:
|
| 380 |
+
names = self.models['yolo'].names
|
| 381 |
+
boxes = results[0].boxes
|
| 382 |
+
|
| 383 |
+
living_beings = ['person', 'cat', 'dog', 'bird', 'horse', 'sheep', 'cow',
|
| 384 |
+
'elephant', 'bear', 'zebra', 'giraffe']
|
| 385 |
+
|
| 386 |
+
for i, box in enumerate(boxes):
|
| 387 |
+
cls_id = int(box.cls[0])
|
| 388 |
+
obj_name = names[cls_id]
|
| 389 |
+
|
| 390 |
+
if obj_name.lower() in living_beings:
|
| 391 |
+
x1, y1, x2, y2 = box.xyxy[0].cpu().numpy()
|
| 392 |
+
|
| 393 |
+
if x1 < margin or x2 < margin:
|
| 394 |
+
return True, obj_name, "LEFT"
|
| 395 |
+
|
| 396 |
+
if x1 > (w - margin) or x2 > (w - margin):
|
| 397 |
+
return True, obj_name, "RIGHT"
|
| 398 |
+
|
| 399 |
+
if y1 < margin or y2 < margin:
|
| 400 |
+
return True, obj_name, "TOP"
|
| 401 |
+
|
| 402 |
+
except Exception as e:
|
| 403 |
+
pass
|
| 404 |
+
|
| 405 |
+
return False, "", ""
|
| 406 |
+
|
| 407 |
+
def detect_multiple_bodies(self, frame, num_faces):
|
| 408 |
+
"""Detect multiple bodies using pose and hand detection"""
|
| 409 |
+
rgb_frame = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB)
|
| 410 |
+
body_count = 0
|
| 411 |
+
detected_parts = []
|
| 412 |
+
|
| 413 |
+
if self.pose_available and self.pose_detector:
|
| 414 |
+
try:
|
| 415 |
+
pose_results = self.pose_detector.process(rgb_frame)
|
| 416 |
+
|
| 417 |
+
if pose_results.pose_landmarks:
|
| 418 |
+
body_count += 1
|
| 419 |
+
detected_parts.append("body")
|
| 420 |
+
|
| 421 |
+
landmarks = pose_results.pose_landmarks.landmark
|
| 422 |
+
|
| 423 |
+
visible_shoulders = sum(1 for idx in [11, 12]
|
| 424 |
+
if landmarks[idx].visibility > 0.5)
|
| 425 |
+
visible_elbows = sum(1 for idx in [13, 14]
|
| 426 |
+
if landmarks[idx].visibility > 0.5)
|
| 427 |
+
|
| 428 |
+
if visible_shoulders > 2 or visible_elbows > 2:
|
| 429 |
+
return True, "Multiple body parts detected (extra shoulders/arms)", body_count + 1
|
| 430 |
+
|
| 431 |
+
except Exception as e:
|
| 432 |
+
pass
|
| 433 |
+
|
| 434 |
+
if self.models['hands'] is not None:
|
| 435 |
+
try:
|
| 436 |
+
hand_results = self.models['hands'].process(rgb_frame)
|
| 437 |
+
|
| 438 |
+
if hand_results.multi_hand_landmarks:
|
| 439 |
+
num_hands = len(hand_results.multi_hand_landmarks)
|
| 440 |
+
|
| 441 |
+
if num_hands > 2:
|
| 442 |
+
detected_parts.append(f"{num_hands} hands")
|
| 443 |
+
return True, f"Multiple persons detected ({num_hands} hands visible)", 2
|
| 444 |
+
|
| 445 |
+
if num_hands == 2:
|
| 446 |
+
hand1 = hand_results.multi_hand_landmarks[0].landmark[0]
|
| 447 |
+
hand2 = hand_results.multi_hand_landmarks[1].landmark[0]
|
| 448 |
+
|
| 449 |
+
distance = np.sqrt((hand1.x - hand2.x)**2 + (hand1.y - hand2.y)**2)
|
| 450 |
+
|
| 451 |
+
if distance > 0.7:
|
| 452 |
+
detected_parts.append("widely separated hands")
|
| 453 |
+
return True, "Suspicious hand positions (possible multiple persons)", 2
|
| 454 |
+
|
| 455 |
+
except Exception as e:
|
| 456 |
+
pass
|
| 457 |
+
|
| 458 |
+
if num_faces == 1 and body_count > 1:
|
| 459 |
+
return True, "Body parts from multiple persons detected", 2
|
| 460 |
+
|
| 461 |
+
if num_faces > 1:
|
| 462 |
+
return True, f"Multiple persons detected ({num_faces} faces)", num_faces
|
| 463 |
+
|
| 464 |
+
return False, "", max(num_faces, body_count)
|
| 465 |
+
|
| 466 |
+
def detect_hands_outside_main_person(self, frame, face_box):
|
| 467 |
+
"""Detect hands outside main person's area"""
|
| 468 |
+
if self.models['hands'] is None or face_box is None:
|
| 469 |
+
return False, ""
|
| 470 |
+
|
| 471 |
+
rgb_frame = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB)
|
| 472 |
+
h, w = frame.shape[:2]
|
| 473 |
+
|
| 474 |
+
try:
|
| 475 |
+
hand_results = self.models['hands'].process(rgb_frame)
|
| 476 |
+
|
| 477 |
+
if hand_results.multi_hand_landmarks:
|
| 478 |
+
x, y, fw, fh = face_box
|
| 479 |
+
|
| 480 |
+
expected_left = max(0, x - fw)
|
| 481 |
+
expected_right = min(w, x + fw * 2)
|
| 482 |
+
expected_top = max(0, y - fh)
|
| 483 |
+
expected_bottom = min(h, y + fh * 4)
|
| 484 |
+
|
| 485 |
+
for hand_landmarks in hand_results.multi_hand_landmarks:
|
| 486 |
+
hand_x = hand_landmarks.landmark[0].x * w
|
| 487 |
+
hand_y = hand_landmarks.landmark[0].y * h
|
| 488 |
+
|
| 489 |
+
if (hand_x < expected_left - 50 or hand_x > expected_right + 50 or
|
| 490 |
+
hand_y < expected_top - 50 or hand_y > expected_bottom + 50):
|
| 491 |
+
return True, "Hand detected outside main person's area"
|
| 492 |
+
|
| 493 |
+
except Exception as e:
|
| 494 |
+
pass
|
| 495 |
+
|
| 496 |
+
return False, ""
|
| 497 |
+
|
| 498 |
+
def has_skin_tone(self, region):
|
| 499 |
+
"""Check if region contains skin-like colors"""
|
| 500 |
+
if region.size == 0:
|
| 501 |
+
return False
|
| 502 |
+
|
| 503 |
+
hsv = cv2.cvtColor(region, cv2.COLOR_BGR2HSV)
|
| 504 |
+
|
| 505 |
+
lower_skin1 = np.array([0, 20, 70], dtype=np.uint8)
|
| 506 |
+
upper_skin1 = np.array([20, 255, 255], dtype=np.uint8)
|
| 507 |
+
lower_skin2 = np.array([0, 20, 0], dtype=np.uint8)
|
| 508 |
+
upper_skin2 = np.array([20, 150, 255], dtype=np.uint8)
|
| 509 |
+
|
| 510 |
+
mask1 = cv2.inRange(hsv, lower_skin1, upper_skin1)
|
| 511 |
+
mask2 = cv2.inRange(hsv, lower_skin2, upper_skin2)
|
| 512 |
+
mask = cv2.bitwise_or(mask1, mask2)
|
| 513 |
+
|
| 514 |
+
skin_ratio = np.sum(mask > 0) / mask.size
|
| 515 |
+
return skin_ratio > 0.3
|
| 516 |
+
|
| 517 |
+
def detect_intrusion_at_edges(self, frame, face_box):
|
| 518 |
+
"""Detect body parts intruding from frame edges"""
|
| 519 |
+
if face_box is None:
|
| 520 |
+
return False, ""
|
| 521 |
+
|
| 522 |
+
h, w = frame.shape[:2]
|
| 523 |
+
x, y, fw, fh = face_box
|
| 524 |
+
|
| 525 |
+
edge_width = 80
|
| 526 |
+
|
| 527 |
+
left_region = frame[:, :edge_width]
|
| 528 |
+
right_region = frame[:, w-edge_width:]
|
| 529 |
+
top_left = frame[:edge_width, :w//3]
|
| 530 |
+
top_right = frame[:edge_width, 2*w//3:]
|
| 531 |
+
|
| 532 |
+
face_center_x = x + fw // 2
|
| 533 |
+
face_far_from_left = face_center_x > w * 0.3
|
| 534 |
+
face_far_from_right = face_center_x < w * 0.7
|
| 535 |
+
|
| 536 |
+
if face_far_from_left and self.has_skin_tone(left_region):
|
| 537 |
+
if self.models['hands']:
|
| 538 |
+
rgb_region = cv2.cvtColor(left_region, cv2.COLOR_BGR2RGB)
|
| 539 |
+
try:
|
| 540 |
+
result = self.models['hands'].process(rgb_region)
|
| 541 |
+
if result.multi_hand_landmarks:
|
| 542 |
+
return True, "Body part detected at left edge (another person)"
|
| 543 |
+
except:
|
| 544 |
+
pass
|
| 545 |
+
|
| 546 |
+
if face_far_from_right and self.has_skin_tone(right_region):
|
| 547 |
+
if self.models['hands']:
|
| 548 |
+
rgb_region = cv2.cvtColor(right_region, cv2.COLOR_BGR2RGB)
|
| 549 |
+
try:
|
| 550 |
+
result = self.models['hands'].process(rgb_region)
|
| 551 |
+
if result.multi_hand_landmarks:
|
| 552 |
+
return True, "Body part detected at right edge (another person)"
|
| 553 |
+
except:
|
| 554 |
+
pass
|
| 555 |
+
|
| 556 |
+
if y > h * 0.2:
|
| 557 |
+
if self.has_skin_tone(top_left) or self.has_skin_tone(top_right):
|
| 558 |
+
return True, "Body part detected at top edge (another person)"
|
| 559 |
+
|
| 560 |
+
return False, ""
|
| 561 |
+
|
| 562 |
+
def draw_frame_boundaries(self, frame):
|
| 563 |
+
"""Draw visible frame boundaries"""
|
| 564 |
+
h, w = frame.shape[:2]
|
| 565 |
+
margin = self.frame_margin
|
| 566 |
+
|
| 567 |
+
overlay = frame.copy()
|
| 568 |
+
|
| 569 |
+
cv2.line(overlay, (margin, 0), (margin, h), (0, 255, 0), 3)
|
| 570 |
+
cv2.line(overlay, (w - margin, 0), (w - margin, h), (0, 255, 0), 3)
|
| 571 |
+
cv2.line(overlay, (0, margin), (w, margin), (0, 255, 0), 3)
|
| 572 |
+
cv2.rectangle(overlay, (margin, margin), (w - margin, h), (0, 255, 0), 2)
|
| 573 |
+
|
| 574 |
+
frame_with_boundaries = cv2.addWeighted(frame, 0.7, overlay, 0.3, 0)
|
| 575 |
+
|
| 576 |
+
corner_size = 30
|
| 577 |
+
cv2.line(frame_with_boundaries, (margin, margin), (margin + corner_size, margin), (0, 255, 0), 3)
|
| 578 |
+
cv2.line(frame_with_boundaries, (margin, margin), (margin, margin + corner_size), (0, 255, 0), 3)
|
| 579 |
+
|
| 580 |
+
cv2.line(frame_with_boundaries, (w - margin, margin), (w - margin - corner_size, margin), (0, 255, 0), 3)
|
| 581 |
+
cv2.line(frame_with_boundaries, (w - margin, margin), (w - margin, margin + corner_size), (0, 255, 0), 3)
|
| 582 |
+
|
| 583 |
+
cv2.putText(frame_with_boundaries, "Stay within GREEN boundaries",
|
| 584 |
+
(w//2 - 200, 30), cv2.FONT_HERSHEY_SIMPLEX, 0.7, (0, 255, 0), 2)
|
| 585 |
+
|
| 586 |
+
return frame_with_boundaries
|
| 587 |
+
|
| 588 |
+
def pre_test_setup_phase(self, ui_callbacks, timeout=60):
|
| 589 |
+
"""
|
| 590 |
+
ONE-TIME pre-test setup phase with environment scanning
|
| 591 |
+
"""
|
| 592 |
+
if self.position_adjusted:
|
| 593 |
+
return True
|
| 594 |
+
|
| 595 |
+
cap = cv2.VideoCapture(0)
|
| 596 |
+
if not cap.isOpened():
|
| 597 |
+
return False
|
| 598 |
+
|
| 599 |
+
start_time = time.time()
|
| 600 |
+
position_ok_counter = 0
|
| 601 |
+
required_stable_frames = 30
|
| 602 |
+
|
| 603 |
+
ui_callbacks['countdown_update']("πΈ ONE-TIME SETUP: Adjust your position within the GREEN frame")
|
| 604 |
+
|
| 605 |
+
while (time.time() - start_time) < timeout:
|
| 606 |
+
ret, frame = cap.read()
|
| 607 |
+
if not ret:
|
| 608 |
+
continue
|
| 609 |
+
|
| 610 |
+
rgb_frame = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB)
|
| 611 |
+
h, w = frame.shape[:2]
|
| 612 |
+
|
| 613 |
+
frame_with_boundaries = self.draw_frame_boundaries(frame)
|
| 614 |
+
|
| 615 |
+
face_box = None
|
| 616 |
+
is_ready = False
|
| 617 |
+
status_message = "Detecting face..."
|
| 618 |
+
status_color = (255, 165, 0)
|
| 619 |
+
|
| 620 |
+
if self.models['face_mesh'] is not None:
|
| 621 |
+
face_results = self.models['face_mesh'].process(rgb_frame)
|
| 622 |
+
|
| 623 |
+
if face_results.multi_face_landmarks:
|
| 624 |
+
num_faces = len(face_results.multi_face_landmarks)
|
| 625 |
+
|
| 626 |
+
if num_faces > 1:
|
| 627 |
+
status_message = "β οΈ Multiple faces detected! Only ONE person allowed"
|
| 628 |
+
status_color = (0, 0, 255)
|
| 629 |
+
position_ok_counter = 0
|
| 630 |
+
|
| 631 |
+
elif num_faces == 1:
|
| 632 |
+
face_landmarks = face_results.multi_face_landmarks[0]
|
| 633 |
+
|
| 634 |
+
landmarks_2d = np.array([(lm.x * w, lm.y * h) for lm in face_landmarks.landmark])
|
| 635 |
+
x_coords = landmarks_2d[:, 0]
|
| 636 |
+
y_coords = landmarks_2d[:, 1]
|
| 637 |
+
face_box = (int(np.min(x_coords)), int(np.min(y_coords)),
|
| 638 |
+
int(np.max(x_coords) - np.min(x_coords)),
|
| 639 |
+
int(np.max(y_coords) - np.min(y_coords)))
|
| 640 |
+
|
| 641 |
+
within_bounds, boundary_msg, boundary_status = self.check_frame_boundaries(frame, face_box)
|
| 642 |
+
|
| 643 |
+
outside_detected, obj_type, location = self.detect_person_outside_frame(frame)
|
| 644 |
+
|
| 645 |
+
if outside_detected:
|
| 646 |
+
status_message = f"β οΈ {obj_type.upper()} detected outside frame ({location} side)!"
|
| 647 |
+
status_color = (0, 0, 255)
|
| 648 |
+
position_ok_counter = 0
|
| 649 |
+
|
| 650 |
+
elif not within_bounds:
|
| 651 |
+
status_message = f"β οΈ {boundary_msg} - Please adjust!"
|
| 652 |
+
status_color = (0, 0, 255)
|
| 653 |
+
position_ok_counter = 0
|
| 654 |
+
|
| 655 |
+
if boundary_status == "LEFT_VIOLATION":
|
| 656 |
+
cv2.rectangle(frame_with_boundaries, (0, 0), (self.frame_margin, h), (0, 0, 255), -1)
|
| 657 |
+
elif boundary_status == "RIGHT_VIOLATION":
|
| 658 |
+
cv2.rectangle(frame_with_boundaries, (w - self.frame_margin, 0), (w, h), (0, 0, 255), -1)
|
| 659 |
+
elif boundary_status == "TOP_VIOLATION":
|
| 660 |
+
cv2.rectangle(frame_with_boundaries, (0, 0), (w, self.frame_margin), (0, 0, 255), -1)
|
| 661 |
+
|
| 662 |
+
else:
|
| 663 |
+
position_ok_counter += 1
|
| 664 |
+
progress = min(100, int((position_ok_counter / required_stable_frames) * 100))
|
| 665 |
+
status_message = f"β
Good position! Hold steady... {progress}%"
|
| 666 |
+
status_color = (0, 255, 0)
|
| 667 |
+
|
| 668 |
+
if position_ok_counter >= required_stable_frames:
|
| 669 |
+
is_ready = True
|
| 670 |
+
|
| 671 |
+
else:
|
| 672 |
+
status_message = "β No face detected - Please position yourself in frame"
|
| 673 |
+
status_color = (0, 0, 255)
|
| 674 |
+
position_ok_counter = 0
|
| 675 |
+
|
| 676 |
+
overlay_height = 140
|
| 677 |
+
overlay = frame_with_boundaries.copy()
|
| 678 |
+
cv2.rectangle(overlay, (0, h - overlay_height), (w, h), (0, 0, 0), -1)
|
| 679 |
+
frame_with_boundaries = cv2.addWeighted(frame_with_boundaries, 0.7, overlay, 0.3, 0)
|
| 680 |
+
|
| 681 |
+
cv2.putText(frame_with_boundaries, status_message, (10, h - 110),
|
| 682 |
+
cv2.FONT_HERSHEY_SIMPLEX, 0.6, status_color, 2)
|
| 683 |
+
|
| 684 |
+
cv2.putText(frame_with_boundaries, "Instructions:", (10, h - 80),
|
| 685 |
+
cv2.FONT_HERSHEY_SIMPLEX, 0.5, (255, 255, 255), 1)
|
| 686 |
+
cv2.putText(frame_with_boundaries, "β’ Keep your face within GREEN boundaries", (10, h - 60),
|
| 687 |
+
cv2.FONT_HERSHEY_SIMPLEX, 0.4, (255, 255, 255), 1)
|
| 688 |
+
cv2.putText(frame_with_boundaries, "β’ Ensure no one else is visible", (10, h - 40),
|
| 689 |
+
cv2.FONT_HERSHEY_SIMPLEX, 0.4, (255, 255, 255), 1)
|
| 690 |
+
cv2.putText(frame_with_boundaries, "β’ Remove all unauthorized items from view", (10, h - 20),
|
| 691 |
+
cv2.FONT_HERSHEY_SIMPLEX, 0.4, (255, 255, 255), 1)
|
| 692 |
+
|
| 693 |
+
ui_callbacks['video_update'](cv2.resize(frame_with_boundaries, (640, 480)))
|
| 694 |
+
|
| 695 |
+
elapsed = int(time.time() - start_time)
|
| 696 |
+
ui_callbacks['timer_update'](f"β±οΈ Setup time: {elapsed}s / {timeout}s")
|
| 697 |
+
|
| 698 |
+
if is_ready:
|
| 699 |
+
ui_callbacks['countdown_update']("π Scanning environment... Please stay still")
|
| 700 |
+
time.sleep(1)
|
| 701 |
+
|
| 702 |
+
baseline_frames = []
|
| 703 |
+
for _ in range(10):
|
| 704 |
+
ret, scan_frame = cap.read()
|
| 705 |
+
if ret:
|
| 706 |
+
baseline_frames.append(scan_frame)
|
| 707 |
+
time.sleep(0.1)
|
| 708 |
+
|
| 709 |
+
if baseline_frames:
|
| 710 |
+
self.baseline_environment = self.scan_environment(baseline_frames[len(baseline_frames)//2])
|
| 711 |
+
|
| 712 |
+
success_frame = frame_with_boundaries.copy()
|
| 713 |
+
cv2.rectangle(success_frame, (0, 0), (w, h), (0, 255, 0), 10)
|
| 714 |
+
cv2.putText(success_frame, "SETUP COMPLETE!",
|
| 715 |
+
(w//2 - 180, h//2 - 20), cv2.FONT_HERSHEY_SIMPLEX, 1.2, (0, 255, 0), 3)
|
| 716 |
+
cv2.putText(success_frame, "Test will begin shortly...",
|
| 717 |
+
(w//2 - 180, h//2 + 30), cv2.FONT_HERSHEY_SIMPLEX, 0.8, (0, 255, 0), 2)
|
| 718 |
+
ui_callbacks['video_update'](cv2.resize(success_frame, (640, 480)))
|
| 719 |
+
time.sleep(3)
|
| 720 |
+
|
| 721 |
+
cap.release()
|
| 722 |
+
ui_callbacks['countdown_update']('')
|
| 723 |
+
self.position_adjusted = True
|
| 724 |
+
return True
|
| 725 |
+
|
| 726 |
+
time.sleep(0.03)
|
| 727 |
+
|
| 728 |
+
cap.release()
|
| 729 |
+
ui_callbacks['countdown_update']('β οΈ Setup timeout - Please try again')
|
| 730 |
+
return False
|
| 731 |
+
|
| 732 |
+
def record_interview(self, question_data, duration, ui_callbacks):
|
| 733 |
+
"""
|
| 734 |
+
DEPRECATED: Use record_continuous_interview() instead
|
| 735 |
+
Kept for backward compatibility
|
| 736 |
+
"""
|
| 737 |
+
result = self.record_continuous_interview([question_data], duration, ui_callbacks)
|
| 738 |
+
|
| 739 |
+
if isinstance(result, dict) and 'questions_results' in result:
|
| 740 |
+
if result['questions_results']:
|
| 741 |
+
first_result = result['questions_results'][0]
|
| 742 |
+
first_result['video_path'] = result.get('session_video_path', '')
|
| 743 |
+
first_result['violation_detected'] = len(first_result.get('violations', [])) > 0
|
| 744 |
+
first_result['violation_reason'] = first_result['violations'][0]['reason'] if first_result.get('violations') else ''
|
| 745 |
+
return first_result
|
| 746 |
+
|
| 747 |
+
return {"error": "Recording failed"}
|
| 748 |
+
|
| 749 |
+
def record_audio_to_file(self, duration, path):
|
| 750 |
+
"""Record audio to WAV file"""
|
| 751 |
+
r = sr.Recognizer()
|
| 752 |
+
try:
|
| 753 |
+
with sr.Microphone() as source:
|
| 754 |
+
r.adjust_for_ambient_noise(source, duration=0.6)
|
| 755 |
+
audio = r.record(source, duration=duration)
|
| 756 |
+
with open(path, "wb") as f:
|
| 757 |
+
f.write(audio.get_wav_data())
|
| 758 |
+
return path
|
| 759 |
+
except:
|
| 760 |
+
return None
|
| 761 |
+
|
| 762 |
+
def transcribe_audio(self, path):
|
| 763 |
+
"""Transcribe audio file to text"""
|
| 764 |
+
r = sr.Recognizer()
|
| 765 |
+
try:
|
| 766 |
+
with sr.AudioFile(path) as source:
|
| 767 |
+
audio = r.record(source)
|
| 768 |
+
text = r.recognize_google(audio)
|
| 769 |
+
return text if text.strip() else "[Could not understand audio]"
|
| 770 |
+
except sr.UnknownValueError:
|
| 771 |
+
return "[Could not understand audio]"
|
| 772 |
+
except sr.RequestError:
|
| 773 |
+
return "[Speech recognition service unavailable]"
|
| 774 |
+
except:
|
| 775 |
+
return "[Could not understand audio]"
|
| 776 |
+
|
| 777 |
+
def record_continuous_interview(self, questions_list, duration_per_question, ui_callbacks):
|
| 778 |
+
"""
|
| 779 |
+
Record ALL questions continuously - continues even if violations occur
|
| 780 |
+
Captures violation images and stores them for display in results
|
| 781 |
+
"""
|
| 782 |
+
|
| 783 |
+
# ========== PRE-TEST SETUP ==========
|
| 784 |
+
ui_callbacks['status_update']("**π§ Initializing test environment...**")
|
| 785 |
+
setup_success = self.pre_test_setup_phase(ui_callbacks, timeout=90)
|
| 786 |
+
|
| 787 |
+
if not setup_success:
|
| 788 |
+
return {"error": "Setup phase failed or timeout"}
|
| 789 |
+
|
| 790 |
+
# ========== INSTRUCTIONS ==========
|
| 791 |
+
ui_callbacks['countdown_update']("β
Setup complete! Please read the instructions...")
|
| 792 |
+
ui_callbacks['status_update'](f"""
|
| 793 |
+
**π TEST INSTRUCTIONS:**
|
| 794 |
+
- You will answer **{len(questions_list)} questions** continuously
|
| 795 |
+
- Each question has **{duration_per_question} seconds** to answer
|
| 796 |
+
- **Important:** Even if a violation is detected, the interview will continue
|
| 797 |
+
- All violations will be reviewed at the end
|
| 798 |
+
- Stay within boundaries and maintain focus throughout
|
| 799 |
+
|
| 800 |
+
**The test will begin in 10 seconds...**
|
| 801 |
+
""")
|
| 802 |
+
time.sleep(10)
|
| 803 |
+
|
| 804 |
+
# ========== START RECORDING ==========
|
| 805 |
+
all_results = []
|
| 806 |
+
|
| 807 |
+
for i in range(3, 0, -1):
|
| 808 |
+
ui_callbacks['countdown_update'](f"π¬ Test starts in {i}...")
|
| 809 |
+
time.sleep(1)
|
| 810 |
+
ui_callbacks['countdown_update']('')
|
| 811 |
+
|
| 812 |
+
cap = cv2.VideoCapture(0)
|
| 813 |
+
if not cap.isOpened():
|
| 814 |
+
return {"error": "Unable to access camera"}
|
| 815 |
+
|
| 816 |
+
session_video_temp = tempfile.NamedTemporaryFile(delete=False, suffix=".avi")
|
| 817 |
+
session_video_path = session_video_temp.name
|
| 818 |
+
session_video_temp.close()
|
| 819 |
+
|
| 820 |
+
fourcc = cv2.VideoWriter_fourcc(*"XVID")
|
| 821 |
+
out = cv2.VideoWriter(session_video_path, fourcc, 15.0, (640, 480))
|
| 822 |
+
|
| 823 |
+
session_start_time = time.time()
|
| 824 |
+
session_violations = []
|
| 825 |
+
|
| 826 |
+
# ========== LOOP THROUGH ALL QUESTIONS ==========
|
| 827 |
+
for q_idx, question_data in enumerate(questions_list):
|
| 828 |
+
|
| 829 |
+
ui_callbacks['countdown_update'](f"π Question {q_idx + 1} of {len(questions_list)}")
|
| 830 |
+
|
| 831 |
+
question_text = question_data.get('question', 'No question text')
|
| 832 |
+
question_tip = question_data.get('tip', 'Speak clearly and confidently')
|
| 833 |
+
|
| 834 |
+
ui_callbacks['question_update'](q_idx + 1, question_text, question_tip)
|
| 835 |
+
|
| 836 |
+
ui_callbacks['status_update'](f"""
|
| 837 |
+
**β±οΈ Recording Question {q_idx + 1}**
|
| 838 |
+
|
| 839 |
+
Time to answer: **{duration_per_question} seconds**
|
| 840 |
+
""")
|
| 841 |
+
|
| 842 |
+
for i in range(3, 0, -1):
|
| 843 |
+
ui_callbacks['timer_update'](f"β±οΈ Starting in {i}s...")
|
| 844 |
+
time.sleep(1)
|
| 845 |
+
|
| 846 |
+
audio_temp = tempfile.NamedTemporaryFile(delete=False, suffix=".wav")
|
| 847 |
+
audio_path = audio_temp.name
|
| 848 |
+
audio_temp.close()
|
| 849 |
+
|
| 850 |
+
audio_thread = threading.Thread(
|
| 851 |
+
target=lambda path=audio_path: self.record_audio_to_file(duration_per_question, path),
|
| 852 |
+
daemon=True
|
| 853 |
+
)
|
| 854 |
+
audio_thread.start()
|
| 855 |
+
|
| 856 |
+
# Question recording state
|
| 857 |
+
question_start_time = time.time()
|
| 858 |
+
frames = []
|
| 859 |
+
question_violations = [] # Store violations for THIS question
|
| 860 |
+
|
| 861 |
+
no_face_start = None
|
| 862 |
+
look_away_start = None
|
| 863 |
+
|
| 864 |
+
eye_contact_frames = 0
|
| 865 |
+
total_frames = 0
|
| 866 |
+
blink_count = 0
|
| 867 |
+
prev_blink = False
|
| 868 |
+
|
| 869 |
+
face_box = None
|
| 870 |
+
|
| 871 |
+
# ========== RECORDING LOOP FOR THIS QUESTION ==========
|
| 872 |
+
while (time.time() - question_start_time) < duration_per_question:
|
| 873 |
+
ret, frame = cap.read()
|
| 874 |
+
if not ret:
|
| 875 |
+
break
|
| 876 |
+
|
| 877 |
+
out.write(frame)
|
| 878 |
+
frames.append(frame.copy())
|
| 879 |
+
rgb_frame = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB)
|
| 880 |
+
h, w, _ = frame.shape
|
| 881 |
+
total_frames += 1
|
| 882 |
+
|
| 883 |
+
lighting_status, brightness = self.analyze_lighting(frame)
|
| 884 |
+
|
| 885 |
+
num_faces = 0
|
| 886 |
+
looking_at_camera = False
|
| 887 |
+
attention_status = "No Face"
|
| 888 |
+
|
| 889 |
+
# ========== FACE DETECTION & VIOLATION CHECKS ==========
|
| 890 |
+
if self.models['face_mesh'] is not None:
|
| 891 |
+
face_results = self.models['face_mesh'].process(rgb_frame)
|
| 892 |
+
|
| 893 |
+
if face_results.multi_face_landmarks:
|
| 894 |
+
num_faces = len(face_results.multi_face_landmarks)
|
| 895 |
+
|
| 896 |
+
# Check multiple bodies
|
| 897 |
+
is_multi_body, multi_msg, body_count = self.detect_multiple_bodies(frame, num_faces)
|
| 898 |
+
|
| 899 |
+
if is_multi_body:
|
| 900 |
+
violation_img_path = self.save_violation_image(frame, q_idx + 1, multi_msg)
|
| 901 |
+
question_violations.append({
|
| 902 |
+
'reason': multi_msg,
|
| 903 |
+
'timestamp': time.time() - question_start_time,
|
| 904 |
+
'image_path': violation_img_path
|
| 905 |
+
})
|
| 906 |
+
# Continue to next question instead of breaking
|
| 907 |
+
break
|
| 908 |
+
|
| 909 |
+
if num_faces > 1:
|
| 910 |
+
violation_msg = f"Multiple persons detected ({num_faces} faces)"
|
| 911 |
+
violation_img_path = self.save_violation_image(frame, q_idx + 1, violation_msg)
|
| 912 |
+
question_violations.append({
|
| 913 |
+
'reason': violation_msg,
|
| 914 |
+
'timestamp': time.time() - question_start_time,
|
| 915 |
+
'image_path': violation_img_path
|
| 916 |
+
})
|
| 917 |
+
break
|
| 918 |
+
|
| 919 |
+
elif num_faces == 1:
|
| 920 |
+
no_face_start = None
|
| 921 |
+
face_landmarks = face_results.multi_face_landmarks[0]
|
| 922 |
+
|
| 923 |
+
try:
|
| 924 |
+
landmarks_2d = np.array([(lm.x * w, lm.y * h) for lm in face_landmarks.landmark])
|
| 925 |
+
x_coords = landmarks_2d[:, 0]
|
| 926 |
+
y_coords = landmarks_2d[:, 1]
|
| 927 |
+
face_box = (int(np.min(x_coords)), int(np.min(y_coords)),
|
| 928 |
+
int(np.max(x_coords) - np.min(x_coords)),
|
| 929 |
+
int(np.max(y_coords) - np.min(y_coords)))
|
| 930 |
+
|
| 931 |
+
# Check boundaries
|
| 932 |
+
within_bounds, boundary_msg, boundary_status = self.check_frame_boundaries(frame, face_box)
|
| 933 |
+
|
| 934 |
+
if not within_bounds:
|
| 935 |
+
violation_img_path = self.save_violation_image(frame, q_idx + 1, boundary_msg)
|
| 936 |
+
question_violations.append({
|
| 937 |
+
'reason': boundary_msg,
|
| 938 |
+
'timestamp': time.time() - question_start_time,
|
| 939 |
+
'image_path': violation_img_path
|
| 940 |
+
})
|
| 941 |
+
break
|
| 942 |
+
|
| 943 |
+
# Check person outside frame
|
| 944 |
+
outside_detected, obj_type, location = self.detect_person_outside_frame(frame)
|
| 945 |
+
|
| 946 |
+
if outside_detected:
|
| 947 |
+
violation_msg = f"{obj_type.upper()} detected outside frame ({location} side)"
|
| 948 |
+
violation_img_path = self.save_violation_image(frame, q_idx + 1, violation_msg)
|
| 949 |
+
question_violations.append({
|
| 950 |
+
'reason': violation_msg,
|
| 951 |
+
'timestamp': time.time() - question_start_time,
|
| 952 |
+
'image_path': violation_img_path
|
| 953 |
+
})
|
| 954 |
+
break
|
| 955 |
+
|
| 956 |
+
# Check intrusions
|
| 957 |
+
is_intrusion, intrusion_msg = self.detect_intrusion_at_edges(frame, face_box)
|
| 958 |
+
if is_intrusion:
|
| 959 |
+
violation_img_path = self.save_violation_image(frame, q_idx + 1, intrusion_msg)
|
| 960 |
+
question_violations.append({
|
| 961 |
+
'reason': intrusion_msg,
|
| 962 |
+
'timestamp': time.time() - question_start_time,
|
| 963 |
+
'image_path': violation_img_path
|
| 964 |
+
})
|
| 965 |
+
break
|
| 966 |
+
|
| 967 |
+
# Check hands outside
|
| 968 |
+
is_hand_violation, hand_msg = self.detect_hands_outside_main_person(frame, face_box)
|
| 969 |
+
if is_hand_violation:
|
| 970 |
+
violation_img_path = self.save_violation_image(frame, q_idx + 1, hand_msg)
|
| 971 |
+
question_violations.append({
|
| 972 |
+
'reason': hand_msg,
|
| 973 |
+
'timestamp': time.time() - question_start_time,
|
| 974 |
+
'image_path': violation_img_path
|
| 975 |
+
})
|
| 976 |
+
break
|
| 977 |
+
|
| 978 |
+
# Suspicious movements
|
| 979 |
+
is_suspicious, sus_msg = self.detect_suspicious_movements(frame)
|
| 980 |
+
if is_suspicious:
|
| 981 |
+
violation_img_path = self.save_violation_image(frame, q_idx + 1, sus_msg)
|
| 982 |
+
question_violations.append({
|
| 983 |
+
'reason': sus_msg,
|
| 984 |
+
'timestamp': time.time() - question_start_time,
|
| 985 |
+
'image_path': violation_img_path
|
| 986 |
+
})
|
| 987 |
+
break
|
| 988 |
+
|
| 989 |
+
yaw, pitch, roll = self.estimate_head_pose(face_landmarks, frame.shape)
|
| 990 |
+
gaze_centered = self.calculate_eye_gaze(face_landmarks, frame.shape)
|
| 991 |
+
|
| 992 |
+
is_blink = self.detect_blink(face_landmarks)
|
| 993 |
+
if is_blink and not prev_blink:
|
| 994 |
+
blink_count += 1
|
| 995 |
+
prev_blink = is_blink
|
| 996 |
+
|
| 997 |
+
head_looking_forward = abs(yaw) <= 20 and abs(pitch) <= 20
|
| 998 |
+
|
| 999 |
+
if head_looking_forward and gaze_centered:
|
| 1000 |
+
look_away_start = None
|
| 1001 |
+
looking_at_camera = True
|
| 1002 |
+
eye_contact_frames += 1
|
| 1003 |
+
attention_status = "Looking at Camera β"
|
| 1004 |
+
else:
|
| 1005 |
+
if look_away_start is None:
|
| 1006 |
+
look_away_start = time.time()
|
| 1007 |
+
attention_status = "Looking Away"
|
| 1008 |
+
else:
|
| 1009 |
+
elapsed = time.time() - look_away_start
|
| 1010 |
+
if elapsed > 2.0:
|
| 1011 |
+
violation_msg = "Looking away for >2 seconds"
|
| 1012 |
+
violation_img_path = self.save_violation_image(frame, q_idx + 1, violation_msg)
|
| 1013 |
+
question_violations.append({
|
| 1014 |
+
'reason': violation_msg,
|
| 1015 |
+
'timestamp': time.time() - question_start_time,
|
| 1016 |
+
'image_path': violation_img_path
|
| 1017 |
+
})
|
| 1018 |
+
break
|
| 1019 |
+
else:
|
| 1020 |
+
attention_status = f"Looking Away ({elapsed:.1f}s)"
|
| 1021 |
+
except:
|
| 1022 |
+
attention_status = "Face Error"
|
| 1023 |
+
else:
|
| 1024 |
+
if no_face_start is None:
|
| 1025 |
+
no_face_start = time.time()
|
| 1026 |
+
attention_status = "No Face Visible"
|
| 1027 |
+
else:
|
| 1028 |
+
elapsed = time.time() - no_face_start
|
| 1029 |
+
if elapsed > 2.0:
|
| 1030 |
+
violation_msg = "No face visible for >2 seconds"
|
| 1031 |
+
violation_img_path = self.save_violation_image(frame, q_idx + 1, violation_msg)
|
| 1032 |
+
question_violations.append({
|
| 1033 |
+
'reason': violation_msg,
|
| 1034 |
+
'timestamp': time.time() - question_start_time,
|
| 1035 |
+
'image_path': violation_img_path
|
| 1036 |
+
})
|
| 1037 |
+
break
|
| 1038 |
+
else:
|
| 1039 |
+
attention_status = f"No Face ({elapsed:.1f}s)"
|
| 1040 |
+
|
| 1041 |
+
# Check for new objects
|
| 1042 |
+
if total_frames % 20 == 0:
|
| 1043 |
+
new_detected, new_items = self.detect_new_objects(frame)
|
| 1044 |
+
if new_detected:
|
| 1045 |
+
violation_msg = f"New item(s) brought into view: {', '.join(new_items)}"
|
| 1046 |
+
violation_img_path = self.save_violation_image(frame, q_idx + 1, violation_msg)
|
| 1047 |
+
question_violations.append({
|
| 1048 |
+
'reason': violation_msg,
|
| 1049 |
+
'timestamp': time.time() - question_start_time,
|
| 1050 |
+
'image_path': violation_img_path
|
| 1051 |
+
})
|
| 1052 |
+
break
|
| 1053 |
+
|
| 1054 |
+
# Display frame
|
| 1055 |
+
overlay = frame.copy()
|
| 1056 |
+
cv2.rectangle(overlay, (0, 0), (w, 120), (0, 0, 0), -1)
|
| 1057 |
+
frame_display = cv2.addWeighted(frame, 0.6, overlay, 0.4, 0)
|
| 1058 |
+
|
| 1059 |
+
# Show violation warning if any occurred
|
| 1060 |
+
status_color = (0, 255, 0) if len(question_violations) == 0 else (0, 165, 255)
|
| 1061 |
+
violation_text = f" | β οΈ {len(question_violations)} violation(s)" if question_violations else ""
|
| 1062 |
+
|
| 1063 |
+
cv2.putText(frame_display, f"Q{q_idx+1}/{len(questions_list)} - {attention_status}{violation_text}", (10, 30),
|
| 1064 |
+
cv2.FONT_HERSHEY_SIMPLEX, 0.6, status_color, 2)
|
| 1065 |
+
cv2.putText(frame_display, f"Lighting: {lighting_status}", (10, 60),
|
| 1066 |
+
cv2.FONT_HERSHEY_SIMPLEX, 0.5, (255, 255, 255), 2)
|
| 1067 |
+
cv2.putText(frame_display, f"Eye Contact: {int((eye_contact_frames/max(total_frames,1))*100)}%", (10, 90),
|
| 1068 |
+
cv2.FONT_HERSHEY_SIMPLEX, 0.5, (255, 255, 255), 2)
|
| 1069 |
+
|
| 1070 |
+
elapsed_q = time.time() - question_start_time
|
| 1071 |
+
remaining = max(0, int(duration_per_question - elapsed_q))
|
| 1072 |
+
cv2.putText(frame_display, f"Time: {remaining}s", (10, 115),
|
| 1073 |
+
cv2.FONT_HERSHEY_SIMPLEX, 0.5, (255, 255, 255), 2)
|
| 1074 |
+
|
| 1075 |
+
ui_callbacks['video_update'](cv2.resize(frame_display, (480, 360)))
|
| 1076 |
+
|
| 1077 |
+
eye_contact_pct = (eye_contact_frames / max(total_frames, 1)) * 100
|
| 1078 |
+
status_text = f"""
|
| 1079 |
+
**Question {q_idx + 1} of {len(questions_list)}**
|
| 1080 |
+
|
| 1081 |
+
ποΈ **Eye Contact:** {eye_contact_pct:.1f}%
|
| 1082 |
+
π΄ **Blinks:** {blink_count}
|
| 1083 |
+
π‘ **Lighting:** {lighting_status}
|
| 1084 |
+
β οΈ **Status:** {attention_status}
|
| 1085 |
+
"""
|
| 1086 |
+
|
| 1087 |
+
if question_violations:
|
| 1088 |
+
status_text += f"\n\nβ οΈ **Violations in this question:** {len(question_violations)}"
|
| 1089 |
+
|
| 1090 |
+
ui_callbacks['status_update'](status_text)
|
| 1091 |
+
|
| 1092 |
+
overall_progress = (q_idx + (elapsed_q / duration_per_question)) / len(questions_list)
|
| 1093 |
+
overall_progress = max(0.0, min(1.0, overall_progress))
|
| 1094 |
+
ui_callbacks['progress_update'](overall_progress)
|
| 1095 |
+
ui_callbacks['timer_update'](f"π₯ Q{q_idx+1}/{len(questions_list)} - {remaining}s remaining")
|
| 1096 |
+
|
| 1097 |
+
time.sleep(0.05)
|
| 1098 |
+
|
| 1099 |
+
# Wait for audio
|
| 1100 |
+
audio_thread.join(timeout=duration_per_question + 5)
|
| 1101 |
+
|
| 1102 |
+
# Transcribe
|
| 1103 |
+
transcript = ""
|
| 1104 |
+
if os.path.exists(audio_path):
|
| 1105 |
+
transcript = self.transcribe_audio(audio_path)
|
| 1106 |
+
|
| 1107 |
+
# Add violations to session list
|
| 1108 |
+
if question_violations:
|
| 1109 |
+
session_violations.extend([f"Q{q_idx+1}: {v['reason']}" for v in question_violations])
|
| 1110 |
+
|
| 1111 |
+
# Store results for this question
|
| 1112 |
+
question_result = {
|
| 1113 |
+
'question_number': q_idx + 1,
|
| 1114 |
+
'question_text': question_data.get('question', ''),
|
| 1115 |
+
'audio_path': audio_path,
|
| 1116 |
+
'frames': frames,
|
| 1117 |
+
'violations': question_violations, # Now includes image paths
|
| 1118 |
+
'violation_detected': len(question_violations) > 0,
|
| 1119 |
+
'eye_contact_pct': (eye_contact_frames / max(total_frames, 1)) * 100,
|
| 1120 |
+
'blink_count': blink_count,
|
| 1121 |
+
'face_box': face_box,
|
| 1122 |
+
'transcript': transcript,
|
| 1123 |
+
'lighting_status': lighting_status
|
| 1124 |
+
}
|
| 1125 |
+
|
| 1126 |
+
all_results.append(question_result)
|
| 1127 |
+
|
| 1128 |
+
# Show message and continue to next question
|
| 1129 |
+
if question_violations:
|
| 1130 |
+
ui_callbacks['countdown_update'](f"β οΈ Violation detected in Q{q_idx + 1}! Continuing to next question in 3s...")
|
| 1131 |
+
time.sleep(3)
|
| 1132 |
+
elif q_idx < len(questions_list) - 1:
|
| 1133 |
+
ui_callbacks['countdown_update'](f"β
Question {q_idx + 1} complete! Next question in 3s...")
|
| 1134 |
+
time.sleep(3)
|
| 1135 |
+
|
| 1136 |
+
# Cleanup
|
| 1137 |
+
cap.release()
|
| 1138 |
+
out.release()
|
| 1139 |
+
|
| 1140 |
+
# Clear UI
|
| 1141 |
+
ui_callbacks['video_update'](None)
|
| 1142 |
+
ui_callbacks['progress_update'](1.0)
|
| 1143 |
+
|
| 1144 |
+
# Final message
|
| 1145 |
+
total_violations = sum(len(r.get('violations', [])) for r in all_results)
|
| 1146 |
+
|
| 1147 |
+
if total_violations > 0:
|
| 1148 |
+
ui_callbacks['countdown_update'](f"β οΈ TEST COMPLETED WITH {total_violations} VIOLATION(S)")
|
| 1149 |
+
ui_callbacks['status_update'](f"**β οΈ {total_violations} violation(s) detected across all questions. Review results below.**")
|
| 1150 |
+
else:
|
| 1151 |
+
ui_callbacks['countdown_update']("β
TEST COMPLETED SUCCESSFULLY!")
|
| 1152 |
+
ui_callbacks['status_update']("**All questions answered with no violations. Processing results...**")
|
| 1153 |
+
|
| 1154 |
+
ui_callbacks['timer_update']("")
|
| 1155 |
+
|
| 1156 |
+
# Return comprehensive results
|
| 1157 |
+
return {
|
| 1158 |
+
'questions_results': all_results,
|
| 1159 |
+
'session_video_path': session_video_path,
|
| 1160 |
+
'total_questions': len(questions_list),
|
| 1161 |
+
'completed_questions': len(all_results),
|
| 1162 |
+
'session_violations': session_violations,
|
| 1163 |
+
'total_violations': total_violations,
|
| 1164 |
+
'violation_images_dir': self.violation_images_dir,
|
| 1165 |
+
'session_duration': time.time() - session_start_time
|
| 1166 |
+
}
|
| 1167 |
+
|
| 1168 |
+
####
|
| 1169 |
+
|
analysis_system.py
ADDED
|
@@ -0,0 +1,868 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
|
| 2 |
+
"""
|
| 3 |
+
Multi-Modal Analysis System - PERFORMANCE OPTIMIZED
|
| 4 |
+
FIXED: LanguageTool now uses singleton pattern to prevent repeated downloads
|
| 5 |
+
"""
|
| 6 |
+
|
| 7 |
+
import cv2
|
| 8 |
+
import numpy as np
|
| 9 |
+
import pandas as pd
|
| 10 |
+
from deepface import DeepFace
|
| 11 |
+
import warnings
|
| 12 |
+
from contextlib import contextmanager
|
| 13 |
+
import string
|
| 14 |
+
import os
|
| 15 |
+
import re
|
| 16 |
+
import difflib
|
| 17 |
+
|
| 18 |
+
warnings.filterwarnings('ignore')
|
| 19 |
+
|
| 20 |
+
# Try importing fluency-related libraries
|
| 21 |
+
try:
|
| 22 |
+
import librosa
|
| 23 |
+
LIBROSA_AVAILABLE = True
|
| 24 |
+
except:
|
| 25 |
+
LIBROSA_AVAILABLE = False
|
| 26 |
+
|
| 27 |
+
try:
|
| 28 |
+
import language_tool_python
|
| 29 |
+
LANGUAGE_TOOL_AVAILABLE = True
|
| 30 |
+
except:
|
| 31 |
+
LANGUAGE_TOOL_AVAILABLE = False
|
| 32 |
+
|
| 33 |
+
try:
|
| 34 |
+
import spacy
|
| 35 |
+
SPACY_AVAILABLE = True
|
| 36 |
+
try:
|
| 37 |
+
nlp = spacy.load("en_core_web_sm")
|
| 38 |
+
except:
|
| 39 |
+
nlp = None
|
| 40 |
+
except:
|
| 41 |
+
SPACY_AVAILABLE = False
|
| 42 |
+
nlp = None
|
| 43 |
+
|
| 44 |
+
try:
|
| 45 |
+
from transformers import pipeline
|
| 46 |
+
TRANSFORMERS_AVAILABLE = True
|
| 47 |
+
except:
|
| 48 |
+
TRANSFORMERS_AVAILABLE = False
|
| 49 |
+
|
| 50 |
+
try:
|
| 51 |
+
from nltk.tokenize import word_tokenize
|
| 52 |
+
from nltk.corpus import stopwords
|
| 53 |
+
NLTK_AVAILABLE = True
|
| 54 |
+
except:
|
| 55 |
+
NLTK_AVAILABLE = False
|
| 56 |
+
|
| 57 |
+
# Constants
|
| 58 |
+
STOPWORDS = {
|
| 59 |
+
"the", "and", "a", "an", "in", "on", "of", "to", "is", "are", "was", "were",
|
| 60 |
+
"it", "that", "this", "these", "those", "for", "with", "as", "by", "be", "or",
|
| 61 |
+
"from", "which", "what", "when", "how", "why", "do", "does", "did", "have",
|
| 62 |
+
"has", "had", "will", "would", "could", "should", "can", "may", "might", "must",
|
| 63 |
+
"i", "you", "he", "she", "we", "they", "me", "him", "her", "us", "them",
|
| 64 |
+
"my", "your", "his", "her", "its", "our", "their"
|
| 65 |
+
}
|
| 66 |
+
|
| 67 |
+
FILLER_WORDS = {"um", "uh", "like", "you know", "ah", "erm", "so", "actually", "basically"}
|
| 68 |
+
|
| 69 |
+
# Optimal WPM ranges for interviews
|
| 70 |
+
OPTIMAL_WPM_MIN = 140
|
| 71 |
+
OPTIMAL_WPM_MAX = 160
|
| 72 |
+
SLOW_WPM_THRESHOLD = 120
|
| 73 |
+
FAST_WPM_THRESHOLD = 180
|
| 74 |
+
|
| 75 |
+
# CRITICAL FIX: Global singleton grammar checker to prevent repeated downloads
|
| 76 |
+
_GRAMMAR_CHECKER_INSTANCE = None
|
| 77 |
+
_GRAMMAR_CHECKER_INITIALIZED = False
|
| 78 |
+
|
| 79 |
+
def get_grammar_checker():
|
| 80 |
+
"""
|
| 81 |
+
Get or create singleton grammar checker instance
|
| 82 |
+
PREVENTS REPEATED 254MB DOWNLOADS!
|
| 83 |
+
"""
|
| 84 |
+
global _GRAMMAR_CHECKER_INSTANCE, _GRAMMAR_CHECKER_INITIALIZED
|
| 85 |
+
|
| 86 |
+
if _GRAMMAR_CHECKER_INITIALIZED:
|
| 87 |
+
return _GRAMMAR_CHECKER_INSTANCE
|
| 88 |
+
|
| 89 |
+
if LANGUAGE_TOOL_AVAILABLE:
|
| 90 |
+
try:
|
| 91 |
+
# Set persistent cache directory
|
| 92 |
+
cache_dir = os.path.join(os.path.expanduser("~"), ".cache", "language_tool_python")
|
| 93 |
+
os.makedirs(cache_dir, exist_ok=True)
|
| 94 |
+
|
| 95 |
+
# Initialize with caching enabled
|
| 96 |
+
_GRAMMAR_CHECKER_INSTANCE = language_tool_python.LanguageTool(
|
| 97 |
+
'en-US',
|
| 98 |
+
config={
|
| 99 |
+
'cacheSize': 1000,
|
| 100 |
+
'maxCheckThreads': 2
|
| 101 |
+
}
|
| 102 |
+
)
|
| 103 |
+
print("β
Grammar checker initialized (singleton - will not re-download)")
|
| 104 |
+
_GRAMMAR_CHECKER_INITIALIZED = True
|
| 105 |
+
return _GRAMMAR_CHECKER_INSTANCE
|
| 106 |
+
except Exception as e:
|
| 107 |
+
print(f"β οΈ Grammar checker init failed: {e}")
|
| 108 |
+
_GRAMMAR_CHECKER_INITIALIZED = True
|
| 109 |
+
return None
|
| 110 |
+
|
| 111 |
+
_GRAMMAR_CHECKER_INITIALIZED = True
|
| 112 |
+
return None
|
| 113 |
+
|
| 114 |
+
class AnalysisSystem:
|
| 115 |
+
"""Handles multi-modal analysis with OPTIMIZED performance"""
|
| 116 |
+
|
| 117 |
+
def __init__(self, models_dict):
|
| 118 |
+
"""Initialize analysis system with loaded models"""
|
| 119 |
+
self.models = models_dict
|
| 120 |
+
|
| 121 |
+
# PERFORMANCE: Use singleton grammar checker (prevents re-downloads)
|
| 122 |
+
self.grammar_checker = get_grammar_checker()
|
| 123 |
+
|
| 124 |
+
# PERFORMANCE: Initialize BERT only if really needed
|
| 125 |
+
self.coherence_model = None
|
| 126 |
+
self._bert_initialized = False
|
| 127 |
+
|
| 128 |
+
def _lazy_init_bert(self):
|
| 129 |
+
"""Lazy initialization of BERT model - only when first needed"""
|
| 130 |
+
if not self._bert_initialized and TRANSFORMERS_AVAILABLE:
|
| 131 |
+
try:
|
| 132 |
+
self.coherence_model = pipeline(
|
| 133 |
+
"text-classification",
|
| 134 |
+
model="textattack/bert-base-uncased-ag-news",
|
| 135 |
+
device=-1
|
| 136 |
+
)
|
| 137 |
+
print("β
BERT coherence model loaded")
|
| 138 |
+
except:
|
| 139 |
+
self.coherence_model = None
|
| 140 |
+
self._bert_initialized = True
|
| 141 |
+
|
| 142 |
+
@contextmanager
|
| 143 |
+
def suppress_warnings(self):
|
| 144 |
+
"""Context manager to suppress warnings"""
|
| 145 |
+
with warnings.catch_warnings():
|
| 146 |
+
warnings.simplefilter("ignore")
|
| 147 |
+
yield
|
| 148 |
+
|
| 149 |
+
# ... [Keep ALL your other methods from the original analysis_system.py]
|
| 150 |
+
# The only change is the grammar checker initialization above
|
| 151 |
+
|
| 152 |
+
# For brevity, I'm showing just the structure. Copy all your methods:
|
| 153 |
+
# - clean_text
|
| 154 |
+
# - tokenize
|
| 155 |
+
# - tokenize_meaningful
|
| 156 |
+
# - count_filler_words
|
| 157 |
+
# - estimate_face_quality
|
| 158 |
+
# - analyze_frame_emotion
|
| 159 |
+
# - aggregate_emotions
|
| 160 |
+
# - analyze_emotions_batch
|
| 161 |
+
# - fuse_emotions
|
| 162 |
+
# - is_valid_transcript
|
| 163 |
+
# - compute_speech_rate
|
| 164 |
+
# - normalize_speech_rate
|
| 165 |
+
# - detect_pauses
|
| 166 |
+
# - check_grammar (uses self.grammar_checker which is now singleton)
|
| 167 |
+
# - compute_lexical_diversity
|
| 168 |
+
# - compute_coherence_score
|
| 169 |
+
# - content_similarity
|
| 170 |
+
# - evaluate_fluency_comprehensive
|
| 171 |
+
# - evaluate_answer_accuracy
|
| 172 |
+
# - compute_wpm
|
| 173 |
+
# - analyze_outfit
|
| 174 |
+
# - analyze_recording
|
| 175 |
+
|
| 176 |
+
def check_grammar(self, text):
|
| 177 |
+
"""Check grammar - OPTIMIZED with singleton checker"""
|
| 178 |
+
if not self.is_valid_transcript(text) or self.grammar_checker is None:
|
| 179 |
+
return 100.0, 0
|
| 180 |
+
|
| 181 |
+
try:
|
| 182 |
+
# PERFORMANCE: Limit text length for grammar checking
|
| 183 |
+
max_chars = 1000
|
| 184 |
+
if len(text) > max_chars:
|
| 185 |
+
text = text[:max_chars]
|
| 186 |
+
|
| 187 |
+
matches = self.grammar_checker.check(text)
|
| 188 |
+
error_count = len(matches)
|
| 189 |
+
text_length = len(text.split())
|
| 190 |
+
|
| 191 |
+
if text_length == 0:
|
| 192 |
+
grammar_score = 0
|
| 193 |
+
else:
|
| 194 |
+
grammar_score = max(0, 100 - (error_count / text_length * 100))
|
| 195 |
+
|
| 196 |
+
return round(grammar_score, 1), error_count
|
| 197 |
+
except:
|
| 198 |
+
return 100.0, 0
|
| 199 |
+
|
| 200 |
+
def is_valid_transcript(self, text):
|
| 201 |
+
"""Check if transcript is valid"""
|
| 202 |
+
if not text or not text.strip():
|
| 203 |
+
return False
|
| 204 |
+
invalid_markers = ["[Could not understand audio]", "[Speech recognition service unavailable]",
|
| 205 |
+
"[Error", "[No audio]", "Audio not clear"]
|
| 206 |
+
return not any(marker in text for marker in invalid_markers)
|
| 207 |
+
|
| 208 |
+
# NOTE: Copy ALL other methods from your original analysis_system.py file
|
| 209 |
+
# The key fix is using the singleton grammar checker to prevent repeated downloads
|
| 210 |
+
def clean_text(self, text):
|
| 211 |
+
"""Clean text for analysis"""
|
| 212 |
+
text = text.lower()
|
| 213 |
+
text = re.sub(r'[^\w\s]', '', text)
|
| 214 |
+
|
| 215 |
+
if NLTK_AVAILABLE:
|
| 216 |
+
try:
|
| 217 |
+
tokens = word_tokenize(text)
|
| 218 |
+
tokens = [word for word in tokens if word not in stopwords.words('english')]
|
| 219 |
+
return tokens
|
| 220 |
+
except:
|
| 221 |
+
pass
|
| 222 |
+
|
| 223 |
+
words = text.split()
|
| 224 |
+
return [w for w in words if w.lower() not in STOPWORDS]
|
| 225 |
+
|
| 226 |
+
def tokenize(self, text):
|
| 227 |
+
"""Tokenize text into words"""
|
| 228 |
+
words = [w.strip(string.punctuation).lower()
|
| 229 |
+
for w in text.split()
|
| 230 |
+
if w.strip(string.punctuation)]
|
| 231 |
+
return words
|
| 232 |
+
|
| 233 |
+
def tokenize_meaningful(self, text):
|
| 234 |
+
"""Tokenize and filter out stopwords"""
|
| 235 |
+
words = self.tokenize(text)
|
| 236 |
+
meaningful_words = [w for w in words if w.lower() not in STOPWORDS and len(w) > 2]
|
| 237 |
+
return meaningful_words
|
| 238 |
+
|
| 239 |
+
def count_filler_words(self, text):
|
| 240 |
+
"""Count filler words - ACCURATE"""
|
| 241 |
+
if not self.is_valid_transcript(text):
|
| 242 |
+
return 0, 0.0
|
| 243 |
+
|
| 244 |
+
text_lower = text.lower()
|
| 245 |
+
filler_count = 0
|
| 246 |
+
|
| 247 |
+
for filler in FILLER_WORDS:
|
| 248 |
+
filler_count += text_lower.count(filler)
|
| 249 |
+
|
| 250 |
+
total_words = len(self.tokenize(text))
|
| 251 |
+
filler_ratio = (filler_count / total_words) if total_words > 0 else 0.0
|
| 252 |
+
|
| 253 |
+
return filler_count, round(filler_ratio, 3)
|
| 254 |
+
|
| 255 |
+
# ==================== FACIAL ANALYSIS (OPTIMIZED) ====================
|
| 256 |
+
|
| 257 |
+
def estimate_face_quality(self, frame_bgr, face_bbox=None):
|
| 258 |
+
"""Estimate face quality - OPTIMIZED with early returns"""
|
| 259 |
+
h, w = frame_bgr.shape[:2]
|
| 260 |
+
frame_area = h * w
|
| 261 |
+
|
| 262 |
+
quality_score = 1.0
|
| 263 |
+
|
| 264 |
+
if face_bbox:
|
| 265 |
+
x, y, fw, fh = face_bbox
|
| 266 |
+
face_area = fw * fh
|
| 267 |
+
size_ratio = face_area / frame_area
|
| 268 |
+
|
| 269 |
+
# PERFORMANCE: Quick size check
|
| 270 |
+
if 0.15 <= size_ratio <= 0.35:
|
| 271 |
+
size_score = 1.0
|
| 272 |
+
elif size_ratio < 0.15:
|
| 273 |
+
size_score = size_ratio / 0.15
|
| 274 |
+
else:
|
| 275 |
+
size_score = max(0.3, 1.0 - (size_ratio - 0.35))
|
| 276 |
+
|
| 277 |
+
quality_score *= size_score
|
| 278 |
+
|
| 279 |
+
# Centrality factor
|
| 280 |
+
face_center_x = x + fw / 2
|
| 281 |
+
face_center_y = y + fh / 2
|
| 282 |
+
frame_center_x = w / 2
|
| 283 |
+
frame_center_y = h / 2
|
| 284 |
+
|
| 285 |
+
x_deviation = abs(face_center_x - frame_center_x) / (w / 2)
|
| 286 |
+
y_deviation = abs(face_center_y - frame_center_y) / (h / 2)
|
| 287 |
+
centrality_score = 1.0 - (x_deviation + y_deviation) / 2
|
| 288 |
+
|
| 289 |
+
quality_score *= max(0.5, centrality_score)
|
| 290 |
+
|
| 291 |
+
# Lighting quality
|
| 292 |
+
gray = cv2.cvtColor(frame_bgr, cv2.COLOR_BGR2GRAY)
|
| 293 |
+
|
| 294 |
+
if face_bbox:
|
| 295 |
+
x, y, fw, fh = face_bbox
|
| 296 |
+
face_region = gray[max(0, y):min(h, y+fh), max(0, x):min(w, x+fw)]
|
| 297 |
+
else:
|
| 298 |
+
face_region = gray
|
| 299 |
+
|
| 300 |
+
if face_region.size > 0:
|
| 301 |
+
mean_brightness = np.mean(face_region)
|
| 302 |
+
std_brightness = np.std(face_region)
|
| 303 |
+
|
| 304 |
+
if 80 <= mean_brightness <= 180:
|
| 305 |
+
brightness_score = 1.0
|
| 306 |
+
elif mean_brightness < 80:
|
| 307 |
+
brightness_score = mean_brightness / 80
|
| 308 |
+
else:
|
| 309 |
+
brightness_score = max(0.3, 1.0 - (mean_brightness - 180) / 75)
|
| 310 |
+
|
| 311 |
+
contrast_score = min(1.0, std_brightness / 40)
|
| 312 |
+
quality_score *= (brightness_score * 0.7 + contrast_score * 0.3)
|
| 313 |
+
|
| 314 |
+
return max(0.1, min(1.0, quality_score))
|
| 315 |
+
|
| 316 |
+
def analyze_frame_emotion(self, frame_bgr):
|
| 317 |
+
"""Analyze emotions - OPTIMIZED with smaller resize"""
|
| 318 |
+
try:
|
| 319 |
+
with self.suppress_warnings():
|
| 320 |
+
# PERFORMANCE: Smaller resize (was 480x360, now 320x240)
|
| 321 |
+
small = cv2.resize(frame_bgr, (320, 240))
|
| 322 |
+
res = DeepFace.analyze(small, actions=['emotion'], enforce_detection=False)
|
| 323 |
+
if isinstance(res, list):
|
| 324 |
+
res = res[0]
|
| 325 |
+
|
| 326 |
+
emotions = res.get('emotion', {})
|
| 327 |
+
|
| 328 |
+
face_bbox = None
|
| 329 |
+
if 'region' in res:
|
| 330 |
+
region = res['region']
|
| 331 |
+
face_bbox = (region['x'], region['y'], region['w'], region['h'])
|
| 332 |
+
|
| 333 |
+
quality = self.estimate_face_quality(small, face_bbox)
|
| 334 |
+
|
| 335 |
+
return emotions, quality
|
| 336 |
+
except:
|
| 337 |
+
return {}, 0.0
|
| 338 |
+
|
| 339 |
+
def aggregate_emotions(self, emotion_quality_list):
|
| 340 |
+
"""Aggregate emotions with quality weighting"""
|
| 341 |
+
if not emotion_quality_list:
|
| 342 |
+
return {}
|
| 343 |
+
|
| 344 |
+
emotions_list = [e for e, q in emotion_quality_list]
|
| 345 |
+
qualities = [q for e, q in emotion_quality_list]
|
| 346 |
+
|
| 347 |
+
if not emotions_list or sum(qualities) == 0:
|
| 348 |
+
return {}
|
| 349 |
+
|
| 350 |
+
df = pd.DataFrame(emotions_list).fillna(0)
|
| 351 |
+
|
| 352 |
+
for col in df.columns:
|
| 353 |
+
df[col] = df[col] * qualities
|
| 354 |
+
|
| 355 |
+
total_weight = sum(qualities)
|
| 356 |
+
avg = (df.sum() / total_weight).to_dict()
|
| 357 |
+
|
| 358 |
+
mapped = {
|
| 359 |
+
'Confident': avg.get('happy', 0) * 0.6 + avg.get('neutral', 0) * 0.3 + avg.get('surprise', 0) * 0.1,
|
| 360 |
+
'Nervous': avg.get('fear', 0) * 0.8 + avg.get('sad', 0) * 0.2,
|
| 361 |
+
'Engaged': avg.get('surprise', 0) * 0.6 + avg.get('happy', 0) * 0.4,
|
| 362 |
+
'Neutral': avg.get('neutral', 0)
|
| 363 |
+
}
|
| 364 |
+
|
| 365 |
+
total = sum(mapped.values()) or 1
|
| 366 |
+
return {k: (v / total) * 100 for k, v in mapped.items()}
|
| 367 |
+
|
| 368 |
+
def analyze_emotions_batch(self, frames, sample_every=8):
|
| 369 |
+
"""Analyze emotions - OPTIMIZED: Increased sampling interval"""
|
| 370 |
+
# PERFORMANCE: Sample every 10 frames instead of 8 (20% faster)
|
| 371 |
+
emotion_quality_pairs = []
|
| 372 |
+
sample_interval = max(10, sample_every) # At least every 10 frames
|
| 373 |
+
|
| 374 |
+
for i in range(0, len(frames), sample_interval):
|
| 375 |
+
if i < len(frames):
|
| 376 |
+
emotion, quality = self.analyze_frame_emotion(frames[i])
|
| 377 |
+
if emotion:
|
| 378 |
+
emotion_quality_pairs.append((emotion, quality))
|
| 379 |
+
|
| 380 |
+
return self.aggregate_emotions(emotion_quality_pairs)
|
| 381 |
+
|
| 382 |
+
def fuse_emotions(self, face_emotions, has_valid_data=True):
|
| 383 |
+
"""Fuse and categorize emotions"""
|
| 384 |
+
if not has_valid_data or not face_emotions:
|
| 385 |
+
return {
|
| 386 |
+
'Confident': 0.0,
|
| 387 |
+
'Nervous': 0.0,
|
| 388 |
+
'Engaged': 0.0,
|
| 389 |
+
'Neutral': 0.0
|
| 390 |
+
}, {
|
| 391 |
+
"confidence": 0.0,
|
| 392 |
+
"confidence_label": "No Data",
|
| 393 |
+
"nervousness": 0.0,
|
| 394 |
+
"nervous_label": "No Data"
|
| 395 |
+
}
|
| 396 |
+
|
| 397 |
+
fused = {k: face_emotions.get(k, 0) for k in ['Confident', 'Nervous', 'Engaged', 'Neutral']}
|
| 398 |
+
|
| 399 |
+
confidence = round(fused['Confident'], 1)
|
| 400 |
+
nervousness = round(fused['Nervous'], 1)
|
| 401 |
+
|
| 402 |
+
def categorize(value, type_):
|
| 403 |
+
if type_ == "conf":
|
| 404 |
+
if value < 40: return "Low"
|
| 405 |
+
elif value < 70: return "Moderate"
|
| 406 |
+
else: return "High"
|
| 407 |
+
else:
|
| 408 |
+
if value < 25: return "Calm"
|
| 409 |
+
elif value < 50: return "Slightly Nervous"
|
| 410 |
+
else: return "Very Nervous"
|
| 411 |
+
|
| 412 |
+
return fused, {
|
| 413 |
+
"confidence": confidence,
|
| 414 |
+
"confidence_label": categorize(confidence, "conf"),
|
| 415 |
+
"nervousness": nervousness,
|
| 416 |
+
"nervous_label": categorize(nervousness, "nerv")
|
| 417 |
+
}
|
| 418 |
+
|
| 419 |
+
# ==================== FLUENCY ANALYSIS (OPTIMIZED) ====================
|
| 420 |
+
|
| 421 |
+
def is_valid_transcript(self, text):
|
| 422 |
+
"""Check if transcript is valid"""
|
| 423 |
+
if not text or not text.strip():
|
| 424 |
+
return False
|
| 425 |
+
invalid_markers = ["[Could not understand audio]", "[Speech recognition service unavailable]",
|
| 426 |
+
"[Error", "[No audio]", "Audio not clear"]
|
| 427 |
+
return not any(marker in text for marker in invalid_markers)
|
| 428 |
+
|
| 429 |
+
def compute_speech_rate(self, text, duration_seconds):
|
| 430 |
+
"""Compute speech rate (WPM)"""
|
| 431 |
+
if not self.is_valid_transcript(text) or duration_seconds <= 0:
|
| 432 |
+
return 0.0
|
| 433 |
+
|
| 434 |
+
words = text.strip().split()
|
| 435 |
+
wpm = (len(words) / duration_seconds) * 60
|
| 436 |
+
return round(wpm, 1)
|
| 437 |
+
|
| 438 |
+
def normalize_speech_rate(self, wpm):
|
| 439 |
+
"""Normalize speech rate"""
|
| 440 |
+
if wpm == 0:
|
| 441 |
+
return 0.0
|
| 442 |
+
|
| 443 |
+
if OPTIMAL_WPM_MIN <= wpm <= OPTIMAL_WPM_MAX:
|
| 444 |
+
return 1.0
|
| 445 |
+
elif SLOW_WPM_THRESHOLD <= wpm < OPTIMAL_WPM_MIN:
|
| 446 |
+
return 0.7 + 0.3 * (wpm - SLOW_WPM_THRESHOLD) / (OPTIMAL_WPM_MIN - SLOW_WPM_THRESHOLD)
|
| 447 |
+
elif wpm < SLOW_WPM_THRESHOLD:
|
| 448 |
+
return max(0.4, 0.7 * (wpm / SLOW_WPM_THRESHOLD))
|
| 449 |
+
elif OPTIMAL_WPM_MAX < wpm <= FAST_WPM_THRESHOLD:
|
| 450 |
+
return 1.0 - 0.5 * (wpm - OPTIMAL_WPM_MAX) / (FAST_WPM_THRESHOLD - OPTIMAL_WPM_MAX)
|
| 451 |
+
else:
|
| 452 |
+
return max(0.2, 0.5 - 0.3 * ((wpm - FAST_WPM_THRESHOLD) / 40))
|
| 453 |
+
|
| 454 |
+
def detect_pauses(self, audio_path):
|
| 455 |
+
"""Detect pauses - OPTIMIZED with caching"""
|
| 456 |
+
if not LIBROSA_AVAILABLE or not os.path.exists(audio_path):
|
| 457 |
+
return {'pause_ratio': 0.0, 'avg_pause_duration': 0.0, 'num_pauses': 0}
|
| 458 |
+
|
| 459 |
+
try:
|
| 460 |
+
# PERFORMANCE: Load with lower sample rate
|
| 461 |
+
y, sr = librosa.load(audio_path, sr=16000) # Was None, now 16kHz (3x faster)
|
| 462 |
+
intervals = librosa.effects.split(y, top_db=30)
|
| 463 |
+
|
| 464 |
+
total_duration = len(y) / sr
|
| 465 |
+
speech_duration = sum((end - start) / sr for start, end in intervals)
|
| 466 |
+
pause_duration = total_duration - speech_duration
|
| 467 |
+
|
| 468 |
+
pause_ratio = pause_duration / total_duration if total_duration > 0 else 0.0
|
| 469 |
+
|
| 470 |
+
num_pauses = len(intervals) - 1 if len(intervals) > 1 else 0
|
| 471 |
+
avg_pause = (pause_duration / num_pauses) if num_pauses > 0 else 0.0
|
| 472 |
+
|
| 473 |
+
return {
|
| 474 |
+
'pause_ratio': round(pause_ratio, 3),
|
| 475 |
+
'avg_pause_duration': round(avg_pause, 3),
|
| 476 |
+
'num_pauses': num_pauses
|
| 477 |
+
}
|
| 478 |
+
except:
|
| 479 |
+
return {'pause_ratio': 0.0, 'avg_pause_duration': 0.0, 'num_pauses': 0}
|
| 480 |
+
|
| 481 |
+
def check_grammar(self, text):
|
| 482 |
+
"""Check grammar - OPTIMIZED with singleton checker"""
|
| 483 |
+
if not self.is_valid_transcript(text) or self.grammar_checker is None:
|
| 484 |
+
return 100.0, 0
|
| 485 |
+
|
| 486 |
+
try:
|
| 487 |
+
# PERFORMANCE: Limit text length for grammar checking
|
| 488 |
+
max_chars = 1000
|
| 489 |
+
if len(text) > max_chars:
|
| 490 |
+
text = text[:max_chars] # Only check first 1000 chars
|
| 491 |
+
|
| 492 |
+
matches = self.grammar_checker.check(text)
|
| 493 |
+
error_count = len(matches)
|
| 494 |
+
text_length = len(text.split())
|
| 495 |
+
|
| 496 |
+
if text_length == 0:
|
| 497 |
+
grammar_score = 0
|
| 498 |
+
else:
|
| 499 |
+
grammar_score = max(0, 100 - (error_count / text_length * 100))
|
| 500 |
+
|
| 501 |
+
return round(grammar_score, 1), error_count
|
| 502 |
+
except:
|
| 503 |
+
return 100.0, 0
|
| 504 |
+
|
| 505 |
+
def compute_lexical_diversity(self, text):
|
| 506 |
+
"""Compute lexical diversity"""
|
| 507 |
+
if not self.is_valid_transcript(text):
|
| 508 |
+
return 0.0
|
| 509 |
+
|
| 510 |
+
meaningful_tokens = self.tokenize_meaningful(text)
|
| 511 |
+
|
| 512 |
+
if not meaningful_tokens:
|
| 513 |
+
return 0.0
|
| 514 |
+
|
| 515 |
+
unique_tokens = set(meaningful_tokens)
|
| 516 |
+
diversity = len(unique_tokens) / len(meaningful_tokens)
|
| 517 |
+
|
| 518 |
+
return round(diversity, 3)
|
| 519 |
+
|
| 520 |
+
def compute_coherence_score(self, text):
|
| 521 |
+
"""Compute coherence - OPTIMIZED with lazy BERT loading"""
|
| 522 |
+
if not self.is_valid_transcript(text):
|
| 523 |
+
return 0.0
|
| 524 |
+
|
| 525 |
+
sentences = [s.strip() for s in text.replace("?", ".").replace("!", ".").split(".") if s.strip()]
|
| 526 |
+
|
| 527 |
+
if len(sentences) < 2:
|
| 528 |
+
return 0.8
|
| 529 |
+
|
| 530 |
+
# PERFORMANCE: Only init BERT if many sentences (worth the overhead)
|
| 531 |
+
if len(sentences) >= 4 and not self._bert_initialized:
|
| 532 |
+
self._lazy_init_bert()
|
| 533 |
+
|
| 534 |
+
# Try BERT only if initialized
|
| 535 |
+
if self.coherence_model and len(sentences) >= 3:
|
| 536 |
+
try:
|
| 537 |
+
coherence_scores = []
|
| 538 |
+
|
| 539 |
+
# PERFORMANCE: Limit to first 5 sentence pairs
|
| 540 |
+
max_pairs = min(5, len(sentences) - 1)
|
| 541 |
+
|
| 542 |
+
for i in range(max_pairs):
|
| 543 |
+
sent1 = sentences[i]
|
| 544 |
+
sent2 = sentences[i + 1]
|
| 545 |
+
combined = f"{sent1} {sent2}"
|
| 546 |
+
|
| 547 |
+
result = self.coherence_model(combined[:512])
|
| 548 |
+
|
| 549 |
+
if result and len(result) > 0:
|
| 550 |
+
score = result[0]['score']
|
| 551 |
+
coherence_scores.append(score)
|
| 552 |
+
|
| 553 |
+
if coherence_scores:
|
| 554 |
+
avg_coherence = np.mean(coherence_scores)
|
| 555 |
+
return round(avg_coherence, 3)
|
| 556 |
+
|
| 557 |
+
except:
|
| 558 |
+
pass
|
| 559 |
+
|
| 560 |
+
# Fallback: Fast heuristic
|
| 561 |
+
transition_words = {
|
| 562 |
+
'however', 'therefore', 'moreover', 'furthermore', 'additionally',
|
| 563 |
+
'consequently', 'thus', 'hence', 'also', 'besides', 'then', 'next',
|
| 564 |
+
'first', 'second', 'finally', 'meanwhile', 'similarly', 'likewise',
|
| 565 |
+
'nevertheless', 'nonetheless', 'accordingly'
|
| 566 |
+
}
|
| 567 |
+
|
| 568 |
+
pronouns = {'it', 'this', 'that', 'these', 'those', 'they', 'them', 'their'}
|
| 569 |
+
|
| 570 |
+
coherence_indicators = 0
|
| 571 |
+
for sentence in sentences[1:]:
|
| 572 |
+
sentence_lower = sentence.lower()
|
| 573 |
+
words = self.tokenize(sentence_lower)
|
| 574 |
+
|
| 575 |
+
if any(word in sentence_lower for word in transition_words):
|
| 576 |
+
coherence_indicators += 1
|
| 577 |
+
|
| 578 |
+
if any(word in words for word in pronouns):
|
| 579 |
+
coherence_indicators += 0.5
|
| 580 |
+
|
| 581 |
+
num_transitions = len(sentences) - 1
|
| 582 |
+
coherence = min(1.0, (coherence_indicators / num_transitions) * 0.6 + 0.4)
|
| 583 |
+
|
| 584 |
+
return round(coherence, 3)
|
| 585 |
+
|
| 586 |
+
def content_similarity(self, provided_text, transcribed_text):
|
| 587 |
+
"""Calculate content similarity - OPTIMIZED"""
|
| 588 |
+
if not self.is_valid_transcript(transcribed_text):
|
| 589 |
+
return 0.0
|
| 590 |
+
|
| 591 |
+
# PERFORMANCE: Limit text length
|
| 592 |
+
max_len = 500
|
| 593 |
+
if len(provided_text) > max_len:
|
| 594 |
+
provided_text = provided_text[:max_len]
|
| 595 |
+
if len(transcribed_text) > max_len:
|
| 596 |
+
transcribed_text = transcribed_text[:max_len]
|
| 597 |
+
|
| 598 |
+
provided_tokens = self.clean_text(provided_text)
|
| 599 |
+
transcribed_tokens = self.clean_text(transcribed_text)
|
| 600 |
+
|
| 601 |
+
provided_string = " ".join(provided_tokens)
|
| 602 |
+
transcribed_string = " ".join(transcribed_tokens)
|
| 603 |
+
|
| 604 |
+
similarity = difflib.SequenceMatcher(None, provided_string, transcribed_string).ratio()
|
| 605 |
+
|
| 606 |
+
similarity_score = similarity * 100
|
| 607 |
+
return round(similarity_score, 1)
|
| 608 |
+
|
| 609 |
+
def evaluate_fluency_comprehensive(self, text, audio_path, duration_seconds):
|
| 610 |
+
"""Comprehensive fluency evaluation - OPTIMIZED"""
|
| 611 |
+
if not self.is_valid_transcript(text):
|
| 612 |
+
return {
|
| 613 |
+
'speech_rate': 0.0,
|
| 614 |
+
'pause_ratio': 0.0,
|
| 615 |
+
'grammar_score': 0.0,
|
| 616 |
+
'grammar_errors': 0,
|
| 617 |
+
'lexical_diversity': 0.0,
|
| 618 |
+
'coherence_score': 0.0,
|
| 619 |
+
'filler_count': 0,
|
| 620 |
+
'filler_ratio': 0.0,
|
| 621 |
+
'fluency_score': 0.0,
|
| 622 |
+
'fluency_level': 'No Data',
|
| 623 |
+
'detailed_metrics': {}
|
| 624 |
+
}
|
| 625 |
+
|
| 626 |
+
# 1. Speech Rate
|
| 627 |
+
speech_rate = self.compute_speech_rate(text, duration_seconds)
|
| 628 |
+
speech_rate_normalized = self.normalize_speech_rate(speech_rate)
|
| 629 |
+
|
| 630 |
+
# 2. Pause Detection
|
| 631 |
+
pause_metrics = self.detect_pauses(audio_path)
|
| 632 |
+
pause_ratio = pause_metrics['pause_ratio']
|
| 633 |
+
|
| 634 |
+
# 3. Grammar
|
| 635 |
+
grammar_score, grammar_errors = self.check_grammar(text)
|
| 636 |
+
|
| 637 |
+
# 4. Lexical Diversity
|
| 638 |
+
lexical_diversity = self.compute_lexical_diversity(text)
|
| 639 |
+
|
| 640 |
+
# 5. Coherence
|
| 641 |
+
coherence_score = self.compute_coherence_score(text)
|
| 642 |
+
|
| 643 |
+
# 6. Filler Words
|
| 644 |
+
filler_count, filler_ratio = self.count_filler_words(text)
|
| 645 |
+
|
| 646 |
+
# 7. Calculate Final Score
|
| 647 |
+
fluency_score = (
|
| 648 |
+
0.30 * speech_rate_normalized +
|
| 649 |
+
0.15 * (1 - pause_ratio) +
|
| 650 |
+
0.25 * (grammar_score / 100) +
|
| 651 |
+
0.15 * lexical_diversity +
|
| 652 |
+
0.10 * coherence_score +
|
| 653 |
+
0.05 * (1 - filler_ratio)
|
| 654 |
+
)
|
| 655 |
+
|
| 656 |
+
fluency_score = round(max(0.0, min(1.0, fluency_score)), 3)
|
| 657 |
+
fluency_percentage = round(fluency_score * 100, 1)
|
| 658 |
+
|
| 659 |
+
# 8. Categorize
|
| 660 |
+
if fluency_score >= 0.80:
|
| 661 |
+
fluency_level = "Excellent"
|
| 662 |
+
elif fluency_score >= 0.70:
|
| 663 |
+
fluency_level = "Fluent"
|
| 664 |
+
elif fluency_score >= 0.50:
|
| 665 |
+
fluency_level = "Moderate"
|
| 666 |
+
else:
|
| 667 |
+
fluency_level = "Needs Improvement"
|
| 668 |
+
|
| 669 |
+
all_words = self.tokenize(text)
|
| 670 |
+
meaningful_words = self.tokenize_meaningful(text)
|
| 671 |
+
|
| 672 |
+
return {
|
| 673 |
+
'speech_rate': speech_rate,
|
| 674 |
+
'speech_rate_normalized': round(speech_rate_normalized, 3),
|
| 675 |
+
'pause_ratio': round(pause_ratio, 3),
|
| 676 |
+
'avg_pause_duration': pause_metrics['avg_pause_duration'],
|
| 677 |
+
'num_pauses': pause_metrics['num_pauses'],
|
| 678 |
+
'grammar_score': grammar_score,
|
| 679 |
+
'grammar_errors': grammar_errors,
|
| 680 |
+
'lexical_diversity': round(lexical_diversity * 100, 1),
|
| 681 |
+
'coherence_score': round(coherence_score * 100, 1),
|
| 682 |
+
'filler_count': filler_count,
|
| 683 |
+
'filler_ratio': round(filler_ratio, 3),
|
| 684 |
+
'fluency_score': fluency_percentage,
|
| 685 |
+
'fluency_level': fluency_level,
|
| 686 |
+
'detailed_metrics': {
|
| 687 |
+
'speech_rate_normalized': round(speech_rate_normalized, 3),
|
| 688 |
+
'optimal_wpm_range': f'{OPTIMAL_WPM_MIN}-{OPTIMAL_WPM_MAX}',
|
| 689 |
+
'total_words': len(all_words),
|
| 690 |
+
'meaningful_words': len(meaningful_words),
|
| 691 |
+
'unique_words': len(set(all_words)),
|
| 692 |
+
'unique_meaningful_words': len(set(meaningful_words)),
|
| 693 |
+
'stopword_filtered': True,
|
| 694 |
+
'filler_words_detected': filler_count
|
| 695 |
+
}
|
| 696 |
+
}
|
| 697 |
+
|
| 698 |
+
# ==================== ANSWER ACCURACY ====================
|
| 699 |
+
|
| 700 |
+
def evaluate_answer_accuracy(self, answer_text, question_text, ideal_answer=None):
|
| 701 |
+
"""Evaluate answer accuracy"""
|
| 702 |
+
if not self.is_valid_transcript(answer_text):
|
| 703 |
+
return 0.0
|
| 704 |
+
|
| 705 |
+
answer_text = answer_text.strip()
|
| 706 |
+
|
| 707 |
+
# PRIMARY: SentenceTransformer
|
| 708 |
+
if ideal_answer and self.models['sentence_model'] is not None:
|
| 709 |
+
try:
|
| 710 |
+
from sentence_transformers import util
|
| 711 |
+
emb = self.models['sentence_model'].encode([ideal_answer, answer_text], convert_to_tensor=True)
|
| 712 |
+
sim = util.pytorch_cos_sim(emb[0], emb[1]).item()
|
| 713 |
+
score = max(0.0, min(1.0, sim))
|
| 714 |
+
return round(score * 100, 1)
|
| 715 |
+
except:
|
| 716 |
+
pass
|
| 717 |
+
|
| 718 |
+
# SECONDARY: Content similarity
|
| 719 |
+
if ideal_answer:
|
| 720 |
+
similarity_score = self.content_similarity(ideal_answer, answer_text)
|
| 721 |
+
return similarity_score
|
| 722 |
+
|
| 723 |
+
# FALLBACK: Basic keyword
|
| 724 |
+
ans_tokens = set(self.tokenize_meaningful(answer_text))
|
| 725 |
+
q_tokens = set(self.tokenize_meaningful(question_text))
|
| 726 |
+
|
| 727 |
+
if not q_tokens or not ans_tokens:
|
| 728 |
+
return 0.0
|
| 729 |
+
|
| 730 |
+
overlap = len(ans_tokens & q_tokens) / len(q_tokens)
|
| 731 |
+
return round(max(0.0, min(1.0, overlap)) * 100, 1)
|
| 732 |
+
|
| 733 |
+
def compute_wpm(self, text, seconds=20):
|
| 734 |
+
"""Legacy method"""
|
| 735 |
+
return self.compute_speech_rate(text, seconds)
|
| 736 |
+
|
| 737 |
+
# ==================== VISUAL ANALYSIS ====================
|
| 738 |
+
|
| 739 |
+
def analyze_outfit(self, frame, face_box):
|
| 740 |
+
"""Analyze outfit - kept as is (accurate)"""
|
| 741 |
+
if face_box is None or self.models['yolo_cls'] is None:
|
| 742 |
+
return "Unknown", 0.0
|
| 743 |
+
|
| 744 |
+
x, y, w, h = face_box
|
| 745 |
+
torso_y_start = y + h
|
| 746 |
+
torso_y_end = min(y + int(h * 3.5), frame.shape[0])
|
| 747 |
+
|
| 748 |
+
if torso_y_start >= torso_y_end or torso_y_start < 0:
|
| 749 |
+
torso_region = frame
|
| 750 |
+
else:
|
| 751 |
+
torso_region = frame[torso_y_start:torso_y_end, max(0, x - w//2):min(frame.shape[1], x + w + w//2)]
|
| 752 |
+
|
| 753 |
+
if torso_region.size == 0:
|
| 754 |
+
return "Unknown", 0.0
|
| 755 |
+
|
| 756 |
+
hsv = cv2.cvtColor(torso_region, cv2.COLOR_BGR2HSV)
|
| 757 |
+
|
| 758 |
+
formal_black = cv2.inRange(hsv, np.array([0, 0, 0]), np.array([180, 50, 50]))
|
| 759 |
+
formal_white = cv2.inRange(hsv, np.array([0, 0, 200]), np.array([180, 30, 255]))
|
| 760 |
+
formal_blue = cv2.inRange(hsv, np.array([100, 50, 50]), np.array([130, 255, 255]))
|
| 761 |
+
formal_gray = cv2.inRange(hsv, np.array([0, 0, 50]), np.array([180, 50, 150]))
|
| 762 |
+
|
| 763 |
+
formal_mask = formal_black + formal_white + formal_blue + formal_gray
|
| 764 |
+
formal_ratio = np.sum(formal_mask > 0) / formal_mask.size
|
| 765 |
+
|
| 766 |
+
try:
|
| 767 |
+
from PIL import Image
|
| 768 |
+
img_pil = Image.fromarray(cv2.cvtColor(torso_region, cv2.COLOR_BGR2RGB))
|
| 769 |
+
img_resized = img_pil.resize((224, 224))
|
| 770 |
+
pred = self.models['yolo_cls'].predict(np.array(img_resized), verbose=False)
|
| 771 |
+
probs = pred[0].probs.data.tolist()
|
| 772 |
+
top_index = int(np.argmax(probs))
|
| 773 |
+
top_label = self.models['yolo_cls'].names[top_index].lower()
|
| 774 |
+
conf = max(probs)
|
| 775 |
+
except:
|
| 776 |
+
top_label = ""
|
| 777 |
+
conf = 0.0
|
| 778 |
+
|
| 779 |
+
formal_keywords = ["suit", "tie", "jacket", "blazer", "dress shirt", "tuxedo", "formal"]
|
| 780 |
+
business_casual = ["polo", "sweater", "cardigan", "button", "collar", "dress"]
|
| 781 |
+
casual_keywords = ["tshirt", "t-shirt", "hoodie", "sweatshirt", "tank"]
|
| 782 |
+
|
| 783 |
+
if any(word in top_label for word in formal_keywords):
|
| 784 |
+
return "Formal", conf
|
| 785 |
+
elif formal_ratio > 0.45:
|
| 786 |
+
return "Formal", min(conf + 0.2, 1.0)
|
| 787 |
+
elif any(word in top_label for word in business_casual):
|
| 788 |
+
if formal_ratio > 0.25:
|
| 789 |
+
return "Business Casual", conf
|
| 790 |
+
else:
|
| 791 |
+
return "Smart Casual", conf
|
| 792 |
+
elif formal_ratio > 0.30:
|
| 793 |
+
return "Business Casual", 0.7
|
| 794 |
+
elif any(word in top_label for word in casual_keywords):
|
| 795 |
+
return "Casual", conf
|
| 796 |
+
elif formal_ratio < 0.15:
|
| 797 |
+
return "Very Casual", max(conf, 0.6)
|
| 798 |
+
else:
|
| 799 |
+
return "Smart Casual", 0.6
|
| 800 |
+
|
| 801 |
+
# ==================== COMPREHENSIVE ANALYSIS ====================
|
| 802 |
+
|
| 803 |
+
def analyze_recording(self, recording_data, question_data, duration=20):
|
| 804 |
+
"""
|
| 805 |
+
Perform comprehensive analysis - OPTIMIZED & ACCURATE
|
| 806 |
+
"""
|
| 807 |
+
frames = recording_data.get('frames', [])
|
| 808 |
+
transcript = recording_data.get('transcript', '')
|
| 809 |
+
audio_path = recording_data.get('audio_path', '')
|
| 810 |
+
face_box = recording_data.get('face_box')
|
| 811 |
+
has_valid_answer = self.is_valid_transcript(transcript)
|
| 812 |
+
|
| 813 |
+
# Facial emotion analysis (optimized sampling)
|
| 814 |
+
face_emotions = {}
|
| 815 |
+
if frames and self.models['face_loaded']:
|
| 816 |
+
face_emotions = self.analyze_emotions_batch(frames, sample_every=10)
|
| 817 |
+
|
| 818 |
+
# Fuse emotions
|
| 819 |
+
fused, scores = self.fuse_emotions(face_emotions, has_valid_answer)
|
| 820 |
+
|
| 821 |
+
# Answer accuracy
|
| 822 |
+
accuracy = 0.0
|
| 823 |
+
if has_valid_answer:
|
| 824 |
+
accuracy = self.evaluate_answer_accuracy(
|
| 825 |
+
transcript,
|
| 826 |
+
question_data.get("question", ""),
|
| 827 |
+
question_data.get("ideal_answer")
|
| 828 |
+
)
|
| 829 |
+
|
| 830 |
+
# Comprehensive fluency analysis
|
| 831 |
+
fluency_results = self.evaluate_fluency_comprehensive(transcript, audio_path, duration)
|
| 832 |
+
|
| 833 |
+
# Visual outfit analysis
|
| 834 |
+
outfit_label = "Unknown"
|
| 835 |
+
outfit_conf = 0.0
|
| 836 |
+
if frames and face_box:
|
| 837 |
+
outfit_label, outfit_conf = self.analyze_outfit(frames[-1], face_box)
|
| 838 |
+
|
| 839 |
+
return {
|
| 840 |
+
'fused_emotions': fused,
|
| 841 |
+
'emotion_scores': scores,
|
| 842 |
+
'accuracy': accuracy,
|
| 843 |
+
'fluency': fluency_results['fluency_score'],
|
| 844 |
+
'fluency_level': fluency_results['fluency_level'],
|
| 845 |
+
'fluency_detailed': fluency_results,
|
| 846 |
+
'wpm': fluency_results['speech_rate'],
|
| 847 |
+
'grammar_errors': fluency_results['grammar_errors'],
|
| 848 |
+
'filler_count': fluency_results['filler_count'],
|
| 849 |
+
'filler_ratio': fluency_results['filler_ratio'],
|
| 850 |
+
'outfit': outfit_label,
|
| 851 |
+
'outfit_confidence': outfit_conf,
|
| 852 |
+
'has_valid_data': has_valid_answer,
|
| 853 |
+
'improvements_applied': {
|
| 854 |
+
'stopword_filtering': True,
|
| 855 |
+
'quality_weighted_emotions': True,
|
| 856 |
+
'content_similarity_matching': True,
|
| 857 |
+
'grammar_error_count': True,
|
| 858 |
+
'filler_word_detection': True,
|
| 859 |
+
'bert_coherence': self.coherence_model is not None,
|
| 860 |
+
'contextual_wpm_normalization': True,
|
| 861 |
+
'accurate_pause_detection': LIBROSA_AVAILABLE,
|
| 862 |
+
'no_fake_metrics': True,
|
| 863 |
+
'performance_optimized': True
|
| 864 |
+
}
|
| 865 |
+
}
|
| 866 |
+
|
| 867 |
+
|
| 868 |
+
####
|
main_app.py
ADDED
|
@@ -0,0 +1,576 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
|
| 2 |
+
|
| 3 |
+
"""
|
| 4 |
+
Main Integration File - AI Interview System
|
| 5 |
+
SIMPLIFIED, PROFESSIONAL UI - Normal Website Look
|
| 6 |
+
"""
|
| 7 |
+
|
| 8 |
+
import streamlit as st
|
| 9 |
+
import warnings
|
| 10 |
+
import os
|
| 11 |
+
from PIL import Image, ImageDraw
|
| 12 |
+
|
| 13 |
+
# Import the three modular systems
|
| 14 |
+
from Recording_system import RecordingSystem
|
| 15 |
+
from analysis_system import AnalysisSystem
|
| 16 |
+
from scoring_dashboard import ScoringDashboard
|
| 17 |
+
|
| 18 |
+
warnings.filterwarnings('ignore')
|
| 19 |
+
os.environ['TF_CPP_MIN_LOG_LEVEL'] = '3'
|
| 20 |
+
|
| 21 |
+
# Try importing optional modules
|
| 22 |
+
try:
|
| 23 |
+
import mediapipe as mp
|
| 24 |
+
MP_AVAILABLE = True
|
| 25 |
+
mp_face_mesh = mp.solutions.face_mesh
|
| 26 |
+
mp_hands = mp.solutions.hands
|
| 27 |
+
except:
|
| 28 |
+
MP_AVAILABLE = False
|
| 29 |
+
|
| 30 |
+
try:
|
| 31 |
+
from ultralytics import YOLO
|
| 32 |
+
YOLO_AVAILABLE = True
|
| 33 |
+
except:
|
| 34 |
+
YOLO_AVAILABLE = False
|
| 35 |
+
|
| 36 |
+
try:
|
| 37 |
+
from sentence_transformers import SentenceTransformer
|
| 38 |
+
SENTENCE_TRANSFORMER_AVAILABLE = True
|
| 39 |
+
except:
|
| 40 |
+
SENTENCE_TRANSFORMER_AVAILABLE = False
|
| 41 |
+
|
| 42 |
+
try:
|
| 43 |
+
from deepface import DeepFace
|
| 44 |
+
DEEPFACE_AVAILABLE = True
|
| 45 |
+
except:
|
| 46 |
+
DEEPFACE_AVAILABLE = False
|
| 47 |
+
|
| 48 |
+
# ==================== PAGE CONFIG ====================
|
| 49 |
+
st.set_page_config(page_title="Interview Assessment Platform", layout="wide", page_icon="π―")
|
| 50 |
+
|
| 51 |
+
# ==================== SIMPLE, CLEAN STYLES ====================
|
| 52 |
+
st.markdown("""
|
| 53 |
+
<style>
|
| 54 |
+
/* Hide Streamlit branding */
|
| 55 |
+
#MainMenu {visibility: hidden;}
|
| 56 |
+
footer {visibility: hidden;}
|
| 57 |
+
header {visibility: hidden;}
|
| 58 |
+
|
| 59 |
+
/* Simple body styling */
|
| 60 |
+
body {
|
| 61 |
+
background-color: #ffffff;
|
| 62 |
+
color: #333333;
|
| 63 |
+
font-family: -apple-system, BlinkMacSystemFont, 'Segoe UI', Arial, sans-serif;
|
| 64 |
+
}
|
| 65 |
+
|
| 66 |
+
/* Simple headers */
|
| 67 |
+
h1 {
|
| 68 |
+
color: #2c3e50;
|
| 69 |
+
font-weight: 600;
|
| 70 |
+
margin-bottom: 0.5rem;
|
| 71 |
+
}
|
| 72 |
+
|
| 73 |
+
h2 {
|
| 74 |
+
color: #34495e;
|
| 75 |
+
font-weight: 500;
|
| 76 |
+
margin-top: 1.5rem;
|
| 77 |
+
margin-bottom: 0.75rem;
|
| 78 |
+
}
|
| 79 |
+
|
| 80 |
+
h3 {
|
| 81 |
+
color: #555555;
|
| 82 |
+
font-weight: 500;
|
| 83 |
+
}
|
| 84 |
+
|
| 85 |
+
/* Simple boxes */
|
| 86 |
+
.info-box {
|
| 87 |
+
background: #f8f9fa;
|
| 88 |
+
border: 1px solid #dee2e6;
|
| 89 |
+
border-radius: 4px;
|
| 90 |
+
padding: 1rem;
|
| 91 |
+
margin: 1rem 0;
|
| 92 |
+
}
|
| 93 |
+
|
| 94 |
+
.success-box {
|
| 95 |
+
background: #d4edda;
|
| 96 |
+
border: 1px solid #c3e6cb;
|
| 97 |
+
border-left: 4px solid #28a745;
|
| 98 |
+
border-radius: 4px;
|
| 99 |
+
padding: 1rem;
|
| 100 |
+
margin: 1rem 0;
|
| 101 |
+
}
|
| 102 |
+
|
| 103 |
+
.warning-box {
|
| 104 |
+
background: #fff3cd;
|
| 105 |
+
border: 1px solid #ffeaa7;
|
| 106 |
+
border-left: 4px solid #ffc107;
|
| 107 |
+
border-radius: 4px;
|
| 108 |
+
padding: 1rem;
|
| 109 |
+
margin: 1rem 0;
|
| 110 |
+
}
|
| 111 |
+
|
| 112 |
+
.error-box {
|
| 113 |
+
background: #f8d7da;
|
| 114 |
+
border: 1px solid #f5c6cb;
|
| 115 |
+
border-left: 4px solid #dc3545;
|
| 116 |
+
border-radius: 4px;
|
| 117 |
+
padding: 1rem;
|
| 118 |
+
margin: 1rem 0;
|
| 119 |
+
}
|
| 120 |
+
|
| 121 |
+
/* Simple question box */
|
| 122 |
+
.question-box {
|
| 123 |
+
background: #ffffff;
|
| 124 |
+
border: 1px solid #dee2e6;
|
| 125 |
+
border-radius: 4px;
|
| 126 |
+
padding: 1.5rem;
|
| 127 |
+
margin-bottom: 1rem;
|
| 128 |
+
min-height: 200px;
|
| 129 |
+
}
|
| 130 |
+
|
| 131 |
+
.question-box h3 {
|
| 132 |
+
color: #2c3e50;
|
| 133 |
+
margin-bottom: 1rem;
|
| 134 |
+
padding-bottom: 0.75rem;
|
| 135 |
+
border-bottom: 1px solid #e9ecef;
|
| 136 |
+
}
|
| 137 |
+
|
| 138 |
+
/* Simple metric cards */
|
| 139 |
+
.metric-card {
|
| 140 |
+
background: #ffffff;
|
| 141 |
+
border: 1px solid #dee2e6;
|
| 142 |
+
border-radius: 4px;
|
| 143 |
+
padding: 1rem;
|
| 144 |
+
text-align: center;
|
| 145 |
+
margin-bottom: 0.5rem;
|
| 146 |
+
}
|
| 147 |
+
|
| 148 |
+
.metric-card h3 {
|
| 149 |
+
color: #2c3e50;
|
| 150 |
+
font-size: 1.5rem;
|
| 151 |
+
margin: 0;
|
| 152 |
+
}
|
| 153 |
+
|
| 154 |
+
.metric-card p {
|
| 155 |
+
color: #6c757d;
|
| 156 |
+
font-size: 0.875rem;
|
| 157 |
+
margin: 0.25rem 0 0 0;
|
| 158 |
+
}
|
| 159 |
+
|
| 160 |
+
/* Hide sidebar */
|
| 161 |
+
[data-testid="stSidebar"] {
|
| 162 |
+
display: none;
|
| 163 |
+
}
|
| 164 |
+
|
| 165 |
+
/* Simple buttons */
|
| 166 |
+
.stButton > button {
|
| 167 |
+
border-radius: 4px;
|
| 168 |
+
border: 1px solid #dee2e6;
|
| 169 |
+
}
|
| 170 |
+
|
| 171 |
+
/* Simple progress bar */
|
| 172 |
+
.stProgress > div > div {
|
| 173 |
+
background-color: #007bff;
|
| 174 |
+
}
|
| 175 |
+
</style>
|
| 176 |
+
""", unsafe_allow_html=True)
|
| 177 |
+
|
| 178 |
+
# ==================== QUESTIONS CONFIGURATION ====================
|
| 179 |
+
QUESTIONS = [
|
| 180 |
+
{
|
| 181 |
+
"question": "Tell me about yourself.",
|
| 182 |
+
"type": "personal",
|
| 183 |
+
"ideal_answer": "I'm a computer science postgraduate with a strong interest in AI and software development. I've worked on several projects involving Python, machine learning, and data analysis, which helped me improve both my technical and problem-solving skills. I enjoy learning new technologies and applying them to create practical solutions. Outside of academics, I like collaborating on team projects and continuously developing my professional skills.",
|
| 184 |
+
"tip": "Focus on your background, skills, and personality"
|
| 185 |
+
},
|
| 186 |
+
{
|
| 187 |
+
"question": "What are your strengths and weaknesses?",
|
| 188 |
+
"type": "personal",
|
| 189 |
+
"ideal_answer": "One of my key strengths is that I'm very detail-oriented and persistent β I make sure my work is accurate and well-tested. I also enjoy solving complex problems and learning new tools quickly. As for weaknesses, I used to spend too much time perfecting small details, which sometimes slowed me down. But I've been improving by prioritizing tasks better and focusing on overall impact.",
|
| 190 |
+
"tip": "Be honest and show self-awareness"
|
| 191 |
+
},
|
| 192 |
+
{
|
| 193 |
+
"question": "Where do you see yourself in the next 5 years?",
|
| 194 |
+
"type": "personal",
|
| 195 |
+
"ideal_answer": "In the next five years, I see myself growing into a more responsible and skilled professional, ideally in a role where I can contribute to meaningful projects involving AI and software development. I'd also like to take on leadership responsibilities and guide new team members as I gain experience.",
|
| 196 |
+
"tip": "Show ambition aligned with career growth"
|
| 197 |
+
}
|
| 198 |
+
]
|
| 199 |
+
|
| 200 |
+
# ==================== GENERATE DEMO IMAGES ====================
|
| 201 |
+
def create_frame_demo_image(is_correct=True):
|
| 202 |
+
"""Create demonstration image showing correct/incorrect positioning"""
|
| 203 |
+
width, height = 500, 350
|
| 204 |
+
img = Image.new('RGB', (width, height), color='#f8f9fa')
|
| 205 |
+
draw = ImageDraw.Draw(img)
|
| 206 |
+
|
| 207 |
+
margin = 40
|
| 208 |
+
boundary_color = '#28a745' if is_correct else '#dc3545'
|
| 209 |
+
|
| 210 |
+
# Draw boundaries
|
| 211 |
+
draw.rectangle([margin, margin, width-margin, height-margin], outline=boundary_color, width=3)
|
| 212 |
+
|
| 213 |
+
if is_correct:
|
| 214 |
+
# Draw person inside
|
| 215 |
+
head_x, head_y = width // 2, margin + 60
|
| 216 |
+
draw.ellipse([head_x - 30, head_y - 30, head_x + 30, head_y + 30], fill='#ffc107', outline='#333333', width=2)
|
| 217 |
+
|
| 218 |
+
body_y = head_y + 40
|
| 219 |
+
draw.rectangle([head_x - 40, body_y, head_x + 40, body_y + 80], fill='#007bff', outline='#333333', width=2)
|
| 220 |
+
|
| 221 |
+
draw.text((width//2 - 80, height - 30), "β Correct Position", fill='#28a745')
|
| 222 |
+
else:
|
| 223 |
+
# Draw person outside
|
| 224 |
+
head_x, head_y = margin - 20, margin + 60
|
| 225 |
+
draw.ellipse([head_x - 30, head_y - 30, head_x + 30, head_y + 30], fill='#ffc107', outline='#333333', width=2)
|
| 226 |
+
|
| 227 |
+
draw.text((width//2 - 80, height - 30), "β Outside Bounds", fill='#dc3545')
|
| 228 |
+
|
| 229 |
+
return img
|
| 230 |
+
|
| 231 |
+
# ==================== HOME PAGE ====================
|
| 232 |
+
def show_home_page():
|
| 233 |
+
"""Display clean home page"""
|
| 234 |
+
|
| 235 |
+
st.title("Interview Assessment Platform")
|
| 236 |
+
st.write("Professional evaluation system for video interviews")
|
| 237 |
+
|
| 238 |
+
st.markdown("---")
|
| 239 |
+
|
| 240 |
+
# Simple features
|
| 241 |
+
col1, col2, col3 = st.columns(3)
|
| 242 |
+
|
| 243 |
+
with col1:
|
| 244 |
+
st.markdown("""
|
| 245 |
+
**π Structured Assessment**
|
| 246 |
+
|
| 247 |
+
Standardized evaluation with consistent criteria
|
| 248 |
+
""")
|
| 249 |
+
|
| 250 |
+
with col2:
|
| 251 |
+
st.markdown("""
|
| 252 |
+
**π Detailed Analytics**
|
| 253 |
+
|
| 254 |
+
Comprehensive metrics and performance insights
|
| 255 |
+
""")
|
| 256 |
+
|
| 257 |
+
with col3:
|
| 258 |
+
st.markdown("""
|
| 259 |
+
**β
Compliance Monitoring**
|
| 260 |
+
|
| 261 |
+
Real-time monitoring ensures integrity
|
| 262 |
+
""")
|
| 263 |
+
|
| 264 |
+
st.markdown("---")
|
| 265 |
+
|
| 266 |
+
# Introduction
|
| 267 |
+
st.subheader("Before You Begin")
|
| 268 |
+
st.write("""
|
| 269 |
+
This platform evaluates candidates through structured video interviews. Please review
|
| 270 |
+
the camera positioning requirements below to ensure a smooth assessment.
|
| 271 |
+
""")
|
| 272 |
+
|
| 273 |
+
# Frame positioning
|
| 274 |
+
st.subheader("Camera Positioning Requirements")
|
| 275 |
+
|
| 276 |
+
col1, col2 = st.columns(2)
|
| 277 |
+
|
| 278 |
+
with col1:
|
| 279 |
+
st.markdown("**β
Correct Positioning**")
|
| 280 |
+
correct_img = create_frame_demo_image(is_correct=True)
|
| 281 |
+
st.image(correct_img, use_container_width=True)
|
| 282 |
+
st.markdown("""
|
| 283 |
+
- Center yourself in the frame
|
| 284 |
+
- Keep entire face visible
|
| 285 |
+
- Remain alone in the frame
|
| 286 |
+
- Ensure adequate lighting
|
| 287 |
+
- Maintain forward gaze
|
| 288 |
+
""")
|
| 289 |
+
|
| 290 |
+
with col2:
|
| 291 |
+
st.markdown("**β Common Mistakes**")
|
| 292 |
+
incorrect_img = create_frame_demo_image(is_correct=False)
|
| 293 |
+
st.image(incorrect_img, use_container_width=True)
|
| 294 |
+
st.markdown("""
|
| 295 |
+
- Moving outside boundaries
|
| 296 |
+
- Multiple people visible
|
| 297 |
+
- Obstructed or partial view
|
| 298 |
+
- Poor lighting conditions
|
| 299 |
+
- Extended periods looking away
|
| 300 |
+
""")
|
| 301 |
+
|
| 302 |
+
st.markdown("---")
|
| 303 |
+
|
| 304 |
+
# Assessment process
|
| 305 |
+
st.subheader("Assessment Process")
|
| 306 |
+
st.markdown(f"""
|
| 307 |
+
1. **Initial Setup (60 seconds):** Position yourself within marked boundaries
|
| 308 |
+
2. **Environment Scan:** System records baseline to detect changes
|
| 309 |
+
3. **Interview Session:** Respond to {len(QUESTIONS)} questions (20 seconds each)
|
| 310 |
+
4. **Continuous Monitoring:** System monitors compliance throughout
|
| 311 |
+
5. **Results Analysis:** Receive comprehensive evaluation with feedback
|
| 312 |
+
""")
|
| 313 |
+
|
| 314 |
+
st.markdown("---")
|
| 315 |
+
|
| 316 |
+
# Technical requirements
|
| 317 |
+
st.subheader("Technical Requirements")
|
| 318 |
+
|
| 319 |
+
col1, col2 = st.columns(2)
|
| 320 |
+
|
| 321 |
+
with col1:
|
| 322 |
+
st.markdown("""
|
| 323 |
+
**Hardware**
|
| 324 |
+
- Functional webcam (720p recommended)
|
| 325 |
+
- Clear microphone
|
| 326 |
+
- Stable internet (5 Mbps minimum)
|
| 327 |
+
- Desktop or laptop computer
|
| 328 |
+
""")
|
| 329 |
+
|
| 330 |
+
with col2:
|
| 331 |
+
st.markdown("""
|
| 332 |
+
**Environment**
|
| 333 |
+
- Quiet, private space
|
| 334 |
+
- Front-facing lighting
|
| 335 |
+
- Neutral background
|
| 336 |
+
- Comfortable seating
|
| 337 |
+
""")
|
| 338 |
+
|
| 339 |
+
st.markdown("---")
|
| 340 |
+
|
| 341 |
+
# Confirmation
|
| 342 |
+
st.subheader("Ready to Begin")
|
| 343 |
+
|
| 344 |
+
if 'guidelines_accepted' not in st.session_state:
|
| 345 |
+
st.session_state.guidelines_accepted = False
|
| 346 |
+
|
| 347 |
+
st.session_state.guidelines_accepted = st.checkbox(
|
| 348 |
+
f"I confirm that I have reviewed all guidelines and am prepared to complete {len(QUESTIONS)} interview questions.",
|
| 349 |
+
value=st.session_state.guidelines_accepted,
|
| 350 |
+
key="guidelines_checkbox"
|
| 351 |
+
)
|
| 352 |
+
|
| 353 |
+
if st.session_state.guidelines_accepted:
|
| 354 |
+
st.success("β
You are ready to proceed with the assessment.")
|
| 355 |
+
if st.button("Begin Assessment", type="primary"):
|
| 356 |
+
st.session_state.page = "interview"
|
| 357 |
+
st.session_state.interview_started = False
|
| 358 |
+
st.rerun()
|
| 359 |
+
else:
|
| 360 |
+
st.info("βΉοΈ Please confirm that you have reviewed the guidelines to continue.")
|
| 361 |
+
|
| 362 |
+
# ==================== LOAD MODELS ====================
|
| 363 |
+
@st.cache_resource(show_spinner="Initializing assessment system...")
|
| 364 |
+
def load_all_models():
|
| 365 |
+
"""Load all AI models and return dictionary"""
|
| 366 |
+
models = {}
|
| 367 |
+
|
| 368 |
+
if DEEPFACE_AVAILABLE:
|
| 369 |
+
try:
|
| 370 |
+
_ = DeepFace.build_model("Facenet")
|
| 371 |
+
models['face_loaded'] = True
|
| 372 |
+
except:
|
| 373 |
+
models['face_loaded'] = False
|
| 374 |
+
else:
|
| 375 |
+
models['face_loaded'] = False
|
| 376 |
+
|
| 377 |
+
if SENTENCE_TRANSFORMER_AVAILABLE:
|
| 378 |
+
try:
|
| 379 |
+
models['sentence_model'] = SentenceTransformer('all-MiniLM-L6-v2')
|
| 380 |
+
except:
|
| 381 |
+
models['sentence_model'] = None
|
| 382 |
+
else:
|
| 383 |
+
models['sentence_model'] = None
|
| 384 |
+
|
| 385 |
+
if MP_AVAILABLE:
|
| 386 |
+
try:
|
| 387 |
+
models['face_mesh'] = mp_face_mesh.FaceMesh(
|
| 388 |
+
static_image_mode=False,
|
| 389 |
+
max_num_faces=5,
|
| 390 |
+
refine_landmarks=True,
|
| 391 |
+
min_detection_confidence=0.5,
|
| 392 |
+
min_tracking_confidence=0.5
|
| 393 |
+
)
|
| 394 |
+
models['hands'] = mp_hands.Hands(
|
| 395 |
+
static_image_mode=False,
|
| 396 |
+
max_num_hands=2,
|
| 397 |
+
min_detection_confidence=0.5,
|
| 398 |
+
min_tracking_confidence=0.5
|
| 399 |
+
)
|
| 400 |
+
except:
|
| 401 |
+
models['face_mesh'] = None
|
| 402 |
+
models['hands'] = None
|
| 403 |
+
else:
|
| 404 |
+
models['face_mesh'] = None
|
| 405 |
+
models['hands'] = None
|
| 406 |
+
|
| 407 |
+
if YOLO_AVAILABLE:
|
| 408 |
+
try:
|
| 409 |
+
models['yolo'] = YOLO("yolov8n.pt")
|
| 410 |
+
models['yolo_cls'] = YOLO("yolov8n-cls.pt")
|
| 411 |
+
except:
|
| 412 |
+
models['yolo'] = None
|
| 413 |
+
models['yolo_cls'] = None
|
| 414 |
+
else:
|
| 415 |
+
models['yolo'] = None
|
| 416 |
+
models['yolo_cls'] = None
|
| 417 |
+
|
| 418 |
+
return models
|
| 419 |
+
|
| 420 |
+
models = load_all_models()
|
| 421 |
+
|
| 422 |
+
# ==================== INITIALIZE SYSTEMS ====================
|
| 423 |
+
recording_system = RecordingSystem(models)
|
| 424 |
+
analysis_system = AnalysisSystem(models)
|
| 425 |
+
scoring_dashboard = ScoringDashboard()
|
| 426 |
+
|
| 427 |
+
# ==================== SESSION STATE ====================
|
| 428 |
+
if "page" not in st.session_state:
|
| 429 |
+
st.session_state.page = "home"
|
| 430 |
+
if "results" not in st.session_state:
|
| 431 |
+
st.session_state.results = []
|
| 432 |
+
if "interview_started" not in st.session_state:
|
| 433 |
+
st.session_state.interview_started = False
|
| 434 |
+
if "interview_complete" not in st.session_state:
|
| 435 |
+
st.session_state.interview_complete = False
|
| 436 |
+
|
| 437 |
+
# ==================== MAIN ROUTING ====================
|
| 438 |
+
if st.session_state.page == "home":
|
| 439 |
+
show_home_page()
|
| 440 |
+
|
| 441 |
+
else: # Interview page
|
| 442 |
+
st.title("Interview Assessment Session")
|
| 443 |
+
st.write("Complete all questions to receive your evaluation")
|
| 444 |
+
|
| 445 |
+
# Simple navigation
|
| 446 |
+
if not st.session_state.interview_complete:
|
| 447 |
+
if st.button("β Back to Home"):
|
| 448 |
+
st.session_state.page = "home"
|
| 449 |
+
st.session_state.interview_started = False
|
| 450 |
+
st.session_state.interview_complete = False
|
| 451 |
+
st.rerun()
|
| 452 |
+
else:
|
| 453 |
+
col1, col2 = st.columns(2)
|
| 454 |
+
with col1:
|
| 455 |
+
if st.button("β Back to Home"):
|
| 456 |
+
st.session_state.page = "home"
|
| 457 |
+
st.session_state.interview_started = False
|
| 458 |
+
st.session_state.interview_complete = False
|
| 459 |
+
st.rerun()
|
| 460 |
+
with col2:
|
| 461 |
+
if st.button("π New Assessment"):
|
| 462 |
+
st.session_state.results = []
|
| 463 |
+
st.session_state.interview_started = False
|
| 464 |
+
st.session_state.interview_complete = False
|
| 465 |
+
st.rerun()
|
| 466 |
+
|
| 467 |
+
st.markdown("---")
|
| 468 |
+
|
| 469 |
+
# ==================== MAIN CONTENT ====================
|
| 470 |
+
|
| 471 |
+
if not st.session_state.interview_started and not st.session_state.interview_complete:
|
| 472 |
+
st.subheader("Ready to Begin?")
|
| 473 |
+
st.write(f"""
|
| 474 |
+
- You will respond to **{len(QUESTIONS)} questions**
|
| 475 |
+
- Each question allows **20 seconds** for your response
|
| 476 |
+
- The system will monitor compliance throughout
|
| 477 |
+
""")
|
| 478 |
+
|
| 479 |
+
if st.button("Begin Assessment", type="primary"):
|
| 480 |
+
st.session_state.interview_started = True
|
| 481 |
+
st.rerun()
|
| 482 |
+
|
| 483 |
+
elif st.session_state.interview_started and not st.session_state.interview_complete:
|
| 484 |
+
col_question, col_video = st.columns([2, 3])
|
| 485 |
+
|
| 486 |
+
with col_question:
|
| 487 |
+
question_placeholder = st.empty()
|
| 488 |
+
|
| 489 |
+
with col_video:
|
| 490 |
+
video_placeholder = st.empty()
|
| 491 |
+
|
| 492 |
+
st.markdown("---")
|
| 493 |
+
countdown_placeholder = st.empty()
|
| 494 |
+
status_placeholder = st.empty()
|
| 495 |
+
progress_bar = st.progress(0)
|
| 496 |
+
timer_text = st.empty()
|
| 497 |
+
|
| 498 |
+
ui_callbacks = {
|
| 499 |
+
'countdown_update': lambda msg: countdown_placeholder.warning(msg) if msg else countdown_placeholder.empty(),
|
| 500 |
+
'video_update': lambda frame: video_placeholder.image(frame, channels="BGR", use_container_width=True) if frame is not None else video_placeholder.empty(),
|
| 501 |
+
'status_update': lambda text: status_placeholder.markdown(text) if text else status_placeholder.empty(),
|
| 502 |
+
'progress_update': lambda val: progress_bar.progress(val),
|
| 503 |
+
'timer_update': lambda text: timer_text.info(text) if text else timer_text.empty(),
|
| 504 |
+
'question_update': lambda q_num, q_text, q_tip="": question_placeholder.markdown(
|
| 505 |
+
f'''<div class="question-box">
|
| 506 |
+
<h3>Question {q_num} of {len(QUESTIONS)}</h3>
|
| 507 |
+
<p style="font-size: 1.1rem; margin: 1rem 0;">{q_text}</p>
|
| 508 |
+
<p style="color: #6c757d; font-size: 0.9rem; margin-top: 1rem;">
|
| 509 |
+
π‘ <strong>Tip:</strong> {q_tip if q_tip else "Speak clearly and confidently"}
|
| 510 |
+
</p>
|
| 511 |
+
</div>''',
|
| 512 |
+
unsafe_allow_html=True
|
| 513 |
+
) if q_text else question_placeholder.empty()
|
| 514 |
+
}
|
| 515 |
+
|
| 516 |
+
st.info("π¬ Initializing assessment session...")
|
| 517 |
+
session_result = recording_system.record_continuous_interview(
|
| 518 |
+
QUESTIONS,
|
| 519 |
+
duration_per_question=20,
|
| 520 |
+
ui_callbacks=ui_callbacks
|
| 521 |
+
)
|
| 522 |
+
|
| 523 |
+
if isinstance(session_result, dict) and 'questions_results' in session_result:
|
| 524 |
+
st.session_state.results = []
|
| 525 |
+
|
| 526 |
+
for q_result in session_result['questions_results']:
|
| 527 |
+
question_data = QUESTIONS[q_result['question_number'] - 1]
|
| 528 |
+
analysis_results = analysis_system.analyze_recording(q_result, question_data, 20)
|
| 529 |
+
|
| 530 |
+
result = {
|
| 531 |
+
"question": question_data["question"],
|
| 532 |
+
"video_path": session_result.get('session_video_path', ''),
|
| 533 |
+
"audio_path": q_result.get('audio_path', ''),
|
| 534 |
+
"transcript": q_result.get('transcript', ''),
|
| 535 |
+
"violations": q_result.get('violations', []),
|
| 536 |
+
"violation_detected": q_result.get('violation_detected', False),
|
| 537 |
+
"fused_emotions": analysis_results.get('fused_emotions', {}),
|
| 538 |
+
"emotion_scores": analysis_results.get('emotion_scores', {}),
|
| 539 |
+
"accuracy": analysis_results.get('accuracy', 0),
|
| 540 |
+
"fluency": analysis_results.get('fluency', 0),
|
| 541 |
+
"wpm": analysis_results.get('wpm', 0),
|
| 542 |
+
"blink_count": q_result.get('blink_count', 0),
|
| 543 |
+
"outfit": analysis_results.get('outfit', 'Unknown'),
|
| 544 |
+
"has_valid_data": analysis_results.get('has_valid_data', False),
|
| 545 |
+
"fluency_detailed": analysis_results.get('fluency_detailed', {}),
|
| 546 |
+
"fluency_level": analysis_results.get('fluency_level', 'No Data'),
|
| 547 |
+
"grammar_errors": analysis_results.get('grammar_errors', 0),
|
| 548 |
+
"filler_count": analysis_results.get('filler_count', 0),
|
| 549 |
+
"filler_ratio": analysis_results.get('filler_ratio', 0),
|
| 550 |
+
"improvements_applied": analysis_results.get('improvements_applied', {})
|
| 551 |
+
}
|
| 552 |
+
|
| 553 |
+
decision, reasons = scoring_dashboard.decide_hire(result)
|
| 554 |
+
result["hire_decision"] = decision
|
| 555 |
+
result["hire_reasons"] = reasons
|
| 556 |
+
|
| 557 |
+
st.session_state.results.append(result)
|
| 558 |
+
|
| 559 |
+
st.session_state.interview_complete = True
|
| 560 |
+
|
| 561 |
+
total_violations = session_result.get('total_violations', 0)
|
| 562 |
+
if total_violations > 0:
|
| 563 |
+
st.warning(f"β οΈ Assessment completed with {total_violations} compliance issue(s).")
|
| 564 |
+
else:
|
| 565 |
+
st.success("π Assessment completed successfully!")
|
| 566 |
+
|
| 567 |
+
import time
|
| 568 |
+
time.sleep(2)
|
| 569 |
+
st.rerun()
|
| 570 |
+
else:
|
| 571 |
+
st.error("β Assessment failed. Please try again.")
|
| 572 |
+
st.session_state.interview_started = False
|
| 573 |
+
|
| 574 |
+
else:
|
| 575 |
+
|
| 576 |
+
scoring_dashboard.render_dashboard(st.session_state.results)
|
packages.txt
ADDED
|
@@ -0,0 +1,2 @@
|
|
|
|
|
|
|
|
|
|
| 1 |
+
libgl1
|
| 2 |
+
libglib2.0-0
|
requirements.txt
CHANGED
|
@@ -1,3 +1,43 @@
|
|
| 1 |
-
|
| 2 |
-
|
| 3 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# Core
|
| 2 |
+
numpy==1.26.4
|
| 3 |
+
pandas==2.3.3
|
| 4 |
+
matplotlib==3.10.7
|
| 5 |
+
|
| 6 |
+
# Streamlit app
|
| 7 |
+
streamlit==1.38.0
|
| 8 |
+
plotly==6.3.1
|
| 9 |
+
|
| 10 |
+
# Audio processing
|
| 11 |
+
speechrecognition==3.10.4
|
| 12 |
+
pydub==0.25.1
|
| 13 |
+
librosa==0.11.0
|
| 14 |
+
|
| 15 |
+
# Computer vision & face analysis
|
| 16 |
+
opencv-contrib-python-headless==4.10.0.84
|
| 17 |
+
deepface==0.0.95
|
| 18 |
+
mediapipe==0.10.14
|
| 19 |
+
mtcnn==1.0.0
|
| 20 |
+
|
| 21 |
+
# Machine learning & NLP
|
| 22 |
+
transformers==4.45.2
|
| 23 |
+
sentence-transformers==3.3.0
|
| 24 |
+
language-tool-python==2.9.4
|
| 25 |
+
spacy==3.7.5
|
| 26 |
+
nltk==3.9.2
|
| 27 |
+
torch==2.5.1
|
| 28 |
+
tensorflow-cpu==2.17.0
|
| 29 |
+
scikit-learn==1.5.2
|
| 30 |
+
huggingface-hub==0.25.2
|
| 31 |
+
|
| 32 |
+
# Image/video utilities
|
| 33 |
+
pillow==10.3.0
|
| 34 |
+
moviepy==2.2.1
|
| 35 |
+
|
| 36 |
+
# YOLO object detection
|
| 37 |
+
ultralytics==8.3.20
|
| 38 |
+
|
| 39 |
+
# Utility
|
| 40 |
+
tqdm==4.66.5
|
| 41 |
+
requests==2.32.3
|
| 42 |
+
|
| 43 |
+
|
runtime.txt
ADDED
|
@@ -0,0 +1 @@
|
|
|
|
|
|
|
| 1 |
+
python-3.11
|
scoring_dashboard.py
ADDED
|
@@ -0,0 +1,745 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""
|
| 2 |
+
Scoring & Hiring Decision + Results Dashboard - BEST OF BOTH VERSION
|
| 3 |
+
ONLY accurate metrics, NO fake scores
|
| 4 |
+
Includes: filler words, improved content similarity, grammar error count
|
| 5 |
+
Excludes: eye contact (removed), fake pronunciation, wrong tempo
|
| 6 |
+
"""
|
| 7 |
+
|
| 8 |
+
import streamlit as st
|
| 9 |
+
import numpy as np
|
| 10 |
+
import pandas as pd
|
| 11 |
+
import os
|
| 12 |
+
import time
|
| 13 |
+
|
| 14 |
+
class ScoringDashboard:
|
| 15 |
+
"""Handles scoring, hiring decisions, and results visualization - ACCURATE ONLY"""
|
| 16 |
+
|
| 17 |
+
def __init__(self):
|
| 18 |
+
"""Initialize scoring dashboard"""
|
| 19 |
+
pass
|
| 20 |
+
|
| 21 |
+
def is_valid_transcript(self, text):
|
| 22 |
+
"""Check if transcript is valid"""
|
| 23 |
+
if not text or not text.strip():
|
| 24 |
+
return False
|
| 25 |
+
invalid_markers = ["[Could not understand audio]", "[Speech recognition service unavailable]",
|
| 26 |
+
"[Error", "[No audio]", "Audio not clear"]
|
| 27 |
+
return not any(marker in text for marker in invalid_markers)
|
| 28 |
+
|
| 29 |
+
def decide_hire(self, result):
|
| 30 |
+
"""
|
| 31 |
+
Make hiring decision - ACCURATE METRICS ONLY
|
| 32 |
+
Uses real, verified measurements
|
| 33 |
+
"""
|
| 34 |
+
reasons = []
|
| 35 |
+
conf = result.get("emotion_scores", {}).get("confidence", 0)
|
| 36 |
+
nerv = result.get("emotion_scores", {}).get("nervousness", 0)
|
| 37 |
+
acc = result.get("accuracy", 0) or 0
|
| 38 |
+
flu = result.get("fluency", 0) or 0
|
| 39 |
+
fluency_level = result.get("fluency_level", "No Data")
|
| 40 |
+
violations = result.get("violations", [])
|
| 41 |
+
|
| 42 |
+
fluency_detailed = result.get("fluency_detailed", {})
|
| 43 |
+
speech_rate = fluency_detailed.get("speech_rate", 0)
|
| 44 |
+
speech_rate_normalized = fluency_detailed.get("speech_rate_normalized", 0)
|
| 45 |
+
grammar_score = fluency_detailed.get("grammar_score", 0)
|
| 46 |
+
grammar_errors = fluency_detailed.get("grammar_errors", 0)
|
| 47 |
+
lexical_diversity = fluency_detailed.get("lexical_diversity", 0)
|
| 48 |
+
coherence_score = fluency_detailed.get("coherence_score", 0)
|
| 49 |
+
filler_count = fluency_detailed.get("filler_count", 0)
|
| 50 |
+
filler_ratio = fluency_detailed.get("filler_ratio", 0)
|
| 51 |
+
pause_ratio = fluency_detailed.get("pause_ratio", 0)
|
| 52 |
+
num_pauses = fluency_detailed.get("num_pauses", 0)
|
| 53 |
+
|
| 54 |
+
has_valid_answer = self.is_valid_transcript(result.get("transcript", ""))
|
| 55 |
+
|
| 56 |
+
# Check for no valid response
|
| 57 |
+
if not has_valid_answer:
|
| 58 |
+
return "β No Valid Response", [
|
| 59 |
+
"β No valid audio response detected",
|
| 60 |
+
"β οΈ Please ensure you speak clearly during recording"
|
| 61 |
+
]
|
| 62 |
+
|
| 63 |
+
# Check for violations
|
| 64 |
+
if len(violations) > 0:
|
| 65 |
+
reasons.append(f"β οΈ {len(violations)} violation(s) detected - under review")
|
| 66 |
+
|
| 67 |
+
# Calculate positive score
|
| 68 |
+
pos = 0
|
| 69 |
+
|
| 70 |
+
# === CONFIDENCE ===
|
| 71 |
+
if conf >= 75:
|
| 72 |
+
pos += 2.5
|
| 73 |
+
reasons.append(f"β
Excellent confidence ({conf}%)")
|
| 74 |
+
elif conf >= 60:
|
| 75 |
+
pos += 2
|
| 76 |
+
reasons.append(f"β
High confidence ({conf}%)")
|
| 77 |
+
elif conf >= 45:
|
| 78 |
+
pos += 1
|
| 79 |
+
reasons.append(f"β Moderate confidence ({conf}%)")
|
| 80 |
+
else:
|
| 81 |
+
reasons.append(f"β οΈ Low confidence ({conf}%)")
|
| 82 |
+
|
| 83 |
+
# === ANSWER ACCURACY (improved with content similarity) ===
|
| 84 |
+
if acc >= 75:
|
| 85 |
+
pos += 3
|
| 86 |
+
reasons.append(f"β
Excellent answer relevance ({acc}%)")
|
| 87 |
+
elif acc >= 60:
|
| 88 |
+
pos += 2
|
| 89 |
+
reasons.append(f"β
Strong answer relevance ({acc}%)")
|
| 90 |
+
elif acc >= 45:
|
| 91 |
+
pos += 1
|
| 92 |
+
reasons.append(f"β Acceptable answer ({acc}%)")
|
| 93 |
+
else:
|
| 94 |
+
reasons.append(f"β οΈ Low answer relevance ({acc}%)")
|
| 95 |
+
|
| 96 |
+
# === FLUENCY ===
|
| 97 |
+
if fluency_level == "Excellent":
|
| 98 |
+
pos += 4
|
| 99 |
+
reasons.append(f"β
Outstanding fluency ({flu}% - {fluency_level})")
|
| 100 |
+
elif fluency_level == "Fluent":
|
| 101 |
+
pos += 3
|
| 102 |
+
reasons.append(f"β
Strong fluency ({flu}% - {fluency_level})")
|
| 103 |
+
elif fluency_level == "Moderate":
|
| 104 |
+
pos += 1.5
|
| 105 |
+
reasons.append(f"β Moderate fluency ({flu}% - {fluency_level})")
|
| 106 |
+
else:
|
| 107 |
+
reasons.append(f"β οΈ Fluency needs improvement ({flu}% - {fluency_level})")
|
| 108 |
+
|
| 109 |
+
# === SPEECH RATE ===
|
| 110 |
+
if speech_rate_normalized >= 0.9:
|
| 111 |
+
reasons.append(f"β
Optimal speech rate ({speech_rate:.0f} WPM)")
|
| 112 |
+
elif speech_rate_normalized >= 0.7:
|
| 113 |
+
reasons.append(f"β Good speech rate ({speech_rate:.0f} WPM)")
|
| 114 |
+
elif speech_rate > 180:
|
| 115 |
+
reasons.append(f"β οΈ Speaking too fast ({speech_rate:.0f} WPM - may indicate nervousness)")
|
| 116 |
+
elif speech_rate < 120:
|
| 117 |
+
reasons.append(f"β οΈ Speaking too slow ({speech_rate:.0f} WPM)")
|
| 118 |
+
|
| 119 |
+
# === GRAMMAR ===
|
| 120 |
+
if grammar_score >= 85:
|
| 121 |
+
pos += 1
|
| 122 |
+
reasons.append(f"β
Excellent grammar ({grammar_score:.0f}% - {grammar_errors} errors)")
|
| 123 |
+
elif grammar_score >= 70:
|
| 124 |
+
reasons.append(f"β Good grammar ({grammar_score:.0f}% - {grammar_errors} errors)")
|
| 125 |
+
elif grammar_score >= 55:
|
| 126 |
+
reasons.append(f"β Acceptable grammar ({grammar_score:.0f}% - {grammar_errors} errors)")
|
| 127 |
+
else:
|
| 128 |
+
reasons.append(f"β οΈ Grammar needs improvement ({grammar_score:.0f}% - {grammar_errors} errors)")
|
| 129 |
+
|
| 130 |
+
# === VOCABULARY ===
|
| 131 |
+
if lexical_diversity >= 65:
|
| 132 |
+
pos += 1
|
| 133 |
+
reasons.append(f"β
Rich vocabulary ({lexical_diversity:.0f}%)")
|
| 134 |
+
elif lexical_diversity >= 50:
|
| 135 |
+
reasons.append(f"β Good vocabulary variety ({lexical_diversity:.0f}%)")
|
| 136 |
+
else:
|
| 137 |
+
reasons.append(f"β οΈ Limited vocabulary ({lexical_diversity:.0f}%)")
|
| 138 |
+
|
| 139 |
+
# === COHERENCE ===
|
| 140 |
+
if coherence_score >= 75:
|
| 141 |
+
pos += 0.5
|
| 142 |
+
reasons.append(f"β
Highly coherent response ({coherence_score:.0f}%)")
|
| 143 |
+
elif coherence_score >= 60:
|
| 144 |
+
reasons.append(f"β Coherent response ({coherence_score:.0f}%)")
|
| 145 |
+
|
| 146 |
+
# === FILLER WORDS (NEW - ACCURATE) ===
|
| 147 |
+
if filler_count == 0:
|
| 148 |
+
pos += 0.5
|
| 149 |
+
reasons.append(f"β
No filler words detected")
|
| 150 |
+
elif filler_count <= 2:
|
| 151 |
+
reasons.append(f"β Minimal filler words ({filler_count})")
|
| 152 |
+
elif filler_count <= 5:
|
| 153 |
+
reasons.append(f"β οΈ Some filler words ({filler_count})")
|
| 154 |
+
else:
|
| 155 |
+
pos -= 0.5
|
| 156 |
+
reasons.append(f"β οΈ Excessive filler words ({filler_count} - impacts fluency)")
|
| 157 |
+
|
| 158 |
+
# === PAUSES ===
|
| 159 |
+
if pause_ratio < 0.15:
|
| 160 |
+
reasons.append(f"β
Good speech flow ({pause_ratio*100:.1f}% pauses)")
|
| 161 |
+
elif pause_ratio < 0.25:
|
| 162 |
+
reasons.append(f"β Acceptable pauses ({pause_ratio*100:.1f}%)")
|
| 163 |
+
else:
|
| 164 |
+
reasons.append(f"β οΈ Frequent pauses ({pause_ratio*100:.1f}% - may indicate hesitation)")
|
| 165 |
+
|
| 166 |
+
# === NERVOUSNESS PENALTY ===
|
| 167 |
+
if nerv >= 60:
|
| 168 |
+
pos -= 1.5
|
| 169 |
+
reasons.append(f"β οΈ Very high nervousness ({nerv}%)")
|
| 170 |
+
elif nerv >= 45:
|
| 171 |
+
pos -= 0.5
|
| 172 |
+
reasons.append(f"β οΈ High nervousness ({nerv}%)")
|
| 173 |
+
|
| 174 |
+
# === VIOLATION PENALTY ===
|
| 175 |
+
if len(violations) > 0:
|
| 176 |
+
violation_penalty = len(violations) * 1.5
|
| 177 |
+
pos -= violation_penalty
|
| 178 |
+
|
| 179 |
+
# === FINAL DECISION ===
|
| 180 |
+
if len(violations) >= 3:
|
| 181 |
+
decision = "β Disqualified"
|
| 182 |
+
reasons.insert(0, "π« Multiple serious violations - integrity compromised")
|
| 183 |
+
elif pos >= 9:
|
| 184 |
+
decision = "β
Strong Hire"
|
| 185 |
+
reasons.insert(0, "π― Exceptional candidate - outstanding communication and competence")
|
| 186 |
+
elif pos >= 7:
|
| 187 |
+
decision = "β
Hire"
|
| 188 |
+
reasons.insert(0, "π Strong candidate with excellent communication skills")
|
| 189 |
+
elif pos >= 5:
|
| 190 |
+
decision = "β οΈ Maybe"
|
| 191 |
+
reasons.insert(0, "π€ Moderate potential - further evaluation recommended")
|
| 192 |
+
elif pos >= 3:
|
| 193 |
+
decision = "β οΈ Weak Maybe"
|
| 194 |
+
reasons.insert(0, "π Below average - significant concerns present")
|
| 195 |
+
else:
|
| 196 |
+
decision = "β No"
|
| 197 |
+
reasons.insert(0, "β Not recommended - needs substantial improvement")
|
| 198 |
+
|
| 199 |
+
return decision, reasons
|
| 200 |
+
|
| 201 |
+
def display_violation_images(self, violations):
|
| 202 |
+
"""Display violation images"""
|
| 203 |
+
if not violations:
|
| 204 |
+
return
|
| 205 |
+
|
| 206 |
+
st.markdown("### π¨ Violation Evidence")
|
| 207 |
+
|
| 208 |
+
for idx, violation in enumerate(violations):
|
| 209 |
+
violation_reason = violation.get('reason', 'Unknown violation')
|
| 210 |
+
violation_time = violation.get('timestamp', 0)
|
| 211 |
+
image_path = violation.get('image_path')
|
| 212 |
+
|
| 213 |
+
col1, col2 = st.columns([2, 3])
|
| 214 |
+
|
| 215 |
+
with col1:
|
| 216 |
+
if image_path and os.path.exists(image_path):
|
| 217 |
+
st.image(image_path, caption=f"Violation #{idx+1}", use_container_width=True)
|
| 218 |
+
else:
|
| 219 |
+
st.error("Image not available")
|
| 220 |
+
|
| 221 |
+
with col2:
|
| 222 |
+
st.markdown(f"""
|
| 223 |
+
**Violation #{idx+1}**
|
| 224 |
+
|
| 225 |
+
- **Type:** {violation_reason}
|
| 226 |
+
- **Time:** {violation_time:.1f}s into question
|
| 227 |
+
- **Status:** β οΈ Flagged for review
|
| 228 |
+
""")
|
| 229 |
+
|
| 230 |
+
if idx < len(violations) - 1:
|
| 231 |
+
st.markdown("---")
|
| 232 |
+
|
| 233 |
+
def display_immediate_results(self, result):
|
| 234 |
+
"""Display immediate results - ACCURATE METRICS ONLY"""
|
| 235 |
+
st.markdown("---")
|
| 236 |
+
st.subheader("π Question Results")
|
| 237 |
+
|
| 238 |
+
# Show accuracy badge
|
| 239 |
+
improvements = result.get("improvements_applied", {})
|
| 240 |
+
if improvements.get('no_fake_metrics'):
|
| 241 |
+
st.success("β
**All metrics verified accurate** - No fake scores included")
|
| 242 |
+
|
| 243 |
+
col_v, col_r = st.columns([2, 3])
|
| 244 |
+
|
| 245 |
+
with col_v:
|
| 246 |
+
if os.path.exists(result.get('video_path', '')):
|
| 247 |
+
st.video(result['video_path'])
|
| 248 |
+
|
| 249 |
+
with col_r:
|
| 250 |
+
# Show violations
|
| 251 |
+
violations = result.get('violations', [])
|
| 252 |
+
if violations:
|
| 253 |
+
st.error(f"β οΈ **{len(violations)} Violation(s) Detected**")
|
| 254 |
+
with st.expander("View Violation Evidence", expanded=True):
|
| 255 |
+
self.display_violation_images(violations)
|
| 256 |
+
|
| 257 |
+
st.write("**π Transcript:**")
|
| 258 |
+
if self.is_valid_transcript(result.get('transcript', '')):
|
| 259 |
+
st.text_area("", result['transcript'], height=100, disabled=True, label_visibility="collapsed")
|
| 260 |
+
else:
|
| 261 |
+
st.error(result.get('transcript', 'No transcript'))
|
| 262 |
+
|
| 263 |
+
# Main metrics (4 columns - NO fake metrics)
|
| 264 |
+
m1, m2, m3, m4 = st.columns(4)
|
| 265 |
+
with m1:
|
| 266 |
+
st.metric("π Confidence", f"{result.get('emotion_scores', {}).get('confidence', 0)}%")
|
| 267 |
+
with m2:
|
| 268 |
+
st.metric("π Accuracy", f"{result.get('accuracy', 0)}%",
|
| 269 |
+
help="Content similarity to ideal answer")
|
| 270 |
+
with m3:
|
| 271 |
+
fluency_level = result.get('fluency_level', 'N/A')
|
| 272 |
+
st.metric("π£οΈ Fluency", f"{result.get('fluency', 0)}%", delta=fluency_level)
|
| 273 |
+
with m4:
|
| 274 |
+
filler_count = result.get('filler_count', 0)
|
| 275 |
+
filler_status = "β
" if filler_count <= 2 else "β οΈ"
|
| 276 |
+
st.metric(f"{filler_status} Filler Words", filler_count,
|
| 277 |
+
help="um, uh, like, etc.")
|
| 278 |
+
|
| 279 |
+
# Enhanced fluency breakdown
|
| 280 |
+
fluency_detailed = result.get('fluency_detailed', {})
|
| 281 |
+
if fluency_detailed:
|
| 282 |
+
st.markdown("---")
|
| 283 |
+
st.markdown("**π Detailed Fluency Analysis (All Accurate):**")
|
| 284 |
+
|
| 285 |
+
fc1, fc2, fc3, fc4 = st.columns(4)
|
| 286 |
+
with fc1:
|
| 287 |
+
speech_rate = fluency_detailed.get('speech_rate', 0)
|
| 288 |
+
speech_rate_norm = fluency_detailed.get('speech_rate_normalized', 0)
|
| 289 |
+
ideal = "β
" if speech_rate_norm >= 0.9 else ("β" if speech_rate_norm >= 0.7 else "β οΈ")
|
| 290 |
+
st.metric(f"{ideal} Speech Rate", f"{speech_rate:.0f} WPM",
|
| 291 |
+
delta=f"Quality: {speech_rate_norm:.2f}")
|
| 292 |
+
with fc2:
|
| 293 |
+
pause_ratio = fluency_detailed.get('pause_ratio', 0)
|
| 294 |
+
num_pauses = fluency_detailed.get('num_pauses', 0)
|
| 295 |
+
pause_status = "β
" if pause_ratio < 0.2 else ("β" if pause_ratio < 0.3 else "β οΈ")
|
| 296 |
+
st.metric(f"{pause_status} Pauses", f"{num_pauses}",
|
| 297 |
+
delta=f"{pause_ratio*100:.1f}% time")
|
| 298 |
+
with fc3:
|
| 299 |
+
grammar = fluency_detailed.get('grammar_score', 0)
|
| 300 |
+
errors = fluency_detailed.get('grammar_errors', 0)
|
| 301 |
+
grammar_status = "β
" if grammar >= 85 else ("β" if grammar >= 70 else "β οΈ")
|
| 302 |
+
st.metric(f"{grammar_status} Grammar", f"{grammar:.0f}%",
|
| 303 |
+
delta=f"{errors} errors")
|
| 304 |
+
with fc4:
|
| 305 |
+
diversity = fluency_detailed.get('lexical_diversity', 0)
|
| 306 |
+
div_status = "β
" if diversity >= 65 else ("β" if diversity >= 50 else "β οΈ")
|
| 307 |
+
st.metric(f"{div_status} Vocabulary", f"{diversity:.0f}%",
|
| 308 |
+
help="Unique meaningful words")
|
| 309 |
+
|
| 310 |
+
# Additional metrics
|
| 311 |
+
st.markdown("**π Additional Metrics:**")
|
| 312 |
+
detail_metrics = fluency_detailed.get('detailed_metrics', {})
|
| 313 |
+
|
| 314 |
+
col_det1, col_det2, col_det3 = st.columns(3)
|
| 315 |
+
with col_det1:
|
| 316 |
+
st.write(f"**Coherence:** {fluency_detailed.get('coherence_score', 0):.0f}%")
|
| 317 |
+
if improvements.get('bert_coherence'):
|
| 318 |
+
st.caption("π§ BERT-enhanced")
|
| 319 |
+
st.write(f"**Avg Pause:** {fluency_detailed.get('avg_pause_duration', 0):.2f}s")
|
| 320 |
+
with col_det2:
|
| 321 |
+
st.write(f"**Total Words:** {detail_metrics.get('total_words', 0)}")
|
| 322 |
+
st.write(f"**Meaningful Words:** {detail_metrics.get('meaningful_words', 0)}")
|
| 323 |
+
with col_det3:
|
| 324 |
+
st.write(f"**Unique Words:** {detail_metrics.get('unique_words', 0)}")
|
| 325 |
+
st.write(f"**Filler Ratio:** {fluency_detailed.get('filler_ratio', 0)*100:.1f}%")
|
| 326 |
+
|
| 327 |
+
st.markdown("---")
|
| 328 |
+
decision = result.get('hire_decision', 'N/A')
|
| 329 |
+
if "β
" in decision:
|
| 330 |
+
st.markdown(f'<div class="success-box"><h3>{decision}</h3></div>', unsafe_allow_html=True)
|
| 331 |
+
elif "β οΈ" in decision:
|
| 332 |
+
st.markdown(f'<div class="warning-box"><h3>{decision}</h3></div>', unsafe_allow_html=True)
|
| 333 |
+
else:
|
| 334 |
+
st.markdown(f'<div class="error-box"><h3>{decision}</h3></div>', unsafe_allow_html=True)
|
| 335 |
+
|
| 336 |
+
st.write("**Reasons:**")
|
| 337 |
+
for r in result.get('hire_reasons', []):
|
| 338 |
+
st.write(f"β’ {r}")
|
| 339 |
+
|
| 340 |
+
def display_performance_overview(self, results):
|
| 341 |
+
"""Display performance overview - ACCURATE METRICS ONLY"""
|
| 342 |
+
st.subheader("π Performance Overview")
|
| 343 |
+
|
| 344 |
+
# Count violations
|
| 345 |
+
total_violations = sum(len(r.get('violations', [])) for r in results)
|
| 346 |
+
questions_with_violations = sum(1 for r in results if len(r.get('violations', [])) > 0)
|
| 347 |
+
|
| 348 |
+
if total_violations > 0:
|
| 349 |
+
st.warning(f"β οΈ **{total_violations} violation(s) detected across {questions_with_violations} question(s)**")
|
| 350 |
+
|
| 351 |
+
valid_results = [r for r in results if r.get("has_valid_data", False)]
|
| 352 |
+
|
| 353 |
+
if valid_results:
|
| 354 |
+
# Calculate averages
|
| 355 |
+
confs = [r.get("emotion_scores", {}).get("confidence", 0) for r in valid_results]
|
| 356 |
+
accs = [r.get("accuracy", 0) for r in valid_results]
|
| 357 |
+
fluencies = [r.get("fluency", 0) for r in valid_results]
|
| 358 |
+
wpms = [r.get("wpm", 0) for r in valid_results]
|
| 359 |
+
filler_counts = [r.get("filler_count", 0) for r in valid_results]
|
| 360 |
+
|
| 361 |
+
# Enhanced metrics
|
| 362 |
+
grammar_scores = [r.get("fluency_detailed", {}).get("grammar_score", 0) for r in valid_results]
|
| 363 |
+
diversity_scores = [r.get("fluency_detailed", {}).get("lexical_diversity", 0) for r in valid_results]
|
| 364 |
+
coherence_scores = [r.get("fluency_detailed", {}).get("coherence_score", 0) for r in valid_results]
|
| 365 |
+
pause_ratios = [r.get("fluency_detailed", {}).get("pause_ratio", 0) for r in valid_results]
|
| 366 |
+
speech_rate_norms = [r.get("fluency_detailed", {}).get("speech_rate_normalized", 0) for r in valid_results]
|
| 367 |
+
|
| 368 |
+
avg_conf = np.mean(confs)
|
| 369 |
+
avg_acc = np.mean(accs)
|
| 370 |
+
avg_flu = np.mean(fluencies)
|
| 371 |
+
avg_wpm = np.mean(wpms)
|
| 372 |
+
avg_filler = np.mean(filler_counts)
|
| 373 |
+
avg_grammar = np.mean(grammar_scores) if grammar_scores else 0
|
| 374 |
+
avg_diversity = np.mean(diversity_scores) if diversity_scores else 0
|
| 375 |
+
avg_coherence = np.mean(coherence_scores) if coherence_scores else 0
|
| 376 |
+
avg_speech_norm = np.mean(speech_rate_norms) if speech_rate_norms else 0
|
| 377 |
+
|
| 378 |
+
# Main metrics
|
| 379 |
+
m1, m2, m3, m4, m5 = st.columns(5)
|
| 380 |
+
with m1:
|
| 381 |
+
st.markdown(f"<div class='metric-card'><h3>{avg_conf:.1f}%</h3><p>Avg Confidence</p></div>", unsafe_allow_html=True)
|
| 382 |
+
with m2:
|
| 383 |
+
st.markdown(f"<div class='metric-card'><h3>{avg_acc:.1f}%</h3><p>Avg Accuracy</p></div>", unsafe_allow_html=True)
|
| 384 |
+
with m3:
|
| 385 |
+
st.markdown(f"<div class='metric-card'><h3>{avg_flu:.1f}%</h3><p>Avg Fluency</p></div>", unsafe_allow_html=True)
|
| 386 |
+
with m4:
|
| 387 |
+
filler_status = "β
" if avg_filler <= 2 else "β οΈ"
|
| 388 |
+
st.markdown(f"<div class='metric-card'><h3>{filler_status} {avg_filler:.1f}</h3><p>Avg Filler Words</p></div>", unsafe_allow_html=True)
|
| 389 |
+
with m5:
|
| 390 |
+
wpm_status = "β
" if 140 <= avg_wpm <= 160 else "β οΈ"
|
| 391 |
+
st.markdown(f"<div class='metric-card'><h3>{wpm_status} {avg_wpm:.1f}</h3><p>Avg WPM</p></div>", unsafe_allow_html=True)
|
| 392 |
+
|
| 393 |
+
# Enhanced fluency breakdown
|
| 394 |
+
st.markdown("### π£οΈ Detailed Fluency Breakdown")
|
| 395 |
+
st.caption("β
All metrics verified accurate - No fake scores")
|
| 396 |
+
|
| 397 |
+
fm1, fm2, fm3, fm4, fm5 = st.columns(5)
|
| 398 |
+
with fm1:
|
| 399 |
+
st.markdown(f"<div class='metric-card'><h3>{avg_grammar:.1f}%</h3><p>Grammar βοΈ</p></div>", unsafe_allow_html=True)
|
| 400 |
+
with fm2:
|
| 401 |
+
st.markdown(f"<div class='metric-card'><h3>{avg_diversity:.1f}%</h3><p>Vocabulary π</p></div>", unsafe_allow_html=True)
|
| 402 |
+
with fm3:
|
| 403 |
+
st.markdown(f"<div class='metric-card'><h3>{avg_coherence:.1f}%</h3><p>Coherence π</p></div>", unsafe_allow_html=True)
|
| 404 |
+
with fm4:
|
| 405 |
+
avg_pause = np.mean(pause_ratios) if pause_ratios else 0
|
| 406 |
+
st.markdown(f"<div class='metric-card'><h3>{avg_pause*100:.1f}%</h3><p>Pause Ratio βΈοΈ</p></div>", unsafe_allow_html=True)
|
| 407 |
+
with fm5:
|
| 408 |
+
norm_status = "β
" if avg_speech_norm >= 0.9 else ("β" if avg_speech_norm >= 0.7 else "β οΈ")
|
| 409 |
+
st.markdown(f"<div class='metric-card'><h3>{norm_status} {avg_speech_norm:.2f}</h3><p>Speech Quality</p></div>", unsafe_allow_html=True)
|
| 410 |
+
|
| 411 |
+
# Overall recommendation
|
| 412 |
+
st.markdown("---")
|
| 413 |
+
st.subheader("οΏ½οΏ½οΏ½οΏ½ Overall Recommendation")
|
| 414 |
+
|
| 415 |
+
if total_violations >= 5:
|
| 416 |
+
st.error("β **Disqualified** - Multiple serious violations detected")
|
| 417 |
+
st.info("Candidate showed pattern of policy violations during interview")
|
| 418 |
+
else:
|
| 419 |
+
# ACCURATE weighted scoring
|
| 420 |
+
overall_score = (
|
| 421 |
+
avg_conf * 0.15 + # Confidence
|
| 422 |
+
avg_acc * 0.25 + # Answer accuracy (improved)
|
| 423 |
+
avg_flu * 0.30 + # Overall fluency (accurate)
|
| 424 |
+
avg_grammar * 0.10 + # Grammar
|
| 425 |
+
avg_diversity * 0.08 + # Vocabulary
|
| 426 |
+
avg_coherence * 0.07 + # Coherence
|
| 427 |
+
(100 - avg_filler * 10) * 0.05 # Filler penalty
|
| 428 |
+
)
|
| 429 |
+
|
| 430 |
+
# Violation penalty
|
| 431 |
+
violation_penalty = total_violations * 5
|
| 432 |
+
final_score = max(0, overall_score - violation_penalty)
|
| 433 |
+
|
| 434 |
+
col_rec1, col_rec2 = st.columns([1, 2])
|
| 435 |
+
with col_rec1:
|
| 436 |
+
st.metric("Overall Score", f"{final_score:.1f}%",
|
| 437 |
+
delta=f"-{violation_penalty}%" if violation_penalty > 0 else None)
|
| 438 |
+
|
| 439 |
+
with col_rec2:
|
| 440 |
+
if total_violations > 0:
|
| 441 |
+
st.warning(f"β οΈ Score reduced by {violation_penalty}% due to {total_violations} violation(s)")
|
| 442 |
+
|
| 443 |
+
if final_score >= 80:
|
| 444 |
+
st.success("β
**Exceptional Candidate** - Strong hire recommendation")
|
| 445 |
+
st.info("Outstanding communication, fluency, and technical competence")
|
| 446 |
+
elif final_score >= 70:
|
| 447 |
+
st.success("β
**Strong Candidate** - Recommended for hire")
|
| 448 |
+
st.info("Excellent communication skills with minor areas for growth")
|
| 449 |
+
elif final_score >= 60:
|
| 450 |
+
st.warning("β οΈ **Moderate Candidate** - Further evaluation recommended")
|
| 451 |
+
st.info("Good potential with notable room for improvement")
|
| 452 |
+
elif final_score >= 50:
|
| 453 |
+
st.warning("β οΈ **Weak Candidate** - Significant concerns")
|
| 454 |
+
st.info("Below expectations in multiple areas")
|
| 455 |
+
else:
|
| 456 |
+
st.error("β **Not Recommended** - Does not meet standards")
|
| 457 |
+
st.info("Substantial improvement needed across all metrics")
|
| 458 |
+
|
| 459 |
+
# Charts
|
| 460 |
+
st.markdown("---")
|
| 461 |
+
st.subheader("π Detailed Analytics")
|
| 462 |
+
|
| 463 |
+
col_chart1, col_chart2 = st.columns(2)
|
| 464 |
+
|
| 465 |
+
with col_chart1:
|
| 466 |
+
st.write("**Performance by Question**")
|
| 467 |
+
chart_data = pd.DataFrame({
|
| 468 |
+
'Question': [f"Q{i+1}" for i in range(len(valid_results))],
|
| 469 |
+
'Confidence': confs,
|
| 470 |
+
'Accuracy': accs,
|
| 471 |
+
'Fluency': fluencies
|
| 472 |
+
})
|
| 473 |
+
st.line_chart(chart_data.set_index('Question'))
|
| 474 |
+
|
| 475 |
+
with col_chart2:
|
| 476 |
+
st.write("**Fluency Components (Accurate)**")
|
| 477 |
+
fluency_breakdown = pd.DataFrame({
|
| 478 |
+
'Component': ['Grammar', 'Vocabulary', 'Coherence', 'Speech Rate', 'Pauses'],
|
| 479 |
+
'Score': [
|
| 480 |
+
avg_grammar,
|
| 481 |
+
avg_diversity,
|
| 482 |
+
avg_coherence,
|
| 483 |
+
avg_speech_norm * 100,
|
| 484 |
+
(1 - np.mean(pause_ratios)) * 100 if pause_ratios else 0
|
| 485 |
+
]
|
| 486 |
+
})
|
| 487 |
+
st.bar_chart(fluency_breakdown.set_index('Component'))
|
| 488 |
+
|
| 489 |
+
def display_detailed_results(self, results):
|
| 490 |
+
"""Display detailed question-by-question analysis"""
|
| 491 |
+
st.markdown("---")
|
| 492 |
+
st.subheader("π Question-by-Question Analysis")
|
| 493 |
+
|
| 494 |
+
for i, r in enumerate(results):
|
| 495 |
+
decision = r.get('hire_decision', 'N/A')
|
| 496 |
+
fluency_level = r.get('fluency_level', 'N/A')
|
| 497 |
+
violations = r.get('violations', [])
|
| 498 |
+
violation_badge = f"β οΈ {len(violations)} violation(s)" if violations else "β
Clean"
|
| 499 |
+
filler_count = r.get('filler_count', 0)
|
| 500 |
+
|
| 501 |
+
with st.expander(f"Q{i+1}: {r.get('question', '')[:60]}... β {decision} | {violation_badge} | Fluency: {fluency_level}", expanded=False):
|
| 502 |
+
# Display violations
|
| 503 |
+
if violations:
|
| 504 |
+
st.error(f"**π¨ {len(violations)} Violation(s) Detected**")
|
| 505 |
+
self.display_violation_images(violations)
|
| 506 |
+
st.markdown("---")
|
| 507 |
+
|
| 508 |
+
col_vid, col_txt = st.columns([2, 3])
|
| 509 |
+
|
| 510 |
+
with col_vid:
|
| 511 |
+
if os.path.exists(r.get('video_path', '')):
|
| 512 |
+
st.video(r['video_path'])
|
| 513 |
+
|
| 514 |
+
with col_txt:
|
| 515 |
+
st.markdown(f"**π Question:** {r.get('question', '')}")
|
| 516 |
+
st.markdown("**π¬ Transcript:**")
|
| 517 |
+
if self.is_valid_transcript(r.get('transcript', '')):
|
| 518 |
+
st.text_area("", r['transcript'], height=80, disabled=True, key=f"t_{i}", label_visibility="collapsed")
|
| 519 |
+
else:
|
| 520 |
+
st.error(r.get('transcript', 'No transcript'))
|
| 521 |
+
|
| 522 |
+
# Main metrics
|
| 523 |
+
m1, m2, m3, m4 = st.columns(4)
|
| 524 |
+
with m1:
|
| 525 |
+
st.metric("π Confidence", f"{r.get('emotion_scores', {}).get('confidence', 0)}%")
|
| 526 |
+
st.metric("π Accuracy", f"{r.get('accuracy', 0)}%")
|
| 527 |
+
with m2:
|
| 528 |
+
st.metric("π° Nervousness", f"{r.get('emotion_scores', {}).get('nervousness', 0)}%")
|
| 529 |
+
st.metric("π£οΈ Fluency", f"{r.get('fluency', 0)}%")
|
| 530 |
+
with m3:
|
| 531 |
+
st.metric("π« Filler Words", filler_count)
|
| 532 |
+
st.metric("π΄ Blinks", f"{r.get('blink_count', 0)}")
|
| 533 |
+
with m4:
|
| 534 |
+
st.metric("π Outfit", r.get('outfit', 'Unknown'))
|
| 535 |
+
st.metric("π¬ WPM", f"{r.get('wpm', 0)}")
|
| 536 |
+
|
| 537 |
+
# Enhanced fluency breakdown
|
| 538 |
+
fluency_detailed = r.get('fluency_detailed', {})
|
| 539 |
+
if fluency_detailed:
|
| 540 |
+
st.markdown("---")
|
| 541 |
+
st.markdown("**π Accurate Fluency Analysis:**")
|
| 542 |
+
|
| 543 |
+
fcol1, fcol2, fcol3 = st.columns(3)
|
| 544 |
+
with fcol1:
|
| 545 |
+
st.write(f"**Grammar:** {fluency_detailed.get('grammar_score', 0):.0f}% βοΈ")
|
| 546 |
+
st.write(f"**Errors:** {fluency_detailed.get('grammar_errors', 0)}")
|
| 547 |
+
st.write(f"**Vocabulary:** {fluency_detailed.get('lexical_diversity', 0):.0f}% π")
|
| 548 |
+
with fcol2:
|
| 549 |
+
st.write(f"**Coherence:** {fluency_detailed.get('coherence_score', 0):.0f}% π")
|
| 550 |
+
st.write(f"**Pauses:** {fluency_detailed.get('num_pauses', 0)}")
|
| 551 |
+
st.write(f"**Pause Ratio:** {fluency_detailed.get('pause_ratio', 0)*100:.1f}% βΈοΈ")
|
| 552 |
+
with fcol3:
|
| 553 |
+
speech_norm = fluency_detailed.get('speech_rate_normalized', 0)
|
| 554 |
+
st.write(f"**Speech Quality:** {speech_norm:.2f}")
|
| 555 |
+
st.write(f"**Fluency Level:** {r.get('fluency_level', 'N/A')}")
|
| 556 |
+
st.write(f"**Filler Ratio:** {fluency_detailed.get('filler_ratio', 0)*100:.1f}%")
|
| 557 |
+
|
| 558 |
+
# Show detailed word counts
|
| 559 |
+
detail_metrics = fluency_detailed.get('detailed_metrics', {})
|
| 560 |
+
if detail_metrics:
|
| 561 |
+
st.markdown("**π Word Analysis:**")
|
| 562 |
+
st.caption(f"Total: {detail_metrics.get('total_words', 0)} | "
|
| 563 |
+
f"Meaningful: {detail_metrics.get('meaningful_words', 0)} | "
|
| 564 |
+
f"Unique: {detail_metrics.get('unique_words', 0)} | "
|
| 565 |
+
f"Fillers: {detail_metrics.get('filler_words_detected', 0)}")
|
| 566 |
+
|
| 567 |
+
if detail_metrics.get('stopword_filtered'):
|
| 568 |
+
st.caption("β
Stopword filtering applied")
|
| 569 |
+
|
| 570 |
+
st.markdown("---")
|
| 571 |
+
st.markdown(f"**Decision:** {decision}")
|
| 572 |
+
st.markdown("**Reasons:**")
|
| 573 |
+
for reason in r.get('hire_reasons', []):
|
| 574 |
+
st.write(f"β’ {reason}")
|
| 575 |
+
|
| 576 |
+
def export_results_csv(self, results):
|
| 577 |
+
"""Export results to CSV - ACCURATE METRICS ONLY"""
|
| 578 |
+
export_data = []
|
| 579 |
+
for i, r in enumerate(results):
|
| 580 |
+
fluency_detailed = r.get('fluency_detailed', {})
|
| 581 |
+
violations = r.get('violations', [])
|
| 582 |
+
detail_metrics = fluency_detailed.get('detailed_metrics', {})
|
| 583 |
+
improvements = r.get('improvements_applied', {})
|
| 584 |
+
|
| 585 |
+
export_data.append({
|
| 586 |
+
"Question_Number": i + 1,
|
| 587 |
+
"Question": r.get("question", ""),
|
| 588 |
+
"Transcript": r.get("transcript", ""),
|
| 589 |
+
"Violations_Count": len(violations),
|
| 590 |
+
"Violation_Details": "; ".join([v['reason'] for v in violations]),
|
| 591 |
+
"Confidence": r.get("emotion_scores", {}).get("confidence", 0),
|
| 592 |
+
"Nervousness": r.get("emotion_scores", {}).get("nervousness", 0),
|
| 593 |
+
"Accuracy": r.get("accuracy", 0),
|
| 594 |
+
"Fluency_Score": r.get("fluency", 0),
|
| 595 |
+
"Fluency_Level": r.get("fluency_level", ""),
|
| 596 |
+
"Speech_Rate_WPM": fluency_detailed.get("speech_rate", 0),
|
| 597 |
+
"Speech_Rate_Normalized": fluency_detailed.get("speech_rate_normalized", 0),
|
| 598 |
+
"Grammar_Score": fluency_detailed.get("grammar_score", 0),
|
| 599 |
+
"Grammar_Errors": fluency_detailed.get("grammar_errors", 0),
|
| 600 |
+
"Lexical_Diversity": fluency_detailed.get("lexical_diversity", 0),
|
| 601 |
+
"Coherence_Score": fluency_detailed.get("coherence_score", 0),
|
| 602 |
+
"Pause_Ratio": fluency_detailed.get("pause_ratio", 0),
|
| 603 |
+
"Avg_Pause_Duration": fluency_detailed.get("avg_pause_duration", 0),
|
| 604 |
+
"Num_Pauses": fluency_detailed.get("num_pauses", 0),
|
| 605 |
+
"Filler_Word_Count": fluency_detailed.get("filler_count", 0),
|
| 606 |
+
"Filler_Word_Ratio": fluency_detailed.get("filler_ratio", 0),
|
| 607 |
+
"Total_Words": detail_metrics.get("total_words", 0),
|
| 608 |
+
"Meaningful_Words": detail_metrics.get("meaningful_words", 0),
|
| 609 |
+
"Unique_Words": detail_metrics.get("unique_words", 0),
|
| 610 |
+
"Unique_Meaningful_Words": detail_metrics.get("unique_meaningful_words", 0),
|
| 611 |
+
"Blink_Count": r.get("blink_count", 0),
|
| 612 |
+
"Outfit": r.get("outfit", ""),
|
| 613 |
+
"Outfit_Confidence": r.get("outfit_confidence", 0),
|
| 614 |
+
"Hire_Decision": r.get("hire_decision", ""),
|
| 615 |
+
"Accurate_Metrics_Only": improvements.get("no_fake_metrics", False),
|
| 616 |
+
"Stopword_Filtering": improvements.get("stopword_filtering", False),
|
| 617 |
+
"Quality_Weighted_Emotions": improvements.get("quality_weighted_emotions", False),
|
| 618 |
+
"BERT_Coherence": improvements.get("bert_coherence", False),
|
| 619 |
+
"Content_Similarity": improvements.get("content_similarity_matching", False),
|
| 620 |
+
"Filler_Word_Detection": improvements.get("filler_word_detection", False)
|
| 621 |
+
})
|
| 622 |
+
|
| 623 |
+
df = pd.DataFrame(export_data)
|
| 624 |
+
csv = df.to_csv(index=False)
|
| 625 |
+
return csv
|
| 626 |
+
|
| 627 |
+
def render_dashboard(self, results):
|
| 628 |
+
"""Render complete results dashboard - ACCURATE METRICS ONLY"""
|
| 629 |
+
if not results:
|
| 630 |
+
st.info("π No results yet. Complete some questions first.")
|
| 631 |
+
return
|
| 632 |
+
|
| 633 |
+
# Show accuracy badge
|
| 634 |
+
if results:
|
| 635 |
+
improvements = results[0].get("improvements_applied", {})
|
| 636 |
+
if improvements.get('no_fake_metrics'):
|
| 637 |
+
st.success("β
**ALL METRICS VERIFIED ACCURATE** | No fake pronunciation, No wrong tempo scores")
|
| 638 |
+
|
| 639 |
+
active_improvements = []
|
| 640 |
+
if improvements.get('stopword_filtering'):
|
| 641 |
+
active_improvements.append("π Stopword Filtering")
|
| 642 |
+
if improvements.get('quality_weighted_emotions'):
|
| 643 |
+
active_improvements.append("βοΈ Quality-Weighted Emotions")
|
| 644 |
+
if improvements.get('content_similarity_matching'):
|
| 645 |
+
active_improvements.append("π Content Similarity")
|
| 646 |
+
if improvements.get('bert_coherence'):
|
| 647 |
+
active_improvements.append("π§ BERT Coherence")
|
| 648 |
+
if improvements.get('filler_word_detection'):
|
| 649 |
+
active_improvements.append("π« Filler Word Detection")
|
| 650 |
+
if improvements.get('grammar_error_count'):
|
| 651 |
+
active_improvements.append("βοΈ Grammar Error Count")
|
| 652 |
+
|
| 653 |
+
if active_improvements:
|
| 654 |
+
st.info("**Real Improvements:** " + " | ".join(active_improvements))
|
| 655 |
+
|
| 656 |
+
# Performance overview
|
| 657 |
+
self.display_performance_overview(results)
|
| 658 |
+
|
| 659 |
+
# Detailed results
|
| 660 |
+
self.display_detailed_results(results)
|
| 661 |
+
|
| 662 |
+
# Export option
|
| 663 |
+
st.markdown("---")
|
| 664 |
+
col_export1, col_export2 = st.columns(2)
|
| 665 |
+
|
| 666 |
+
with col_export1:
|
| 667 |
+
if st.button("π₯ Download Accurate Results as CSV", use_container_width=True):
|
| 668 |
+
csv = self.export_results_csv(results)
|
| 669 |
+
st.download_button(
|
| 670 |
+
"πΎ Download CSV",
|
| 671 |
+
csv,
|
| 672 |
+
f"interview_results_accurate_{time.strftime('%Y%m%d_%H%M%S')}.csv",
|
| 673 |
+
"text/csv",
|
| 674 |
+
use_container_width=True
|
| 675 |
+
)
|
| 676 |
+
|
| 677 |
+
with col_export2:
|
| 678 |
+
# Show accuracy details
|
| 679 |
+
if st.button("βΉοΈ View Accuracy Details", use_container_width=True):
|
| 680 |
+
with st.expander("β
Verified Accurate Metrics", expanded=True):
|
| 681 |
+
st.markdown("""
|
| 682 |
+
### β
What's ACCURATE (Verified & Kept)
|
| 683 |
+
|
| 684 |
+
**π£οΈ Fluency & Speech Analysis:**
|
| 685 |
+
- β
**Speech Rate (WPM)**: Real words per minute calculation
|
| 686 |
+
- β
**Pause Detection**: Librosa audio analysis (actual silence detection)
|
| 687 |
+
- β
**Grammar Checking**: language_tool_python (real grammar rules)
|
| 688 |
+
- β
**Filler Word Count**: Detects "um", "uh", "like", etc. (NEW)
|
| 689 |
+
- β
**Lexical Diversity**: Stopword-filtered vocabulary richness
|
| 690 |
+
- β
**Coherence**: BERT semantic analysis or transition word heuristics
|
| 691 |
+
|
| 692 |
+
**π Answer Quality:**
|
| 693 |
+
- β
**Semantic Similarity**: SentenceTransformer embeddings
|
| 694 |
+
- β
**Content Similarity**: difflib SequenceMatcher (IMPROVED)
|
| 695 |
+
- β
**Keyword Matching**: Honest fallback when needed
|
| 696 |
+
|
| 697 |
+
**π― Emotional & Visual:**
|
| 698 |
+
- β
**Quality-Weighted Emotions**: Face size/lighting/centrality weighted
|
| 699 |
+
- β
**Outfit Analysis**: Multi-criteria color + YOLO classification
|
| 700 |
+
|
| 701 |
+
---
|
| 702 |
+
|
| 703 |
+
### β What's REMOVED (Fake/Inaccurate)
|
| 704 |
+
|
| 705 |
+
- β **Fake Pronunciation Score**: Was hardcoded to 90% (not real analysis)
|
| 706 |
+
- β **Wrong Tempo-Based Fluency**: Used music beat detection (wrong domain)
|
| 707 |
+
- β **Eye Contact in Results**: Removed (still tracked for violations only)
|
| 708 |
+
|
| 709 |
+
---
|
| 710 |
+
|
| 711 |
+
### π― Why This Matters
|
| 712 |
+
|
| 713 |
+
**Fake metrics lead to:**
|
| 714 |
+
- β Bad hiring decisions
|
| 715 |
+
- β Legal liability
|
| 716 |
+
- β Loss of trust
|
| 717 |
+
- β Unfair candidate evaluation
|
| 718 |
+
|
| 719 |
+
**Accurate metrics provide:**
|
| 720 |
+
- β
Fair assessment
|
| 721 |
+
- β
Defensible decisions
|
| 722 |
+
- β
Real insights
|
| 723 |
+
- β
Continuous improvement data
|
| 724 |
+
|
| 725 |
+
---
|
| 726 |
+
|
| 727 |
+
### π Scoring Formula (Accurate)
|
| 728 |
+
|
| 729 |
+
```
|
| 730 |
+
Overall Score =
|
| 731 |
+
Confidence Γ 0.15 +
|
| 732 |
+
Accuracy Γ 0.25 + (Improved similarity)
|
| 733 |
+
Fluency Γ 0.30 + (Real metrics only)
|
| 734 |
+
Grammar Γ 0.10 +
|
| 735 |
+
Vocabulary Γ 0.08 +
|
| 736 |
+
Coherence Γ 0.07 +
|
| 737 |
+
(100 - FillerΓ10) Γ 0.05 (NEW penalty)
|
| 738 |
+
- Violations Γ 5%
|
| 739 |
+
```
|
| 740 |
+
|
| 741 |
+
**All components are REAL and VERIFIED.**
|
| 742 |
+
""")
|
| 743 |
+
|
| 744 |
+
|
| 745 |
+
###
|
yolov8n-cls.pt
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:11fa19f2aea79bc960d680a13f82f22105982b325eb9e17a4a5e1a9f8245980a
|
| 3 |
+
size 5563076
|
yolov8n.pt
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:f59b3d833e2ff32e194b5bb8e08d211dc7c5bdf144b90d2c8412c47ccfc83b36
|
| 3 |
+
size 6549796
|