Secure AI Data Annotation and Governance Platform

PROJECTS

SHARE

The case study below details the technical architecture, implementation methodology, challenges overcome, and quantifiable business results of this project.

A 90-day, multi-phased implementation, delivered extraordinary results:

400% increase in annotation throughput (80 → 300 images/day)
35% improvement in label accuracy (65% → 92% agreement)
6-month schedule acceleration for AI model deployment
60% reduction in annotation costs
40% fewer annotations required via active learning
IL5 compliance achieved for classified data
120 concurrent annotators supported
95% object detection accuracy achieved

Our customer, a defense contractor supporting U.S. Department of Defense AI initiatives, needed to annotate 2M+ images and 50K+ hours of video for object detection and activity recognition models. Manual annotation processes lacked quality control, annotator accountability, and security controls required for classified defense data.

Remaker Digital built a secure, enterprise-grade data annotation platform that increased annotation throughput by 400%, improved label accuracy by 35%, and achieved IL5 compliance for classified data handling—enabling the customer to deliver defense AI models 6 months ahead of schedule.

Business Context

The defense contractor was developing computer vision models for autonomous systems, threat detection, and intelligence analysis. Their AI training programs required massive annotated datasets:

Image annotation: 2M+ satellite, drone, and sensor images
Video annotation: 50K+ hours of surveillance and activity footage
Text annotation: 500K+ intelligence reports and communications
Annotation team: 120 annotators (mix of in-house and cleared contractors)
Project timeline: 18-month model development program
Security requirement: Impact Level 5 (IL5) for classified data

Data Annotation Challenges

The contractor faced critical bottlenecks in annotation workflows:

Low Throughput: Manual processes limited to 50-80 images per annotator per day
Quality Inconsistency: Inter-annotator agreement at 65% (target: 90%+)
No Performance Tracking: Couldn’t identify low-performing annotators or skill gaps
Security Risks: Annotators accessed data via unsecured file shares and consumer tools
Inefficient Task Assignment: Manual task distribution wasted time on low-value annotations
Limited Collaboration: No structured review or consensus mechanisms

Without a scalable annotation platform, the AI model development program risked missing critical deadlines for operational deployment.

Existing Infrastructure

The contractor’s annotation environment included:

Annotation Tools: Mix of CVAT, LabelImg, and custom scripts (no centralized platform)
Data Storage: On-premises file shares with manual access control
Quality Control: Spreadsheet-based sampling and manual review
Security: VPN access to internal network, no IL5-compliant environment
Task Management: Jira for high-level tracking, manual annotator assignment
ML Pipeline: Kubeflow for model training once annotations complete

Technical Requirements

The solution needed to:

Support image, video, text, and sensor data annotation
Achieve IL5 compliance for classified defense data (Impact Level 5)
Integrate with existing Kubeflow ML pipeline for active learning
Scale to 120+ concurrent annotators with performance tracking
Implement multi-stage quality assurance workflows (review, consensus, arbitration)
Deploy in air-gapped, on-premises environment (no internet access)
Provide real-time dashboards for program managers and ML engineers
Support custom annotation schemas for defense-specific use cases

Elapsed time (days): 14

Discovery and Planning

Discovery & Compliance Assessment (2 weeks)

Conducted stakeholder interviews with annotation team leads, ML engineers, and security officers. Analyzed existing annotation workflows and quality metrics. Documented IL5 compliance requirements and security controls. Assessed Kubeflow integration requirements for active learning. Reviewed 1,000+ existing annotations to understand quality challenges.

Elapsed time (days): 21

Architecture Design

Architecture Design & Security Framework (3 weeks)

Designed annotation platform architecture for air-gapped deployment. Created quality assurance workflow with multi-stage review. Designed skill-based task routing and performance tracking system. Developed IL5 security controls documentation (200+ page ATO package). Defined custom annotation schemas for defense-specific use cases. Planned Kubeflow integration for active learning loop.

Elapsed time (days): 42

Development and Integration

Development & Security Hardening (6 weeks)

Built React annotation interface supporting image, video, and text modalities. Developed Django REST Framework backend with PostgreSQL database. Implemented multi-stage quality assurance workflows with inter-annotator agreement calculations. Created annotator performance tracking and leaderboard system. Built Kubeflow connector for active learning integration. Hardened platform for IL5 compliance (encryption, audit logging, access controls).

Elapsed time (days): 21

Testing and Training

Testing & Compliance Validation (3 weeks)

Conducted security testing and penetration testing for IL5 compliance. Tested annotation workflows with 20 pilot annotators across all modalities. Validated quality assurance workflows with golden dataset samples. Load tested platform for 120 concurrent users. Achieved 92% inter-annotator agreement in pilot testing. Completed ATO documentation review with security team.

Elapsed time (days): 14

Deployment

Pilot Deployment & Training (2 weeks)

Deployed platform in classified environment with 40 annotators. Conducted training sessions on annotation interface and quality standards. Integrated with Kubeflow ML pipeline for active learning. Processed 50K annotations during pilot phase. Achieved 300 images/day per annotator throughput. Refined task routing and performance metrics based on feedback.

Elapsed time (days): 8

Handoff to Operations

Full Rollout & IL5 ATO (1 week)

Full rollout to 120 annotators across all defense AI programs. Obtained IL5 Authority to Operate (ATO) from security authorization official. Established weekly quality review meetings with ML engineering team. Deployed active learning integration for all annotation tasks. Achieved target metrics: 300 images/day, 92% agreement, IL5 compliance. Delivered comprehensive documentation and training materials.

IL5 Compliance in Air-Gapped Environment

Deploying a modern web application in an air-gapped, classified environment created unique challenges:

No External Dependencies: All npm packages, Docker images, and Python libraries bundled offline
Manual Updates: Security patches delivered via encrypted physical media
Certificate Management: Internal PKI infrastructure for TLS certificates
Compliance Documentation: 200+ page ATO package covering security controls, system architecture, and risk assessment

Annotator Performance Without Punitive Culture

Tracking performance while maintaining annotator morale required careful design:

Positive Reinforcement: Leaderboards and skill certifications emphasize growth, not punishment
Training, Not Termination: Low performers receive targeted feedback and retraining modules
Transparency: Annotators see their own metrics and understand quality standards
Fair Comparisons: Normalized metrics account for task difficulty (complex vs. simple annotations)

Active Learning Integration Complexity

Integrating with Kubeflow ML pipeline for uncertainty-based annotation prioritization:

Model Uncertainty Scores: Kubeflow exports prediction confidence for each image
Dynamic Prioritization: Annotation queue automatically reprioritizes based on model feedback
Feedback Loop: New annotations trigger model retraining, updating uncertainty scores
Performance Gains: Active learning reduced required annotations by 40% compared to random sampling

Multi-Modal Annotation Consistency

Ensuring consistent quality across image, video, and text annotations:

Unified Quality Metrics: Common accuracy thresholds across all modalities
Cross-Modal Review: Video annotators periodically review image annotations to maintain standards
Schema Standardization: Consistent label taxonomies across image, video, and text tasks

Annotation Throughput & Quality Gains

The annotation platform transformed defense AI data preparation:

400% increase in throughput: From 50-80 images/day to 200-300 images/day per annotator
35% improvement in label accuracy: Inter-annotator agreement increased from 65% to 92%
6-month schedule acceleration: AI model development completed ahead of schedule
120 concurrent annotators: Seamlessly scaled to full team capacity
IL5 compliance achieved: Full ATO granted for classified data handling

Cost Efficiency

Annotation efficiency gains delivered significant cost savings:

60% reduction in annotation costs: Through automation and workflow efficiency
40% fewer annotations required: Active learning eliminated low-value annotations
Reduced rework: Quality assurance workflows prevented costly error propagation

Program Success Impact

The platform enabled successful delivery of defense AI program:

Model performance targets met: 95% object detection accuracy achieved
Operational deployment: AI models deployed to field systems 6 months early
Follow-on contracts: Platform reused for 3 additional defense AI programs
Industry recognition: Featured in DoD AI innovation showcase

Lessons Learned

Quality Assurance is Non-Negotiable: Multi-stage review workflows and inter-annotator agreement metrics are essential for high-stakes defense applications. 92% agreement vs. 65% baseline justified the investment.
Annotator Performance Tracking Drives Improvement: Transparent metrics and skill-based task routing increased productivity while maintaining team morale through positive reinforcement.
Active Learning Reduces Annotation Requirements: Prioritizing uncertain samples cut required annotations by 40%, demonstrating the value of ML-human collaboration.
IL5 Compliance Requires Upfront Design: Security controls can’t be retrofitted. Air-gapped deployment, audit logging, and compliance documentation must be architectural foundations.
Scalable Platforms Enable Program Success: The annotation platform became a strategic asset, reused across multiple defense AI programs and generating long-term value beyond the initial project.

Appendices

Integration Overview

The system integrates with defense AI infrastructure through:

Kubeflow ML Pipeline: Exports annotated data in COCO, YOLO, and custom formats; receives model uncertainty scores for active learning
MinIO Object Storage: S3-compatible storage for images, videos, and sensor data in air-gapped environment
PKI Infrastructure: Internal certificate authority for TLS certificates and mutual authentication
Active Directory: LDAP integration for user authentication and role-based access control
Secure Workstations: Annotators access platform via hardened Windows 10 workstations with no removable media

Model Selection Rationale

The platform itself does not use LLMs for annotation (human annotators perform labeling). However, it integrates with defense AI models for active learning:

Active Learning Integration: ML models (typically YOLOv8, Faster R-CNN, or custom architectures) trained in Kubeflow generate uncertainty scores for each unlabeled sample. Platform prioritizes annotations for high-uncertainty samples, reducing total annotations required by 40%.

Quality Assurance Automation: Automated flagging uses statistical outlier detection (not ML) to identify low-confidence annotations for expert review.

Cost Analysis

Platform development and operational costs:

Development Costs: $450K (4-month project with 6-person team)
IL5 Compliance Costs: $120K (ATO documentation, security testing, penetration testing)
Infrastructure Costs: $80K (on-premises Kubernetes cluster, storage, networking)
Operational Costs: $15K/month (system administration, maintenance, security updates)

Security Architecture (IL5 Compliance)

Impact Level 5 security controls for classified defense data:

Air-Gapped Deployment: No external network connectivity; all updates via encrypted physical media
Access Control: Multi-factor authentication (CAC + PIN), role-based access control (RBAC)
Data Encryption: AES-256 encryption at rest, TLS 1.3 in transit, FIPS 140-2 compliant cryptography
Audit Logging: All user actions logged to write-once audit database with 7-year retention
Network Segmentation: Platform deployed in isolated VLAN with firewall rules
Secure Workstations: Annotators use hardened Windows 10 workstations with disabled USB ports, no removable media, full disk encryption
Compliance: NIST 800-53 High baseline controls, CMMC Level 3, FedRAMP High equivalent
Availability: 99.9% uptime SLA, Kubernetes high availability with 3 control plane nodes, daily backups with 30-day retention
Incident Response: 24/7 security monitoring, incident response playbooks, annual penetration testing