Lead Data Annotation & AI Evaluation Specialist

Role: Lead Data Annotation & AI Evaluation Specialist

Experience: 5–8+ years

Location: Chennai

Notice Period : Immediate

Role Summary

Objectways is seeking a highly technical Lead Data Annotation & AI Evaluation Specialist with at least 5 years of hands-on experience in complex data annotation, AI/ML data operations, or model evaluation.

This role will lead technically complex AI data projects involving LLMs, agentic AI, multimodal data, computer vision, NLP, model evaluation, safety evaluation, and human-in-the-loop workflows.

The ideal candidate must be able to take a detailed customer specification, read it end-to-end, understand the technical and operational intent, identify ambiguities and risks, design the annotation/evaluation workflow, create examples and edge cases, define QA criteria, and guide an annotation team through successful execution.

This is not simply a people-management or annotation-production role. We need someone who can think technically and act as the bridge between the customer specification, engineering/ML teams, and annotation operations.

Key Responsibilities

• Read and interpret complex customer annotation, data-generation, and AI-evaluation specifications.
• Convert requirements into clear SOPs, annotation guidelines, decision trees, examples, edge cases, and QA checklists.
• Independently determine how a project should be executed rather than waiting for step-by-step instructions.
• Design annotation schemas, taxonomies, labels, metadata, acceptance criteria, and review workflows.
• Understand complex multi-turn conversations, model behavior, tool calls, system prompts, agent trajectories, and contextual dependencies.
• Work with JSON/JSONL, structured data, APIs, tool-call traces, logs, model outputs, and annotation platforms.
• Create gold-standard examples and benchmark datasets before production begins.
• Conduct pilot annotations personally and identify gaps in customer guidelines before scaling to the annotation team.
• Define inter-annotator agreement, arbitration, QC sampling, error taxonomy, and acceptance thresholds.
• Analyze disagreements and distinguish annotator error, guideline ambiguity, tooling issues, and genuinely ambiguous data.
• Train annotators and reviewers on technically complex projects and certify readiness before production.
• Work closely with engineering and ML teams to understand model inputs/outputs and improve annotation tooling and workflows.
• Communicate directly with customers or internal project teams to raise technically meaningful clarification questions.
• Monitor production quality and identify systematic errors rather than simply reporting aggregate accuracy.
• Perform root-cause analysis and recommend changes to guidelines, workflows, tooling, or training.
• Own annotation quality from requirements pilot production QC delivery.

Technical Skills

The candidate should be comfortable with:

• LLMs and Generative AI
• AI agents and multi-turn conversational systems
• Prompt/response evaluation
• AI safety and model behavior evaluation
• NLP annotation and classification
• Annotation schema and taxonomy design
• JSON/JSONL and structured datasets
• Tool calls, API concepts, and structured model outputs
• Python or SQL at a working level
• Regular expressions and basic scripting/data analysis
• Annotation platforms such as CVAT, Label Studio, Encord, SageMaker Ground Truth, or equivalent
• Quality metrics including accuracy, precision/recall, confusion matrices, IAA/Cohen's Kappa/Fleiss Kappa
• Dataset validation and error analysis