AI Evaluation Engineer – Design Real‑World Benchmark Tasks

About the role

Gramian Consultancy seeks an AI Evaluation Engineer to create realistic, terminal‑based benchmark tasks that assess how large language models reason through debugging, operational failures, and complex multi‑step workflows. The role is fully remote and can be performed full‑time or part‑time over a five‑week contract.

Key responsibilities

• Design technically deep debugging and investigation scenarios for AI evaluation systems.
• Develop task specifications that involve infrastructure, pipelines, and operational failure modes.
• Write clear solution approaches and deterministic evaluation criteria.
• Identify realistic edge cases, failure modes, and system constraints.
• Craft multi‑step reasoning challenges across complex technical environments.
• Collaborate with reviewers and researchers to refine benchmark quality and validation logic.

Required profile

• 3‑10 years of experience in software engineering or related technical domains.
• Strong analytical, debugging, and systems‑reasoning abilities.
• Good understanding of system architecture, dependencies, and operational processes.
• Experience with terminal, CLI, automation, or developer‑tooling workflows.
• Exposure to AI systems, LLMs, benchmarking, or evaluation frameworks is a plus.

Required skills

• Backend engineering
• Infrastructure
• DevOps
• Data systems
• MLOps
• Cybersecurity
• Platform engineering
• Terminal / CLI
• Automation
• Developer tooling
• AI systems
• Large language models (LLMs)
• Benchmarking
• Evaluation frameworks