Python Engineer – Evaluator Library

About the role

We are seeking a Python Engineer to design and implement reusable evaluation libraries that ensure the quality, safety, and compliance of enterprise AI agents and LLM‑powered workflows. The role focuses on building deterministic evaluation components that integrate with our AI platform and support automated quality validation across multiple levels of the workflow.

Key responsibilities

• Design, develop, and maintain reusable Python‑based evaluator libraries for AI agents and LLM‑powered workflows.
• Build AWS Lambda functions that perform deterministic quality, compliance, and validation checks.
• Implement LLM‑as‑a‑Judge logic to assess relevance, helpfulness, consistency, and response quality.
• Develop regex‑based PII detection and integrate AWS Bedrock Guardrails.
• Create TOOL_CALL‑level evaluators for JSON Schema compliance and tool response correctness.
• Construct SESSION‑level evaluators to verify workflow contract compliance and execution integrity.
• Implement TRACE‑level evaluators for numerical accuracy and deterministic result validation.
• Integrate evaluation components with AWS AgentCore evaluation workflows and enterprise AI quality pipelines.
• Collaborate with AI Platform Engineers, ML Engineers, and DevOps teams to define evaluation standards and scoring methodologies.
• Write unit, integration, and validation tests and expose evaluation results to Amazon CloudWatch.

Required profile

• Bachelor's degree in Computer Science, Software Engineering, Information Technology or a related field.
• 4+ years of professional experience in Python software engineering.
• Hands‑on experience with LLM evaluation, AI quality assurance, or AI/ML validation frameworks.
• Experience developing and deploying AWS Lambda functions.
• Proven ability to design reusable software libraries and write maintainable, testable Python code.

Required skills

• Python
• AWS Lambda development
• AWS Bedrock Guardrails
• LLM‑as‑a‑Judge prompt engineering
• Regex‑based PII detection
• JSON Schema validation
• Unit and integration testing
• Amazon CloudWatch integration