Platform Engineer – CI/CD Gate & Online Evaluation

About the role

The Platform Engineer – CI/CD Gate & Online Evaluation will join the Core Architecture Team to embed automated quality evaluation into AI deployment pipelines. You will ensure AI agents meet defined quality standards before release and continue to perform reliably in production environments.

Key responsibilities

• Design and implement CI/CD quality gates using AWS AgentCore Evaluation to validate AI agents before deployment.
• Integrate evaluation execution into GitHub Actions and deployment pipelines to automate release decisions.
• Configure and maintain on‑demand evaluation workflows for pre‑release validation and regression detection.
• Implement online evaluation capabilities, including production sampling strategies and scheduling.
• Build Amazon CloudWatch dashboards to visualize evaluation scores, quality trends, and AI performance.
• Set up CloudWatch alarms and notifications to detect quality degradation or performance regressions.
• Collaborate with AI Engineering, Platform Engineering, and DevOps teams to define evaluation thresholds and release criteria.
• Analyze evaluation outcomes to identify regressions across AI agent versions, prompts, tools, and workflows.

Required profile

• Proven experience in platform engineering and CI/CD pipeline development.
• Strong background working with AWS services, especially CloudWatch and custom evaluation tools.
• Hands‑on experience integrating automated checks into GitHub Actions or similar CI systems.
• Ability to work closely with AI, DevOps, and governance teams to define and enforce quality standards.

Required skills

• AWS
• GitHub Actions
• Amazon CloudWatch
• CI/CD pipelines
• Automation scripting
• AI evaluation workflows