Senior Python Engineer – AI Coding Agent Evaluation (Freelance)

About the role

Mindrift is looking for a senior Python engineer to help build a dataset for evaluating AI coding agents. This freelance, project‑based role involves designing realistic developer environments, crafting challenging tasks, and writing robust evaluation tests.

Key responsibilities

• Create virtual company environments with codebases, infrastructure, tickets, documentation, and conversation histories.
• Design intermediate‑state tasks, define clear success criteria, and ensure they are solvable by AI agents.
• Write functional and integration tests that accept all correct solutions while rejecting incorrect ones.
• Iterate on tasks and tests based on QA feedback, analysing agent failures and refining evaluation criteria.

Required profile

• 8+ years of professional software development experience.
• Strong understanding of where current AI coding models succeed and fail.
• Ability to design fair, robust evaluation scenarios for advanced models.
• Proficient English (minimum B2 level).

Required skills

• Python (FastAPI)
• JavaScript / TypeScript (React)
• Docker
• PostgreSQL
• Kafka
• Redis
• Writing functional and integration tests

What we offer

• Up to $150 per hour, based on experience and pace.
• Fully remote work with flexible scheduling.
• Opportunity to influence cutting‑edge AI evaluation methodology.