We are looking for an Agentic Development & Evals Ops Engineer to design and develop AI agents and support evaluation operations for AI systems. The role involves building agent-based workflows, monitoring AI performance, and creating evaluation frameworks to ensure the reliability and accuracy of AI models.

Key Responsibilities:

• Design and develop AI agents and agentic workflows using modern AI/LLM frameworks.
• Build and maintain evaluation pipelines (Evals Ops) to test AI agent performance, accuracy, and reliability.
• Implement automated evaluation frameworks for LLM outputs and agent behaviours.
• Collaborate with engineering and product teams to improve AI model quality and performance.
• Develop monitoring systems for AI agents to track metrics such as response quality, latency, and accuracy.
• Create and maintain documentation, evaluation reports, and testing datasets.
• Work on continuous improvement loops by integrating feedback into agent training and system updates.
• Ensure scalability, reliability, and governance of AI systems.

Required Skills

• Experience with Agentic Development frameworks (LangChain, AutoGen, CrewAI, etc.).
• Knowledge of LLMs and AI systems such as OpenAI, Anthropic, or similar platforms.
• Experience with evaluation frameworks for AI/LLMs (Evals Ops).
• Strong programming skills in Python.
• Experience with prompt engineering and AI workflow automation.
• Knowledge of data analysis and model performance evaluation.
• Familiarity with CI/CD pipelines and experimentation workflows.

Preferred Skills

• Experience with AI observability tools.
• Knowledge of vector databases (Pinecone, Weaviate, FAISS).
• Experience working with RAG (Retrieval Augmented Generation) systems.
• Understanding of AI safety, evaluation metrics, and benchmarking.

Qualifications

• Bachelor’s or Master’s degree in Computer Science, AI, Data Science, or related field.

2+ years of experience in AI/ML development or related roles.