Senior AI Tools Testing Engineer

Job Title: Senior / Lead AI Tools & Testing Engineer (Agentic AI & LLM Evaluation)
Location: Dallas, TX (Hybrid)
Duration: LongTerm
Experience:10 + Years
Interview : Client Round In-person

We seek a senior/lead AI engineer with handson experience building, orchestrating, monitoring, and evaluating productiongrade Generative AI and Agentic AI applications. The ideal candidate is expert in Amazon Bedrock, AgentCore, LangGraph, LangChain, RAG, LLM observability, and evaluation frameworks. This role focuses on creating reliable, scalable, and productionready AI applications, emphasizing agent orchestration, AI testing, LLM evaluation, and continuous quality monitoring.

The candidate will collaborate with engineering, QA, and platform teams to establish best practices for building, testing, monitoring, and improving AIpowered applications.

Key Responsibilities

1. Agentic AI & GenAI Engineering
- Design, develop, and deploy productiongrade agentic and generative AI applications.
- Build and orchestrate AI agents using LangGraph, LangChain, Amazon Bedrock, and AgentCore.
- Implement multiagent workflows, tool calling, state management, memory, and orchestration.
- Create scalable LLMpowered applications and reusable AI components.

2. RAG & LLM Application Development
- Design and implement RetrievalAugmented Generation (RAG) solutions.
- Develop pipelines for document ingestion, chunking, embedding, retrieval, reranking, and context management.
- Optimize retrieval accuracy, response quality, latency, and cost.
- Apply grounding, hallucination reduction, and contextrelevance techniques.

3. LLM Evaluation & AI Testing
- Build evaluation frameworks for LLMbased applications and agentic workflows.
- Automate evaluation with DeepEval and other LLM evaluation tools.
- Use LLMasaJudge methods to assess response quality, relevance, accuracy, faithfulness, and reasoning.
- Create evaluation datasets, test scenarios, benchmarks, and regressiontesting pipelines.
- Evaluate agent behavior, tool execution, workflow accuracy, and response consistency.
- Identify failure patterns and continuously improve AI application quality.

4. AI Observability & Monitoring
- Implement tracing, monitoring, and observability with OpenTelemetry, LangSmith, and Langfuse.
- Track LLM requests, agent execution flows, tool calls, latency, token consumption, errors, and performance.
- Establish debugging mechanisms for complex multistep agentic workflows.
- Develop monitoring practices to detect performance degradation and operational issues.

5. AWS AI Platform & Production Engineering
- Deploy AI applications using Amazon Bedrock and Bedrock AgentCore.
- Integrate foundation models, agent frameworks, APIs, and enterprise data sources.
- Apply productionreadiness practices covering scalability, reliability, security, and performance.
- Support CI/CD integration and automated evaluation of AI applications.
- Optimize model selection, inference costs, latency, and resource utilization.

6. Technical Leadership & Collaboration
- Provide technical guidance to engineering and QA teams.
- Establish reusable frameworks, implementation patterns, and best practices.
- Collaborate with architecture, DevOps, QA, and engineering teams.
- Participate in design discussions, code reviews, proofofconcept development, and implementation.
- Mentor engineers and help teams adopt effective AI development and testing practices.

Required Technical Skills

Agentic AI & Orchestration
- LangGraph
- LangChain
- Amazon Bedrock
- Amazon Bedrock AgentCore
- AI agents, multiagent orchestration, tool calling, workflow management

Generative AI & RAG
- Large Language Models (LLMs)
- RetrievalAugmented Generation (RAG)
- Embeddings and vector search
- Prompt engineering and context management
- LLM application development

LLM Evaluation & Testing
- DeepEval
- LLMasaJudge techniques
- Evaluation metrics and benchmarking
- Automated evaluation and regression testing
- Assessment of hallucination, faithfulness, relevance, and response quality

AI Observability & Monitoring
- OpenTelemetry
- LangSmith
- Langfuse
- LLM tracing, debugging, and performance monitoring

Production Engineering
- Deploying and operating productiongrade GenAI applications
- API integration and backend engineering
- CI/CD and automated testing integration
- Scalability, reliability, security, and performance optimization

Preferred Qualifications
- 8+ years of software engineering or IT experience.
- Handson experience developing and deploying GenAI or Agentic AI applications.
- Experience with enterprisescale AI implementations.
- Familiarity with AWS cloud services and enterprise AI security practices.
- Experience establishing AI quality standards and evaluation methodologies.
- Strong problemsolving, communication, and technical collaboration skills.

What Success Looks Like
- Development and deployment of productiongrade agentic AI applications.
- Reliable orchestration and execution of AI agents and workflows.
- Effective LLM evaluation and automated AI testing frameworks.
- Improved AI application accuracy, reliability, observability, and performance.
- Reusable engineering practices for building and maintaining enterprise GenAI solutions.

Important: This is a handson engineering position. Candidates with practical experience building and evaluating AI applications are preferred over those whose experience is limited to AI strategy, architecture documentation, conventional QA automation, or using AI coding assistants.

Flexible work from home options available.