Agentic AI Developer

Summary

The Agentic AI Developer designs, builds, and operates agentic AI process automations and reusable

LLM‑powered components on our Azure-first stack. Partnering with Business Technology Partners (BTPs)

and teams across the company, you will convert business requirements and process maps into

production-grade automations that are observable, governed, and reusable. A core responsibility is

rigorous quality control: you will develop evaluation workflows (test prompts, regression suites),

implement hallucination detection and mitigation, measure and reduce bias, enforce guardrails and

schema validation, and ensure outputs are grounded, reliable, compliant, and auditable—consistent

with SEC governance and U.S. legal requirements for registered financial advisors.

Who you are

You're a hands-on builder fluent in modern AI tooling who can translate business intent into robust

agentic workflows. You communicate clearly with technical and non-technical stakeholders, execute

with precision, and bring a strong quality mindset to everything you release. You balance speed with

governance, instrument your solutions for observability and cost control, and document decisions for

traceability. You are adaptable, eager to learn new technologies, and able to manage multiple projects

under time constraints.

What you'll do

 Design agentic AI systems that automate multi-step workflows; orchestrate LLM tools/skills and

APIs to deliver reliable outcomes.

 Build reusable components (prompt libraries, tools/functions, skills, RAG pipelines, evaluation

harnesses) for enterprise reuse across teams.

 Integrate LLMs (e.g., ChatGPT, Claude) with Azure services (Azure OpenAI, Azure AI

Search/Vector, Functions, Logic Apps) and Microsoft 365/Copilot Studio.

 Use n8n (and/or Power Automate) for orchestration, eventing, and integrations across line-of-

business systems.

 Implement RAG (retrieval‑augmented generation) with vector search and grounding/citation

policies; enforce structured outputs (JSON schemas).

 Build evaluation & QC: red-team test prompts, regression suites, groundedness checks, bias

assessments, output validation, cost/latency monitoring.

 Instrument observability (logging, trace, metrics), set SLOs (e.g., first‑pass accuracy,

hallucination rate, bias metrics, latency, cost‑per‑task), and implement guardrails.

 Follow intake and prioritization through BTPs/supervisor; collaborate directly with business

stakeholders on requirements and acceptance tests.

 Ensure security & compliance (PII handling, RBAC, data minimization, audit trails) aligned to SEC

governance and company policies (e.g., prompt/response archiving).

 Create documentation and runbooks; contribute to code reviews and best practices; support

end‑users and BTPs with adoption and training.

 Troubleshoot and resolve issues related to AI automations and integrations.

 Stay current on agentic AI frameworks and Azure/M365 capabilities; propose improvements to

platform, tooling, and standards.

 Other duties, responsibilities, and special projects as assigned.

What you'll bring

Experience

 5+ years professional programming experience; 2+ years building AI tool based solutions (LLM

APIs, Copilot, ChatGPT/Claude, orchestration).

 Proven delivery of production automations on Azure and Microsoft 365 ecosystems.

 At least 2 years of experience developing solutions with AI tools (e.g., Copilot, ChatGPT, Claude,

n8n, Foundry).

Technical skills (core)

 Languages: Python, C#, R (plus familiarity with JavaScript/TypeScript helpful).

 Frameworks/tools: LangChain, LangGraph, LlamaIndex, AutoGen (or similar agent frameworks);

Copilot Studio, GitHub, n8n.

 Azure: Azure OpenAI, Azure AI Search/Vector, Functions, Logic Apps, Key Vault, Storage,

DevOps/GitHub Actions.

 RAG pipelines, vector databases (e.g., Azure AI Search; familiarity with Pinecone/Weaviate a

plus).

 API design & integration, structured outputs (JSON schema), function/tool calling.

 Observability & cost control: logging, tracing, metrics, rate/usage monitoring.

Quality control & evaluation (explicit)

 Ability to design and run test prompts and regression suites covering normal, edge, and

adversarial scenarios.

 Skill in hallucination detection/mitigation (groundedness checks, citation requirements, retrieval

confidence thresholds, fallback policies).

 Experience measuring and reducing bias (fairness metrics, scenario/corpus checks, cohort A/B

testing, safe‑prompting).

 Output validation using schema enforcement, tool/function results verification, and

business‑rule assertions.

 Building evaluation harnesses with clear SLOs/KPIs (first‑pass accuracy, hallucination rate, bias

indicators, latency, cost‑per‑task) and reporting dashboards.

 Implementing guardrails (content filters, allow/deny lists, sensitive-topic controls) and red‑team

testing before deployment.

 Documenting test plans, evaluation results, and traceability from requirements to acceptance

criteria.

Collaboration & compliance

 Strong communication with BTPs/end‑users; aptitude for translating requirements into technical

designs and acceptance tests.

 Working knowledge of SEC governance and U.S. regulatory expectations for AI usage in financial

advisory contexts; commitment to Responsible AI practices.

 Familiarity with enterprise archiving/monitoring of prompts and outputs per policy (e.g.,

Smarsh, Theta Lake) and change‑control practices.

Nice to have

 Experience with real‑estate AI platforms (e.g., V7, Kolena, Cadastral) or domain workflows.

 Knowledge of GenAIOps/MLOps concepts for lifecycle and evaluation (not model training).

 Containers (Docker) and CI/CD; Power Platform experience.

Additional Information for Remote Role

 Work Hours: Must have at least 4 hours overlap with Mountain Time zone (Denver, CO).

 Communication: Strong internet connection and reliable communication tools are essential.

 Remote Work Environment: Must have a dedicated workspace free from distractions to ensure

productivity.

 Cultural Sensitivity: Ability to work effectively with a diverse team and understand cultural

differences.