ashby
Agent Architect
Pencil
Remote, GBR
Full-time
Mid · 3+ years experience
Remote
Discovered 2 weeks ago
RAGAPILLM-as-a-judgepromptfooDSPyPython
Free
Job Fit Check
Base Career helps you apply smarter for this job.
?%
Ready to ScanKey skills for this role
RAGAPILLM-as-a-judge
Key Skills for This Role
RAGAPILLM-as-a-judgepromptfooDSPyPython
Full Job Posting
The Role: Architecting the Creative Brain
You won’t just be "prompting"; you will be engineering behavior . You will bridge the gap between creative intent and machine execution, ensuring that our agents are robust, predictable, and capable of high-fidelity output across text, image, and video.
Your work will fall into two high-impact pillars
Core Systems: Designing and scaling the foundational agents that power the Pencil SaaS platform.
Client Solutions: Architecting custom workflows for world-class brands that require specific "brand DNA" and complex creative logic.
Job Responsibilities
- Agent Architecture: Design and implement multi-agent workflows, including task decomposition, state management, and tool-use (RAG, API integration, etc.).
- Systematic Optimization: Move beyond "vibe-based" testing. Implement rigorous evaluation frameworks (e.g., using LLM-as-a-judge, promptfoo, or DSPy) to measure and improve agent performance at scale.
- Scalable Frameworks: Develop reusable agents and prompt libraries that allow the platform to serve 1,000+ brands with unique voices simultaneously.
- Cross-Functional Engineering: Partner with AI Engineering and Product teams to determine which behaviors should be handled via prompting, RAG, or fine-tuning.
- Client Architecture: Act as the technical lead for complex client deployments, translating high-level creative briefs into deterministic AI workflows.
What We’re Looking For
3+ Years of Direct GenAI Experience: You have a deep, intuitive, and technical understanding of LLMs (GPT-4, Claude, Gemini) and multimodal models (Stable Diffusion, Midjourney, Video Gen).
Apply for this job in 1 click
Skip the repetitive application forms
Install the Base Career Chrome Extension and autofill job applications across major job boards with your profile.
Trusted by over 500,000 job seekers on Base Career
More from this employer
More jobs at Pencil
Enterprise Sales Development Representative
, GBR
Discovered 1 weeks agoFull-time
Senior Customer Success Manager
, USA
Discovered 1 weeks agoFull-time
Enterprise Sales Development Representative
, CAN
Discovered 1 weeks agoFull-time
Enterprise Sales Development Representative (SDR)
, GBR
Discovered 1 weeks agoFull-time
Senior Product Manager - EMEA - Remote
, GBR
Discovered 2 weeks agoFull-time
People Operations Specialist
, GBR
Discovered 2 weeks agoFull-time
Head of Legal
, GBR
Discovered 2 weeks agoFull-time
Security Engineer - UK - Remote
, GBR
Discovered 2 weeks agoFull-time