Software Engineer - Evaluation (Remote)
Job Fit Check
Base Career helps you apply smarter for this job.
Key skills for this role
About the Role
We are seeking a Senior Software Engineer for a part-time role focused on LLM evaluation and repository validation. You will analyze GitHub issues, set up code repositories, and evaluate LLM performance on software engineering tasks.
Key Skills for This Role
Responsibilities
- Analyze and triage GitHub issues across trending open source libraries to assess LLM performance
- Set up and configure code repositories, including Dockerization and environment setup for testing
- Evaluate unit test coverage and quality across complex codebases
- Modify and run codebases locally to assess how LLMs perform in bug fixing scenarios
- Collaborate with researchers to design and identify repositories and issues that challenge LLMs
Requirements
- Minimum 3+ years of overall software engineering experience
- Strong proficiency in Go and familiarity with Git, Docker, and basic software pipeline setup
- Ability to understand and navigate complex codebases efficiently
- Experience running, modifying, and testing real world projects locally
- Comfortable contributing to or evaluating open source projects
Full Job Posting
Role Overview
- We are hiring for one of our clients, seeking a Senior Software Engineer – Go (LLM Evaluation & Repository Validation) to work on a part time basis.
- The project involves building LLM evaluation and training datasets to train models on realistic software engineering problems.
- This includes developing verifiable software engineering tasks based on public repository histories using a synthetic approach with human in the loop input.
Key Responsibilities
- Analyze and triage GitHub issues across trending open source libraries to assess LLM performance.
- Set up and configure code repositories, including Dockerization and environment setup for testing.
- Evaluate unit test coverage and quality across complex codebases.
- Modify and run codebases locally to assess how LLMs perform in bug fixing scenarios.
- Collaborate with researchers to design and identify repositories and issues that challenge LLMs.
Required Skills & Qualifications
- Minimum 3+ years of overall software engineering experience.
- Strong proficiency in Go and familiarity with Git, Docker, and basic software pipeline setup.
- Ability to understand and navigate complex codebases efficiently.
- Experience running, modifying, and testing real world projects locally.
- Comfortable contributing to or evaluating open source projects.
More About the Opportunity
- This role offers a unique opportunity to work with a global leader in the technology industry.
- Contributing to the advancement of AI assisted software development.
- The position blends practical software engineering with AI research, enabling hands on impact in evaluating LLMs against real world coding challenges.
Equal Opportunity Employer
- We hire based on skills and expertise.
- All qualified candidates are welcome regardless of background, experience, or prior employment history.
- Applications are reviewed solely on demonstrated technical ability and qualifications.
Apply for this job in 1 click
Skip the repetitive application forms
Install the Base Career Chrome Extension and autofill job applications across major job boards with your profile.
Trusted by over 500,000 job seekers on Base Career
More from this employer
More jobs at Hire Feed
Design Expert (Remote)
Abu Dhabi, UAE
Role : Design Expert (Remote) Location : Remote (Work from Anywhere) Job Type : Contract Payout : $15 - $40/hour Role Overview: We are hiring for one of our clients, seeking a Design Expert (Multidisciplinary) to work
Image/Video Annotator (Remote)
Abu Dhabi, UAE
Role : Image/Video Annotator (Remote) Location : Remote (Work from Anywhere) Job Type : Part-Time Payout : Competitive, based on experience Role Overview: We are hiring for one of our clients to assist in advancing la
Electrical Systems Specialist (Remote)
Abu Dhabi, UAE
Hire Feed seeks an Electrical Systems Specialist for a contract role involving on-site inspections, diagnostics, and field operations of electrical systems across multiple locations. Requires experience with electrical s
Network Security Engineer (Hybrid)
Abu Dhabi, UAE
We are hiring a Network Security Engineer for a contract role. You will design, deploy, and maintain secure network infrastructure, monitor traffic, and respond to incidents. Requires 3+ years of experience and proficien
Mathematics Expert (Remote)
Abu Dhabi, UAE
We are hiring a Mathematics Expert (Master's/Ph.D.) to work part-time on projects that improve and evaluate large language models by applying advanced mathematical reasoning, problem-solving, and clear written communicat
Math Specialist (Remote)
Abu Dhabi, UAE
We are hiring a Math Expert (PhD) to develop and optimize mathematical models for AI-driven platforms on a contract basis. The role involves designing algorithms, analyzing datasets, collaborating with cross-functional t
Management Consultant (Remote)
Abu Dhabi, UAE
We are hiring a Management Consultant for a contract role with a global Information Services client. You will analyze operational processes, implement efficiency improvements, and develop standardized procedures for remo
Business Consultant (Remote)
Abu Dhabi, UAE
We are hiring a Business Consultant for a contract role with a global Information Services client. You will analyze business processes, identify inefficiencies, and recommend strategic improvements. Requires management c
Design Expert (Remote)
Abu Dhabi, UAE
Image/Video Annotator (Remote)
Abu Dhabi, UAE
Electrical Systems Specialist (Remote)
Abu Dhabi, UAE
Network Security Engineer (Hybrid)
Abu Dhabi, UAE
Mathematics Expert (Remote)
Abu Dhabi, UAE
Math Specialist (Remote)
Abu Dhabi, UAE
Management Consultant (Remote)
Abu Dhabi, UAE
Business Consultant (Remote)
Abu Dhabi, UAE