Nowadays the hiring landscape in the market is more competitive than ever. With an increasing number of candidates applying for each role, recruiters and hiring managers are often overwhelmed by the enormous volume of resumes to review. Efficient candidate assessment has become a critical factor in identifying the right talent, especially for roles where specific skills and experience are essential.
In this blogpost, I combine RAG (Retrieval-Augmented Generation) and LLMs (Large Language Models), two cutting-edge technologies to showcase a way to streamline the candidate assesment (from candidate extraction to resume summarization) and having a human-in-the-loop to validate the final results. This approach prioritizes privacy protection through data anonymization and offers an in-premise solution, eliminating reliance on third-party software.
Unlike traditional ATS systems, it uses advanced AI analytics to deliver faster, more accurate, and efficient hiring. By automating tedious tasks and enabling human oversight, this solution ensures a balanced, secure, and future-proof recruitment process. Discover how cutting-edge AI can transform your hiring strategy!
The Challenge of Candidate Selection
Traditional candidate selection processes often involve manually sifting through hundreds or even thousands of resumes! This approach is not only time-consuming but also prone to human error and bias. Recruiters may overlook qualified candidates due to the overwhelming volume of applications or fail to identify the best fit for highly specialized roles.
The stakes are high in the hiring business. A single bad hire can cost a company significant time and resources, while the right candidate can drive innovation and growth. Finding the perfect match requires a more efficient and data-driven approach, which is where RAG and LLMs come into play.
Applicant tracking system (ATS) vs AI Systems
Non-AI systems for candidate assessment and hiring processes often fall short in several key areas compared to AI-driven solutions like RAG and LLMs.
Non-AI Systems:
Often use simple keyword matching or rigid criteria (e.g., "must have 5 years of experience") to filter resumes. This approach can miss qualified candidates who use different terminology or have non-traditional career paths.
Provide limited insights into candidate suitability. Recruiters often rely on gut feelings or surface-level information, which may not accurately predict job performance.
AI Systems:
Use advanced natural language processing (NLP) to understand context, synonyms, and related skills. For example, RAG can identify that "Python programming" and "Python development" refer to the same skill, ensuring more accurate matching.
Offer deeper insights by analyzing patterns in candidate data. For example, LLMs can identify trends like which skills are most common among top performers or which candidates are likely to excel in specific roles.
Implementing RAG + LLM pipeline for candidate asseseement
RAG is a hybrid AI model that combines retrieval-based methods wfith generative capabilities (I have reviewed RAG in a previous post). In my implementation, I consider the following steps:
Data anonymization for privacy protection (I follow this guide that combines NER + LLMs to anonymize textual data effectively)
Create encodings from resumes to build vector database for relevant resume fetching
Retrieving relevant resumes from a database based on the job description.
Relevance score of the resume with respect to the job description from an ensemble of LLMs
Generating insights or summaries of the top 3 candidates for the role.
Once candidates are extracted, LLMs (Large Language Models) step in to evaluate their suitability to the job description. For instance, an LLM might score a candidate highly for their expertise in Python and cloud computing but note a lack of experience with DevOps tools. These scores provide hiring teams with actionable insights, allowing them to prioritize candidates who are the best fit for the role. The scoring process also uncovers patterns and trends, such as which skills are most common among top candidates or where gaps exist in the applicant pool. This information can inform future hiring strategies and training programs.
Resume Summarization
After scoring, the next step is summarizing the top 3 candidate resumes. LLMs can generate concise, easy-to-read summaries that highlight key qualifications, experiences, and skills as well as lack of skills.
These summaries enable hiring managers to quickly review candidates and make informed decisions without having to read through lengthy resumes. This is particularly valuable in high-volume hiring scenarios, where time is of the essence.
Prompting
In the aforementioned steps, I implemented two prompts that worked generally good across 4 different LLMs which will be mentioned later. The first prompt is about the scoring of the candidate in accordance to the job description. The latter is about summarizing the resume and highlight pros and cons again based on the resume. Each prompt does not consider more than one resumes; therefore, each candidate evaluation is independent of the others.
Scoring prompt:
You are an expert in evaluating job candidates. Analyze the JOB DESCRIPTION and RESUME below.
### INSTRUCTIONS:
Assign a SCORE from 1 to 5, where:
1 = Not qualified at all (minimal or no match to the job requirements).
2 = Slightly qualified (some relevance but significant gaps).
3 = Moderately qualified (meets some key requirements but has significant gaps).
4 = Highly qualified (strong match with minor gaps).
5 = Perfect match (exceeds or fully meets all key requirements).
The SCORE must be an integer. Return the score ONLY in this JSON format:
{{
"score": <integer>
}}
### JOB DESCRIPTION:
{}
### RESUME:
{}
### Response: Summazing prompt:
You are an expert in evaluating job candidates. Your task is to analyze a candidate's resume in the context of a job description and provide a brief summary of their pros (strengths) and cons (weaknesses) for the role.
### INSTRUCTIONS:
1. Carefully review the JOB DESCRIPTION and the RESUME provided.
2. Identify the candidate's key strengths (pros) that align with the job requirements.
3. Identify the candidate's key weaknesses (cons) that may hinder their performance in the role.
4. Focus only on relevant skills, experiences, and qualifications mentioned in the resume and job description.
5. Be objective and avoid unnecessary commentary.
6. Provide a brief summary.
### JOB DESCRIPTION:
{}
### RESUME:
{}
### Response:Employed Models
The LLMs that I used for this usecase were deployed through ollama locally. In order to have a good and diverse evaluation, I selected 4 LLMs of varying size and architecture. The list of employed LLMs from ollama's repository is the following:
Llama3.2, 3 billion parameters
Qwen2.5, 14 billion parameters
Phi4, 14 billion parameters
Qwen2.5, 32 billion parameters
As for the sentence-embeddings model which encodes the resumes, I use:
Alibaba-NLP/gte-Qwen2-1.5B-instruct
Using an ensemble of LLMs is better than using a single LLM because it combines the strengths of multiple models, reducing the risk of errors or biases from any one model. This approach improves accuracy, robustness, and reliability by leveraging diverse perspectives and capabilities.
From Input Data to Output Results: The RAG + LLM Pipeline in Action
For the system to be usable, it is essential to have resumes for fetching! For the sake of visualization, I employed an open-source dataset from Kaggle for resumes which contains about 2,5K resumes from people in different professions (pdf format). This data has been parsed using the best pdf parser (see blogpost) and then anonymized using NER+LLM techniques to remove any PII information that can make the process biased.
For the job descriptions, I have manually crawled a couple of ads from various domains to see the retrieved results and outcomes. I will give an example below, including all the system's outputs from the top candidate only (job posting is anonymised).
Job description
Bachelors degree or equivalent practical experience years of experience in saas or productivity tools businessexperience managing enterprise accounts with sales cycles\npreferred qualifications\n years of experience building strategic business partnerships with enterprise customersability to work through and with a reseller ecosystem to scale the businessability to plan pitch and execute a territory business strategyability to build relationships and to deliver results in a crossfunctionalmatrixed environmentability to identify crosspromoting and uppromoting opportunities within the existing account baseexcellent account management writtenverbal communication strategic and analyticalthinking skills\nabout the job\nas a member of the google cloud team you inspire leading companies schools and government agencies to work smarter with google tools like google workspace search and chrome you advocate the innovative power of our products to make organizations more productive collaborative and mobile your guiding light is doing whats right for the customer you will meet customers exactly where they are at and provide them the best solutions for innovation using your passion for google products you help spread the magic of google to organizations around the world\nthe google workspace team helps customers transform and evolve their business through the use of googles productivity collaboration and content management suite of applications as part of an entrepreneurial team in this growing business you will help shape the future of businesses use technology to connect with customers employees and partners\nas a google workspace sales specialist you will be responsible for maintenance and expansion of google workspace business growth across the region with customers in this role youll create and execute the strategy and provide unique insights on applying google workspace solutions to enterprisesyou will build an excellent pipeline and work with the account teams to build out the customer solution and establish partnerships you will strategize with partners to increase account and territory business growth you will work directly with customers coordinate internal resources and construct successful strategies at account and territory level\ngoogle cloud accelerates organizations ability to digitally transform their business with the best infrastructure platform industry solutions and expertise we deliver enterprisegrade solutions that leverage googles cuttingedge technology all on the cleanest cloud in the industry customers in more than countries and territories turn to google cloud as their trusted partner to enable growth and solve their most critical business problems'LLM's summarization:
**Brief Summary:**
The candidate, a seasoned Business Development Executive with 20 years of experience in managing enterprise accounts and cultivating strong customer relationships, is well-suited for the Sales Specialist role. Key strengths include:
* Proven track record of growing account bases, generating new business acquisition, and delivering profitable contracts
* Excellent negotiation, financial analysis, and relationship-building skills
* Experience with CRM software (Salesforce) and office productivity tools like Office 365
However, potential weaknesses to consider are:
* Limited direct experience with SaaS or productivity tools, which may require additional training or adaptation
* No explicit mention of strategic business partnership building or reseller ecosystem management, although experience in managing enterprise accounts and GPOs suggests transferable skills
Overall, the candidate's extensive sales experience, customer relationship-building expertise, and familiarity with CRM software make them a strong fit for this role.The system was able to pull out of 2,5K resumes the most relevant ones which had been graded by 4 LLMs and then the top resume was selected for summarization. This flow speeds up exhaustive search from HR professionals and makes the whole process easier for organizations!
Challenges and Considerations
While RAG and LLMs offer significant benefits, there are challenges to consider:
Limitations: AI models may struggle with highly nuanced or unconventional resumes.
Ethical Concerns: Automated systems must be carefully designed to avoid bias and ensure fairness.
Human Oversight: AI should augment, not replace, human judgment. Recruiters play a crucial role in the final selection process.
By addressing these challenges, companies can maximize the value of RAG and LLMs while maintaining ethical and inclusive hiring practices.
Conclusion
The deployment of RAG and LLMs in the hiring process represents a significant leap forward in candidate assessment. By automating time-consuming tasks like resume screening and summarization, these technologies enable recruiters to focus on what truly matters: finding the best talent for their organizations.
Another idea is to flip the script and instead of looking for resumes, is to look for jobs! In this scenario, the pool is the job descriptions where the candidate is looking for the most relevant job based on their skills and experiences. That would be probably a function of an advanced job search engine where the candidate uploads their CV and the system shows them the most relevant jobs.
I hope my insights and tips help pave the way for your success in fine-tuning LLMs for real-world applications. Happy coding!
** If you are interested in other ML use-cases, please contact me using the form (and also include a publicly available dataset for this case, I'm always curious to explore new problems).