HackerRank open sourced its ATS. My resume scored 90/100. Oh wait 74. No – 88
The open‑source ATS from HackerRank grades resumes out of 100, but the same document can receive different scores — from 66 to 99 — depending on random LLM outputs. The author tested it repeatedly, saw an initial 90, then 74 after removing debug prints, and after 100 runs scores clustered between 78 and 88; a cutoff of 85 leads to a 65 % failure rate.
The evaluation breaks down into five weighted blocks — open‑source contributions (35 points), personal projects (30), work experience (25), technical skills (10) and up to 20 bonus points for startup experience, a portfolio site or technical blog. Technical skills show consistency (8/10 in 98 of 100 runs) because they are checklist items, while project and open‑source scores vary greatly, making the overall result feel like a luck filter. Experience scores cluster tightly around 22‑23, whereas projects and open‑source contributions swing widely. The default model is gemma3:4b at temperature 0.1, yet even temperature 0 does not eliminate the variance. Takeaway: hiring decisions based on such a stochastic score are a roll of the dice.