How to configure AI-Assisted Coding - Admin View

What this feature does

The Coding Assessment with AI Assistant extends your existing iMocha coding assessment. Candidates solve the same coding and database problems in the same editor, with one addition: a built-in AI assistant they can consult while they work. When they finish, you get two independent readings on every candidate — their coding result, scored exactly as your coding tests are today, and their AI fluency.

AI fluency is the ability to understand, collaborate with, and strategically guide the AI Assistant as a thinking partner during a coding task, assessed through how well candidates provide context, think critically, and collaborate with AI.

The assistant is hint-only. It gives one hint, explanation, or guiding question at a time and never writes code that would pass a test case. AI fluency is scored on the candidate's own messages and choices, never on whether their final code worked. The two scores do not move together — a strong coder can be a weak AI collaborator, and vice versa.

💡 Note: The AI Assistant and AI Fluency evaluation are available only for assessments that include coding questions. For general assessments without coding questions, AI Fluency is not applicable and will not appear in the report.

How to enable it

1. Open your test and go to the Settings tab.
2. Under Test Settings, find Enable AI Assistant For Coding Questions.
3. Select Yes. The default is No.
4. Click Save Test Settings.
   
   

The setting is per test and applies at the invitation level. With the setting on No, the test behaves and scores exactly as it does today — no assistant, no AI fluency anywhere in reporting. Enabling it for one test does not affect any other test, existing assessments, or historical results. Past attempts are never re-scored.

Note: The AI Assistant and AI Fluency evaluation are available only for assessments that include coding questions. For general assessments without coding questions, AI Fluency is not applicable and will not appear in the report.

What the candidate sees

The familiar coding-test layout — problem statement, code editor, run, and submit — with an AI assistant chat in a collapsible panel alongside. The assistant has the problem statement pre-loaded, but it cannot see the candidate's code, run results, or editor. The candidate can share their current code or test cases with the assistant using the attach (@) action in the chat. How well they brief the assistant is part of what is measured.

Viewing results on the Reports list

When the feature is enabled, the Reports tab gains two additions:

• AI Fluency badge on each candidate row — displayed beneath the candidate's score, showing their overall level: High, Medium, or Low. Hover over the badge for the AI fluency definition. This lets you scan AI fluency across all candidates without opening individual reports.
• AI Fluency filter — an AI Fluency (All) dropdown in the right-hand filter panel lets you filter the candidate list by level to surface top performers or candidates needing follow-up.

A badge showing AI Fluency: NA indicates there is no AI fluency evaluation for that attempt — for example, an attempt taken before the assistant was enabled for the test, or a test that was left incomplete.

Known issue: The AI Fluency tag currently appears on the Reports grid for all assessments by default, including those without coding questions. This is a display issue and will be fixed shortly. After the fix, the tag will appear on the grid only for assessments that include coding questions. ****

Reading the candidate report

Opening a candidate's report shows five tabs: Summary, Questions, AI Fluency, Proctoring Details, and Audit Log.

Summary tab

Alongside the usual Score Analysis, a condensed AI Fluency For Coding Questions strip shows the overall level badge and the three parameter grades — Context Quality, Critical Thinking, and Collaboration — with a View Details link that opens the full AI Fluency tab.

AI Fluency tab

The dedicated tab contains, top to bottom:

Gauge and overall level. A gauge showing where the candidate lands (Low / Medium / High), with a one-line statement, e.g. "Amit has Low AI Fluency in AI Assisted Coding."

AI Fluency Summary. A plain-language narrative of how the candidate worked with the assistant across the whole test.

Three parameter cards, each with its own A / B / C grade and a written summary of the candidate's behaviour:

• Context Quality — how clearly they briefed the assistant: the right code or query, the exact error, expected-vs-actual behaviour, relevant attachments.
• Critical Thinking — whether they reasoned for themselves: forming hypotheses, attempting first, questioning suggestions rather than extracting answers.
• Collaboration — how they progressed across the conversation: acting on hints, refining, and building on earlier replies instead of restarting.

Token Usage. Input tokens (candidate to AI), output tokens (AI to candidate), and total conversation tokens, showing conversation volume.

AI Fluency Scoring Definitions. A reference panel spelling out what each level and grade means, so anyone reading the report can interpret it without training.

The full candidate–assistant conversation is also available in the report, so you can read the actual exchange behind the grade rather than only the summaries.

The report is generated automatically when the candidate finishes — no manual review step is needed. The Proctoring Details tab reflects whatever proctoring configuration the test already has; this feature does not add or change proctoring.

How to interpret the levels

• High — Demonstrates strong proficiency in leveraging AI tools: providing clear context, exercising critical thinking, and collaborating productively to achieve high-quality outcomes.
• Medium — Shows moderate ability in leveraging AI tools, with adequate context and collaboration but room to improve critical evaluation of AI outputs.
• Low — Demonstrates limited ability to work effectively with AI tools: prompts lack precision, AI outputs are rarely evaluated critically, and collaborative engagement is minimal.

Each parameter carries its own A / B / C grade: A means the behaviour was consistent throughout, B means it was genuine but uneven, and C means it was largely absent.

Two things to keep in mind when comparing candidates:

• The scores are independent. Do not expect a strong coding score to imply a strong AI fluency level or the reverse. You are getting two separate reads on purpose.
• Not using the assistant produces the lowest AI fluency level. A candidate who solves everything without engaging the assistant provides no AI-collaboration evidence, so their AI fluency comes out Low with C on all three parameters. Their coding correctness is scored normally and separately. Per-question engagement matters too: questions where the candidate did not engage the assistant pull the overall read down.

Scenarios at a glance

Downloading the report

The downloaded PDF report includes a dedicated AI Fluency Analysis section: the overall level with the generated narrative, the three parameter cards with grades and summaries, the token-usage strip, and the level and parameter definitions. Use the Download button at the top of the candidate report.

Not available in this release

• Excel export of AI fluency results, and pushing results into an ATS or LMS.
• A practice or preview version of the AI-assisted experience.
• Re-evaluation of a grade on request.

If the assistant is unavailable during a test

The coding part keeps working — candidates can still write, run, and submit code, and submitted work is saved. If the assistant cannot be reached, the candidate should refresh and contact support with the test name and time if the issue persists.