Building the future of virtual lab assessments

Virtual labs have become essential in modern IT training, providing scalable, hands-on experiences without the need for physical infrastructure. However, evaluating user performance remotely remains a complex challenge, requiring reliable validation across various environments and interfaces.

The Challenge

Our client, a leading provider of IT training and certification, relied on curated virtual environments to deliver hands-on labs. Scaling the business to meet demand was lagging because each assessment required either expert human review or bespoke PowerShell scripts. Both approaches were slow, costly, and demanded specialised scripting skills beyond typical content creation.

Our Solution

nTheories partnered with the client to explore possible solutions to this challenge. Our strategic cooperation delivered an evolving assessment automation in three major phases.

Phase 1: Custom Workflow Engine

Our first solution predated the genAI revolution we are currently experiencing. Using a more conventional approach, we solved the need for a human review and significantly reduced the reliance on PowerShell scripts for validation.

  • We created a bespoke solution to capture Windows OS interface interactions at a low level, ensuring speed and accuracy.
  • We created a validation engine with a workflow-based UI, allowing the client’s content team to visually map different solutions to assessment tasks, with complex logic, branching and support for multiple scenarios.
  • We maintained the capability for two-stage validation via PowerShell scripts, incorporated within the scenario definitions.

While the benefit of this approach was tremendous for the client, it still required both effort and training, which limited its potential.

Phase 2: LLM-Powered Automation

Building on the foundation of Phase 1, we recognised the opportunity to leverage emerging capabilities of large language models to streamline the assessment process further. This transition marked a significant shift from manual validation to intelligent automation.

  • Using the same OS signals capture in phase one, the new automation was able to determine task completion by just comparing user actions to the task description as provided to the student.
  • The combination of state-of-the-art models and careful tuning enabled us to achieve error rates below the human baseline. In some situations, the model could determine an outcome that a human might have overlooked with our phase one solution.

This phase dramatically reduced the need for human intervention, allowing content creators to focus solely on instructional design without worrying about validation logic.

Phase 3: Multi-Modal Vision and Agentic Capabilities

As the technology matured, we moved beyond simple model use to a fully agentic system capable of autonomously handling complex validation tasks. This phase introduced a new level of intelligence and flexibility to the platform.

  • We developed agents capable of creating, validating, and executing shell scripts within target environments, eliminating the need for manual scripting. The new architecture maintained the critical capability of dual verification and reduced running costs by only using vision models when needed, as decided by the agent.
  • We took a radically new approach to UI validation - using multi-modal models to identify user actions across different environments visually. We transformed our solution into a truly environment-agnostic assessment platform, able to cover Windows, Linux, MacOS, terminals and device-specific interfaces.

This evolution empowered the platform to deliver scalable, adaptive assessments across a wide range of technical domains, with minimal overhead.

Implementation Highlights

  • Seamless integration with existing lab infrastructure
  • ReAct agent architecture enabling advanced reasoning and dynamic tool invocation
  • Vision-based understanding enabled environment-agnostic assessments

Future Directions

nTheories is extending the platform to include:

  • Fully automated lab provisioning, offloading DevOps teams
  • Real-time adaptive tutoring agents that provide hints and step-by-step walkthroughs

As new AI technologies emerge, we continue to advance the existing products developed for the client, as well as help them identify new opportunities and refine their AI strategy.

Get in touch

Are you exploring how Artificial Intelligence can help you automate your processes and find new avenues of expansion? nTheories can help you navigate the complexity of AI development and integration, leaving you to focus on what really matters – growing your business.

Contact Us