Importance of Hackathon

Hackathons have become a vital platform for driving innovation, collaboration, and skill development across diverse industries. These intense, time-bound events bring together programmers, designers, and problem solvers to rapidly prototype new ideas and tackle real-world challenges. Beyond fostering creativity, hackathons enable participants to expand their technical skills, network with like-minded professionals, and gain industry recognition.

The “DevOps for GenAI” hackathon, recently held in Ottawa, showcased 15 competitive projects at the intersection of DevOps and Generative AI. The event highlighted the growing convergence of AI observability and prompt engineering, with projects split between these two domains. This white paper outlines the intent, technical gaps, strengths, future perspectives, and collaboration requirements of the presented projects, drawing from the available repositories and event summaries.

• AI Observability: Roughly half the projects focused on monitoring, debugging, and optimizing AI systems. These solutions aimed to provide real-time insights into model performance, resource utilization, and anomaly detection, reflecting the industry’s need for robust AI system management.
• Prompt Evaluation: The remaining teams targeted prompt engineering and evaluation tools for GenAI systems, aiming to improve prompt quality, automate testing, and provide feedback loops for continuous improvement.

The rapid adoption of large language models (LLMs) and generative AI (GenAI) systems in enterprise and research settings has elevated the need for advanced observability, monitoring, and accessibility solutions. This white paper explores the combined strengths and future potential of open-source projects produced during the DevOps for GenAI Hackathon, Ottawa edition. It particularly explores three projects in the AI Observability space —InsightAI_Minions, AI Observability Monitoring InnerAI, and GenA11yHelper—to deliver a unified, extensible, and accessible framework for GenAI deployment and operation.

AI Observability & Monitoring (Inner AI)

Presented by James Drayton Beninger, Amogh Thoutu, Farhan Mahamud, Shubhangi Singh, Bardia Azami, Inner AI Group, Canada DevOps Community 2025

AI Observability & Monitoring (Inner AI) middleware solution, designed to enhance transparency, reliability, and accountability in text-based generative AI (GenAI) applications. The solution integrates seamlessly into developer workflows, providing real-time monitoring, metrics, and testing capabilities for large language model (LLM) interactions. We discuss the system’s core strengths, outline future directions for research and development, and identify opportunities for collaboration within the broader AI and DevOps communities. The goal is to foster robust, trustworthy AI deployments across industries by advancing observability standards and practices. The Inner AI solution comprises four main modules:

• API Wrapper: Intercepts and augments LLM prompt/response cycles.
• Metrics: Collects detailed telemetry on LLM performance and behavior.
• Dashboard: Visualizes real-time and historical metrics for developers.
• Testing Module: Simulates end-user interactions to validate system robustness.

Key Strengths

• Seamless Integration: Designed as middleware, the system requires minimal changes to existing application code, supporting rapid adoption.
• Comprehensive Monitoring: Captures both quantitative (latency, error rates) and qualitative (response quality, prompt drift) metrics.
• Automated Testing: Simulates diverse user scenarios, enabling proactive detection of issues such as hallucinations or degraded model performance.
• Extensibility: Built using widely adopted frameworks (e.g., FastAPI, Prometheus, LangChain), facilitating customization and scalability.
• Developer-Centric Design: Provides actionable insights through an intuitive dashboard, accelerating debugging and model iteration cycles.

InsightAI_Minions: Enabling Observability and KPI Monitoring for LLM Deployment

By Denis Shleifman, Morteza Mirzaei, Dorian Gerdes, Manav Isrrani, Sachin Kumar, Canada DevOps Community 2025

InsightAI_Minions, an open-source framework for deploying, monitoring, and benchmarking large language models (LLMs) using DevOps best practices. The system provides automated KPI measurement, advanced alerting, and extensible observability for LLM deployments. We discuss the strengths of the platform, future development directions, and the need for collaborative efforts across the AI and DevOps communities to advance the state of LLM observability and reliability.

The rapid adoption of LLMs in production environments requires robust tools for deployment, monitoring, and performance evaluation. InsightAI_Minions addresses this gap by providing a comprehensive solution for automated LLM deployment and observability, leveraging modern DevOps pipelines and open-source monitoring stacks.

Strengths of InsightAI_Minions

• Automated Deployment: Utilizes GitHub Actions and Docker Compose for seamless LLM deployment, supporting continuous integration and delivery workflows.
• Comprehensive KPI Measurement: Simulates diverse test prompts (from a set of 50,000) to measure latency, token throughput, and success rates, providing actionable insights into LLM performance.
• Advanced Observability: Integrates OpenTelemetry (OTEL) for metrics collection and VictoriaMetrics for time-series storage, with visualization via Grafana dashboards.
• Proactive Alerting: Implements advanced alerting (warning and critical) with configurable Slack notifications, enabling rapid incident response.
• User-Defined KPIs: Allows users to define custom metrics and dashboards, enhancing flexibility for varied deployment scenarios.

InsightAI_Minions represents a significant step forward in operationalizing LLM deployments with robust observability, benchmarking, and alerting capabilities. Continued collaboration across industry, academia, and standards organizations is essential to address evolving challenges and ensure reliable, scalable, and transparent AI systems.

Future Perspectives

• Expanded Integration: Support for additional deployment platforms (e.g., Kubernetes, cloud-native environments) and CI/CD tools beyond GitHub Actions.
• Enhanced Metric Coverage: Inclusion of more granular performance and reliability metrics, such as GPU utilization, memory footprint, and prompt-specific analytics.
• AI-Driven Anomaly Detection: Incorporation of machine learning models for predictive analytics and automated anomaly detection in LLM behavior.
• Scalability Improvements: Optimization for large-scale, multi-model benchmarking and distributed deployments.

GenA11yHelper: DevOps-Powered Prompt Testing for Smarter GenAI Applications

By Yassine Sami, Mihir Soni, Dondre Samuels, Vaishali Jaiswal, Prompt Pilot Group, Canada DevOps Community 2025

GenA11yHelper is a lightweight, DevOps-driven tool designed to streamline prompt experimentation and optimization for large language models (LLMs). By integrating user feedback, automated prompt promotion, and robust versioning within a cloud-native workflow, GenA11yHelper empowers teams to iteratively enhance GenAI applications. This white paper highlights the project’s core strengths, outlines future development directions, and identifies key areas for community and industry collaboration to maximize the impact of accessible, effective GenAI systems.

• Integrated DevOps Workflow: Utilizes Terraform, Docker, and GitHub Actions to automate deployment, prompt versioning, and CI/CD, ensuring repeatability and scalability.
• User-Centric Feedback Loop: Incorporates real-time user ratings to identify and promote the most effective prompts, directly aligning model outputs with end-user needs.
• Cloud-Native and Lightweight: Deployed on AWS using EC2 and S3, with a minimal resource footprint, making it accessible for a wide range of teams and organizations.
• Open and Modular Architecture: Built with Streamlit, LangChain, and OpenAI API, enabling rapid prototyping and easy integration with other GenAI tools and workflows.

Future Perspectives

• Enhanced Feedback Analytics: Planned integration with Weights & Biases will provide advanced visualization and tracking of prompt performance over time, enabling data-driven optimization.
• Expanded Accessibility Features: Future releases aim to incorporate accessibility testing for prompts and outputs, ensuring GenAI solutions are inclusive for users with diverse needs.
• Support for Multiple LLM Providers: Extending compatibility beyond OpenAI APIs to include other LLM backends, fostering broader experimentation and resilience.
• Community-Driven Prompt Libraries: Establishing shared repositories of high-quality, versioned prompts to accelerate GenAI adoption and best practices across domains.

GenA11yHelper demonstrates a practical, DevOps-powered approach to prompt engineering for GenAI applications, with strong potential for future growth and impact. Continued collaboration across open source, industry, and academia will be essential to realize its vision of accessible, effective, and user-driven GenAI systems.

Strengths of Combined Approach

The integration of InsightAI_Minions, AIObservability-Monitoring_InnerAI, and GenA11yHelper represents a significant advancement in the operationalization of GenAI systems. By uniting performance monitoring, real-time observability, and accessibility, this approach lays the foundation for robust, transparent, and inclusive AI deployments—meeting the demands of modern enterprises and diverse user communities.

Combining InsightAI_Minions’ KPI benchmarking with InnerAI’s real-time middleware monitoring provides both macro (system-level) and micro (transaction-level) insights into LLM performance. Integration enables unified dashboards (Grafana, Prometheus) and customizable alerts, facilitating rapid incident response and proactive maintenance.

Unified AI Operations Platform

• Single pane of glass: The integration can evolve into a centralized AI operations platform, offering unified monitoring, alerting, and accessibility management for all GenAI assets.
• Self-healing AI systems: With comprehensive observability and alerting, automated remediation (e.g., rollback, scaling) becomes feasible, reducing downtime and manual intervention.

Overall DevOps for GenAI Hackathon Strengths

• Innovation: Teams demonstrated creative approaches to AI observability and prompt evaluation, including novel dashboards, feedback tools, and automated benchmarking.
• Community Collaboration: The hackathon fostered cross-disciplinary teamwork, with developers, data scientists, and DevOps engineers collaborating closely.
• Rapid Prototyping: Projects showcased the ability to rapidly prototype and iterate on ideas, leveraging open-source tools and cloud-native technologies.

Strategic Gaps in Coding and Further Considerations

Despite promising prototypes, several strategic coding gaps were observed:

• Integration Depth: Many solutions demonstrated basic integrations between DevOps pipelines and AI models but lacked end-to-end automation or seamless CI/CD integration.
• Scalability: Most projects were at the proof-of-concept stage, with limited attention to scalability, multi-cloud deployment, or enterprise-grade robustness.
• Security & Compliance: Few projects addressed data privacy, secure model deployment, or compliance with industry standards—critical for production AI systems.
• Testing Coverage: Automated testing, especially for AI-specific edge cases and prompt variability, was often incomplete or absent.

Future Perspective

• Production-Ready Solutions: Moving forward, projects should focus on hardening prototypes for real-world deployment, emphasizing scalability, security, and maintainability.
• Standardization: There is a need for standardized interfaces and APIs for AI observability and prompt evaluation tools, facilitating broader adoption within enterprise DevOps workflows.
• Continuous Learning: Integrating feedback loops to enable models and prompts to evolve based on real-world usage and monitoring data will be crucial.
• AI Ethics & Trust: As GenAI systems become more pervasive, embedding ethical guidelines and transparency into observability and evaluation tools will be essential.

Collaboration Required

• Industry Partnerships: Collaboration with cloud providers, AI platform vendors, and enterprises will help transition prototypes to production environments.
• Open-Source Community: Continued engagement with the open-source community can accelerate development, testing, and adoption of these tools.
• Academic Involvement: Partnerships with universities can drive research into advanced observability metrics, prompt evaluation methodologies, and AI system reliability.

Importance of Community Building

Beyond enhancing individual skills, hackathons foster strong community building by bringing together diverse talents and perspectives, which boosts networking, teamwork, and knowledge sharing. This sense of community not only accelerates professional development but also inspires collective innovation, creating environments where participants support each other and co-create impactful solutions that might not emerge from solitary efforts.

Hackathons as a Way to Bridge Academia and Students

Hackathons act as a key educational strategy linking academic theory with practical, industry-relevant experience. They cultivate a culture of creativity, collaboration, and innovation that benefits students by preparing them for the complexities of modern professional environments while also enriching academic programs through close ties with industry.

Conclusion

The DevOps for GenAI hackathon has energized the local tech community and highlighted both the promise and challenges of integrating GenAI into DevOps workflows. Addressing the identified coding gaps and fostering broader collaboration will be essential for maturing these innovations into impactful, production-ready solutions.