<center> # Reflector: An Open-Source, Real-Time Translation Solution <big> **Explore Monadical’s Open-Source Solution for Online Meetings** </big> *Written by Hanna Jodrey and Gokul Mohanarangan. Originally published 2024-04-18 on the [Monadical blog](https://monadical.com/blog.html).* </center> ## Executive Summary In today's digital landscape, audio content's sheer volume and complexity pose significant challenges for analysis, accessibility, and meaningful data extraction. Monadical's Reflector platform aims to address these challenges by providing an open-source solution designed to transform speech, audio, or text into actionable knowledge while ensuring data security, privacy, and organization. This technical overview begins by highlighting Reflector's current capabilities and deployment options. It then provides a technical description of the platform's architecture and the AI technologies underpinning its core functionalities. Furthermore, it examines how Reflector addresses specific user needs across various domains and discusses its competitive advantages over existing solutions. Potential real-world applications and the platform's anticipated impact are also explored. Finally, it outlines Reflector's future enhancements and long-term objectives. ## Background Reflector originated from a need to analyze and derive insights from presentations or idea pitches. Initially a simple NLP program for keyword detection, it has evolved into a robust speech-to-text (STT) translation tool with advanced capabilities. ## Current Capabilities and Deployment Options Reflector offers real-time transcription services (predominantly in English) and the ability to simultaneously translate transcriptions into over 100 languages. It provides customizable settings to filter out profane language and segments audio content into live topics with concise summaries and transcriptions. Additionally, it generates high-level summaries, identifies individual speakers through diarization, and allows users to edit transcriptions and summaries. Reflector supports three primary deployment options: the Monadical Instance for internal use, the Media instance for external audiences, and the Projector Reflector for large-scale events and conferences. ## Technical Overview ### Infrastructure Reflector's infrastructure caters to diverse user requirements through its three deployment options: the Monadical Instance, the Media Instance, and Projector Reflector. The following outlines what each deployment option offers: - **Monadical Instance:** Built for Monadical’s internal use, this instance keeps our audio data secure while recording and summarizing key insights and takeaways from meetings, pitches, and presentations. - **Media Instance:** Expanding Reflector's reach, the Media instance targets external audiences in other organizations, fostering greater accessibility and insight generation. <figure> <img src="https://docs.monadical.com/uploads/c4b23631-444e-4fac-b704-8ff93d804a21.png"> <figcaption> The Reflector Media Instance. </figcaption> </img> </figure> <style> figcaption { text-align: center; /* This centers the caption text */ } </style> - **Projector Reflector:** Specifically engineered for large-scale events and conferences, this deployment option engages audiences through live, on-screen displays of essential content, including critical points and translations. It showcases Reflector's ability to enhance the delivery and reception of information in public settings and ensures that content is accessible and engaging for diverse audiences. <figure> <img src="https://docs.monadical.com/uploads/26935151-89a9-4434-b41f-0c55fc08be4a.png"> <figcaption> Showcasing Project Reflector on the big screen at <a href="https://allinevent.ai/"> ALL IN </a> in 2023. </figcaption> </img> </figure> ### Architecture Reflector's architecture uses [React](https://react.dev/), [NextJS](https://nextjs.org/), [ChakraUI](https://chakra-ui.com/), and [Tailwind CSS](https://tailwindcss.com/) for the UI and an [OpenAPI](https://www.openapis.org/) back-end API. It utilizes [SQLite](https://www.sqlite.org/) and [Alembic](https://alembic.sqlalchemy.org/en/latest/) for data management and leverages APIs, Websockets, and [WebRTC](https://webrtc.org/) for real-time interactions. <figure> <img src="https://docs.monadical.com/uploads/e4a81ed8-6d33-4de6-9534-5c3be1d588cc.png"> <figcaption> Reflector’s current architecture. </figcaption> </img> </figure> <style> figcaption { text-align: center; /* This centers the caption text */ } </style> ### AI Technologies Reflector employs open-source AI technologies selected through R&D. Key models include [Facebook seamlessM4T](https://ai.meta.com/blog/seamless-m4t/) for translation, vicuna-13b-v1.5 and Zephyr for content analysis, and [OpenAI Whisper](https://openai.com/research/whisper) and [Faster Whisper](https://github.com/SYSTRAN/faster-whisper) for transcription. <figure> <img src="https://docs.monadical.com/uploads/af2ec358-704d-4736-9b83-b08283ac1f1e.png"> <figcaption> Some of the AI technologies currently used by Reflector. </figcaption> </img> </figure> <style> figcaption { text-align: center; /* This centers the caption text */ } </style> ### Addressing Specific User Needs Through Features Reflector enhances communication and knowledge transfer across various applications. The platform accommodates diverse input mediums, including text, real-time speech, and recorded audio in multiple languages, enabling real-time processing and relay of information with clarity and accuracy. Transcription and translation capabilities help eliminate communication barriers posed by language differences. The platform simplifies input through real-time generative features like topic segmentation, titling, and summarization, aiding comprehension and knowledge retention. Users can customize these outputs to suit their specific requirements. Speaker diarization distinguishes between speakers, refining content clarity and context understanding. Users can edit and refine generated outputs, minimizing errors and creating a repository of ground-truth annotations for model fine-tuning. Reflector allows users to query the data for holistic answers and easily share insights, promoting collaboration and informed decision-making. By addressing and anticipating user needs, Reflector provides an intuitive platform for managing the complexities of communication and information across diverse applications. <figure> <img src="https://docs.monadical.com/uploads/91161ca3-41c8-4f90-8dda-c1c0e36fed4c.png "> <figcaption> A sample session with the <a href="https://reflector.media/transcripts/new"> Reflector Media Instance </a> demonstrating live topics, speaker diarization, and meeting summary features. </figcaption> </img> </figure> ## Competitive Edge Reflector's unique advantages include its open-source nature, self-hosting capability for enhanced privacy and control, balanced approach to generalization and customization, and focused R&D for continuous improvement. A detailed description of each advantage: **Open-Source:** Reflector utilizes open-source models, aligning with Monadical's sharing of knowledge and expertise. **Self-Hosting Capability:** Reflector is designed for self-hosting by end-users, providing autonomy to deploy the platform within their infrastructure, whether on private servers or cloud providers. This self-hosted nature grants complete control over deployment and operation, which is crucial for custom configurations and stringent data governance. It also enhances privacy by ensuring sensitive data remains within the user's environment, preventing unauthorized access. **Balanced Approach:** Reflector balances general applicability across various meeting types with the ability for users to customize granularity, style, and data extraction formats to suit their specific needs and use cases. **Focused R&D:** Continuous improvement efforts include refining core features like speaker diarization for clarity, enhancing transcription and translation accuracy through custom post-processing, and developing real-time dynamic topic detection, auto-merging topics, and live Q&A capabilities. ## Real-World Applications and Impact Reflector has the potential to benefit users across various sectors: **Academic Domain:** Language inclusivity, efficiency in learning, and support for diverse learning needs. **Communication Domain:** Facilitating multilingual engagement and catering to varied communicative demands. **Business Environment:** Meeting efficiency, creation of a knowledge repository, and dynamic data interactions. Reflector has also received positive reception, demonstrated at the [ALL IN](https://allinevent.ai/) summit in 2023, indicating a promising trajectory. <figure> <img src="https://docs.monadical.com/uploads/59fb16ee-5091-4b03-8a6d-f137b940a76e.jpg"> <figcaption> Monadical founder Max McCrea demonstrates Reflector at ALL IN. </figcaption> </img> </figure> <style> figcaption { text-align: center; /* This centers the caption text */ } </style> ## Future Enhancements and Roadmap Reflector's short-term enhancements will include fine-tuned domain-specific transcription, speaker recognition, dynamic topic detection, multimedia content integration, on-demand generative features, and edit history and revisions. Long-term enhancements include insight generation by meeting type, enhanced intent extraction, extension and platform integration, refinement through user feedback, and a voice-driven NLP interface. A detailed description of each enhancement is outlined below. ### Short-term Enhancements **Fine-tuned domain-specific transcription:** Develop specialized transcription models that cater to specific fields and enhance accuracy and relevance. **Speaker recognition:** Implement advanced speaker recognition technologies, utilizing fine-tuned speaker segmentation models alongside voice signatures to improve diarization. **Dynamic topic detection:** Introduce sophisticated algorithms for real-time identification and categorizing discussion topics, elevating content summarization and analysis. **Multimedia content integration:** This feature enables the upload and display of video content within Reflector, broadening the spectrum of usable media formats. **On-demand generative features:** Users can re-trigger Reflector's generative capabilities, such as artifact creation and summarization, ensuring dynamic content adaptation. **Edit history and revisions:** Establish a comprehensive revision tracking system to facilitate transparency and accountability in content modification. ### Long-term Objectives **Insight generation by meeting type:** Tailoring analytics and insight extraction based on the specific nature of meetings, offering customized information processing. **Enhanced intent extraction:** Advancements in NLP enable accurate interpretation of user intent from spoken or written inputs, improving interaction quality.Extension and integration: A browser extension will be created to simplify access to Reflector's features and integrate with platforms like Meet and Whereby for a seamless user experience. **Refinement through feedback:** Continuous optimization of LLM workflows, informed by direct user feedback to ensure alignment with user expectations. **Voice-driven NLP interface:** Introduction of a voice-activated interface, making Reflector more intuitive and accessible to a broader user base. Reflector prioritizes enhancements based on their impact, technical feasibility, and value to the ecosystem. Decisions on development and resource allocation are made collaboratively with periodic reviews. User feedback and annotated data are critical in refining and advancing Reflector's capabilities. This plan reflects Reflector's commitment to staying ahead in audio content analysis technology and its responsive approach to meeting users' changing needs. ## Conclusion While Reflector is well-positioned to address challenges in audio content analysis, its journey is just beginning. Its objectives outline practical milestones for enhancing accessibility in education, streamlining business operations, and fostering inclusive communication. Monadical welcomes insights, critiques, and collaboration opportunities from stakeholders and the tech community as it refines and tests Reflector's capabilities. Together, we can approach the future of audio content analysis with balanced optimism and the potential to create a meaningful impact.