Integrating AI Models into Zulip Chatbots Using FastAPI, Part 2 - HedgeDoc
  648 views
 owned this note
<center> # Integrating AI Models into Zulip Chatbots Using FastAPI: Part 2 <big> **A comprehensive guide to building a scalable Zulip bot with an N-tier architecture.** </big> *Written by [Cesar Uribe](https://github.com/curibe) and [Juan David Arias](https://github.com/juanArias8). Originally published 2023-08-09 on the [Monadical blog](https://monadical.com/blog.html).* </center> Most developers have had the misfortune of experiencing a project quickly becoming an irredeemable mess. All it takes is a few critical ingredients for this potential disaster: poorly defined user stories, unclear business logic abstraction, lack of communication on the team, poorly implemented code with no documentation, and so on and so forth. It’s very easy to fall into one of these traps, and just a few of these could be enough for a project to become a "big ball of mud", or, in simpler terms, a Frankenstein system that, if it even works, does so for reasons that are either mysterious or, in the worst cases, miraculous. Fortunately for us mortals, this problem is not new, and thanks to the expertise of some software philosophers, we now have tools to tackle this problem. When we write code, we try to abstract real problems by representing them through programming languages. With a bit of practice, we try to encapsulate common behaviors in functions and modules, which saves us from writing tons of duplicated code and helps give a little structure to our project. And yeah, that’s ok for small projects, but this approach alone might not be sufficient for medium and large-sized systems; these larger systems have the potential to turn our system into one with high coupling, where one module calls or depends on others, which in turn depends on many more. Under this approach, it is easy, for example, to end up with redundant or cyclical imports, where module A depends on B, and B depends on A. This necessitates a higher level of abstraction to decouple the different parts of a system, which is where [N-tier architecture](https://stackify.com/n-tier-architecture/) - also known as [layered architecture](https://www.cosmicpython.com/book/part1.html) - comes into play. Basically, n-tier architecture is a software design pattern that splits an app into different components based on a separation of concerns, where each layer or tier is in charge of a specific logical task, making it more robust, secure, and easy to maintain. Under a layered architecture, the different modules (files) of a system are categorized according to common purposes. Files responsible for data access? Data access layer. Files handling business logic? Services layer. Files that present the interface to the user or some service to consumers? Presentation layer. The previous example is just the standard configuration of an N-tier architecture, but an application can contain as many layers as necessary, such as integration, security, logging, etc. N-tier architecture also proposes a rule of upward dependency, indicating how the layers interact with each other, and stating that an upper layer depends only on immediately lower layers to obtain services and functionalities. This approach offers a clear separation of responsibilities, allowing for increased cohesion and reduced coupling of our system, which should be the goal. That’s where N-tier architecture enters the Delphos bot project. This approach can help you create a bot that is not only functional, but also easy to scale, maintain, and extend as your project grows. In [Part 1](https://monadical.com/posts/zulip-ia-bot-1.html) of this 3-part series, we showed you how to set up the basic structure and components of a Zulip bot server. In this part, we’ll show you how to add features to the bot, such as retrieving messages, links, and images from the streams and topics in Zulip. This will prepare us for the final part, where we’ll integrate some AI models to make the bot smarter and more creative. ## Project architecture This specific project is a Python backend using the FastAPI framework. It’s implemented using a services-oriented (n-tier) architecture, where each file is separated by its purpose (presentation, services, repository, etc.). The architectural layers for this project include: - **Controller:** This is the bot entry point. It consists of the file `server.py`, which is the file previously migrated to FastAPI. This controller allows the communication between the client (Zulip) and the bot. - **Operators:** This layer validates the command inputs and runs the domain logic for each command. - **Integrations:** This layer interacts directly with external service providers. In this example, it will connect directly to the Zulip API to retrieve stored messages and upload files. In the next blog post, this layer will be used to integrate some machine learning models for image generation and natural language processing. - **Models:** These are the entities’ abstractions and the schemas for the application. We’ll use the [Pydantic library](https://pydantic-docs.helpmanual.io/) to create the models and schemas. Pydantic is awesome because it automatically validates the data types, ensuring that they’re all correct at runtime and displaying a helpful message if something goes wrong. Here’s a diagram of the proposed architecture: ![](https://docs.monadical.com/uploads/ded6dc2b-004b-43d2-890f-f418d651c02b.png) In the N-tier architecture, each lower layer is considered to be the provider of the upper one. In this case, the data flow is: - Using the Zulip app, a user requests an action from the bot by entering a command like <span style="color:red;">`@delphos messages 100`</span>. - The client generates a POST request, which is attended by the controller layer; in this case, it’s the FastAPI app server defined in the `server.py` file, where we have the API entry point. - The request body is checked, and the bot handler picks the right operator for the command. - The operator makes sure the command is valid and runs the instruction. Frequently, the operator obtains data from the integrations layer as well, processing it according to business logic. - Once the process is complete, the operator generates the response sent to the user. The response is then returned to the upper layers up to the bot handler. - Finally, the bot handler sends the response to the Zulip user using the `bot_handler.send_reply(message, response)` call. The final project structure will look like this: ``` delphos ├── src/ │ ├── bots/ │ | ├── delphos/ │ ├── config/ │ ├── models/ │ ├── operators/ │ ├── integrations/ │ └── utils/ ├── tests/ ├── zuliprc-api ├── README.md ├── requirements.txt ├── server.py ``` ## Adding features to the bot Now that we know what we’re doing, let’s add some cool features to the bot. But first, make sure you’ve followed [Part 1](https://monadical.com/posts/zulip-ia-bot-1.html) or clone this repo and use it as a starting point. ```shell git clone git@github.com:juanArias8/zulip-delphos-bot.git ``` Now, we’re ready to actually get to work demonstrating the following functionalities: 1. Displaying a help message to the user on how to use the commands in the bot. 2. Extracting the latest messages from a Zulip topic and making a file with them ordered by date. 3. Extracting the links from a Zulip topic and displaying them on the screen, ordered by date. 4. Extracting the images that have been shared in a Zulip topic and displaying them chronologically. This demonstrates how to use the Zulip API in the bot and get data from Zulip. In the following (and final) post of this series, we’ll use this data to train AI models and do NLP tasks like summarization, sentiment analysis, event extraction and creation, autocomplete systems, and more. ## Building the bot handler The bot handler is what makes the bot work. It gets the data sent by the user from Zulip, and chooses, based on the specified command, which operator will do the task and process the request. A user request comes in the form <span style="color:red;">`@delphos messages 100`</span>, where `messages` is the command, and the representation of the instruction being called (that is, to extract messages from a Zulip topic). Data after the command are called arguments. Here, the argument is the number of messages the user wants from Zulip. Since our bot will have a lot of features , we need to find a way to handle them independently and be able to add or remove them easily. In other words, we need to keep them decoupled. To do this, we’ll use the [factory pattern](https://refactoring.guru/design-patterns/factory-method). We’ll make a function that handles different commands and an operator for each command in the bot. [Remember](https://zulip.com/api/writing-bots#writing-a-bot) that we assign the bot handler two methods: `usage` and `handle_message`. The `usage` function just returns a string with general information about the bot. The `handle_message` function performs the requested operation and sends a response to the user. Let’s start by creating the `DelphosHandler` class inside the `src/bots/delphos/delphos.py` file: ```python import shlex from typing import Any, Dict from src.utils.constants import ERROR_HELP from zulip_bots.lib import BotHandler from src.models.schemas import Command class DelphosHandler: @staticmethod def usage() -> str: return """ Zulip Bot for advanced search """ def handle_message(self, message: Dict[str, Any], bot_handler: BotHandler): message_type = message.get("type", None) # private or stream content = message.get("content", None) # message content if not (message_type and content): return [ERROR_HELP] # split content ["@Delphos", "messages", "10"] content = shlex.split(content.strip()) if len(content) == 0: return [ERROR_HELP] mentions = ["Delphos", "@**Delphos**"] starts_with_mention = content[0] in mentions if message_type == "private" and starts_with_mention: content = content[1:] command = Command(instruction=content[0], params=content[1:]) if message_type == "stream": command.stream = message.get("display_recipient", None) command.topic = message.get("subject", None) instance = self.get_operator_instance(command) response = instance.get_response(command) for item in response: bot_handler.send_reply(message, item) handler_class = DelphosHandler ``` The `handle_message` function validates that the call to the bot has been well-elaborated. If the validation fails, the function will send a help message to the user explaining how the instruction should be written. If the instruction is well-structured, the requested command will be cleaned and standardized, and each part of it will be stored in an instance of the class `Command`. The Command class represents a command entered by a user in the Delphos bot. Let's take a closer look at the class structure. To begin, create a new file `src/models/schemas.py` and enter the following: ```python from enum import Enum from typing import List from pydantic import BaseModel class Instruction(str, Enum): HELP = "help" MESSAGES = "messages" LINKS = "links" class Command(BaseModel): instruction: Instruction params: List[str] = [] stream: str = None topic: str = None ``` If the user enters the command `@delphos messages 100` from the topic `tests` in the stream `bots`, the instance of the `Command` object will look like this: ```python instruction: "messages" params: ["10"] stream: "bots" topic: "test" ``` Back in the <span style="color:red;">`delphos.py`</span> file, add the <span style="color:red;">`get_operator_instance`</span> function, which uses the factory pattern to pick the right operator for the command. If there’s no match, it will use the help operator as a default. ```python from src.models.schemas import Command, Instruction from src.operators.base import BaseOperator from src.operators.help import HelpOperator class DelphosHandler: … @staticmethod def get_operator_instance(command: Command) -> BaseOperator: """ Generates a response based on the given command. """ instructions = Instruction available_operators = { instructions.HELP: HelpOperator, } operator_class = available_operators.get(command.instruction) return HelpOperator() if operator_class is None else operator_class() ``` With the handler built, the first four features can now be added 🎉. ## Implementing Operators Operators are responsible for executing the instructions corresponding to the command requested by the user. First, they make sure that the arguments entered are valid and consistent with the requested command. Then, they run the logic for the request, using integrations to get the data they need. In our case, we have an integration that connects to the Zulip API and obtains the data required by the operator. To standardize the handling of operators, we’ve created an interface that will be used as a template for the different operators we create. To create the interface, first create a new file `delphos/src/operators/base.py` and define the `BaseOperator` class within it. This class should implement the `get_response` method, which generates the response to be sent to the user. ```python from typing import Optional, Protocol, List from src.models.schemas import Command class BaseOperator(Protocol): def get_response(self, commands: Optional[Command]) -> List[str]: ... ``` Using this interface as a template ensures that all operators have a consistent structure and follow the same good programming practices. With the `BaseOperator` interface in place, we can now start implementing our operators. ### Operator 1: Help Before we dive into the more complex operators, let's start with a fundamental operator that will greatly improve the user experience of our bot: the Help operator. By providing easy-to-follow instructions on how to use each command, the Help operator will make it easier for users to interact with the bot and get the most out of its features. In addition to improving the UX, implementing the Help operator will also demonstrate how to implement subsequent operators following this architecture. Building and testing the Help operator will also give insight into the principles behind creating operators that are easy to maintain and extend. This is crucial if you want to build a bot that users will love, and not find intolerably irritating. #### Step 1: Create the Help operator Define the `HelpOperator`class in a new file called `src/operators/help.py`. This operator will be used when the user types “@delphos help” or when they type something wrong. ```python from typing import List from src.models.schemas import Command from src.utils.constants import HELP_MESSAGE class HelpOperator: @staticmethod def get_response(command: Command) -> List[str]: return [HELP_MESSAGE] ``` Also, define two new constants - `HELP_MESSAGE` and `ERROR_HELP` - in a new `src/utils/constants.py` file. `ERROR_HELP` is a message with some general info on how to use the bot. It will show up when the user types the wrong command or parameters. ```python ERROR_HELP = """ It seems that there was an error processing your request. Please check the format of your message and try again. If you're not sure how to use Delphos, you can use the @delphos help command to see a list of available commands and how to use them. Here's a general guide to using Delphos: In a stream, use the format @delphos <command> <arguments> In a private message, use the format <command> <arguments> If you continue to experience issues, please contact the bot administrator for further assistance. """ ``` `HELP_MESSAGE` is a message that gives more info on how to use each command. It will show up when the user types “@delphos help”. This message should tell them how to format each command, give some examples, and include any notes or warnings. ```python MESSAGES_HELP = """ ## Messages Prints the last <amount> messages of the topic. Example: @delphos messages 10 PM Example: messages 10 learning css """ LINKS_HELP = """ ## Links Prints all the links in the topic. Example: @delphos links PM Example: links learning css """ HELP_MESSAGE = f""" Welcome to Delphos, a bot that helps you manage your topic! Use the following commands to get started: {MESSAGES_HELP} {LINKS_HELP} ## Help Displays a list of available commands and provides examples for each one. Example: @delphos help PM Example: help """ ``` Note that for `HELP_MESSAGE`, info on all the commands that the bot has needs to be included. This means that when a new operator is added to the bot, the`HELP_MESSAGE` section needs to be updated to let users know how to use the new feature. #### Step 2: Test the operator Now that the bot’s first operator has been added, it’s time to test it. To do so, we’ll run the bot locally and then use ngrok to expose the server to the internet. Once we have the ngrok URL that our server runs on, the endpoint URL in the bot configuration on Zulip will need to be updated in order to integrate them. Let's look at this in more detail. From the terminal, run the command to start the server locally. Remember to have the `zuliprc` file ready, which has the Zulip login info. ```shell pipenv install pipenv shell python3 server.py --config-file zuliprc-delphos --hostname="0.0.0.0" --port=8000 ``` Next, we’ll use ngrok to make our local server available online. Run the following command in a new terminal: ```shell ngrok http 8000 ``` After that step, ngrok will return a URL where the system is exposed online. Copy this URL and update the endpoint URL field in the bot configuration. ![](https://docs.monadical.com/uploads/08dd3a2a-ff7e-439d-ba09-ff73ce5d630c.png) ![](https://docs.monadical.com/uploads/9f47f4a2-ec36-4730-aea8-993cbb9e6e5e.png) Now, from any topic in Zulip, run the `@delphos help` command to obtain the help message defined earlier. ![](https://docs.monadical.com/uploads/7d2a8627-41af-456a-9ee9-a41167b09375.png) If it works, your bot is up and running. Now, on to the next feature! ### Operator 2: Messages The following feature allows the bot to retrieve the latest messages from a stream topic and return it as a text file to the user. The [Zulip API](https://zulip.com/api/rest) is used to get the messages, as well as some utility functions to clean up the response before sending it to the user. #### Step 1: Create the Messages operator To implement this feature, start by creating a `MessagesOperator` class in `src/operators/search/messages.py`. The operator implements the `get_response` function from the `BaseOperator` class. The `get_response` method takes a command as input, checks that it’s valid, and calls the `search_messages` function to get the last N messages from the specified stream and topic. ```python from typing import List from src.models.schemas import Command from src.utils.constants import MESSAGES_HELP class MessagesOperator: def get_response(self, command: Command) -> List[str]: # From Stream chat => messages 100 limit = command.params[0] if len(command.params) > 0 else 1000 # From private chat => messages 100 stream topic if len(command.params) == 3: command.stream = command.params[1] command.topic = command.params[2] # Validate command if not (command.stream and command.topic): return [MESSAGES_HELP] return self.search_messages( stream=command.stream, topic=command.topic, limit=limit ) ``` The `get_response` method parses the command parameters to determine the stream, topic, and limit for the search. It then checks that they are in the correct format and calls the `search_messages` method to retrieve the requested messages. #### Step 2: Define the 'search messages' method Add a new function to the `MessagesOperator` class, this time creating the `search_messages` method, which will use a `self.zulip_integration` instance (it’ll be defined later on) to retrieve the last `N` messages from a topic in Zulip: ```python from src.integrations.search.zulip import ZulipIntegration class MessagesOperator: def __init__(self): self.zulip_integration = ZulipIntegration() def get_response(self, command: Command) -> str[]: … def search_messages(self, *, stream: str, topic: str, limit: int) -> List[str]: request = self.zulip_integration.build_search_narrow( stream=stream, topic=topic, limit=limit, apply_markdown=False ) response = self.zulip_integration.get_zulip_messages(request) if response.result == "success": content = format_messages(response.messages) file_name = self.zulip_integration.upload_file_to_zulip( f"{stream}-{topic}.txt", content ) return [file_name or "Error uploading file to Zulip."] return [f"Error searching messages in Zulip: {response.msg}"] ``` The previous method creates a request object with the specified stream, topic, and number of messages, and then sends the request to the Zulip API using the `zulip_integration` instance. If the API call is successful, the method formats the retrieved messages using the `format_messages` function. It then uploads them to Zulip as a text file, with the name based on the stream and topic. Finally, it sends the URL where the user can download the file in the topic’s chat. #### Step 3: Create the Zulip Integration The third step is to make the integration that connects to Zulip or other services. To begin, add the Zulip dependency to the project. This is done by executing the following command: ```shell pipenv install zulip ``` Next, create a new schema to handle the responses from the Zulip server. Inside the `src/models/schemas.py` file, add this model: ```python class ZulipResponse(BaseModel): result: str msg: str found_anchor: bool = None found_oldest: bool = None found_newest: bool = None history_limited: bool = None anchor: int = None messages: List[Message] = None ``` Finally, create a new file called `src/integrations/zulip_integration.py` and define some functions within it. ```python import io from typing import List, Union, Any, Dict from zulip import Client from src.models.schemas import ZulipResponse class ZulipIntegration: def __init__(self): self.client = Client(config_file="./zuliprc-api") def get_zulip_messages(self, request: Any): messages = self.client.get_messages(request) return ZulipResponse(**messages) def upload_file_to_zulip(self, filename: str, content: List[str]) -> Union[str, None]: with io.StringIO("\n".join(content)) as fstream: fstream.name = filename result = self.client.upload_file(fstream) uri = result.get("uri") return f"[{filename}]({uri})" if uri else None @staticmethod def build_search_narrow( *, stream: str, topic: str, limit: int = 1000, content: str = None, apply_markdown: bool = True, ) -> Dict[str, Any]: narrow = { "anchor": "newest", "num_before": limit, "num_after": 0, "apply_markdown": apply_markdown, "narrow": [ {"operator": "stream", "operand": stream}, {"operator": "topic", "operand": topic}, ], } if content: narrow["narrow"].append({"operator": "has", "operand": content}) return narrow ``` The `ZulipIntegration` class handles all interactions with Zulip. In the `__init__` method, a `Client` object is initialized from Zulip. A `zuliprc` file is used again for logging into Zulip. But note that a new authentication file is needed, so make sure to add a new `zuliprc-api` file to the main directory. You can reuse the bot's credentials or create new ones from your Zulip application. The file should look like this: ```shell [api] email=delphos-bot@zulip.domain.com key=xxxxxkeyxxxxx site=https://zulip.domain.com token=xxxxxtokenxxxxx ``` The first function we’ll define is `get_zulip_messages`. It uses the Zulip client instance and gets the latest messages from a stream-topic based on what we put in the request parameter. The `upload_file_to_zulip` function generates a file in memory based on the messages in the content parameter. It then uploads the file to the Zulip cloud and returns the file name and URL, so it can be seen in the browser. Finally, the `build_search_narrow` function helps build a dictionary with the parameters of the request that will be made to the Zulip API. Here, the desired stream and the topic are included, as well as the number of messages to be retrieved, whether a specific format, like links or images, is sought, and finally, if markdown should be applied to the response. #### Step 4: Clean the Zulip messages data The messages include a lot of extraneous information as-is, so cleaning and filtering them makes for a much better user experience. For our use case, we wanted to get the messages in this format: ***2020-11-11 16:17:54 Juan: hello, there 2020-11-11 16:18:20 Cesar: hello, world*** We needed to get rid of messages and characters that don’t add useful content to the text, such as messages sent by the Delphos bot itself, or the responses generated by it. We also noticed that when someone replies to a message, Zulip adds a quote block with the text they’re replying to, so we needed to get rid of those as well. And finally, to improve readability, we wanted to remove the ** characters from mentions. Zulip stores this information in the format ***@ ** juan ** hello, there***, and we wanted it to say ***Juan: hello, there.*** To accomplish the cleanup, create a new file ` src/operators/search/utils/messages.py`, and add the following functions. The entry point is `format_messages`, which is responsible for formatting the messages in the ***2020-11-11 16:17:54 Juan: hello there*** format. ```python import re from datetime import datetime from typing import List from src.models.schemas import Message def format_messages(messages: List[Message]) -> List[str]: all_items = [] for message in messages: cleaned = remove_delphos_messages(message.content, message.sender_email) cleaned = remove_quotes(cleaned) cleaned = remove_mentions(cleaned) if cleaned: all_items.append( f"{datetime.fromtimestamp(message.timestamp)} " f"{message.sender_full_name}: {cleaned}" ) return all_items def remove_delphos_messages(message: str, sender_email: str) -> str: is_delphos_sender = sender_email == "delphos-bot@zulip.monadical.com" is_delphos_message = message.startswith("@**Delphos**") if is_delphos_sender or is_delphos_message: return "" return message def remove_quotes(message: str) -> str: if "** [said](" in message: message = re.sub(r"@_\*\*(.*?)\):", "", message) message = re.sub(r"```quote\n(.*?)\n```", "", message) message = re.sub(r"^\s+|\s+$", "", message) return message def remove_mentions(message: str) -> str: return message.replace("**", "") ``` With the previous steps completed, the functionality of the Messages operator can be tested. #### Step 5: Register the operator Alright, we’re almost done with the operator! The last thing to do is to let the bot know that it exists. To do that, update the `src/bots/delphos/delphos.py` file and include the Messages operator. ```python … from src.operators.search.messages import MessagesOperator class DelphosHandler: … @staticmethod def get_operator_instance(command: Command) -> BaseOperator: instructions = Instruction available_operators = { instructions.HELP: HelpOperator, instructions.MESSAGES: MessagesOperator, } … ``` #### Step 6: Test the search messages service Now, the Messages operator can be used in two ways: by sending a private message to the bot, or by typing it in a stream topic. In both cases, write: ![](https://docs.monadical.com/uploads/9b93599b-bfd3-4b0b-b680-f4f890156406.png) If the name of the stream or the topic has more than one word, surround it with quotes so that the bot understands that it’s a full sentence. If the bot call is made from a stream, just writing the command and the number of messages works. The bot will automatically get the stream and topic from which the request is sent. Here’s an example, using a private message with Delphos: ![](https://docs.monadical.com/uploads/722a6ef7-f55d-411d-bab0-285fc1ca3157.png) The bot in this case returns the content as a downloadable text file named in the previously-mentioned format, <span style="color:red;">`<stream>-<topic>.txt`</span>. ### Operator 3: Links Next, let’s turn our attention to the Links operator, which is used to retrieve all the different links that users have shared within a specific topic. We’ve found that it can be annoying and time-consuming to scroll back through hundreds of messages in Zulip to find a shared link, so this helps solve that issue. #### Step 1: Create the Links operator Start by creating a new file `src/operators/search/links.py` and a new class - `LinksOperator` - in it. ```python from typing import List from src.integrations.zulip_integration import ZulipIntegration from src.models.schemas import Command from src.operators.search.utils.links import format_links from src.utils.constants import LINKS_HELP class LinksOperator: def __init__(self): self.zulip_integration = ZulipIntegration() def get_response(self, command: Command) -> List[str]: if len(command.params) == 2: command.stream = command.params[0] command.topic = command.params[1] if not (command.stream and command.topic): return [LINKS_HELP] return self.search_links(stream=command.stream, topic=command.topic) def search_links(self, *, stream: str, topic: str) -> List[str]: request = self.zulip_integration.build_search_narrow( stream=stream, topic=topic, content="link", ) response = self.zulip_integration.get_zulip_messages(request) if response.result == "success": return format_links(response.messages) return [f"Error searching links in Zulip: {response.msg}"] ``` The `LinksOperator` class also uses a `ZulipIntegration` instance to retrieve links from the Zulip server. The `LinksOperator` class also implements the `get_response` method from the `BaseOperator` interface, where it checks that the command is well-structured. If so, it goes on to search for links within Zulip. The `search_links function` builds the request that will go to the Zulip API, requesting the last 100 links shared within the topic. If a response exists, it formats the response that will be sent to the user with the `format_links function`. #### Step 2: Clean the Zulip links data Just like in the messages example, Zulip responds with a lot of unnecessary information. The message returned by Zulip includes, for example, the entire message related to the link, as well as the links themselves (in HTML). It’s also returned in a format that’s not super useful. For our use case, we wanted to get all the links and generate a list of links in the form ["https://link1.com", "https://link2.com"], etc. To do this, the link needs to be separated from the message, obtained, and added to the list of links. It’s also important to know that there is a character and line limit for printing a message in Zulip. So to prevent the message delivery from failing, divide the final set of results into chunks of size `amount`. Create a new file `src/operators/search/utils/links.py` and add the following functions: ```python import math import re from typing import Union, List from src.models.schemas import Message def format_links(messages: List[Message]) -> List[str]: amount = 10 all_items = [] for message in messages: all_items.extend(extract_links_from_string(message.content)) unique_items = list(set(all_items)) sliced_list = [ format_links_response(unique_items[i * amount: (i + 1) * amount]) for i in range(math.ceil(len(unique_items) / amount)) ] return sliced_list def extract_links_from_string(input_string: str) -> Union[List[str], None]: regex = re.compile(r'href=[\'"]?(http[^\'" >]+)', re.I) match = regex.findall(input_string) return match def format_links_response(links: List[str]): return '\n'.join(f"[]({link})" for link in links) ``` These utility functions are called by the operator to clean the data. And there we go - that’s it for the Links operator! Now it just needs to be integrated. #### Step 3: Register the Links operator As with the message operator, we need to tell the bot about the Links operator. To do this, go to the `src/bots/delphos/delphos.py` file and update the `get_operator_instance` function with the `LinksOperator`. ```python … from src.operators.search.links import LinksOperator class DelphosHandler: … @staticmethod def get_operator_instance(command: Command) -> BaseOperator: available_operators = { instructions.HELP: HelpOperator, instructions.MESSAGES: MessagesOperator, instructions.LINKS: LinksOperator, } … ``` #### Step 4: Test the Links operator Once the operator is integrated, it's time to test it. To do this, head to any Zulip topic and type the command “@delphos links”. This command should return all the different links shared by users within the topic: ![](https://docs.monadical.com/uploads/9a4a7f68-f4f5-4d12-89e7-94ce9229642b.png) ### Operator 4: Images As in the previous two examples, it’s possible to add a new operator to get the images shared on a topic, but we’ll leave this task to you, curious reader, to complete. Here are a few hints to help you out: - Create the new Image operator. - Get the images from Zulip and specify "image" in the Zulip narrow. - Clean the data and create the format required for Zulip. - Send the data back to the user. ## Conclusion We hope you’ve enjoyed this detailed guide on how to build a scalable Zulip bot with an n-tier architecture. We’ve looked at how to organize your project using this method, so that your bot is not only working, but also easy to scale and maintain as your project grows. We’ve covered the creation of a well-designed project architecture, including the controller, operators, integrations, and model layers. We’ve also looked at how to add features to the bot, such as displaying a help message and extracting messages, links, and images from Zulip topics. But! We’re not done quite yet. In the next and final blog post in this series, we’ll look at how to take your bot to the next level by adding some AI features, providing it with more capabilities as a result. We’ll cover how to perform NLP tasks on the messages you obtained with the messages command from this blog post, and how to generate images with Stable Diffusion. In the meantime, though, be sure to [check out the repo on GitHub](https://github.com/juanArias8/zulip-delphos-bot) and let us know what you think! Remember, a well-structured project is key to the long-term success and maintainability of any software project. Adopt best practices and a robust architecture, and you can ensure that your Zulip bot will be easily updated and maintained as your needs evolve, and ready to tackle anything you throw at it. Stay connected for our third and final post coming soon! ### References * https://stackify.com/n-tier-architecture/ * https://www.cosmicpython.com/book/chapter_04_service_layer.html * https://www.cosmicpython.com/book/chapter_02_repository.html * https://pydantic-docs.helpmanual.io/ * https://zulip.com/api/rest * https://godatadriven.com/blog/protocols-in-python-why-you-need-them/ * https://realpython.com/factory-method-python/ * https://refactoring.guru/design-patterns/factory-method/python/example



Recent posts:


Back to top