diff --git a/docs/Makefile b/docs/Makefile index d4bb2cbb..115749bf 100644 --- a/docs/Makefile +++ b/docs/Makefile @@ -18,3 +18,6 @@ help: # "make mode" option. $(O) is meant as a shortcut for $(SPHINXOPTS). %: Makefile @$(SPHINXBUILD) -M $@ "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O) + +serve_docs: ## serve the built docs locally to preview the site in the browser + sphinx-autobuild . $(BUILDDIR)/html diff --git a/docs/_images/event-bus-overview.png b/docs/_images/event-bus-overview.png new file mode 100644 index 00000000..f3e20b2a Binary files /dev/null and b/docs/_images/event-bus-overview.png differ diff --git a/docs/_images/event-bus-vs-asynchronous-tasks.png b/docs/_images/event-bus-vs-asynchronous-tasks.png new file mode 100644 index 00000000..5e1ac697 Binary files /dev/null and b/docs/_images/event-bus-vs-asynchronous-tasks.png differ diff --git a/docs/_images/event-bus-workflow-service-a.png b/docs/_images/event-bus-workflow-service-a.png new file mode 100644 index 00000000..f8101fc1 Binary files /dev/null and b/docs/_images/event-bus-workflow-service-a.png differ diff --git a/docs/_images/event-bus-workflow-service-b.png b/docs/_images/event-bus-workflow-service-b.png new file mode 100644 index 00000000..91393f12 Binary files /dev/null and b/docs/_images/event-bus-workflow-service-b.png differ diff --git a/docs/concepts/event-bus.rst b/docs/concepts/event-bus.rst new file mode 100644 index 00000000..9c84cceb --- /dev/null +++ b/docs/concepts/event-bus.rst @@ -0,0 +1,129 @@ +Open edX Event Bus +================== + +Overview +-------- + +The suggested strategy for cross-service communication in the Open edX ecosystem is through an event-based architecture implemented via the :term:`Event Bus`. This functionality used for asynchronous communication between services is built on top of sending Open edX Events (Open edX-specific Django signals) within a service. + +What is the Open edX Event Bus? +------------------------------- + +The :term:`Event Bus` implements an event-driven architecture that enables asynchronous communication between services using the `publish/subscribe messaging pattern`_ (pub/sub). In the Open edX ecosystem, the event bus is used to broadcast Open edX Events to multiple services, allowing them to react to changes or actions in the system. The event bus is a key component of the Open edX architecture, enabling services to communicate without direct dependencies on each other. + +.. image:: ../_images/event-bus-overview.png + :alt: Open edX Event Bus Overview + :align: center + +Why use the Open edX Event Bus? +------------------------------- + +The :term:`Event Bus` can help us achieve loose coupling between services, replacing blocking requests between services and large sync jobs, leading to a faster, more reliable, and more extensible system. See event messaging architectural goals highlighted in :doc:`openedx-proposals:architectural-decisions/oep-0041-arch-async-server-event-messaging` to read more about its benefits. Here's a brief summary of some key points: + +* **Eliminate Blocking, Synchronous Requests**: reduce site outages when services become overloaded with requests by replacing synchronous calls with asynchronous communication. +* **Eliminate Expensive, Delayed, Batch Synchronization**: replace expensive batch processing with near real-time data updates. +* **Reduce the need for Plugins**: reduce the computational load for plugins that don't need to run in the same process by allowing cross-service communication of lifecycle events. + +How Does the Open edX Event Bus Work? +------------------------------------- + +The Open edX platform uses the ``OpenEdxPublicSignals`` (Open edX-specific Django Signals) to send events within a service. The event bus extends these signals, allowing them to be broadcasted and handled across multiple services. That's how Open edX Events are used for internal and external communication. For more details on how these Open edX-specific Django Signals are used by the event bus, refer to the :doc:`../decisions/0004-external-event-bus-and-django-signal-events` Architectural Decision Record (ADR). + +Open edX Events provides an abstract implementation of the `publish/subscribe messaging pattern`_ (pub/sub) which is the chosen pattern for the event bus implementation as explained in :doc:`openedx-proposals:architectural-decisions/oep-0052-arch-event-bus-architecture`. It implements two abstract classes, `EventProducer`_ and `EventConsumer`_ which allow concrete implementations of the event bus based on different message brokers, such as Pulsar. + +This abstraction allows for developers to implement their own concrete implementations of the event bus in their own plugins. There are currently two implementations of the event bus, Redis (`event-bus-redis`_) and Kafka (`event-bus-kafka`_). Redis streams is a data structure that acts like an append-only log, and Kafka is a distributed event streaming application. Both implementations handle event production and consumption with technology specific methods. + +Architectural Diagram +********************* + +These diagrams show what happens when an event is sent to the event bus. The event sending workflow follows the same steps as explained in :ref:`events-architecture`, with a key difference: when configured, the event bus recognizes events and publish them to the message broker for consumption by other services. + +Components +~~~~~~~~~~ + +* **Service A (Producer)**: The service that emits the event. Developers may have emitted this event in a key section of the application logic, signaling that a specific action has occurred. +* **Service B (Consumer)**: The service that listens for the event and executes custom logic in response. +* **OpenEdxPublicSignal**: The class that implements all methods used to manage sending the event. This class inherits from Django's Signals class and adds Open edX-specific metadata and behaviors. +* **Event Producer**: The class in the :term:`Producer` service that sends events to the event bus. The producer serializes the event data and enriches it with relevant metadata for the consumer. +* **Event Consumer**: The class in the :term:`Consumer` service that receives events from the event bus. The consumer deserializes the :term:`message ` and re-emits it as an event with the data that was transmitted. +* **Message Broker**: The :term:`message broker ` is responsible for storing and delivering messages between the producer and consumer. +* **Event Bus Plugin**: The concrete implementation of the event bus (EventProducer and EventConsumer) based on a specific :term:`message broker `, such as Pulsar. The plugin handles event production and consumption with technology-specific methods. + +Workflow +~~~~~~~~ + +.. image:: ../_images/event-bus-workflow-service-a.png + :alt: Open edX Event Bus Workflow (Service A) + :align: center + +**From Service A (Producer)** + +1. When the event is sent, a registered event receiver `general_signal_handler`_ is called to send the event to the event bus. This receiver is registered by the Django Signal mechanism when the ``openedx-events`` app is installed, and it listens for all Open edX Events. +2. The receiver checks the ``EVENT_BUS_PRODUCER_CONFIG`` to look for the ``event_type`` of the event that was sent. +3. If the event type is found and it's enabled for publishing in the configuration, the receiver obtains the configured producer class (``EventProducer``) from the concrete event bus implementation. For example, the producer class for Redis or Kafka implemented in their respective plugins. +4. The ``EventProducer`` serializes the event data and enriches it with relevant metadata, and then transforms it into a message that can be transmitted. +5. The producer uses its technology-specific ``send`` method to publish a message to the configured broker (e.g., Redis or Kafka). + +.. image:: ../_images/event-bus-workflow-service-b.png + :alt: Open edX Event Bus Workflow (Service B) + :align: center + +**From Service B (Consumer)** + +1. A :term:`Worker` process in Service B runs indefinitely, checking the broker for new messages. +2. When a new message is found, the ``EventConsumer`` deserializes the message and re-emits it as an event with the data that was transmitted. +3. The event sending and processing workflow repeats in Service B. + +This approach of producing events via settings with the generic handler was chosen to allow for flexibility in the event bus implementation. It allows developers to choose the event bus implementation that best fits their needs, and to easily switch between implementations if necessary. See more details in the :doc:`../decisions/0012-producing-to-event-bus-via-settings` Architectural Decision Record (ADR). + +Event Bus vs Asynchronous Tasks +------------------------------- + +Asynchronous tasks are used for long-running, resource-intensive operations that should not block the main thread of a service. The event bus is used for broadcasting messages to multiple services, allowing them to react to changes or actions in the system. Both can be used for asynchronous communication, but they serve different purposes and have different workflows. + +In this diagram, you can see the difference between the two when it comes to asynchronous communication: + +.. image:: ../_images/event-bus-vs-asynchronous-tasks.png + :alt: Open edX Event Bus vs Asynchronous Tasks + :align: center + +In the upper part of the diagram, we have Service A and Service B. Service A is the producer of the event, and a :term:`Worker` of Service B is the consumer. This is the event bus workflow which allows asynchronous non-blocking communication between services: + +- Service A sends the event as a message to the event bus and continues its execution, as we previously explained. +- Service B polls the message broker for new messages and converts them into ``OpenEdxPublicSignal``. +- Service B emits the event locally and registered Django Signal receivers can react to it. +- Service C can also listen to the event and react to it. + +In the lower part of the diagram, we have the asynchronous tasks workflow: + +- A worker of Service A picks up a task which communicates with Service B via a request-response mechanism, such as HTTP. +- The worker of Service A sends a request to Service B and waits for a response. +- Service B processes the request and sends a response back to the worker. +- The worker receives the response and continues with the next step in its processing. + +If we were to introduce a Service C in this scenario, it would need to wait for the worker of Service A to finish processing the response from Service B and receive a response before it could continue. + +This is an example of an asynchronous approach (from the producer point of view) to send messages to another services but with a blocking nature. + +Use the Open edX Event bus instead of asynchronous tasks when: + +- You want to send a message but don't need a response. For example, notifying other services of an event that occurred. +- You need to send a high volume of messages to a single or multiple services. For example, notifying when users visit a unit in a course or watch a video. +- You want to decouple services and avoid direct dependencies. +- You want to send events out of the Open edX ecosystem. For example, external databases or services that can consume events to update their own state. + +When you need to send a message to a particular service and wait their response for further processing, use asynchronous tasks. + +How is the Open edX Event Bus Used? +----------------------------------- + +The event bus is used to broadcast Open edX Events to multiple services, allowing them to react to changes or actions in the system. + +We encourage you to review the :doc:`../reference/real-life-use-cases` page for examples of how the event bus is used in the Open edX ecosystem by the community. Also, see the :doc:`../how-tos/using-the-event-bus` guide to get start sending events to the event bus. + +.. _general_signal_handler: https://github.com/openedx/openedx-events/blob/main/openedx_events/apps.py#L16-L44 +.. _EventProducer: https://github.com/openedx/openedx-events/blob/main/openedx_events/event_bus/__init__.py#L71-L91 +.. _EventConsumer: https://github.com/openedx/openedx-events/blob/main/openedx_events/event_bus/__init__.py#L128-L139 +.. _publish/subscribe messaging pattern: https://en.wikipedia.org/wiki/Publish%E2%80%93subscribe_pattern +.. _event-bus-redis: https://github.com/openedx/event-bus-redis/ +.. _event-bus-kafka: https://github.com/openedx/event-bus-kafka/ diff --git a/docs/concepts/index.rst b/docs/concepts/index.rst index 25dc09c9..2eec03d9 100644 --- a/docs/concepts/index.rst +++ b/docs/concepts/index.rst @@ -6,3 +6,4 @@ Concepts :caption: Contents: openedx-events + event-bus diff --git a/docs/concepts/openedx-events.rst b/docs/concepts/openedx-events.rst index 43a07004..350dcea4 100644 --- a/docs/concepts/openedx-events.rst +++ b/docs/concepts/openedx-events.rst @@ -15,13 +15,15 @@ These events are built on top of Django signals, inheriting their behavior while Django includes a "signal dispatcher" which helps decoupled applications get notified when actions occur elsewhere in the framework. In a nutshell, signals allow certain senders to notify a set of receivers that some action has taken place. They're especially useful when many pieces of code may be interested in the same events. -Events are primarily used as a communication method between internal services by leveraging Django Signals and external services using the `Event Bus technology`_, making them the standard communication mechanism within the Open edX ecosystem. +Events are primarily used as a communication method between internal services by leveraging Django Signals and external services using the :doc:`../concepts/event-bus`, making them the standard communication mechanism within the Open edX ecosystem. How do Open edX Events work? ---------------------------- Open edX Events are implemented by a class called `OpenEdxPublicSignal`_, which inherits from `Django's Signals class` and adds behaviors specific to the Open edX ecosystem. Thanks to this design, ``OpenEdxPublicSignal`` leverages the functionality of Django signals, allowing developers to apply their existing knowledge of the Django framework. +.. _events-architecture: + Architectural Diagram ********************* @@ -71,14 +73,13 @@ The `Django Signals Documentation`_ provides a more detailed explanation of how How are Open edX Events used? ----------------------------- -As mentioned previously, developers can listen to Open edX Events by registering signal receivers from their Open edX Django plugins that respond to the emitted events or by using the `Event Bus technology`_ to send events to external services. +As mentioned previously, developers can listen to Open edX Events by registering signal receivers from their Open edX Django plugins that respond to the emitted events or by using the :doc:`../concepts/event-bus` to send events to external services. For more information on using Open edX Events, refer to the :doc:`../how-tos/using-events` how-to guide. We also encourage you to explore the :doc:`../reference/real-life-use-cases` section for real-life examples of how Open edX Events are used by the community. .. _Django Signals Documentation: https://docs.djangoproject.com/en/4.2/topics/signals/ .. _triggering the COURSE_ENROLLMENT_CREATED event: https://github.com/openedx/edx-platform/blob/master/common/djangoapps/student/models/course_enrollment.py#L777-L795 .. _course_enrollment_post_save receiver: https://github.com/openedx/edx-platform/blob/master/openedx/core/djangoapps/notifications/handlers.py#L38-L53 -.. _Event Bus technology: https://openedx.atlassian.net/wiki/spaces/AC/pages/3508699151/How+to+start+using+the+Event+Bus .. _Django signals registry mechanism: https://docs.djangoproject.com/en/4.2/topics/signals/#listening-to-signals .. _signal receivers in their plugins: https://edx.readthedocs.io/projects/edx-django-utils/en/latest/edx_django_utils.plugins.html#edx_django_utils.plugins.constants.PluginSignals .. _Open edX Django plugins: https://edx.readthedocs.io/projects/edx-django-utils/en/latest/plugins/readme.html diff --git a/docs/conf.py b/docs/conf.py index 27cffd79..77ad5adc 100644 --- a/docs/conf.py +++ b/docs/conf.py @@ -38,6 +38,7 @@ 'sphinxcontrib.mermaid', 'code_annotations.contrib.sphinx.extensions.openedx_events', 'sphinx.ext.intersphinx', + 'code_annotations.contrib.sphinx.extensions.settings', ] # Add any paths that contain templates here, relative to this directory. @@ -131,4 +132,10 @@ f"https://docs.openedx.org/{rtd_language}/{rtd_version}", None, ), + "openedx-proposals": ( + # Not setting the version on purpose because we always want the latest version + # of OEPs + f"https://docs.openedx.org/projects/openedx-proposals/{rtd_language}/latest", + None, + ), } diff --git a/docs/how-tos/adding-events-to-event-bus.rst b/docs/how-tos/adding-events-to-event-bus.rst deleted file mode 100644 index 89e45da1..00000000 --- a/docs/how-tos/adding-events-to-event-bus.rst +++ /dev/null @@ -1,42 +0,0 @@ -Using the Open edX Event bus -============================ - -After creating a new Open edX Event, you might need to send it across services -instead of just within the same process. For this kind of use-cases, you might want -to use the Open edX Event Bus. Here, we list useful information about -adding a new event to the event bus: - -- `How to start using the Event Bus`_ - - -.. _How to start using the Event Bus: https://openedx.atlassian.net/wiki/spaces/AC/pages/3508699151/How+to+start+using+the+Event+Bus - - -Producing to event bus -^^^^^^^^^^^^^^^^^^^^^^ - -In the producing/host application, include ``openedx_events`` in ``INSTALLED_APPS`` settings and add ``EVENT_BUS_PRODUCER_CONFIG`` setting. For example, below snippet is to push ``XBLOCK_PUBLISHED`` to two different topics and ``XBLOCK_DELETED`` signal to one topic in event bus. - -.. code-block:: python - - # .. setting_name: EVENT_BUS_PRODUCER_CONFIG - # .. setting_default: {} - # .. setting_description: Dictionary of event_types to dictionaries for topic-related configuration. - # Each topic configuration dictionary uses the topic as a key and contains: - # * A flag called `enabled` denoting whether the event will be published. - # * The `event_key_field` which is a period-delimited string path to event data field to use as event key. - # Note: The topic names should not include environment prefix as it will be dynamically added based on - # EVENT_BUS_TOPIC_PREFIX setting. - EVENT_BUS_PRODUCER_CONFIG = { - 'org.openedx.content_authoring.xblock.published.v1': { - 'content-authoring-xblock-lifecycle': {'event_key_field': 'xblock_info.usage_key', 'enabled': True}, - 'content-authoring-xblock-published': {'event_key_field': 'xblock_info.usage_key', 'enabled': True} - }, - 'org.openedx.content_authoring.xblock.deleted.v1': { - 'content-authoring-xblock-lifecycle': {'event_key_field': 'xblock_info.usage_key', 'enabled': True}, - }, - } - -The ``EVENT_BUS_PRODUCER_CONFIG`` is read by openedx_events and a handler is -attached which does the leg work of reading the configuration again and pushing -to appropriate handlers. diff --git a/docs/how-tos/index.rst b/docs/how-tos/index.rst index 3c210b0c..45b871bf 100644 --- a/docs/how-tos/index.rst +++ b/docs/how-tos/index.rst @@ -7,6 +7,6 @@ How-tos creating-new-events adding-events-to-a-service - adding-events-to-event-bus using-events + using-the-event-bus add-new-event-bus-concrete-implementation diff --git a/docs/how-tos/using-the-event-bus.rst b/docs/how-tos/using-the-event-bus.rst new file mode 100644 index 00000000..17a9256d --- /dev/null +++ b/docs/how-tos/using-the-event-bus.rst @@ -0,0 +1,87 @@ +Using the Open edX Event Bus +============================ + +After creating a new Open edX Event, you might need to send it across services instead of just within the same process. For this kind of use-cases, you might want to use the Open edX Event Bus. Here :doc:`../concepts/event-bus`, you can find useful information to start getting familiar with the Open edX Event Bus. + +The Open edX Event Bus is a key component of the Open edX architecture, enabling services to communicate without direct dependencies on each other. This guide provides an overview of how to use the event bus to broadcast Open edX Events to multiple services, allowing them to react to changes or actions in the system. + +Setup +----- + +To start producing and consuming events using the Open edX Event Bus, follow these steps: + +Step 1: Install the Open edX Event Bus Plugin +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +First, you need to install the Open edX Event Bus plugin in both the producing and consuming services. The plugin is a Django app that provides the necessary tools and configurations to produce and consume events. You could install the Redis plugin by running: + +.. code-block:: bash + + pip install edx-event-bus-redis + +.. note:: There are currently two community-supported concrete implementations of the Open edX Events Bus, Redis (`event-bus-redis`_) and Kafka (`event-bus-kafka`_). Redis is the default plugin for the Open edX Event Bus, but you can also use Kafka depending on your requirements. If none of these implementations meet your needs, you can implement your own plugin by following the :doc:`../how-tos/add-new-event-bus-concrete-implementation` documentation. + +Step 2: Configure the Event Bus +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +In :doc:`../reference/event-bus-configurations`, you can find the available configurations for the event bus that are used to set up the event bus in the Open edX platform. + +In both the producing and consuming services, you need to configure the event bus. This includes setting up the :term:`event types `, :term:`topics `, and other configurations for the :term:`Event Bus` to work with. In this step, you should configure the producer and consumer classes for the event bus implementation you are using: + +- In the :term:`producing ` service, configure the producer class by setting the ``EVENT_BUS_PRODUCER`` setting. E.g., ``edx_event_bus_redis.create_producer``. +- In the :term:`consuming ` service, configure the consumer class by setting the ``EVENT_BUS_CONSUMER`` setting. E.g., ``edx_event_bus_redis.RedisEventConsumer``. + +By configuring these settings, you are telling Open edX Events which concrete implementation to use for producing and consuming events. + +Step 3: Produce the Event +~~~~~~~~~~~~~~~~~~~~~~~~~ + +In the producing/host application, include ``openedx_events`` in ``INSTALLED_APPS`` settings if necessary and add ``EVENT_BUS_PRODUCER_CONFIG`` setting. This setting is a dictionary of :term:`event types ` to dictionaries for :term:`Topic` related configuration. Each :term:`Topic` configuration dictionary uses the topic as a key and contains: + +- A flag called ``enabled`` denoting whether the event will be published. +- The ``event_key_field`` which is a period-delimited string path to event data field to use as event key. + +.. note:: The topic names should not include environment prefix as it will be dynamically added based on ``EVENT_BUS_TOPIC_PREFIX`` setting. + +Here's an example of the producer configuration which will publish events for XBlock published and deleted events to the specified :term:`Topic`: + +.. code-block:: python + + EVENT_BUS_PRODUCER_CONFIG = { + 'org.openedx.content_authoring.xblock.published.v1': { + 'content-authoring-xblock-lifecycle': {'event_key_field': 'xblock_info.usage_key', 'enabled': True}, + 'content-authoring-xblock-published': {'event_key_field': 'xblock_info.usage_key', 'enabled': True} + }, + 'org.openedx.content_authoring.xblock.deleted.v1': { + 'content-authoring-xblock-lifecycle': {'event_key_field': 'xblock_info.usage_key', 'enabled': True}, + }, + } + +The ``EVENT_BUS_PRODUCER_CONFIG`` is read by ``openedx_events`` and a handler (`general_signal_handler`_) is attached which does the leg work of reading the configuration again and pushing to appropriate handlers. + +Step 4: Consume the Event +~~~~~~~~~~~~~~~~~~~~~~~~~ + +In the consuming service, include ``openedx_events`` in ``INSTALLED_APPS`` settings if necessary and add ``EVENT_BUS_CONSUMER_CONFIG`` setting. Then, you should implement a receiver for the event type you are interested in. In this example, we are interested in the XBlock deleted event: + +.. code-block:: python + + @receiver(XBLOCK_DELETED) + def update_some_data(sender, **kwargs): + ... do things with the data in kwargs ... + ... log the event for debugging purposes ... + +Step 5: Run the Consumer +~~~~~~~~~~~~~~~~~~~~~~~~ + +To consume events, Open edX Events provides a management command called `consume_events`_ which can be called from the command line, how to run this command will depend on your deployment strategy. This command will start a process that listens to the message broker for new messages, processes them and emits the event. Here is an example using of a `consumer using Tutor hosted in Kubernetes`_. + +You can find more a concrete example of how to produce and consume events in the `event-bus-redis`_ documentation. + +.. _consume_events: https://github.com/openedx/openedx-events/blob/main/openedx_events/management/commands/consume_events.py +.. _event-bus-redis: https://github.com/openedx/event-bus-redis +.. _event-bus-kafka: https://github.com/openedx/event-bus-kafka +.. _run the consumer locally without tutor: https://github.com/openedx/event-bus-redis/?tab=readme-ov-file#testing-locally +.. _run the consumer locally with tutor: https://github.com/openedx/event-bus-redis/blob/main/docs/tutor_installation.rst#setup-example-with-openedx-course-discovery-and-tutor +.. _general_signal_handler: https://github.com/openedx/openedx-events/blob/main/openedx_events/apps.py#L16-L44 +.. _consumer using Tutor hosted in Kubernetes: https://github.com/openedx/tutor-contrib-aspects/blob/master/tutoraspects/patches/k8s-deployments#L535-L588 diff --git a/docs/reference/event-bus-configurations.rst b/docs/reference/event-bus-configurations.rst new file mode 100644 index 00000000..c5313f3c --- /dev/null +++ b/docs/reference/event-bus-configurations.rst @@ -0,0 +1,7 @@ +Event Bus Configuration +======================= + +Here are the available configurations for the event bus that are used to setup the event bus in the Open edX platform. + +.. settings:: + :folder_path: openedx_events/event_bus diff --git a/docs/reference/glossary.rst b/docs/reference/glossary.rst index 44f2b593..6c118c19 100644 --- a/docs/reference/glossary.rst +++ b/docs/reference/glossary.rst @@ -21,21 +21,24 @@ An event has multiple components that are used to define, trigger, and handle th An event is a signal that is emitted when a specific action occurs in the platform. The event definition is the instantiation of the ``OpenEdxPublicSignal`` class that defines the structure and metadata of an event. This definition includes information such as the event name, description, payload, and version. Event definitions are used to create events which are later imported into the services and are triggered by using the ``send_event`` method. Event Bus - To be added soon. + Code that handles asynchronous message transfer between services. - Event Bus Producer - To be added soon. + Message + A unit of information sent from one service to another via a message broker. - Event Bus Consumer - To be added soon. + Message Broker + A service that receives, organizes, and stores messages so other services can query and retrieve them. - Event Bus Topic - To be added soon. + Worker + A machine or process that runs service code without hosting the web application. Workers are used to process messages from the message broker. - Event Bus Producer Config - To be added soon. + Producer + A service that sends events to the event bus. The producer serializes the event data and enriches it with relevant metadata for the consumer. - Event Bus Settings - To be added soon. + Consumer + A service that receives events from the event bus. The consumer deserializes the message and re-emits it as an event with the data that was transmitted. + + Topic + How the event bus implementation groups related events, such as streams in Redis. Producers publish events to topics, and consumers subscribe to topics to receive events. .. _Events Payload ADR: :doc: `/decisions/0003-events-payload` diff --git a/docs/reference/index.rst b/docs/reference/index.rst index 0a1895ab..b7167d78 100644 --- a/docs/reference/index.rst +++ b/docs/reference/index.rst @@ -7,6 +7,7 @@ References events glossary + event-bus-configurations oeps architecture-subdomains real-life-use-cases diff --git a/docs/reference/real-life-use-cases.rst b/docs/reference/real-life-use-cases.rst index 7c1a1ff4..10eeebb7 100644 --- a/docs/reference/real-life-use-cases.rst +++ b/docs/reference/real-life-use-cases.rst @@ -16,9 +16,7 @@ The following list of real-life use cases showcases the different ways Open edX Cross-services communication **************************** -The suggested strategy for cross-service communication efficiently in the Open edX ecosystem is through an event-based architecture implemented via the Event Bus. This functionality used for asynchronous communication between services is built on top of sending Open edX Events (Open edX-specific Django signals) within a service. The Event Bus extends these signals, allowing them to be broadcast and handled across multiple services. For more details on the Event Bus, please see `How to Start Using the Event Bus`_. - -.. TODO: replace event bus confluence page with native docs +As mentioned in :doc:`../concepts/event-bus`, the suggested strategy for cross-service communication in the Open edX ecosystem is through an event-based architecture implemented via the :doc:`../concepts/event-bus`. This functionality used for asynchronous communication between services is built on top of sending Open edX Events (Open edX-specific Django signals) within a service. For more details on the Event Bus, please see :doc:`../how-tos/using-the-event-bus`. Here are some examples of how the Event Bus can be used to facilitate communication between IDAs: @@ -132,7 +130,6 @@ Here are some additional use cases that can be implemented using Open edX Events .. note:: If you have implemented a solution using Open edX Events and would like to share it with the community, please submit a pull request to add it to this list! -.. _How to Start Using the Event Bus: https://openedx.atlassian.net/wiki/spaces/AC/pages/3508699151/How+to+start+using+the+Event+Bus .. _edx-exams: https://github.com/edx/edx-exams .. _edx-proctoring: https://github.com/openedx/edx-proctoring .. _ADR Implementation of Event Driven Architecture for Exam Downstream Effects: https://github.com/edx/edx-exams/blob/main/docs/decisions/0004-downstream-effect-events.rst diff --git a/openedx_events/event_bus/__init__.py b/openedx_events/event_bus/__init__.py index e0fcc332..3f07e3cd 100644 --- a/openedx_events/event_bus/__init__.py +++ b/openedx_events/event_bus/__init__.py @@ -186,6 +186,16 @@ def _reset_state(sender, **kwargs): # pylint: disable=unused-argument get_producer.cache_clear() +# .. setting_name: EVENT_BUS_PRODUCER_CONFIG +# .. setting_default: {} +# .. setting_description: Dictionary of event_types to lists of dictionaries for topic related configuration. +# Each topic configuration dictionary contains a flag called `enabled` denoting whether the event will be +# published to the topic, topic/stream name called `topic` where the event will be pushed to, +# `event_key_field` which is a period-delimited string path to event data field to use as event key. +# The topic names should not include environment prefix as it will be dynamically added based on +# EVENT_BUS_TOPIC_PREFIX setting. See 0012-producing-to-event-bus-via-settings for more details. + + def merge_producer_configs(producer_config_original, producer_config_overrides): """ Merge two EVENT_BUS_PRODUCER_CONFIG maps.