Update ideas-20250101.md

docs/internships/gsoc/ideas-20250101.md

@@ -163,18 +163,18 @@ This created additional complexity in the configuration and management of the ap
          1. Reduce the learning curve to contribute.
       1. Your plan must include an approach to convert the existing repository to support only the web front end.
    1. Installation
-      1. The current version requires the installation of system daemons. A method that uses more modern containerization for simplicity is needed.
-      1. Interaction between the equivalent replacements for the daemons must only communicate via existing GraphQL API calls for data.
-      1. We need an easy to use installation and configuration script for the web UI.
-         1. The structure of the configuration files must be maintained
-   1. Documentation
+      1. The current version requires the installation of system daemons. A method that improves any existing containerization for simplicity is needed.
+      2. Interaction between the equivalent replacements for the daemons must only communicate via existing GraphQL API calls for data.
+      3. We need an easy to use installation and configuration script for the web UI.
+         1. The structure of the configuration files could be improved.
+   2. Documentation
       1. Create autogenerated markdown documentation for developers from the code base based on code comments.
          1. This is already done in our Talawa\* repositories.
-         1. GitHub actions must be created to ensure that the auto-generated documentation is formatted according to our standards.
-      1. The site must be autoupdated whenever the markdown mentioned in this section is changed.
-   1. Testing
+         2. GitHub actions must be created to ensure that the auto-generated documentation is formatted according to our standards.
+      2. The site must be autoupdated whenever the markdown mentioned in this section is changed.
+   3. Testing
       1. We want to achieve 100% test code coverage for the repository using GitHub actions
-      1. We also want GitHub actions to fail if codecov.io code coverage criteria are not met and coderabbit.io does not approve the PR.
+      2. We also want GitHub actions to fail if codecov.io code coverage criteria are not met and coderabbit.io does not approve the PR.
 3. **Repos to Update:** SwitchMap-NG
 4. **Skills Required:** Proficiency in the code stacks related to the repository. Refer to the introduction section for more details.
 5. **Depends on Project:** N/A
@@ -190,35 +190,41 @@ This created additional complexity in the configuration and management of the ap
 2. **Expected Outcomes:**
    1. Operation
       1. The app isn’t suitable for a distributed deployment with groups of pollers that may not have access to the central API server. There needs to be an intuitive way for:
-         1. pollers to send data to an aggregator in their region
-         1. each regional aggregator to send data to the central API server
-         1. configuring this.
-   1. Asynchronous Polling
-      1. Replace threaded polling with an asynchronous polling mechanism using Python’s asyncio or aiohttp for better scalability. This would allow polling of larger networks without blocking and improve the overall speed and resource usage. This would facilitate faster data collection, more efficient resource utilization, and the ability to scale to large networks.
-   1. Historical Data Storage
-      1. Use the current state database to store and analyze historical network data, such as port utilization, device status, and ARP data over time. This would allow users to analyze trends and detect issues that may have developed gradually.
-   1. Additional Data Collection
-      1. Extend SNMP support by parsing additional MIBs for more detailed device information, such as CPU usage, memory stats, or interface statistics. This would make the inventory system more comprehensive.
-   1. Repository Strategy
+         1. pollers to reliably store and forward data to an aggregator in their region.
+         2. each regional aggregator to send data to the central API server
+         3. configuring this.
+      2. The solution must tolerate poor connectivity between the pollers and the API without loss of data for a configurable amount of time.
+   2. Asynchronous Polling
+      1. The current python EasySNMP package is very resource intensive and must be replaced.
+      2. Consider replacing multiprocessing / threaded polling with a faster asynchronous polling mechanism using Python’s asyncio or aiohttp for better scalability. This would allow polling of larger networks without blocking and improve the overall speed and resource usage. This would facilitate faster data collection, more efficient resource utilization, and the ability to scale to large networks.
+      3. Another option would be to use a purpose built high performance SNMP application like MRTG/RRDTool to poll and/or evaluate the available SNMP parameters on the target devices.
+   3. Realtime data updates
+      1. The current approach stores posted data to disk with periodic database updates. 
+      2. The speed of processing the data may have optimization potential.
+   4. Historical Data Storage
+      1. Use the current state database to more effectively store and analyze historical network data, such as port utilization, device status, and ARP data over time. This would allow users to analyze trends and detect issues that may have developed gradually.
+   5. Additional Data Collection
+      1. Extend SNMP support by parsing additional OIDs (MIB translations can be slower) for more detailed device information, such as CPU usage, memory stats, or interface statistics. This would make the inventory system more comprehensive.
+   6. Repository Strategy
       1. We want to migrate from a monorepo to a polyrepo strategy. We feel this will:
          1. Allow optimization of each subsystem's code using the appropriate best practices.
-         1. Reduce the learning curve to contribute.
-      1. Your plan must include approaches to:
+         2. Reduce the learning curve to contribute.
+      2. Your plan must include approaches to:
          1. Create a separate poller repo
-         1. Create a separate API repo while maintaining the existing endpoint functionality.
-   1. Installation
-      1. The current version requires the installation of system daemons. A method that uses more modern containerization for simplicity is needed.
-      1. Interaction between the equivalent replacements for the daemons must only communicate via existing API calls. The RESTful posting of data to the DB API server’s file system should remain to reduce the potential overload unless a suitable alternative can be found.
-      1. We need an easy to use installation and configuration script for the web UI.
-         1. The structure of the configuration files must be maintained
-   1. Documentation
+         2. Create a separate API repo while maintaining the existing endpoint functionality.
+   7. Installation
+      1. The current version requires the installation of system daemons. A method that improves any existing containerization for simplicity is needed.
+      2. Interaction between the equivalent replacements for the daemons must only communicate via existing API calls. The RESTful posting of data to the DB API server’s file system should remain to reduce the potential overload unless a suitable alternative can be found.
+      3. We need an easy to use installation and configuration script for the web UI.
+         1. The structure of the configuration files could be improved.
+   8.  Documentation
       1. Create autogenerated markdown documentation for developers from the code base based on code comments.
          1. This is already done in our Talawa\* repositories.
-         1. GitHub actions must be created to ensure that the auto-generated documentation is formatted according to our standards.
-      1. The site must be autoupdated whenever the markdown mentioned in this section is changed.
-   1. Testing
+         2. GitHub actions must be created to ensure that the auto-generated documentation is formatted according to our standards.
+      2. The site must be autoupdated whenever the markdown mentioned in this section is changed.
+   9.  Testing
       1. We want to achieve 100% test code coverage for the repository using GitHub actions
-      1. We also want GitHub actions to fail if codecov.io code coverage criteria are not met and coderabbit.io does not approve the PR.
+      2. We also want GitHub actions to fail if codecov.io code coverage criteria are not met and coderabbit.io does not approve the PR.
 3. **Repos to Update:** SwitchMap-NG
 4. **Skills Required:** Proficiency in the code stacks related to the repository. Refer to the introduction section for more details.
 5. **Depends on Project:** N/A