✨ Add shutdown event and save per page option (#102)

* ✨ Add shutdown event and save per page option

* Update documentation and tests

* Lint

* Add docstring

* Bump version

* Update feature list

* Add test for shutdown

Author: roniemartinez

README.md

@@ -113,7 +113,9 @@ The output in `data.json` should contain the actual URL and the metadata prepend
   - [lxml](https://roniemartinez.github.io/dude/advanced/11_lxml.html) - `pip install pydude[lxml]`
   - [Pyppeteer](https://roniemartinez.github.io/dude/advanced/12_pyppeteer.html) - `pip install pydude[pyppeteer]`
   - [Selenium](https://roniemartinez.github.io/dude/advanced/13_selenium.html) - `pip install pydude[selenium]`
-- Option to follow all links indefinitely (Crawler/Spider). WARNING: Do not use yet until https://github.com/roniemartinez/dude/pull/27 has been implemented.
+- Option to follow all links indefinitely (Crawler/Spider).
+- Events - attach functions to startup, pre-setup, post-setup and shutdown events.
+- Option to save data on every page.
 
 ## Supported Parser Backends
 
@@ -219,14 +221,6 @@ Here is the summary of features supported by each parser backend.
 Read the complete documentation at [https://roniemartinez.github.io/dude/](https://roniemartinez.github.io/dude/).
 All the advanced and useful features are documented there.
 
-## Support
-
-This project is at a very early stage. This dude needs some love! ❤️
-
-Contribute to this project by feature requests, idea discussions, reporting bugs, opening pull requests, or through Github Sponsors. Your help is highly appreciated.
-
-[![Github Sponsors](https://img.shields.io/github/sponsors/roniemartinez?label=github%20sponsors&logo=github%20sponsors&style=for-the-badge)](https://github.com/sponsors/roniemartinez)
-
 ## Requirements
 
 - ✅ Any dude should know how to work with selectors (CSS or XPath).

docs/advanced/06_custom_storage.md

@@ -42,6 +42,43 @@ The custom storage above can then be called using any of the options below.
         dude.run(urls=["<url>"], pages=2, format="table")
     ```
 
+## Saving on every page
+
+It is possible to call the save functions after each page.
+This is useful when running in spider mode to prevent lost of data.
+To make use of this option, the flag `is_per_page` in the `@save()` should be set to `True`.
+
+=== "Python"
+
+    ```python
+    @save("table", is_per_page=True)
+    def save_table(data, output) -> bool:
+        ...
+    ```
+
+To run the scraper in per-page save, pass `--save-per-page` argument.
+
+=== "Terminal"
+
+    ```bash
+    dude scrape --url "<url>" path/to/script.py --format table --save-per-page
+    ```
+
+=== "Python"
+
+    ```python
+    if __name__ == "__main__":
+        import dude
+
+        dude.run(urls=["<url>"], pages=2, format="table", save_per_page=True)
+    ```
+
+!!! note
+
+    The option `--save-per-page` is best used with events to make sure that connections or file handles are opened 
+    and closed properly. Check the examples below.
+
 ## Examples
 
-A more extensive example can be found at [examples/custom_storage.py](https://github.com/roniemartinez/dude/tree/master/examples/custom_storage.py).
+A more extensive example can be found at [examples/custom_storage.py](https://github.com/roniemartinez/dude/tree/master/examples/custom_storage.py) and
+[examples/save_per_page.py](https://github.com/roniemartinez/dude/tree/master/examples/save_per_page.py).

docs/advanced/14_events.md

@@ -67,3 +67,20 @@ def print_pdf(page):
     unique_name = str(uuid.uuid4())
     page.pdf(path=SAVE_DIR / f"{unique_name}.pdf")
 ```
+
+## Shutdown Event
+
+The Shutdown event is executed before terminating the application.
+
+The `@shutdown()` decorator can be used to register a function for shutdown. 
+
+```python
+import shutil
+
+from dude import shutdown
+
+
+@shutdown()
+def zip_all():
+    shutil.make_archive("images-and-pdfs", "zip", SAVE_DIR)
+```

docs/cli.md

@@ -3,10 +3,10 @@
 === "CLI"
 
     ```commandline
-    usage: dude scrape [-h] --url URL [--playwright | --bs4 | --parsel | --lxml | --pyppeteer | --selenium] [--headed] [--browser {chromium,firefox,webkit}] [--pages PAGES] [--output OUTPUT]
-                   [--format FORMAT] [--proxy-server PROXY_SERVER] [--proxy-user PROXY_USER] [--proxy-pass PROXY_PASS] [--follow-urls]
-                   PATH [PATH ...]
-
+    usage: dude scrape [-h] --url URL [--playwright | --bs4 | --parsel | --lxml | --pyppeteer | --selenium] [--headed] [--browser {chromium,firefox,webkit}] [--pages PAGES] [--output OUTPUT] [--format FORMAT] [--proxy-server PROXY_SERVER] [--proxy-user PROXY_USER]
+                       [--proxy-pass PROXY_PASS] [--follow-urls] [--save-per-page]
+                       PATH [PATH ...]
+    
     Run the dude scraper.
 
     optional arguments:
@@ -28,13 +28,13 @@
                             Browser type to use.
       --pages PAGES         Maximum number of pages to crawl before exiting (default=1). This is only valid when a navigate handler is defined.
       --output OUTPUT       Output file. If not provided, prints into the terminal.
-      --format FORMAT       Output file format. If not provided, uses the extension of the output file or defaults to "json". Supports "json", "yaml/yml", and "csv" but can be extended using the @save()
-                            decorator.
+      --format FORMAT       Output file format. If not provided, uses the extension of the output file or defaults to "json". Supports "json", "yaml/yml", and "csv" but can be extended using the @save() decorator.
       --proxy-server PROXY_SERVER
                             Proxy server.
       --proxy-user PROXY_USER
                             Proxy username.
       --proxy-pass PROXY_PASS
                             Proxy password.
       --follow-urls         Automatically follow URLs.
+      --save-per-page       Flag to save data on every page extraction or not. If not, saves all the data at the end.If --follow-urls is set to true, this variable will be automatically set to true.
     ```

docs/diagrams/events.png

@@ -15,4 +15,6 @@
     - [lxml](https://lxml.de/) - `pip install pydude[lxml]`
     - [Pyppeteer](https://github.com/pyppeteer/pyppeteer) - `pip install pydude[pyppeteer]`
     - [Selenium](https://github.com/SeleniumHQ/Selenium) - `pip install pydude[selenium]`
-- Option to follow all links indefinitely (Crawler/Spider). WARNING: Do not use yet until https://github.com/roniemartinez/dude/pull/27 has been implemented.
+- Option to follow all links indefinitely (Crawler/Spider).
+- Events - attach functions to startup, pre-setup, post-setup and shutdown events.
+- Option to save data on every page.

docs/features.md

@@ -3,10 +3,10 @@
 from pathlib import Path
 from typing import Any
 
-from .context import group, post_setup, pre_setup, run, save, select, startup  # noqa: F401
+from .context import group, post_setup, pre_setup, run, save, select, shutdown, startup  # noqa: F401
 from .scraper import Scraper  # noqa: F401
 
-__al__ = ["Scraper", "group", "run", "save", "select", "startup", "pre_setup", "post_setup"]
+__al__ = ["Scraper", "group", "run", "save", "select", "startup", "shutdown", "pre_setup", "post_setup"]
 
 
 logging.basicConfig(stream=sys.stdout, level=logging.INFO, format="%(asctime)s %(levelname)s %(message)s")
@@ -148,6 +148,14 @@ def cli() -> None:  # pragma: no cover
         action="store_true",
         help="Automatically follow URLs.",
     )
+    optional.add_argument(
+        "--save-per-page",
+        dest="save_per_page",
+        default=False,
+        action="store_true",
+        help="Flag to save data on every page extraction or not. If not, saves all the data at the end."
+        "If --follow-urls is set to true, this variable will be automatically set to true.",
+    )
     arguments = parser.parse_args()
 
     if arguments.version:
@@ -204,4 +212,5 @@ def cli() -> None:  # pragma: no cover
         output=arguments.output,
         format=arguments.format,
         follow_urls=arguments.follow_urls,
+        save_per_page=arguments.save_per_page,
     )

dude/__init__.py

@@ -43,14 +43,14 @@ def __init__(
         self,
         rules: List[Rule] = None,
         groups: Dict[Callable, Selector] = None,
-        save_rules: Dict[str, Any] = None,
+        save_rules: Dict[Tuple[str, bool], Any] = None,
         events: Optional[DefaultDict] = None,
         has_async: bool = False,
         scraper: Optional["ScraperAbstract"] = None,
     ) -> None:
         self.rules: List[Rule] = rules or []
         self.groups: Dict[Callable, Selector] = groups or {}
-        self.save_rules: Dict[str, Any] = save_rules or {"json": save_json}
+        self.save_rules: Dict[Tuple[str, bool], Any] = save_rules or {("json", False): save_json}
         self.events: DefaultDict = events or collections.defaultdict(list)
         self.has_async = has_async
         self.scraper = scraper
@@ -67,6 +67,7 @@ def run(
         output: Optional[str],
         format: str,
         follow_urls: bool = False,
+        save_per_page: bool = False,
     ) -> None:
         """
         Abstract method for executing the scraper.
@@ -77,6 +78,7 @@ def run(
         :param output: Output file. If not provided, prints in the terminal.
         :param format: Output file format. If not provided, uses the extension of the output file or defaults to json.
         :param follow_urls: Automatically follow URLs.
+        :param save_per_page: Flag to save data on every page extraction or not. If not, saves all the data at the end.
         """
         raise NotImplementedError  # pragma: no cover
 
@@ -188,19 +190,20 @@ def wrapper(func: Callable) -> Union[Callable, Coroutine]:
 
         return wrapper
 
-    def save(self, format: str) -> Callable:
+    def save(self, format: str, is_per_page: bool = False) -> Callable:
         """
         Decorator to register a save function to a format.
 
         :param format: Format (json, csv, or any custom string).
+        :param is_per_page: Flag to identify if func will be called after each page.
         """
 
         def wrapper(func: Callable) -> Callable:
             if asyncio.iscoroutinefunction(func):
                 self.has_async = True
 
             save_rules = self.scraper.save_rules if self.scraper else self.save_rules
-            save_rules[format] = func
+            save_rules[format, is_per_page] = func
             return func
 
         return wrapper
@@ -258,6 +261,24 @@ def wrapper(func: Callable) -> Callable:
 
         return wrapper
 
+    def shutdown(self) -> Callable:
+        """
+        Decorator to register a function to the shutdown events.
+
+        Shutdown events are executed before terminating the application for cleaning up or closing resources like
+        files and database sessions.
+        """
+
+        def wrapper(func: Callable) -> Callable:
+            if asyncio.iscoroutinefunction(func):
+                self.has_async = True
+
+            events = self.scraper.events if self.scraper else self.events
+            events["shutdown"].append(func)
+            return func
+
+        return wrapper
+
     def iter_urls(self) -> Iterable[str]:
         try:
             while True:
@@ -291,11 +312,20 @@ def event_startup(self) -> None:
         """
         Run all startup events
         """
+        self.run_event("startup")
+
+    def event_shutdown(self) -> None:
+        """
+        Run all shutdown events
+        """
+        self.run_event("shutdown")
+
+    def run_event(self, event_name: str) -> None:
         loop = None
         if self.has_async:
             loop = asyncio.get_event_loop()
 
-        for func in self.events["startup"]:
+        for func in self.events[event_name]:
             if asyncio.iscoroutinefunction(func):
                 assert loop is not None
                 loop.run_until_complete(func())
@@ -308,7 +338,7 @@ def __init__(
         self,
         rules: List[Rule] = None,
         groups: Dict[Callable, Selector] = None,
-        save_rules: Dict[str, Any] = None,
+        save_rules: Dict[Tuple[str, bool], Any] = None,
         events: Optional[DefaultDict] = None,
         has_async: bool = False,
     ) -> None:
@@ -441,29 +471,39 @@ def get_flattened_data(self) -> List[Dict]:
             items.append(item)
         return items
 
-    def _save(self, format: str, output: Optional[str] = None) -> None:
+    def _save(self, format: str, output: Optional[str] = None, save_per_page: bool = False) -> None:
         if output:
             extension = Path(output).suffix.lower()[1:]
             format = extension
-
-        data = self.get_flattened_data()
         try:
-            if self.save_rules[format](data, output):
+            handler = self.save_rules[format, save_per_page]
+            data = self.get_flattened_data()
+            if not len(data):
+                logger.info(
+                    "No data was scraped. Skipped saving %s.",
+                    dict(format=format, output=format, save_per_page=save_per_page),
+                )
+                return
+            if handler(data, output):
                 self.collected_data.clear()
             else:
                 raise Exception("Failed to save output %s.", {"output": output, "format": format})
         except KeyError:
-            self.collected_data.clear()
             raise
 
-    async def _save_async(self, format: str, output: Optional[str] = None) -> None:
+    async def _save_async(self, format: str, output: Optional[str] = None, save_per_page: bool = False) -> None:
         if output:
             extension = Path(output).suffix.lower()[1:]
             format = extension
-
-        data = self.get_flattened_data()
         try:
-            handler = self.save_rules[format]
+            handler = self.save_rules[format, save_per_page]
+            data = self.get_flattened_data()
+            if not len(data):
+                logger.info(
+                    "No data was scraped. Skipped saving %s.",
+                    dict(format=format, output=format, save_per_page=save_per_page),
+                )
+                return
             if asyncio.iscoroutinefunction(handler):
                 is_successful = await handler(data, output)
             else:
@@ -473,7 +513,6 @@ async def _save_async(self, format: str, output: Optional[str] = None) -> None:
             else:
                 raise Exception("Failed to save output %s.", {"output": output, "format": format})
         except KeyError:
-            self.collected_data.clear()
             raise
 
     @staticmethod

dude/base.py

@@ -11,6 +11,7 @@
 save = _scraper.save
 select = _scraper.select
 startup = _scraper.startup
+shutdown = _scraper.shutdown
 pre_setup = _scraper.pre_setup
 post_setup = _scraper.post_setup