You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
HTTP API for scheduling `Scrapy <https://scrapy.org/>`_ spiders and receiving their items in response.
26
25
27
-
HTTP server which provides API for scheduling `Scrapy <https://scrapy.org/>`_ spiders and
28
-
making requests with spiders.
29
-
30
-
Features
31
-
========
32
-
* Allows you to easily add HTTP API to your existing Scrapy project
33
-
* All Scrapy project components (e.g. middleware, pipelines, extensions) are supported out of the box.
34
-
* You simply run Scrapyrt in Scrapy project directory and it starts HTTP server allowing you to schedule your spiders and get spider output in JSON format.
35
-
36
-
Note
37
-
====
38
-
* Project is not a replacement for `Scrapyd <https://scrapyd.readthedocs.io/en/stable/>`_ or `Scrapy Cloud <https://www.zyte.com/scrapy-cloud/>`_ or other infrastructure to run long running crawls
39
-
* Not suitable for long running spiders, good for spiders that will fetch one response from some website and return response
40
-
41
-
Getting started
26
+
Quickstart
42
27
===============
43
28
44
-
To install Scrapyrt::
29
+
**1. install**
45
30
46
-
pip install scrapyrt
31
+
> pip install scrapyrt
47
32
48
-
Now you can run Scrapyrt from within Scrapy project by just typing::
33
+
**2. switch to Scrapy project (e.g. quotesbot project)**
49
34
50
-
scrapyrt
35
+
> cd ../quotesbot
51
36
52
-
in Scrapy project directory.
37
+
**3. launch ScrapyRT**
53
38
54
-
Scrapyrt will look for ``scrapy.cfg`` file to determine your project settings,
55
-
and will raise error if it won't find one. Note that you need to have all
56
-
your project requirements installed.
39
+
> scrapyrt
57
40
58
-
Scrapyrt supports endpoint ``/crawl.json`` that can be requested
59
-
with two methods: GET and POST.
41
+
**4. run your spiders**
60
42
61
-
To run sample toscrape-css spider from `Quotesbot <https://github.com/scrapy/quotesbot>`_
* Allows you to easily add HTTP API to existing Scrapy project
56
+
* All Scrapy project components (e.g. middleware, pipelines, extensions) are supported out of the box.
57
+
* You simply run Scrapyrt in Scrapy project directory and it starts HTTP server allowing you to schedule your spiders and get spider output in JSON format.
71
58
59
+
Note
60
+
====
61
+
* Project is not a replacement for `Scrapyd <https://scrapyd.readthedocs.io/en/stable/>`_ or `Scrapy Cloud <https://www.zyte.com/scrapy-cloud/>`_ or other infrastructure to run long running crawls
62
+
* Not suitable for long running spiders, good for spiders that will fetch one response from some website and return response
0 commit comments