Request

Request is used for operating web requests. It returns a Response object.

Methods:

Core arguments

  • url: the resource link
  • method: request method, shoud be GET or `POST
  • callback: callback function
  • encoding: html encode
  • headers: request headers
  • metadata: some data that need pass to next request
  • request_config: the configure of the request
    • RETRIES: number of retries before failing (default: 3)
    • DELAY: delay (seconds) between each request (default: 0)
    • RETRY_DELAY: delay (seconds) between each retry (default: 0)
    • TIMEOUT: time (seconds) to presist with request before failing/retrying (default: 10)
    • RETRY_FUNC: function to call on retry
    • VALID: function to call after retrieving data
  • request_session: aiohttp.ClientSession
  • aiohttp_kwargs: aiohttp arguments for request

Usage

From the arguments above, we can see that Request can be used both associated with Spider and standalone.

import asyncio

from ruia import Request

request = Request("https://news.ycombinator.com/")
response = asyncio.get_event_loop().run_until_complete(request.fetch())

# Output
# [2018-07-25 11:23:42,620]-Request-INFO  <GET: https://news.ycombinator.com/>
# <Response url[text]: https://news.ycombinator.com/ status:200 metadata:{}>

How It Works?

Request class will send asynchronous http request by packaging aiohttp and pyppeteer.