Python Requests Library

The Python Requests library simplifies making HTTP requests to web servers and working with their responses. In this article, we will learn how to install this library, Understand the structure of a request, Make GET and POST requests, Read and extract elements of the HTML of a web page, and how to Improve your requests.

What is Python Requests Library?

The Requests library offers a basic API for interacting with HTTP activities like GET, POST, etc. The Requests library’s methods perform HTTP operations against a specific web server given by its URL. It also allows you to give extra information to a web server via arguments and headers, encode server responses, identify faults, and handle redirection.

What is HTTP?

The Hypertext Transfer Protocol (HTTP) is a client-server request/response protocol that uses TCP/IP connections to exchange request and response messages. HTTP clients, such as web browsers or mobile applications, submit requests to HTTP servers. Which respond with messages that include a status line, a header, and a body.

Installing the Python Requests Library

Let’s begin by installing the requests library. To do so, run the following command:

$ pip install requests

If you prefer to use Pipenv for managing Python packages, you can run the following:

$ pipenv install requests

Once the requests library is installed, you can use it in your application. Importing requests look like this:

import requests

Python Requests Methods

  1. Get: Request data
  2. Post: Publish data
  3. Put: Replace data
  4. Patch: Make Partial changes to the data
  5. Delete: Delete data
  6. Head: Similar to get request but without the body
  7. Request: Create a request object by specifying the method to choose

Method 01: Python Requests Get

import requests
 
url = 'https://codegrepper.com/'
response = requests.get(url)
 
print('URL: ', response.url)
print('Status code: ', response.status_code)
print('HTTP header: ', response.headers)

Output:

URL:  https://www.codegrepper.com/
Status code:  200
HTTP header:  {'Server': 'nginx/1.15.5 (Ubuntu)', 'Date': 'Tue, 24 May 2022 12:03:48 GMT', 'Content-Type': 'text/html; charset=UTF-8', 'Transfer-Encoding': 'chunked', 'Connection': 'keep-alive', 'Content-Encoding': 'gzip'}

Method 02: Python Requests Post

import requests

url = 'http://httpbin.org/post'
payload = {
    'website':'softhunt.net',
    'courses':['Python','JAVA']
    }
response = requests.post(url, data=payload)
print(response.text)

Output:

{
  "args": {}, 
  "data": "", 
  "files": {}, 
  "form": {
    "courses": [
      "Python", 
      "JAVA"
    ], 
    "website": "softhunt.net"
  }, 
  "headers": {
    "Accept": "*/*", 
    "Accept-Encoding": "gzip, deflate", 
    "Content-Length": "48", 
    "Content-Type": "application/x-www-form-urlencoded", 
    "Host": "httpbin.org", 
    "User-Agent": "python-requests/2.22.0", 
    "X-Amzn-Trace-Id": "Root=1-628cd0b2-0145853f3a8682976a8178b0"
  }, 
  "json": null, 
  "origin": "34.136.57.210", 
  "url": "http://httpbin.org/post"
}

Method 03: Python Requests Put

import requests

url = 'https://httpbin.org/put'
payload = {
    'website':'softhunt.net',
    'courses':['Python','JAVA']
    }
response = requests.put(url, data=payload)
print(response.text)

Output:

{
  "args": {}, 
  "data": "", 
  "files": {}, 
  "form": {
    "courses": [
      "Python", 
      "JAVA"
    ], 
    "website": "softhunt.net"
  }, 
  "headers": {
    "Accept": "*/*", 
    "Accept-Encoding": "gzip, deflate", 
    "Content-Length": "48", 
    "Content-Type": "application/x-www-form-urlencoded", 
    "Host": "httpbin.org", 
    "User-Agent": "python-requests/2.22.0", 
    "X-Amzn-Trace-Id": "Root=1-628cd135-4beba1dc074c5cea549100e7"
  }, 
  "json": null, 
  "origin": "34.136.57.210", 
  "url": "https://httpbin.org/put"
}

Method 04: Python Requests Patch

import requests

url = 'https://httpbin.org/patch'
payload = {
    'website':'softhunt.net',
    'courses':['Python','JAVA']
    }
response = requests.patch(url, data=payload)
print(response.json())

Output:

{'args': {}, 'data': '', 'files': {}, 'form': {'courses': ['Python', 'JAVA'], 'website': 'softhunt.net'}, 'headers': {'Accept': '*/*', 'Accept-Encoding': 'gzip, deflate', 'Content-Length': '48', 'Content-Type': 'application/x-www-form-urlencoded', 'Host': 'httpbin.org', 'User-Agent': 'python-requests/2.22.0', 'X-Amzn-Trace-Id': 'Root=1-628cd1c7-7f6f10dc3bdfacce2d413f9c'}, 'json': None, 'origin': '34.136.57.210', 'url': 'https://httpbin.org/patch'}

Method 05: Python Requests Delete

import requests

url = 'https://httpbin.org/delete'

response = requests.delete(url)
print(response.json())

Output:

{'args': {}, 'data': '', 'files': {}, 'form': {}, 'headers': {'Accept': '*/*', 'Accept-Encoding': 'gzip, deflate', 'Content-Length': '0', 'Host': 'httpbin.org', 'User-Agent': 'python-requests/2.22.0', 'X-Amzn-Trace-Id': 'Root=1-628cd26b-58c47e295b860e0d6dabc580'}, 'json': None, 'origin': '34.136.57.210', 'url': 'https://httpbin.org/delete'}

Response Methods and Attributes

The response object contains the server’s response to the HTTP request. You can investigate the details of the Response object by using help().

import requests
 
url = 'https://codegrepper.com/'
response = requests.get(url)
 
print(help(response))

Output:

Help on Response in module requests.models object:

class R Re es sp po on ns se e(builtins.object)
 |  The :class:`Response <Response>` object, which contains a
 |  server's response to an HTTP request.
 |  
 |  Methods defined here:
 |  
 |  __b bo oo ol l__(self)
 |      Returns True if :attr:`status_code` is less than 400.
 |      
 |      This attribute checks if the status code of the response is between
 |      400 and 600 to see if there was a client error or a server error. If
 |      the status code, is between 200 and 400, this will return True. This
|      is **not** a check to see if the response code is ``200 OK``.
 |  
 |  __e en nt te er r__(self)
 |  
 |  __e ex xi it t__(self, *args)
 |  
 |  __g ge et ts st ta at te e__(self)
 |  
 |  __i in ni it t__(self)
--More--
  • text [data descriptor]: Content of the response, in Unicode.
  • content [data descriptor]: Content of the response, in bytes.
  • URL [attribute]: URL of the request
  • status_code [attribute]: Status code returned by the server
  • headers [attribute]: HTTP headers returned by the server
  • history [attribute]: list of response objects holding the history of request
  • links [attribute]: Returns the parsed header links of the response, if any.
  • JSON [attribute]: Returns the JSON-encoded content of a response, if any.

Access the Response Methods and Attributes

The response from the request is an object in which you can access its methods and attributes. You can access the attributes using the object.attribute notation and the methods using the object.method() notation.

import requests
 
url = 'https://httpbin.org/get'
response = requests.get(url)
 
print(response.text) # access response data atributes and descriptors
print(response.json()) # access response methods

Output:

{
  "args": {}, 
  "headers": {
    "Accept": "*/*", 
    "Accept-Encoding": "gzip, deflate", 
    "Host": "httpbin.org", 
    "User-Agent": "python-requests/2.22.0", 
    "X-Amzn-Trace-Id": "Root=1-628ce673-34ec9faf21c10df06548452c"
  }, 
  "origin": "35.222.138.55", 
  "url": "https://httpbin.org/get"
}

{'args': {}, 'headers': {'Accept': '*/*', 'Accept-Encoding': 'gzip, deflate', 'Host': 'httpbin.org', 'User-Agent': 'python-requests/2.22.0', 'X-Amzn-Trace-Id': 'Root=1-628ce673-34ec9faf21c10df06548452c'}, 'origin': '35.222.138.55', 'url': 'https://httpbin.org/get'}

Process the Response

Process 01: Show Status Code

import requests
 
url = 'https://codegrepper.com/'
response = requests.get(url)
 
print(response.status_code)

Output:

200

Process 02: Get the HTML of the page

import requests
 
url = 'http://httpbin.org'
response = requests.get(url)
 
print(response.text) # get content as a string
print(response.content) # get content as bytes

Output:

<!DOCTYPE html>
<html lang="en">

<head>
    <meta charset="UTF-8">
    <title>httpbin.org</title>
    <link href="https://fonts.googleapis.com/css?family=Open+Sans:400,700|Source+Code+Pro:300,600|Titillium+Web:400,600,700"
        rel="stylesheet">
    <link rel="stylesheet" type="text/css" href="/flasgger_static/swagger-ui.css">
    <link rel="icon" type="image/png" href="/static/favicon.ico" sizes="64x64 32x32 16x16" />
    <style>
        html {
            box-sizing: border-box;
            overflow: -moz-scrollbars-vertical;
            overflow-y: scroll;
        }

        *,
        *:before,
        *:after {
            box-sizing: inherit;
        }

        body {
            margin: 0;
            background: #fafafa;
        }
    </style>
</head>

<body>
    <a href="https://github.com/requests/httpbin" class="github-corner" aria-label="View source on Github">
        <svg width="80" height="80" viewBox="0 0 250 250" style="fill:#151513; color:#fff; position: absolute; top: 0; border: 0; right: 0;"
            aria-hidden="true">
            <path d="M0,0 L115,115 L130,115 L142,142 L250,250 L250,0 Z"></path>
            <path d="M128.3,109.0 C113.8,99.7 119.0,89.6 119.0,89.6 C122.0,82.7 120.5,78.6 120.5,78.6 C119.2,72.0 123.4,76.3 123.4,76.3 C127.3,80.9 125.5,87.3 125.5,87.3 C122.9,97.6 130.6,101.9 134.4,103.2"
                fill="currentColor" style="transform-origin: 130px 106px;" class="octo-arm"></path>
            <path d="M115.0,115.0 C114.9,115.1 118.7,116.5 119.8,115.4 L133.7,101.6 C136.9,99.2 139.9,98.4 142.2,98.6 C133.8,88.0 127.5,74.4 143.8,58.0 C148.5,53.4 154.0,51.2 159.7,51.0 C160.3,49.4 163.2,43.6 171.4,40.1 C171.4,40.1 176.1,42.5 178.8,56.2 C183.1,58.6 187.2,61.8 190.9,65.4 C194.5,69.0 197.7,73.2 200.1,77.6 C213.8,80.2 216.3,84.9 216.3,84.9 C212.7,93.1 206.9,96.0 205.4,96.6 C205.1,102.4 203.0,107.8 198.3,112.5 C181.9,128.9 168.3,122.5 157.7,114.1 C157.9,116.9 156.7,120.9 152.7,124.9 L141.0,136.5 C139.8,137.7 141.6,141.9 141.8,141.8 Z"
                fill="currentColor" class="octo-body"></path>
        </svg>
    </a>
    <svg xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" style="position:absolute;width:0;height:0">
        <defs>
            <symbol viewBox="0 0 20 20" id="unlocked">
                <path d="M15.8 8H14V5.6C14 2.703 12.665 1 10 1 7.334 1 6 2.703 6 5.6V6h2v-.801C8 3.754 8.797 3 10 3c1.203 0 2 .754 2 2.199V8H4c-.553 0-1 .646-1 1.199V17c0 .549.428 1.139.951 1.307l1.197.387C5.672 18.861 6.55 19 7.1 19h5.8c.549 0 1.428-.139 1.951-.307l1.196-.387c.524-.167.953-.757.953-1.306V9.199C17 8.646 16.352 8 15.8 8z"></path>
            </symbol>

            <symbol viewBox="0 0 20 20" id="locked">
                <path d="M15.8 8H14V5.6C14 2.703 12.665 1 10 1 7.334 1 6 2.703 6 5.6V8H4c-.553 0-1 .646-1 1.199V17c0 .549.428 1.139.951 1.307l1.197.387C5.672 18.861 6.55 19 7.1 19h5.8c.549 0 1.428-.139 1.951-.307l1.196-.387c.524-.167.953-.757.953-1.306V9.199C17 8.646 16.352 8 15.8 8zM12 8H8V5.199C8 3.754 8.797 3 10 3c1.203 0 2 .754 2 2.199V8z"
                />
            </symbol>

            <symbol viewBox="0 0 20 20" id="close">
                <path d="M14.348 14.849c-.469.469-1.229.469-1.697 0L10 11.819l-2.651 3.029c-.469.469-1.229.469-1.697 0-.469-.469-.469-1.229 0-1.697l2.758-3.15-2.759-3.152c-.469-.469-.469-1.228 0-1.697.469-.469 1.228-.469 1.697 0L10 8.183l2.651-3.031c.469-.469 1.228-.469 1.697 0 .469.469.469 1.229 0 1.697l-2.758 3.152 2.758 3.15c.469.469.469 1.229 0 1.698z"
                />
            </symbol>

            <symbol viewBox="0 0 20 20" id="large-arrow">
                <path d="M13.25 10L6.109 2.58c-.268-.27-.268-.707 0-.979.268-.27.701-.27.969 0l7.83 7.908c.268.271.268.709 0 .979l-7.83 7.908c-.268.271-.701.27-.969 0-.268-.269-.268-.707 0-.979L13.25 10z"
                />
            </symbol>

            <symbol viewBox="0 0 20 20" id="large-arrow-down">
                <path d="M17.418 6.109c.272-.268.709-.268.979 0s.271.701 0 .969l-7.908 7.83c-.27.268-.707.268-.979 0l-7.908-7.83c-.27-.268-.27-.701 0-.969.271-.268.709-.268.979 0L10 13.25l7.418-7.141z"
                />
            </symbol>


            <symbol viewBox="0 0 24 24" id="jump-to">
                <path d="M19 7v4H5.83l3.58-3.59L8 6l-6 6 6 6 1.41-1.41L5.83 13H21V7z" />
            </symbol>

            <symbol viewBox="0 0 24 24" id="expand">
                <path d="M10 18h4v-2h-4v2zM3 6v2h18V6H3zm3 7h12v-2H6v2z" />
            </symbol>

        </defs>
    </svg>


    <div id="swagger-ui">
        <div data-reactroot="" class="swagger-ui">
            <div>
                <div class="information-container wrapper">
                    <section class="block col-12">
                        <div class="info">
                            <hgroup class="main">
                                <h2 class="title">httpbin.org
                                    <small>
                                        <pre class="version">0.9.2</pre>
                                    </small>
                                </h2>
                                <pre class="base-url">[ Base URL: httpbin.org/ ]</pre>
                            </hgroup>
                            <div class="description">
                                <div class="markdown">
                                    <p>A simple HTTP Request &amp; Response Service.
                                        <br>
                                        <br>
                                        <b>Run locally: </b>
                                        <code>$ docker run -p 80:80 kennethreitz/httpbin</code>
                                    </p>
                                </div>
                            </div>
                            <div>
                                <div>
                                    <a href="https://kennethreitz.org" target="_blank">the developer - Website</a>
                                </div>
                                <a href="mailto:[email protected]">Send email to the developer</a>
                            </div>
                        </div>
                        <!-- ADDS THE LOADER SPINNER -->
                        <div class="loading-container">
                            <div class="loading"></div>
                        </div>

                    </section>
                </div>
            </div>
        </div>
    </div>


    <div class='swagger-ui'>
        <div class="wrapper">
            <section class="clear">
                <span style="float: right;">
                    [Powered by
                    <a target="_blank" href="https://github.com/rochacbruno/flasgger">Flasgger</a>]
                    <br>
                </span>
            </section>
        </div>
    </div>



    <script src="/flasgger_static/swagger-ui-bundle.js"> </script>
    <script src="/flasgger_static/swagger-ui-standalone-preset.js"> </script>
    <script src='/flasgger_static/lib/jquery.min.js' type='text/javascript'></script>
    <script>

        window.onload = function () {
            

            fetch("/spec.json")
                .then(function (response) {
                    response.json()
                        .then(function (json) {
                            var current_protocol = window.location.protocol.slice(0, -1);
                            if (json.schemes[0] != current_protocol) {
                                // Switches scheme to the current in use
                                var other_protocol = json.schemes[0];
                                json.schemes[0] = current_protocol;
                                json.schemes[1] = other_protocol;

                            }
                            json.host = window.location.host;  // sets the current host

                            const ui = SwaggerUIBundle({
                                spec: json,
                                validatorUrl: null,
                                dom_id: '#swagger-ui',
                                deepLinking: true,
                                jsonEditor: true,
                                docExpansion: "none",
                                apisSorter: "alpha",
                                //operationsSorter: "alpha",
                                presets: [
                                    SwaggerUIBundle.presets.apis,
                                    // yay ES6 modules ↘
                                    Array.isArray(SwaggerUIStandalonePreset) ? SwaggerUIStandalonePreset : SwaggerUIStandalonePreset.default
                                ],
                                plugins: [
                                    SwaggerUIBundle.plugins.DownloadUrl
                                ],
            
            // layout: "StandaloneLayout"  // uncomment to enable the green top header
        })

        window.ui = ui

        // uncomment to rename the top brand if layout is enabled
        // $(".topbar-wrapper .link span").replaceWith("<span>httpbin</span>");
        })
    })
}
    </script>  <div class='swagger-ui'>
    <div class="wrapper">
        <section class="block col-12 block-desktop col-12-desktop">
            <div>

                <h2>Other Utilities</h2>

                <ul>
                    <li>
                        <a href="/forms/post">HTML form</a> that posts to /post /forms/post</li>
                </ul>

                <br />
                <br />
            </div>
        </section>
    </div>
</div>
</body>

</html>
b'<!DOCTYPE html>\n<html lang="en">\n\n<head>\n    <meta charset="UTF-8">\n    <title>httpbin.org</title>\n    <link href="https://fonts.googleapis.com/css?family=Open+Sans:400,700|Source+Code+Pro:300,600|Titillium+Web:400,600,700"\n        rel="stylesheet">\n    <link rel="stylesheet" type="text/css" href="/flasgger_static/swagger-ui.css">\n    <link rel="icon" type="image/png" href="/static/favicon.ico" sizes="64x64 32x32 16x16" />\n    <style>\n        html {\n      box-sizing: border-box;\n            overflow: -moz-scrollbars-vertical;\n            overflow-y: scroll;\n        }\n\n        *,\n        *:before,\n        *:after {\n            box-sizing: inherit;\n        }\n\n        body {\n            margin: 0;\n            background: #fafafa;\n        }\n    </style>\n</head>\n\n<body>\n    <a href="https://github.com/requests/httpbin" class="github-corner" aria-label="View source on Github">\n        <svg width="80" height="80" viewBox="0 0 250 250" style="fill:#151513; color:#fff; position: absolute; top: 0; border: 0; right: 0;"\n            aria-hidden="true">\n            <path d="M0,0 L115,115 L130,115 L142,142 L250,250 L250,0 Z"></path>\n            <path d="M128.3,109.0 C113.8,99.7 119.0,89.6 119.0,89.6 C122.0,82.7 120.5,78.6 120.5,78.6 C119.2,72.0 123.4,76.3 123.4,76.3 C127.3,80.9 125.5,87.3 125.5,87.3 C122.9,97.6 130.6,101.9 134.4,103.2"\n                fill="currentColor" style="transform-origin: 130px 106px;" class="octo-arm"></path>\n            <path d="M115.0,115.0 C114.9,115.1 118.7,116.5 119.8,115.4 L133.7,101.6 C136.9,99.2 139.9,98.4 142.2,98.6 C133.8,88.0 127.5,74.4 143.8,58.0 C148.5,53.4 154.0,51.2 159.7,51.0 C160.3,49.4 163.2,43.6 171.4,40.1 C171.4,40.1 176.1,42.5 178.8,56.2 C183.1,58.6 187.2,61.8 190.9,65.4 C194.5,69.0 197.7,73.2 200.1,77.6 C213.8,80.2 216.3,84.9 216.3,84.9 C212.7,93.1 206.9,96.0 205.4,96.6 C205.1,102.4 203.0,107.8 198.3,112.5 C181.9,128.9 168.3,122.5 157.7,114.1 C157.9,116.9 156.7,120.9 152.7,124.9 L141.0,136.5 C139.8,137.7 141.6,141.9 141.8,141.8 Z"\n                fill="currentColor" class="octo-body"></path>\n        </svg>\n    </a>\n    <svg xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" style="position:absolute;width:0;height:0">\n        <defs>\n            <symbol viewBox="0 0 20 20" id="unlocked">\n                <path d="M15.8 8H14V5.6C14 2.703 12.665 1 10 1 7.334 1 6 2.703 6 5.6V6h2v-.801C8 3.754 8.797 3 10 3c1.203 0 2 .754 2 2.199V8H4c-.553 0-1 .646-1 1.199V17c0 .549.428 1.139.951 1.307l1.197.387C5.672 18.861 6.55 19 7.1 19h5.8c.549 0 1.428-.139 1.951-.307l1.196-.387c.524-.167.953-.757.953-1.306V9.199C17 8.646 16.352 8 15.8 8z"></path>\n            </symbol>\n\n            <symbol viewBox="0 0 20 20" id="locked">\n                <path d="M15.8 8H14V5.6C14 2.703 12.665 1 10 1 7.334 1 6 2.703 6 5.6V8H4c-.553 0-1 .646-1 1.199V17c0 .549.428 1.139.951 1.307l1.197.387C5.672 18.861 6.55 19 7.1 19h5.8c.549 0 1.428-.139 1.951-.307l1.196-.387c.524-.167.953-.757.953-1.306V9.199C17 8.646 16.352 8 15.8 8zM12 8H8V5.199C8 3.754 8.797 3 10 3c1.203 0 2 .754 2 2.199V8z"\n                />\n            </symbol>\n\n            <symbol viewBox="0 0 20 20" id="close">\n                <path d="M14.348 14.849c-.469.469-1.229.469-1.697 0L10 11.819l-2.651 3.029c-.469.469-1.229.469-1.697 0-.469-.469-.469-1.229 0-1.697l2.758-3.15-2.759-3.152c-.469-.469-.469-1.228 0-1.697.469-.469 1.228-.469 1.697 0L10 8.183l2.651-3.031c.469-.469 1.228-.469 1.697 0 .469.469.469 1.229 0 1.697l-2.758 3.152 2.758 3.15c.469.469.469 1.229 0 1.698z"\n                />\n            </symbol>\n\n            <symbol viewBox="0 0 20 20" id="large-arrow">\n                <path d="M13.25 10L6.109 2.58c-.268-.27-.268-.707 0-.979.268-.27.701-.27.969 0l7.83 7.908c.268.271.268.709 0 .979l-7.83 7.908c-.268.271-.701.27-.969 0-.268-.269-.268-.707 0-.979L13.25 10z"\n                />\n            </symbol>\n\n            <symbol viewBox="0 0 20 20" id="large-arrow-down">\n                <path d="M17.418 6.109c.272-.268.709-.268.979 0s.271.701 0 .969l-7.908 7.83c-.27.268-.707.268-.979 0l-7.908-7.83c-.27-.268-.27-.701 0-.969.271-.268.709-.268.979 0L10 13.25l7.418-7.141z"\n                />\n            </symbol>\n\n\n            <symbol viewBox="0 0 24 24" id="jump-to">\n                <path d="M19 7v4H5.83l3.58-3.59L8 6l-6 6 6 6 1.41-1.41L5.83 13H21V7z" />\n            </symbol>\n\n            <symbol viewBox="0 0 24 24" id="expand">\n                <path d="M10 18h4v-2h-4v2zM3 6v2h18V6H3zm3 7h12v-2H6v2z" />\n            </symbol>\n\n    </defs>\n    </svg>\n\n\n    <div id="swagger-ui">\n        <div data-reactroot="" class="swagger-ui">\n            <div>\n                <div class="information-container wrapper">\n                    <section class="block col-12">\n                        <div class="info">\n                            <hgroup class="main">\n                                <h2 class="title">httpbin.org\n                                    <small>\n                                        <pre class="version">0.9.2</pre>\n                                    </small>\n                                </h2>\n                                <pre class="base-url">[ Base URL: httpbin.org/ ]</pre>\n                            </hgroup>\n                            <div class="description">\n                                <div class="markdown">\n                                    <p>A simple HTTP Request &amp; Response Service.\n                                        <br>\n                                        <br>\n                                        <b>Run locally: </b>\n                                        <code>$ docker run -p 80:80 kennethreitz/httpbin</code>\n                                    </p>\n                                </div>\n                            </div>\n                            <div>\n                                <div>\n                                    <a href="https://kennethreitz.org" target="_blank">the developer - Website</a>\n                                </div>\n                                <a href="mailto:[email protected]">Send email to the developer</a>\n                            </div>\n                        </div>\n                        <!-- ADDS THE LOADER SPINNER -->\n                        <div class="loading-container">\n                            <div class="loading"></div>\n                        </div>\n\n                    </section>\n                </div>\n            </div>\n        </div>\n    </div>\n\n\n    <div class=\'swagger-ui\'>\n        <div class="wrapper">\n            <section class="clear">\n                <span style="float: right;">\n                    [Powered by\n                    <a target="_blank" href="https://github.com/rochacbruno/flasgger">Flasgger</a>]\n                    <br>\n                </span>\n            </section>\n        </div>\n    </div>\n\n\n\n    <script src="/flasgger_static/swagger-ui-bundle.js"> </script>\n    <script src="/flasgger_static/swagger-ui-standalone-preset.js"> </script>\n    <script src=\'/flasgger_static/lib/jquery.min.js\' type=\'text/javascript\'></script>\n    <script>\n\n        window.onload = function () {\n            \n\n            fetch("/spec.json")\n                .then(function (response) {\n                    response.json()\n                        .then(function (json) {\n                            var current_protocol = window.location.protocol.slice(0, -1);\n                            if (json.schemes[0] != current_protocol) {\n                                // Switches scheme to the current in use\n                                var other_protocol = json.schemes[0];\n                                json.schemes[0] = current_protocol;\n                                json.schemes[1] = other_protocol;\n\n                            }\n                            json.host = window.location.host;  // sets the current host\n\n                            const ui = SwaggerUIBundle({\n                                spec: json,\n                                validatorUrl: null,\n                                dom_id: \'#swagger-ui\',\n                                deepLinking: true,\n                                jsonEditor: true,\n                                docExpansion: "none",\n                                apisSorter: "alpha",\n                                //operationsSorter: "alpha",\n                                presets: [\n                                    SwaggerUIBundle.presets.apis,\n                                    // yay ES6 modules \xe2\x86\x98\n                                    Array.isArray(SwaggerUIStandalonePreset) ? SwaggerUIStandalonePreset : SwaggerUIStandalonePreset.default\n                                ],\n                                plugins: [\n                                    SwaggerUIBundle.plugins.DownloadUrl\n                                ],\n            \n            // layout: "StandaloneLayout"  // uncomment to enable the green top header\n        })\n\n        window.ui = ui\n\n        // uncomment to rename the top brand if layout is enabled\n        // $(".topbar-wrapper .link span").replaceWith("<span>httpbin</span>");\n        })\n    })\n}\n    </script>  <div class=\'swagger-ui\'>\n    <div class="wrapper">\n        <section class="block col-12 block-desktop col-12-desktop">\n            <div>\n\n                <h2>Other Utilities</h2>\n\n                <ul>\n                    <li>\n                        <a href="/forms/post">HTML form</a> that posts to /post /forms/post</li>\n                </ul>\n\n                <br />\n                <br />\n            </div>\n        </section>\n    </div>\n</div>\n</body>\n\n</html>'
> 

Process 03: Show HTTP header

import requests
 
url = 'https://codegrepper.com/'
response = requests.get(url)
 
print(response.headers)

Output:

{'Server': 'nginx/1.15.5 (Ubuntu)', 'Date': 'Tue, 24 May 2022 14:31:29 GMT', 'Content-Type': 'text/html; charset=UTF-8', 'Transfer-Encoding': 'chunked', 'Connection': 'keep-alive', 'Content-Encoding': 'gzip'}

Process 04: Show redirections

import requests
 
url = 'https://codegrepper.com/redirects/redirect_chain_allowed'
response = requests.get(url)
 
for redirect in response.history:
    print(redirect.url, redirect.status_code)
print(response.url, response.status_code)

Output:

https://codegrepper.com/redirects/redirect_chain_allowed 301
https://www.codegrepper.com/redirects/redirect_chain_allowed 404

Improve the Request

Improve the Request 01: Handle Errors

import requests
 
url = 'wrong url'
 
try:
    r = requests.get(url)
except Exception as e:
    print(f'There was an error: {e}')

Output:

There was an error: Invalid URL 'wrong url': No schema supplied. Perhaps you meant http://wrong url?

Improve the Request 02: Change User-Agent

import requests 
 
url = 'https://www.codegrepper.com/'
 
headers = {
    'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/86.0.4240.198 Safari/537.36'
}
 
response = requests.get(url, headers=headers)
print(response)

Output:

<Response [200]>

Improve the Request 03: Add Timeout to request

import requests
 
url = 'http://httpbin.org/basic-auth/user/pass'
 
try:
    response = requests.get(url, timeout=0.1)
except Exception as e:
    print(e)
    
print(response.status_code)

Output:

HTTPConnectionPool(host='httpbin.org', port=80): Max retries exceeded with url: /basic-auth/user/pass (Caused by ConnectTimeoutError(<urllib3.connection.HTTPConnection object at 0x7fb03a7fa290>, 'Connection to httpbin.org timed out. (connect timeout=0.1)'))
401

Improve the Request 04: Use Proxies

import requests 
 
url = 'https://codegrepper.com/'
 
proxies = {
    'http': '128.199.237.57:8080'
}
 
response = requests.get(url, proxies=proxies)
print(response)

Output:

<Response [200]>

Improve the Request 05: Add Headers to Requests

import requests 
 
url = 'http://httpbin.org/headers'
 
access_token = {
    'Authorization': 'Bearer {access_token}'
    }
 
response = requests.get(url, headers=access_token)
print(response.json())

Output:

{'headers': {'Accept': '*/*', 'Accept-Encoding': 'gzip, deflate', 'Authorization': 'Bearer {access_token}', 'Host': 'httpbin.org', 'User-Agent': 'python-requests/2.22.0', 'X-Amzn-Trace-Id': 'Root=1-628cf340-7b562578714150784312004c'}}

Requests Session

The session object is useful when you need to make requests with parameters that persist through all the requests in a single session.

import requests
 
session = requests.Session()
 
url = 'https://httpbin.org/headers'
 
access_token = {
    'Authorization': 'Bearer {access_token}'
    }
 
session.headers.update(access_token)
 
response1 = session.get(url)
response2 = session.get(url)
 
print('response1: ', response1.json()['headers']['Authorization'])
print('response2: ', response2.json()['headers']['Authorization'])

Output:

response1:  Bearer {access_token}
response2:  Bearer {access_token}

Conclusion

That’s all for this article, if you have any confusion contact us through our website or email us at [email protected] or by using LinkedIn

Suggested Articles:

  1. Python Try Except

Leave a Comment