Download a web page (HTTP GET with requests)

[tinmarino]

HTTP GET

Give the capability to the AI to download the content of web pages into the current prompt. This can consume many tokens if used recursively like a wget.

import requests  # HTTP GET
from instructor import OpenAISchema
from pydantic import Field

class Function(OpenAISchema):
    """ Download a web page using HTTP GET from the Python requests library and returns the content as a string. """
    url: str = Field(
        ...,
        example="https://example.com",
        description="URL of the web page to download.",
    )

    class Config:
        title = "download_web_page"

    @classmethod
    def execute(cls, url: str) -> dict:
        response = requests.get(url)
        if response.status_code == 200:
            return response.text
        else:
            return {"error": f"Failed to download the web page. Status code: {response.status_code}"}

Example

sgpt "What is this page: https://example.com"
@FunctionCall download_web_page(url="https://example.com")
The page at https://example.com is a simple HTML document titled "Example Domain." It serves as a placeholder for illustrative examples in documents. The content states that this domain can be used in literature without prior permission. It includes a link for more information about the domain, directing users to the IANA website. The design features a clean layout with a light background and a centered content box.