Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Improvement in web commands functionalities #7068

Merged
merged 10 commits into from
Jul 25, 2024
Merged

Improvement in web commands functionalities #7068

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
10 commits
Select commit Hold shift + click to select a range
78c9d8e
use of different backend for ddg
amirdaaee Apr 3, 2024
7dbe78f
selenium proxy support
amirdaaee Apr 3, 2024
3fcee82
added example in .env.template
amirdaaee Apr 3, 2024
af8862d
Merge branch 'master' into web_functionalities-update
kcze Jul 3, 2024
62bee9a
Update
kcze Jul 3, 2024
2e92436
Merge pull request #1 from Significant-Gravitas/web_functionalities-u…
amirdaaee Jul 6, 2024
115106e
Merge branch 'master' into improvement_in_web_functionalities
ntindle Jul 19, 2024
24c8c2e
Merge branch 'master' into improvement_in_web_functionalities
ntindle Jul 22, 2024
9052105
Merge branch 'master' into improvement_in_web_functionalities
ntindle Jul 22, 2024
f393613
Remove `DUCKDUCKGO_BACKEND` from `.env.template`
kcze Jul 25, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions autogpt/.env.template
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -105,6 +105,7 @@
## HUGGINGFACE_API_TOKEN - HuggingFace API token (Default: None)
# HUGGINGFACE_API_TOKEN=


### Stable Diffusion (IMAGE_PROVIDER=sdwebui)

## SD_WEBUI_AUTH - Stable Diffusion Web UI username:password pair (Default: None)
Expand Down
12 changes: 7 additions & 5 deletions docs/content/forge/components/built-in-components.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -155,11 +155,12 @@ Allows agent to search the web. Google credentials aren't required for DuckDuckG

### `WebSearchConfiguration`

| Config variable | Details | Type | Default |
| -------------------------------- | ----------------------------------------------------------------------- | ----- | ------- |
| `google_api_key` | Google API key, *ENV:* `GOOGLE_API_KEY` | `str` | `None` |
| `google_custom_search_engine_id` | Google Custom Search Engine ID, *ENV:* `GOOGLE_CUSTOM_SEARCH_ENGINE_ID` | `str` | `None` |
| `duckduckgo_max_attempts` | Maximum number of attempts to search using DuckDuckGo | `int` | `3` |
| Config variable | Details | Type | Default |
| -------------------------------- | ----------------------------------------------------------------------- | --------------------------- | ------- |
| `google_api_key` | Google API key, *ENV:* `GOOGLE_API_KEY` | `str` | `None` |
| `google_custom_search_engine_id` | Google Custom Search Engine ID, *ENV:* `GOOGLE_CUSTOM_SEARCH_ENGINE_ID` | `str` | `None` |
| `duckduckgo_max_attempts` | Maximum number of attempts to search using DuckDuckGo | `int` | `3` |
| `duckduckgo_backend` | Backend to be used for DDG sdk | `"api" \| "html" \| "lite"` | `"api"` |

### DirectiveProvider

Expand All @@ -183,6 +184,7 @@ Allows agent to read websites using Selenium.
| `headless` | Run browser in headless mode | `bool` | `True` |
| `user_agent` | User agent used by the browser | `str` | `"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.97 Safari/537.36"` |
| `browse_spacy_language_model` | Spacy language model used for chunking text | `str` | `"en_core_web_sm"` |
| `selenium_proxy` | Http proxy to use with Selenium | `str` | `None` |

### DirectiveProvider

Expand Down
7 changes: 5 additions & 2 deletions forge/forge/components/web/search.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
import json
import logging
import time
from typing import Iterator, Optional
from typing import Iterator, Literal, Optional

from duckduckgo_search import DDGS
from pydantic import BaseModel, SecretStr
Expand All @@ -24,6 +24,7 @@ class WebSearchConfiguration(BaseModel):
None, from_env="GOOGLE_CUSTOM_SEARCH_ENGINE_ID", exclude=True
)
duckduckgo_max_attempts: int = 3
duckduckgo_backend: Literal["api", "html", "lite"] = "api"


class WebSearchComponent(
Expand Down Expand Up @@ -89,7 +90,9 @@ def web_search(self, query: str, num_results: int = 8) -> str:
if not query:
return json.dumps(search_results)

search_results = DDGS().text(query, max_results=num_results)
search_results = DDGS().text(
query, max_results=num_results, backend=self.config.duckduckgo_backend
)

if search_results:
break
Expand Down
5 changes: 5 additions & 0 deletions forge/forge/components/web/selenium.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -68,6 +68,8 @@
"""User agent used by the browser"""
browse_spacy_language_model: str = "en_core_web_sm"
"""Spacy language model used for chunking text"""
selenium_proxy: Optional[str] = None
"""Http proxy to use with Selenium"""


class WebSeleniumComponent(
Expand Down Expand Up @@ -301,6 +303,9 @@
options.add_argument("--headless=new")
options.add_argument("--disable-gpu")

if self.config.selenium_proxy:
options.add_argument(f"--proxy-server={self.config.selenium_proxy}")

Check warning on line 307 in forge/forge/components/web/selenium.py

View check run for this annotation

Codecov / codecov/patch

forge/forge/components/web/selenium.py#L307 

Added line #L307 was not covered by tests

self._sideload_chrome_extensions(options, self.data_dir / "assets" / "crx")

if (chromium_driver_path := Path("/usr/bin/chromedriver")).exists():
Expand Down
Loading