[Demo Enhancement] added storm wiki frontend with themes #135

jaigouk · 2024-08-10T09:36:15Z

why

wish to choose dark / light themes
sometimes, search engine may not return results. and there are search engines that are specialised for specific purpose(arxiv. maybe github in the future). I want to be able to choose them before I send request.
ollama also can fail. So I want to have an option to "fallback" and continue.

changes

themes: dracula soft dark color and other light and dark themes
engines: duckduckgo, searxng and arxiv
llm: ollama, anthropic
users can change search engine before triggering search
users can save primary and fallback llm in settings
save result files as '*.md'
add date to to top of the result file
added arize-phoenix to trace.
added github ci file to test fallback options for search and llm
change number of display columns
pagination in sidebar

note

because this frontend is using "fallbacks", I thought it would be better to have separate example instead of overwriting the existing demo_light.

screenshots

dark and light theme

verified llm fallback. ollama failed and gpt4o-mini kicked in and continued to generate the result.

jaigouk · 2024-08-11T20:50:48Z

I wish to have more features like

articles are sorted by created time by default
sorting and searching articles
deleting articles
updating the existing doc
API to generate articles

From my side, updating the existing doc via API would be a killer feature. but I thought it is better to clean things up first and get code review before I dive into those.

shaoyijia · 2024-08-12T07:00:18Z

Thank you so much for working on this! The screenshots look really good (I think it could be useful for many). I am a bit busy these days 😔 and plan to go through this PR next weekend.

From my side, updating the existing doc via API would be a killer feature. but I thought it is better to clean things up first and get code review first before I dive into those.

I agree.

shaoyijia

Thank you for contributing to this! It's really cool.

I run into issue with the "My Article" page and add some suggestions on making it better.

frontend/storm_wiki/util/storm_runner.py

shaoyijia · 2024-08-17T06:05:53Z

frontend/storm_wiki/README.md

We hope to merge frontend/storm_wiki to our main branch since it's really good. Could you revise the Readme.md? You can add the contributor info / other information (e.g., your GitHub account, terms of use, etc.) to the README.md if you would like to.

frontend/storm_wiki/util/consts.py

shaoyijia · 2024-08-17T06:22:38Z

frontend/storm_wiki/util/storm_runner.py

+                model_settings=model_settings,
+            )
+
+        for lm_type in [


I think we may want to allow people to configure LMs for these components separately. Currently, we can only use the same LM for all these components based on the interface design. One design philosophy of STORM is that it's a LM-empowered system so different modules can be run with different LMs to jointly optimize the cost and quality. I think viewing the system in this way will be better than viewing it as a chain of GPT-4/a specific LM prompts.

No need to adjust it this in this PR though.

shaoyijia · 2024-08-17T06:26:55Z

frontend/storm_wiki/util/search.py

+        logger.info(f"Raw results from {engine}: {results}")
+        return results
+
+    def _search_duckduckgo(self, query: str) -> List[Dict[str, Any]]:


Could we move this part to knowledge_storm/rm.py? Simply setting up a wrapper class for DuckDuckGoSearchAPIWrapper would be good. I think people may want to use it but not sure everyone will be able to find it in this frontend folder.

shaoyijia · 2024-08-17T06:29:18Z

frontend/storm_wiki/util/search.py

+        self.search_engines = self._initialize_search_engines()
+        self._initialize_domain_restrictions()
+
+    def _initialize_search_engines(self):


yourdm is not registered here. This is a bit confusing as I run into the issue in first run because I add YDC_API_KEY and select the search engine as yourdm.

sure. same with bing search.

frontend/storm_wiki/README.md

shaoyijia · 2024-08-17T06:47:24Z

frontend/storm_wiki/pages_util/MyArticles.py

+    return all_articles
+
+
+def display_article_list(page_size, num_columns):


In my test, I can create new article successfully. But "My Articles" does not display properly, it just shows "Showing all 0 articles". I am not sure whether I mess up configuration somewhere.

Also, by reading the code, I have a question: Since st.session_state will be reset every time you restart the streamlit app, how are the articles maintained? Where can I find them?

it is handled with https://github.com/jaigouk/storm_wiki/blob/main/pages_util/MyArticles.py#L55
FileIOHelper.get_output_dir(category) used and ` we load articles

I will resolve issues in the weekend. thanks for the feedbacks.

jaigouk · 2024-09-05T10:18:33Z

@Yucheng-Jiang @shaoyijia it has been a while. i will work on this PR in this weekend. I was thinking about using command pattern to tidy things up. but I think it is better to refactor it with another PR in the future.

shaoyijia · 2024-09-06T16:18:42Z

Sounds great! Thank you :)

jaigouk · 2024-09-09T16:30:14Z

it seems that i need to organise files better before I use rm and lm from the storm library. I converted this PR to draft.

kinoc · 2024-09-24T09:47:23Z

Is the PR still active or planned?

jaigouk · 2024-10-22T07:44:12Z

i guess this is not going to be able to easy to maintain with the codes i added.

jaigouk marked this pull request as ready for review August 10, 2024 15:22

shaoyijia changed the title ~~added storm wiki frontend with themes~~ [Demo Enhancement] added storm wiki frontend with themes Aug 10, 2024

shaoyijia requested changes Aug 17, 2024

View reviewed changes

jaigouk force-pushed the add-themed-frontend branch from 6e29d84 to 4fadac4 Compare August 24, 2024 13:20

Yucheng-Jiang mentioned this pull request Aug 28, 2024

More info on human-ai collaboration mode #151

Closed

jaigouk mentioned this pull request Sep 7, 2024

Error uploading document to Chroma db jaigouk/storm_wiki#20

Open

jaigouk added 5 commits September 7, 2024 12:01

added themed frontend

50c550c

added tests to enhance fallback options

24f886d

use streamlit secrets.toml instead of dotenv

0090724

avoid circular import in db_operations.py

8fc172b

chore: Update theme manager to use default theme constant

c6e217c

jaigouk force-pushed the add-themed-frontend branch from 4fadac4 to c6e217c Compare September 7, 2024 10:24

chore: Update search engine names to match case-sensitive constants

1face9a

jaigouk marked this pull request as draft September 9, 2024 16:29

jaigouk closed this Oct 22, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Demo Enhancement] added storm wiki frontend with themes #135

[Demo Enhancement] added storm wiki frontend with themes #135

jaigouk commented Aug 10, 2024 •

edited

Loading

jaigouk commented Aug 11, 2024 •

edited

Loading

shaoyijia commented Aug 12, 2024 •

edited

Loading

shaoyijia left a comment

shaoyijia Aug 17, 2024

shaoyijia Aug 17, 2024

shaoyijia Aug 17, 2024

shaoyijia Aug 17, 2024

jaigouk Aug 19, 2024

shaoyijia Aug 17, 2024

jaigouk Aug 19, 2024

jaigouk commented Sep 5, 2024 •

edited

Loading

shaoyijia commented Sep 6, 2024

jaigouk commented Sep 9, 2024

kinoc commented Sep 24, 2024

jaigouk commented Oct 22, 2024

		return all_articles


		def display_article_list(page_size, num_columns):

[Demo Enhancement] added storm wiki frontend with themes #135

[Demo Enhancement] added storm wiki frontend with themes #135

Conversation

jaigouk commented Aug 10, 2024 • edited Loading

why

changes

note

screenshots

jaigouk commented Aug 11, 2024 • edited Loading

shaoyijia commented Aug 12, 2024 • edited Loading

shaoyijia left a comment

Choose a reason for hiding this comment

shaoyijia Aug 17, 2024

Choose a reason for hiding this comment

shaoyijia Aug 17, 2024

Choose a reason for hiding this comment

shaoyijia Aug 17, 2024

Choose a reason for hiding this comment

shaoyijia Aug 17, 2024

Choose a reason for hiding this comment

jaigouk Aug 19, 2024

Choose a reason for hiding this comment

shaoyijia Aug 17, 2024

Choose a reason for hiding this comment

jaigouk Aug 19, 2024

Choose a reason for hiding this comment

jaigouk commented Sep 5, 2024 • edited Loading

shaoyijia commented Sep 6, 2024

jaigouk commented Sep 9, 2024

kinoc commented Sep 24, 2024

jaigouk commented Oct 22, 2024

jaigouk commented Aug 10, 2024 •

edited

Loading

jaigouk commented Aug 11, 2024 •

edited

Loading

shaoyijia commented Aug 12, 2024 •

edited

Loading

jaigouk commented Sep 5, 2024 •

edited

Loading