feat: Improved project bootstrapping #538

janbuchar · 2024-09-20T09:12:07Z

This adds a unified crawler template. The original playwright and beautifulsoup templates are kept for compatibility with older versions of the CLI.

The user is now prompted for package manager type (pip, poetry), crawler type, start URL and whether or not Apify integration should be set up.

closes Automatically install dependencies after bootstrapping a project via CLI #317
closes Feature Request: Add More Options to Crawlee CLI for Crawler Creation #414 (http client selection is not implemented)
closes Optionally include Apify integration in project bootstrapping #511

github-actions

⚠️ Pull Request Tookit has failed!

Pull request is neither linked to an issue or epic nor labeled as adhoc!

github-actions

⚠️ Pull Request Tookit has failed!

Pull request is neither linked to an issue or epic nor labeled as adhoc!

janbuchar · 2024-09-20T12:32:14Z

okay, this could use some testing now... plus there's a bunch of stuff to consider

the pip thing is probably not very ergonomic and I'm not sure how to approach it well... should we create a .venv for users? should we have a requirements.txt and use it as a lockfile?
does this work on windows?
isn't the creation dialog too long now? maybe some of the options don't need the prompt and just CLI flags are enough
should we include http client selection?
we should probably scream and shout if poetry is not installed... or we could just disable the option in that case

vdusek

Nice, just a few comments

vdusek · 2024-09-23T09:13:34Z

templates/crawler/cookiecutter.json

@@ -0,0 +1,12 @@
+{
+    "project_name": "crawlee-python-beautifulsoup-project",


Is it correct?

Good point. I'll probably just change it to my-crawler or something.

vdusek · 2024-09-23T09:28:55Z

templates/crawler/{{cookiecutter.project_name}}/README.md

@@ -0,0 +1,37 @@
+# {{cookiecutter.project_name}}
+
+Project skeleton generated by Crawlee (Beautifulsoup template).


is it always BS?

src/crawlee/_cli.py

janbuchar · 2024-09-25T07:16:12Z

Thanks for the review @vdusek! Do you have anything to add regarding those points for consideration?

vdusek · 2024-09-25T08:13:04Z

the pip thing is probably not very ergonomic and I'm not sure how to approach it well... should we create a .venv for users? should we have a requirements.txt and use it as a lockfile?

That is generally how pip works. I wouldn't add additional overhead here. Simply dumping the dependencies into requirements.txt should be enough. Managing the environment and installation should be left to the user - that's the standard approach with pip.

isn't the creation dialog too long now? maybe some of the options don't need the prompt and just CLI flags are enough

IMO the current setup is fine. Also, all options have default values.

should we include http client selection?

That would be a nice feature, but I wouldn't prioritize it at this moment. We have a guide covering that, which should be sufficient for now.

we should probably scream and shout if poetry is not installed... or we could just disable the option in that case

I would suggest throwing an error. If a user wants to use Poetry, the option should be conditionally on Poetry being installed.

vdusek · 2024-09-25T08:19:51Z

templates/crawler/{{cookiecutter.project_name}}/Dockerfile

I know this is copied, but I suggest using Hadolint and improving it according to their suggestions.

We should also pin (or at least limit to < 2) the Poetry (mostly because of python-poetry/poetry#3332).

vdusek · 2024-09-25T08:20:21Z

templates/crawler/{{cookiecutter.project_name}}/README.md

+
+## Usage
+
+{% if cookiecutter.package_manager == 'poetry' %}


this leads to 2 empty lines

vdusek · 2024-09-25T08:20:40Z

templates/crawler/{{cookiecutter.project_name}}/README.md

+## Usage
+
+{% if cookiecutter.package_manager == 'poetry' %}
+To get started, ensure you have [Poetry](https://python-poetry.org/), a package and dependency management system, installed on your machine. We recommend installing it with the following command:


could we wrap it to 120 line width?

vdusek

the Pip versions does not work

janbuchar · 2024-09-25T08:30:56Z

the pip thing is probably not very ergonomic and I'm not sure how to approach it well... should we create a .venv for users? should we have a requirements.txt and use it as a lockfile?

That is generally how pip works. I wouldn't add additional overhead here. Simply dumping the dependencies into requirements.txt should be enough. Managing the environment and installation should be left to the user - that's the standard approach with pip.

I generally agree. But how would you ensure that pipx run crawlee create ... installs the dependencies in a venv? Often people will want to keep the virtualenv in the project directory, which doesn't get created until you run crawlee create.

should we include http client selection?

That would be a nice feature, but I wouldn't prioritize it at this moment. We have a guide covering that, which should be sufficient for now.

It's a 15-minute adventure, the only problem is that it'd make the dialog longer.

Add a unified 'crawler' template

1c54b81

janbuchar added the t-tooling Issues with this label are in the ownership of the tooling team. label Sep 20, 2024

janbuchar requested review from vdusek and B4nan September 20, 2024 09:12

github-actions bot assigned janbuchar Sep 20, 2024

github-actions bot added this to the 98th sprint - Tooling team milestone Sep 20, 2024

github-actions bot reviewed Sep 20, 2024

View reviewed changes

janbuchar added 2 commits September 20, 2024 13:57

Install dependencies in hook

f15275b

Update tests

ef878b5

github-actions bot added the tested Temporary label used only programatically for some analytics. label Sep 20, 2024

github-actions bot reviewed Sep 20, 2024

View reviewed changes

janbuchar marked this pull request as ready for review September 20, 2024 12:33

vdusek requested changes Sep 23, 2024

View reviewed changes

vdusek requested changes Sep 25, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: Improved project bootstrapping #538

feat: Improved project bootstrapping #538

janbuchar commented Sep 20, 2024 •

edited

Loading

github-actions bot left a comment

github-actions bot left a comment

janbuchar commented Sep 20, 2024 •

edited

Loading

vdusek left a comment

vdusek Sep 23, 2024

janbuchar Sep 23, 2024

vdusek Sep 23, 2024

janbuchar commented Sep 25, 2024

vdusek commented Sep 25, 2024

vdusek Sep 25, 2024

vdusek Sep 25, 2024

vdusek Sep 25, 2024

vdusek left a comment

janbuchar commented Sep 25, 2024

		@@ -0,0 +1,12 @@
		{
		"project_name": "crawlee-python-beautifulsoup-project",

		@@ -0,0 +1,37 @@
		# {{cookiecutter.project_name}}

		Project skeleton generated by Crawlee (Beautifulsoup template).

feat: Improved project bootstrapping #538

Are you sure you want to change the base?

feat: Improved project bootstrapping #538

Conversation

janbuchar commented Sep 20, 2024 • edited Loading

github-actions bot left a comment

Choose a reason for hiding this comment

github-actions bot left a comment

Choose a reason for hiding this comment

janbuchar commented Sep 20, 2024 • edited Loading

vdusek left a comment

Choose a reason for hiding this comment

vdusek Sep 23, 2024

Choose a reason for hiding this comment

janbuchar Sep 23, 2024

Choose a reason for hiding this comment

vdusek Sep 23, 2024

Choose a reason for hiding this comment

janbuchar commented Sep 25, 2024

vdusek commented Sep 25, 2024

vdusek Sep 25, 2024

Choose a reason for hiding this comment

vdusek Sep 25, 2024

Choose a reason for hiding this comment

vdusek Sep 25, 2024

Choose a reason for hiding this comment

vdusek left a comment

Choose a reason for hiding this comment

janbuchar commented Sep 25, 2024

janbuchar commented Sep 20, 2024 •

edited

Loading

janbuchar commented Sep 20, 2024 •

edited

Loading