forked from okfn-brasil/querido-diario
-
Notifications
You must be signed in to change notification settings - Fork 0
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Merge branch 'okfn-brasil:main' into connect_qd
- Loading branch information
Showing
198 changed files
with
2,746 additions
and
860 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file was deleted.
Oops, something went wrong.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,40 @@ | ||
| name: Update spider status on production | ||
|
|
||
| on: | ||
| workflow_dispatch: | ||
| inputs: | ||
| spider_name: | ||
| description: 'Spider name' | ||
| required: true | ||
| status: | ||
| type: choice | ||
| description: 'New Spider status in production' | ||
| options: | ||
| - enabled | ||
| - disabled | ||
| required: true | ||
|
|
||
| jobs: | ||
| update_status: | ||
| runs-on: ubuntu-latest | ||
| env: | ||
| QUERIDODIARIO_DATABASE_URL: ${{ secrets.QUERIDODIARIO_DATABASE_URL }} | ||
| steps: | ||
| - uses: actions/checkout@v2 | ||
| - uses: actions/setup-python@v2 | ||
| with: | ||
| python-version: '3.10' | ||
| - name: Prepare environment | ||
| run: | | ||
| python -m pip install --upgrade pip | ||
| pip install click python-decouple scrapinghub SQLAlchemy psycopg2 | ||
| - name: Enable spider in production | ||
| if: ${{ github.event.inputs.status == 'enabled' }} | ||
| run: | | ||
| cd data_collection/ | ||
| python scheduler.py enable-spider --spider_name=${{ github.event.inputs.spider_name }} | ||
| - name: Disable spider in production | ||
| if: ${{ github.event.inputs.status == 'disabled' }} | ||
| run: | | ||
| cd data_collection/ | ||
| python scheduler.py disable-spider --spider_name=${{ github.event.inputs.spider_name }} |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,6 @@ | ||
| AWS_ACCESS_KEY_ID=minio-access-key | ||
| AWS_SECRET_ACCESS_KEY=minio-secret-key | ||
| AWS_ENDPOINT_URL=http://localhost:9000 | ||
| AWS_REGION_NAME=nyc3 | ||
| FILES_STORE=s3://queridodiariobucket/ | ||
| QUERIDODIARIO_DATABASE_URL=postgresql://queridodiario:queridodiario@localhost:5432/queridodiariodb |
Empty file.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,53 @@ | ||
| import datetime | ||
|
|
||
| from scrapy.commands import ScrapyCommand | ||
| from scrapy.exceptions import UsageError | ||
|
|
||
| from gazette.utils import get_enabled_spiders | ||
|
|
||
|
|
||
| class Command(ScrapyCommand): | ||
| requires_project = True | ||
|
|
||
| def add_options(self, parser): | ||
| ScrapyCommand.add_options(self, parser) | ||
| parser.add_argument( | ||
| "--start_date", | ||
| dest="start_date", | ||
| default=None, | ||
| metavar="VALUE", | ||
| help="List spiders enabled from date (format: YYYY-MM-DD)", | ||
| ) | ||
| parser.add_argument( | ||
| "--end_date", | ||
| dest="end_date", | ||
| default=None, | ||
| metavar="VALUE", | ||
| help="List spiders enabled until date (format: YYYY-MM-DD)", | ||
| ) | ||
|
|
||
| def short_desc(self): | ||
| return "List production enabled spiders" | ||
|
|
||
| def run(self, args, opts): | ||
| start_date, end_date = None, None | ||
|
|
||
| if opts.start_date is not None: | ||
| try: | ||
| start_date = datetime.datetime.strptime(opts.start_date, "%Y-%m-%d") | ||
| except ValueError: | ||
| raise UsageError("'start_date' must match YYYY-MM-DD format") | ||
|
|
||
| if opts.end_date is not None: | ||
| try: | ||
| end_date = datetime.datetime.strptime(opts.end_date, "%Y-%m-%d") | ||
| except ValueError: | ||
| raise UsageError("'end_date' must match YYYY-MM-DD format") | ||
|
|
||
| print("\nEnabled spiders\n===============") | ||
| for spider_name in get_enabled_spiders( | ||
| database_url=self.settings["QUERIDODIARIO_DATABASE_URL"], | ||
| start_date=start_date, | ||
| end_date=end_date, | ||
| ): | ||
| print(spider_name) |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.