Feature. Add chapter 'Lists of dictionaries' to docsite

[vbotka]

SUMMARY

Add filter_guide -> abstract_informations -> lists_of_dictionaries:

• keep_keys
• remove_keys
• replace_keys

ISSUE TYPE

• Docs Pull Request

COMPONENT NAME

filter_guide-abstract_informations-lists_of_dictionaries.rst
filter_guide-abstract_informations-lists_of_dictionaries-remove_keys.rst
filter_guide-abstract_informations-lists_of_dictionaries-replace_keys.rst
filter_guide-abstract_informations-lists_of_dictionaries-keep_keys.rst

ADDITIONAL INFORMATION

Created helpers:

• docs/docsite/helper/keep_keys
• docs/docsite/helper/remove_keys
• docs/docsite/helper/replace_keys

Create RST files by running the below command in the helper/*_keys directory

shell> ansible-playbook playbook.yml

[vbotka]

For the record: There seems to be a problem with needs_rebase

shell> git pull --rebase upstream main
From https://github.com/ansible-collections/community.general
 * branch                main       -> FETCH_HEAD
The current branch feature-docsite-keep_keys is up to date.
shell> git push --force-with-lease
Everything up-to-date

[felixfontein]

I think needs_rebase is added because of More than 50 changed files.; usually that number of changed files implies that someone screwed up a rebase and added some random unrelated commits. In this case it's safe to ignore :-)

[vbotka]

... it's safe to ignore :-)

Thank you for the comment!

[felixfontein]

@vbotka this now has a conflict.

[vbotka]

... now has a conflict.

This file shouldn't be included in the PR. I rebased the PR and resolved the conflict. Sorry for the noise. I tested all helpers and docsite. This PR should be ready. Thank you!

[felixfontein]

What's the purpose of this (and similar) files? I'm not sure what ansible-test.cfg files are doing.

[vbotka]

The idea is to use integration tests in the examples:

1. Run the integration tests using ansible-test.cfg
2. Create the examples
3. Run playbook.yml and create RST using ansible.cfg

This scheme is WIP. I'll create README when completed.

1. You can run integration tests in the development copy using ansible-test.cfg. For example, put the development copy into the directory

/scratch/admin/ansible.community.general

The parameter roles_path points to the roles

roles_path = /scratch/admin/ansible.community.general/tests/integration/targets

and the parameter collections_path points to this collection

collections_path = /scratch/collections

shell>ll /scratch/collections/ansible_collections/community/
general -> /scratch/admin/ansible.community.general/

This is an example. Fit the paths to your needs. Then, the below play

shell> cat pb-test.yml 
- name: ansible.community.general/tests/integration/targets/filter_keep_keys
  hosts: localhost
  roles:
    - filter_keep_keys

will run the integration tests

(env) > ANSIBLE_CONFIG=$PWD/ansible-test.cfg ansible-playbook pb-test.yml 

PLAY [ansible.community.general/tests/integration/targets/filter_keep_keys] *****************************************

TASK [filter_keep_keys : Debug ansible_version] *********************************************************************
skipping: [localhost]

TASK [filter_keep_keys : Test keep keys equal (default)] ************************************************************
ok: [localhost]

TASK [filter_keep_keys : Test keep keys regex string] ***************************************************************
ok: [localhost]

TASK [filter_keep_keys : Test keep keys targets1] *******************************************************************
ok: [localhost] => (item=equal: ['k0_x0', 'k1_x1'])
ok: [localhost] => (item=starts_with: ['k0', 'k1'])
ok: [localhost] => (item=ends_with: ['x0', 'x1'])
ok: [localhost] => (item=regex: ['^.*[01]_x.*$'])
ok: [localhost] => (item=regex: ^.*[01]_x.*$)

TASK [filter_keep_keys : Test keep keys targets2] *******************************************************************
ok: [localhost] => (item=equal: k0_x0)
ok: [localhost] => (item=starts_with: k0)
ok: [localhost] => (item=ends_with: x0)
ok: [localhost] => (item=regex: ^.*0_x.*$)

PLAY RECAP **********************************************************************************************************
localhost                  : ok=4    changed=0    unreachable=0    failed=0    skipped=1    rescued=0    ignored=0   

2. Create examples when the integration tests are OK. See:

• examples.yml ........... list of examples
• *.j2 ............................. templates
• playbook.yml ............ playbook

3. Create RST

(env) > ansible-playbook -e @extra-vars.yml playbook.yml

HTH. I propose to consolidate all helpers later (keep_keys, lists_mergeby, remove_keys, and replace_keys). I believe that this can be further simplified when the picture is complete.

[felixfontein]

I don't understand why you don't simply run the integration tests with ansible-test integration filter_keep_keys. That's a lot simpler than having to do all these steps and modify the contents of a file checked into this repository (to adjust the paths).

Also assuming you checked out the collection in the right directory structure that's added to ansible-core's COLLECTIONS_PATH (as described on https://docs.ansible.com/ansible/devel/community/collection_contributors/collection_test_pr_locally.html), you can simply run ansible-playbook playbook.yml in the directory without the special ansible.cfg file.

[vbotka]

Thank you for the clarification! I'll consolidate this mess.

JFYI, links to the clones do not work. In the below example, ansible-test will fail if general is a link

shell> pwd
/scratch/collections/ansible_collections/community/general
shell> ansible-test env
FATAL: The current working directory must be within the source tree being tested.

All is right if I copy general to the path

shell> pwd
/scratch/collections/ansible_collections/community/general
shell> ansible-test env
ansible:
  version: 2.16.3
  ...

[felixfontein]

That's why I generally clone the repo inside the ansible_collections path structure, instead of linking to the 'real' clone from there.

[vbotka]

I see. Thank you!

[vbotka]

Would it be possible to review the PR? The docs are generated from the integration tests.

[felixfontein]

I'm not convinced that coupling the integration tests (which are tests run in CI) are coupled to docs generation (these are NOT run in CI!). Anytime now someone modifies the tests they break the docs build process, without anyone noticing.

IMO docs creation should be completely separate from integration tests.

[vbotka]

I'm not convinced that coupling the integration tests (which are tests run in CI) are coupled to docs generation (these are NOT run in CI!).

This is not a coupling. The tests are not coupled to the docs. The docs take advantage of a friendly test structure if possible. There are only advantages:

• Single source for tests and examples.
• Already tested examples.
• Automatic generation of the documentation = lower rate of typing errors.

The variable tests in tests/integration/targets/filter_keep_keys/vars/main/tests.yml is a list. You can select items in the template, e.g.

{% for i in tests[0:1]|subelements('group') %} 
...

Anytime now someone modifies the tests they break the docs build process, without anyone noticing.

This is not the case. The helper playbooks create the rst file locally in the helper directory. You have to copy it manually to the rst directory. I have no problem if anybody updates the docs files directly in the rst directory. The helper can be synchronized anytime if needed. Use it if you want to. It's not mandatory, of course.

IMO docs creation should be completely separate from integration tests.

OK. Nothing in this PR makes the integration tests not completely separated. Still, you can use the list tests in the docs if you want to. I would go even further and propose creating also the plugin's examples similarly for the same reasons.

Ultimately, you can anytime copy the tests list to the helper if necessary.

[vbotka]

I updated the helpers. The feasibility is up to you.

[felixfontein]

For files that do not support comments, you can add a .license file. See tests/sanity/ignore-*.txt.license for examples.

[vbotka]

For files that do not support comments, you can add a .license file. See tests/sanity/ignore-*.txt.license for examples.

Thank you! I used this and simplified templates.

[patchback]

Backport to stable-9: 💚 backport PR created

✅ Backport PR branch: patchback/backports/stable-9/caecb2297fbf8cfb9ae80926ead10b4fb11a003f/pr-8482

Backported as #8582

🤖 @patchback
I'm built with octomachinery and
my source is open — https://github.com/sanitizers/patchback-github-app.

[felixfontein]

@vbotka thanks for your contribution!

[vbotka]

@vbotka thanks for your contribution!

@felixfontein thank you!