Update method in which external content is ingested.

[mrbobbytables]

Currently, external content is brought into the contributor-site via the hack/gen-content.sh bash script. In its original inception, it was quite small and lean -- but as a bash script usually does, it has grown to be unwieldy.

It'd be far better to replace it with a more supportable (and testable) means of bringing in external content. This means rewriting it something like go or python, or finding some other means of bringing in external content.

/kind feature
cc @vishakhanihore @nimbinatus

[markjacksonfishing]

this is a long term replacement and is not needed for the contributor site.

[markjacksonfishing]

/assign @markyjackson-taulia

[nimbinatus]

Hey @markyjackson-taulia! I had started poking at this issue a couple weeks ago to see what we needed. I'm pretty sure a simple include will work with a bit of tweaking. Did you want to see what I had in draft form?

[markjacksonfishing]

@nimbinatus sorry for the delay in responding. I would be interested to see what you have. I am thinking of converting this in its entirety to go so I can include some testing around it.

[nimbinatus]

@markyjackson-taulia sounds good! I'll try to clean up my code with some comments and get my branch up this week for you to poke at.

[nimbinatus]

@markyjackson-taulia here's what I was poking at. It does require a bit more work to add pages to the repo, and it's not 100% accurate yet regarding some of the internal links on each page. However, it's more native to Hugo, so I was hoping it would be a bit cleaner to test.

https://github.com/nimbinatus/contributor-site/blob/shortcode-import/layouts/shortcodes/content.gohtml

[markjacksonfishing]

Will run some test

[mrbobbytables]

The hardest part that I'm concerned about with the hugo method is the replication of the link expander. It determines if the content is has been synced locally and updates it with the correct path, or if not -- expand it to the full github url.

This blurb is slightly out of date (updated in #110), but it is similar enough to explain.
https://github.com/kubernetes-sigs/contributor-site/blob/fe36c430d92ed0a81cb85e154fa092f55912a3be/hack/gen-content.sh#L68-L89

[markjacksonfishing]

I found that through my testing, using the hugo method did not do fully what I wanted so I am going to redo this. Thank you @nimbinatus for the code

[markjacksonfishing]

/milestone v1.19

[markjacksonfishing]

@nimbinatus I started the work in the above PR, did it in python but there is still more to do.
I would love to pair with you on this.

[markjacksonfishing]

We are going to scrap this python code and move toward go after discussing with @mrbobbytables

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[mrbobbytables]

/lifecycle frozen
This will be a long term issue

[kushthedude]

Hi All, I would like to collaborate on this! Let me know what is the status of the same 😄

[mrbobbytables]

It got put on the back-burner and hasn't really made any progress^^;;;
Ideally it'd be written in go, and use the same markdown library as hugo to detect what links need to be replaced. Right now it's just a bunch of regex's that are sort of best effort >_>

The configs right now are just some CSVs, but ideally we'd switch to a yaml spec - something like:

sources:
- repo: # external repo
   files:
   - include: # regex pattern to match files on a path
     exclude: # regex pattern to remove files from matched path
     dest: # destination to copy files to

Using the community repo csv as an example:

sources:
- repo: [email protected]:kubernetes/community.git
  files:
  - include: "\/contributors\/guide\/README.md"
    dest: "/docs/guide/_index.md"   #  done by itself because the file must be renamed in the process
  - include: "\/contributors\/guide\/*"
    exclude: ".*guide\/(README\.md|cheatsheet.*)"
    dest: "/docs/guide"
  - include: "\/events\/(community-meeting.md|office-hours.md)"
    dest: "/events"
  - include: "\/mentoring\/programs\/meet-our-contributors.md"
    dest:  "/events/meet-our-contributors.md"

Sorta like rsync syntax.

[mrbobbytables]

/help

[mrbobbytables]

/help

[k8s-ci-robot]

@mrbobbytables:
This request has been marked as needing help from a contributor.

Please ensure the request meets the requirements listed here.

If this request no longer meets these requirements, the label can be removed
by commenting with the /remove-help command.

In response to this:

/help

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[whs-dot-hk]

@mrbobbytables Will ansible work in this case?

https://docs.ansible.com/ansible/latest/collections/ansible/posix/synchronize_module.html

[mrbobbytables]

@whs-dot-hk No, this would be best in golang where it could use the same library used by Hugo to generate links. The difficult part is not syncing the content, it's ensuring all links within the pages are updated without potentially modifying other content like code snippets in code fences.

[whs-dot-hk]

@mrbobbytables Should it be a separate go program or a hugo "plugin"?

[mrbobbytables]

@whs-dot-hk it would be a separate program.

[RinkiyaKeDad]

/assign

[Debanitrkl]

/assign