-
Notifications
You must be signed in to change notification settings - Fork 323
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
feat: Community videos scraping (#2441)
* add deserialization * add serialization * swap them * Update data/cookbook/deserialise-from-json/00-deserialize-yojson.ml Co-authored-by: sabine <[email protected]> * Update data/cookbook/deserialise-from-json/00-deserialize-yojson.ml Co-authored-by: sabine <[email protected]> * Update data/cookbook/deserialise-from-json/00-deserialize-yojson.ml Co-authored-by: sabine <[email protected]> * Update data/cookbook/serialise-to-json/00-serialize-yojson.ml Co-authored-by: sabine <[email protected]> * Update data/cookbook/serialise-to-json/00-serialize-yojson.ml Co-authored-by: sabine <[email protected]> * Update data/cookbook/serialise-to-json/00-serialize-yojson.ml Co-authored-by: sabine <[email protected]> * Data type and scraping for YouTube video MRSS * YouTube page * Add UI * Formatting * Per source filters * Only use ocaml as scrape filter * fix typo * Scrape and merge * Formatting * Generate asset/video.rss * Single type for Youtube * minor formatting, etc. * Revert "minor formatting, etc." This reverts commit ce6cfc1. * Use better youtube urls * Formatting * Rebase * Renaming * Formatting * Import watch.ocaml.org video list * WIP * Process watch.ocaml.org videos * Aggregate youtube and watch.ocaml.org * Everything in the right place * Misc * remove video route, videos will render under OCaml Planet in a different PR --------- Co-authored-by: gpopides <[email protected]> Co-authored-by: gpopides <[email protected]> Co-authored-by: sabine <[email protected]> Co-authored-by: Cuihtlauac ALVARADO <[email protected]> Co-authored-by: Christine Rose <[email protected]>
- Loading branch information
1 parent
3712051
commit 35bc6ab
Showing
22 changed files
with
3,489 additions
and
1,413 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Large diffs are not rendered by default.
Oops, something went wrong.
Large diffs are not rendered by default.
Oops, something went wrong.
This file was deleted.
Oops, something went wrong.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,27 @@ | ||
- name: Jane Street - Tech Talks | ||
kind: playlist | ||
id: PLCiAikFFaMJoWyXnJ2BWpse5HuiYibNYs | ||
only_ocaml: true | ||
- name: Jane Street - OCaml Unboxed | ||
kind: playlist | ||
id: PLCiAikFFaMJrgFrWRKn0-1EI3gVZLQJtJ | ||
- name: Jane Street - Signal & Threads | ||
kind: playlist | ||
id: PLCiAikFFaMJouorRXDSfS2UoKV4BfKyQm | ||
only_ocaml: true | ||
- name: Emelle TV | ||
kind: channel | ||
id: UCvVVfCa7-nzSuCdMKXnNJNQ | ||
- name: The Vimeagen | ||
kind: channel | ||
id: UCVk4b-svNJoeytrrlOixebQ | ||
only_ocaml: true | ||
- name: OCAML Workshop at ICFP 2023 | ||
kind: playlist | ||
id: PLyrlk8Xaylp7Tq5-ZN6jkir-sYrhGi_0E | ||
- name: "OCaml Programming: Correct + Efficient + Beautiful" | ||
kind: playlist | ||
id: PLre5AT9JnKShBOPeuiD9b-I4XROIJhkIU | ||
- name: Collège de France - Sciences du logiciel | ||
kind: playlist | ||
id: PLtimy8tnozICbD45yhB7Ha_zIBJTIK3im |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file was deleted.
Oops, something went wrong.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,43 @@ | ||
open Data_intf.Video | ||
|
||
type video_list = t list [@@deriving yaml, show] | ||
|
||
let all () = Youtube.all () @ Watch.all () | ||
|
||
let template () = | ||
Format.asprintf {ocaml| | ||
include Data_intf.Video | ||
let all =%a | ||
|ocaml} | ||
pp_video_list (all ()) | ||
|
||
let create_entry (v : t) = | ||
let url = Uri.of_string v.content in | ||
let source : Syndic.Atom.source = | ||
Syndic.Atom.source ~authors:[] | ||
~id:(Uri.of_string v.source_link) | ||
~title:(Syndic.Atom.Text v.source_title) | ||
~links:[ Syndic.Atom.link (Uri.of_string v.source_link) ] | ||
?updated:None ?categories:None ?contributors:None ?generator:None | ||
?icon:None ?logo:None ?rights:None ?subtitle:None | ||
in | ||
let content = Syndic.Atom.Text v.description in | ||
let id = url in | ||
let authors = | ||
(Syndic.Atom.author ~uri:(Uri.of_string v.author_uri) v.author_name, []) | ||
in | ||
let updated = Syndic.Date.of_rfc3339 v.published in | ||
Syndic.Atom.entry ~content ~source ~id ~authors | ||
~title:(Syndic.Atom.Text v.title) ~updated | ||
~links:[ Syndic.Atom.link id ] | ||
() | ||
|
||
let create_feed () = | ||
let open Rss in | ||
() |> all | ||
|> create_feed ~id:"video.xml" ~title:"OCaml Videos" ~create_entry ~span:3653 | ||
|> feed_to_string | ||
|
||
let scrape () = | ||
Youtube.scrape "data/video-youtube.yml"; | ||
Watch.scrape "data/vide-watch.yml" |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.