From 16b0c933ac3f15bcabf371f16859b60d4dc877e6 Mon Sep 17 00:00:00 2001 From: Karl <5254025+karwa@users.noreply.github.com> Date: Tue, 26 Jul 2022 16:28:57 +0200 Subject: [PATCH] [Docs] Update readme --- README.md | 211 ++++++++++++++++++++++++++---------------------------- 1 file changed, 101 insertions(+), 110 deletions(-) diff --git a/README.md b/README.md index 1bb9fe675..56443b15f 100644 --- a/README.md +++ b/README.md @@ -3,22 +3,42 @@ [![](https://img.shields.io/endpoint?url=https%3A%2F%2Fswiftpackageindex.com%2Fapi%2Fpackages%2Fkarwa%2Fswift-url%2Fbadge%3Ftype%3Dswift-versions)](https://swiftpackageindex.com/karwa/swift-url) [![](https://img.shields.io/endpoint?url=https%3A%2F%2Fswiftpackageindex.com%2Fapi%2Fpackages%2Fkarwa%2Fswift-url%2Fbadge%3Ftype%3Dplatforms)](https://swiftpackageindex.com/karwa/swift-url) -A new URL type for Swift. +A new URL type for Swift. -- ๐ŸŒ **Compliant**. WebURL conforms to the [latest URL Standard](https://url.spec.whatwg.org/), which specifies how modern browsers such as Safari and Chrome interpret URLs. - -- โšก๏ธ **Fast**. Tuned for high performance and low memory use. +

๐ŸŒ  Standards Compliant

-- ๐Ÿญ **Delightful**. The API is designed around modern best practices, and liberal use of Swift features such as generics and zero-cost abstractions make it really expressive and powerful. It's not _just_ easier to use than Foundation's `URL` - more importantly, it's easier to use _correctly_. +WebURL fully supports the [latest URL Standard](https://url.spec.whatwg.org/), which specifies how modern browsers such as Safari and Chrome interpret URLs. It includes support for Unicode domain names (IDNA). -- ๐Ÿงณ **Portable**. The core WebURL library has no dependencies other than the Swift standard library, and no platform-specific behavior. +Foundation's `URL` and `URLComponents` each conform to different standards (about 20 years old) and _neither_ of them match how browsers and other modern software such as NodeJS processes URLs. WebURL does. +
-- ๐Ÿ”— **Interoperable**. Compatibility libraries allow you to use WebURL's modern, standards-compliant parser and API together with `Foundation` and `swift-system`, and our port of `async-http-client` demonstrates how to use WebURL together with `swift-nio`. +

๐Ÿญ  Delightful to Use

-_(and of course, it's written in **100% Swift**)_. +We take full advantage of Swift to offer a rich, expressive API that encourages modern best practices and ways of working with URLs. Common tasks like reading or modifying a URL's path or query are easier to do, more efficient, and help you avoid subtle mistakes. -๐Ÿ“š Check out the [Documentation][weburl-docs] to learn more ๐Ÿ“š +Even asking whether two URLs are `==` is full of surprising edge-cases when working with Foundation's `URL`. WebURL has greatly simplified semantics, which helps make your applications more robust and matches how you probably think about URLs.
+ +

๐Ÿ”—  ๐Ÿงณ  Portable, Interoperable

+ +The core WebURL library has no external dependencies or platform-specific behavior. Everything works the same everywhere, and **everything fully back-deploys**. + +And thanks to integration libraries that come with this package, WebURL still works seamlessly with `Foundation` and `swift-system`. We've also ported `async-http-client` to use WebURL, which shows how easy it is for `swift-nio`-based projects to adopt. +
+ +

โšก๏ธ  Fast

+ +Despite offering very high-level APIs, WebURL _also_ delivers great performance and low memory use. Extra effort has been spent on optimizing common operations such as converting URLs to/from strings (such as from JSON), or making efficient in-place modifications. + +The API uses the concept of write-through views to expose its lower-level generic implementation in a separate scope, without polluting the rest of the API. This allows high-volume workflows to achieve minimal overheads (for example, applications which parse lists of URLs or scan data can do so directly from the bytes of a file). +
+ +_(and it's written in **100% Swift**)_. + +
+ +๐Ÿ“š **Check out the [Documentation][weburl-docs] to learn more** ๐Ÿ“š +
# Using WebURL in your project @@ -28,19 +48,15 @@ To use this package in a SwiftPM project, you need to set it up as a package dep ```swift // Add the package as a dependency. dependencies: [ - .package( - url: "https://github.com/karwa/swift-url", - .upToNextMinor(from: "0.3.1") - ) + .package(url: "https://github.com/karwa/swift-url", .upToNextMinor(from: "0.4.0")) ] // Then add the WebURL library as a target dependency. targets: [ .target( name: "", - dependencies: [ - .product(name: "WebURL", package: "swift-url") - ] + // ๐Ÿ‘‡ Add this line ๐Ÿ‘‡ + dependencies: [ .product(name: "WebURL", package: "swift-url") ] ) ] ``` @@ -56,8 +72,10 @@ url.hostname // "github.com" url.path // "/karwa/swift-url" url.pathComponents.removeLast(2) +// "https://github.com/" + url.pathComponents += ["apple", "swift"] -url // "https://github.com/apple/swift" +// "https://github.com/apple/swift" ``` ๐Ÿ“š Check out the [Documentation][weburl-docs] to learn about WebURL's API ๐Ÿ“š @@ -66,9 +84,9 @@ url // "https://github.com/apple/swift" ## ๐Ÿ”— Integration with Foundation -The `WebURLFoundationExtras` compatibility library allows you to convert between `WebURL` and Foundation `URL` values, and includes convenience wrappers for many Foundation APIs (such as `URLSession`), allowing them to be used directly with `WebURL` values. +The `WebURLFoundationExtras` compatibility library allows you to convert between WebURL and Foundation `URL` values, including making URLSession requests with WebURLs. -To enable Foundation integration, add the compatibility library to your target dependencies and import it from your code. +To make `WebURLFoundationExtras` available, add it to your target dependencies and import it from your code. ```swift targets: [ @@ -83,30 +101,26 @@ targets: [ ] ``` -Now, you can take advantage of `WebURL`'s modern, standards-compliant parser and faster, more convenient API, all while keeping compatibility with existing clients and using `URLSession` to make requests (as is required on Apple platforms). +With that, you can take full advantage of `WebURL`, while keeping compatibility with existing clients. Again, since WebURL is a fully portable package solution, you can even take advantage of features such as Unicode domain names (IDNA), and it all back-deploys, effortlessly. ```swift import Foundation import WebURL import WebURLFoundationExtras -// โ„น๏ธ Make URLSession requests using WebURL. -func makeRequest(to url: WebURL) -> URLSessionDataTask { - return URLSession.shared.dataTask(with: url) { +// Make URLSession requests using WebURL, including IDNA support. +let task = + URLSession.shared.dataTask(with: WebURL("https://๐Ÿ˜€.example.com/")!) { data, response, error in - // ... + // Works! } -} -// โ„น๏ธ Also supports Swift concurrency. -func processData(from url: WebURL) async throws { - let (data, _) = try await URLSession.shared.data(from: url) - // ... -} +// Also supports Swift concurrency. +let (data, _) = + try await URLSession.shared.data(from: WebURL("https://ๆ•ฐๆฎ.example.com/")!) -// โ„น๏ธ For libraries: move to WebURL without breaking -// compatibility with clients using Foundation's URL. -public func processURL(_ url: Foundation.URL) throws { +// URL <-> WebURL conversion allows incremental adoption. +public func connect(to url: Foundation.URL) throws { guard let webURL = WebURL(url) else { throw InvalidURLError() } @@ -114,15 +128,17 @@ public func processURL(_ url: Foundation.URL) throws { } ``` -For more information about why `WebURL` is a great choice even for applications and libraries using Foundation, and a discussion about how to safely work with multiple URL standards, we **highly recommend** reading: [Using WebURL with Foundation][using-weburl-with-foundation]. +For more information about why `WebURL` is a great choice even for applications and libraries using Foundation, and a discussion about how to safely work with multiple URL standards, read: [Using WebURL with Foundation][using-weburl-with-foundation].

## ๐Ÿ”— Integration with swift-system -The `WebURLSystemExtras` compatibility library allows you to convert between `file:` URLs and `FilePath`s, using the `swift-system` package or Apple's `System` framework. It has excellent support for both POSIX and Windows paths, with features for security and legacy compatibility (e.g. non-Unicode file names) modelled on Chrome's implementation. - -To enable `swift-system` integration, add the compatibility library to your target dependencies and import it from your code. +The `WebURLSystemExtras` library allows you to convert between `file:` URLs and `FilePath`s. That's actually quite a complex operation because it has never been standardized and there are lots of legacy issues to understand, but we took the time and created a **great** implementation. The trickiest platform is always Windows, so we based our implementation on Chromium rather than Foundation, with extra security filters inspired by `rust-url` and our own research, and built a [comprehensive test database](/Sources/WebURLTestSupport/TestFilesData/file_url_path_tests.json) to make sure we handled all the known edge-cases. + +That means WebURL has excellent support for both POSIX (Apple/Linux/etc) and Windows paths, including legacy, pre-Unicode file URLs. There are _a lot_ of documents whose names use pre-Unicode text encodings (e.g. SHIFT-JIS in Japan, or EUC-KR in South Korea), and they still need their URLs to work. It's tricky (Unicode is definitely better), but we should support them just as well as Chrome does. We're making a big effort to be the best way to work with `file:` URLs, on all platforms Swift supports. + + To make `WebURLSystemExtras` available, add it to your target dependencies and import it from your code. ```swift .target( @@ -135,7 +151,7 @@ To enable `swift-system` integration, add the compatibility library to your targ ) ``` -And that's it - you're good to go! +And that's it - you're good to go! It does a lot, but it's super easy to use! ```swift import System @@ -143,6 +159,7 @@ import WebURL import WebURLSystemExtras func openFile(at url: WebURL) throws -> FileDescriptor { + // WebURL -> FilePath let path = try FilePath(url: url) return try FileDescriptor.open(path, .readOnly) } @@ -151,11 +168,11 @@ func openFile(at url: WebURL) throws -> FileDescriptor { ## ๐Ÿงช async-http-client Port -Our prototype port of [async-http-client](https://github.com/karwa/async-http-client) uses WebURL for _all_ of its internal URL handling, including processing HTTP redirects. It's the best library available in Swift to ensure your HTTP requests use web-compatible URL processing, and is also a great demonstration of how to adopt WebURL in a library built upon `swift-nio`. +Our port of [async-http-client](https://github.com/karwa/async-http-client) uses WebURL for _all_ of its internal URL handling. It's the best Swift library for ensuring your HTTP requests use web-compatible URL processing, and is a great demonstration of how to adopt WebURL in a library using `swift-nio`. It keeps all the features and even (mostly) keeps compatibility with the existing `Foundation.URL` API. -By default, the port uses WebURL's Foundation compatibility for ease of integration, so you can make requests using either URL type, but it can also be built without any Foundation dependency at all - meaning smaller binaries and faster startup times. +The port also has an experimental mode which allows it to be built without any Foundation dependency at all. This relies on a non-public standard library API to replace the one call we use from Foundation, so for now it is an **opt-in** feature, but everything works and there are some significant performance advantages from it. Lots of developers have been trying to look beyond Foundation for next-generation, pure Swift libraries, but until now URLs have been an obstacle; they're not trivial, and since they get passed around a lot as a currency type, any replacement needs to meet or exceed the capabilities and ergonomics of `URL`, `URLComponents`, and related APIs for things like percent-encoding. WebURL does all of that. -**Note:** We'll be updating the port periodically, so if you wish to use it in an application we recommend making a fork and pulling in changes as you need. +**Note:** We'll be updating this port periodically, so if you wish to use it in an application we recommend making a fork and pulling in changes as you need. ```swift import AsyncHTTPClient @@ -163,74 +180,70 @@ import WebURL let client = HTTPClient(eventLoopGroupProvider: .createNew) -// โ„น๏ธ Supports the traditional NIO EventLoopFuture API. +// The async API uses WebURL behind the scenes (in our port). do { - let url = WebURL("https://github.com/karwa/swift-url/raw/main/README.md")! - - let request = try HTTPClient.Request(url: url) - let response = try client.execute(request: request).map { response in - response.body.map { String(decoding: $0.readableBytesView, as: UTF8.self) } - }.wait() - print(response) + let request = HTTPClientRequest(url: "https://github.com/karwa/swift-url/raw/main/README.md") + let response = try await client.execute(request, timeout: .seconds(30)).body.collect() + print(String(decoding: response.readableBytesView, as: UTF8.self)) // "# WebURL A new URL type for Swift..." } -// โ„น๏ธ Also supports Swift concurrency. +// Also supports the traditional NIO EventLoopFuture API. do { - let request = HTTPClientRequest(url: "https://github.com/karwa/swift-url/raw/main/README.md") - let response = try await client.execute(request, timeout: .seconds(30)).body.collect() - print(String(decoding: response.readableBytesView, as: UTF8.self)) + let url = WebURL("https://github.com/karwa/swift-url/raw/main/README.md")! + + let response = try client.execute(request: try HTTPClient.Request(url: url)) + .map { response in + response.body.map { String(decoding: $0.readableBytesView, as: UTF8.self) } + } + .wait() + print(response) // "# WebURL A new URL type for Swift..." } ```
-# ๐Ÿ“ฐ Project Status +# ๐Ÿ—บ Project Status & Roadmap -WebURL is a complete URL library, implementing the latest version of the URL Standard. It currently does not support Internationalized Domain Names (IDNA), but that support is planned. +## Standards Compliance -It is tested against the [shared `web-platform-tests`](https://github.com/web-platform-tests/wpt/) used by the major browsers and other libraries, and passes all constructor and setter tests (other than those which require IDNA). We pool our implementation experience to ensure there is no divergence, and the fact that WebURL passes these tests should give you confidence that it interprets URLs just like the latest release of Safari or `rust-url`. +WebURL fully implements the latest version of the URL Standard. -WebURL also includes a comprehensive set of APIs for working with URLs: getting/setting components, percent-encoding and decoding, manipulating path components, query strings, file paths, etc. Each has these come with additional, very comprehensive sets of tests. The project is regularly benchmarked and fuzz-tested using the tools available in the `Benchmarks` and `Fuzzers` directories respectively. +We validate conformance using the [shared `web-platform-tests`](https://github.com/web-platform-tests/wpt/) used by the major browsers and other libraries. All constructor, setter, and IDNA tests pass ([code](Tests/WebURLTests/WebPlatformTests.swift)), and our implementation of IDNA is validated by these and Unicode's UTS46 conformance test suite ([code](Tests/IDNATests/UTS46ConformanceTests.swift)). -Being a pre-1.0 package, we reserve the right to make source-breaking changes before committing to a stable API. We'll do our best to keep those to a minimum, and of course, any such changes will be accompanied by clear documentation explaining how to update your code. +If you find any situations where WebURL is not producing the correct result, please open a GitHub issue. -If there's anything you think could be improved, this is a great time to let us know! Either open a GitHub issue or post to the [Swift forums][swift-forums-weburl]. +## API Stability and Roadmap +While the package is still pre-1.0, there are only limited API stability guarantees. -## ๐Ÿ—บ Roadmap +> **Important**: +> +> The 0.x.x versions treat MINOR version numbers like MAJOR version numbers. +> +> That means there will be no source-breaking changes between 0.3.x and 0.3.y, +> but we want the ability to make source-breaking changes between 0.3.x to 0.4.0 if necessary. +> +> For stability, use a `.upToNextMinor(from: ...)` version constraint.
+> For the latest release, use `.upToNextMajor(from: ...)`. -Aside from stabilizing the API, the other priorities for v1.0 are: +Version 1.0 is coming soon. I think the current API has the correct shape and scope, and its behavior should be relatively uncontroversial. -1. More APIs for query parameters. +The one API I'm less sure about is `formParams` (for working with the URL's query). It has some nice ideas, but its behavior comes from Javascript's `URLSearchParams` class. That class has a number of quirks that even JS developers don't really love, so we should rethink how we want to approach query parameters before locking down the API. - A URL's `query` component is often used as a string of key-value pairs. This usage appears to have originated with HTML forms, and WebURL has excellent support for it already via the `formParams` view -- but these days, by popular convention, it is also used with keys and values that are _not strictly_ form-encoded. This can lead to decoding issues, so we should offer a variant of `formParams` that allows for percent-encoding, not just form-encoding. +There are also some areas where the language falls short. The URL Standard is a _living standard_, so we need to be careful about locking-in details that might realistically change. One issue is that they might support new kinds of host one day (currently it is only domains/ipv4/ipv6), and we _really_ want to expose this data as a Swift enum - [`WebURL.Host`][weburl-docs-host]. But enums in Swift are exhaustive, so if we added a new case to that enum in an update, that would be a source-breaking change. The result is that we can only support new kinds of host together with major version increments (e.g. 1.x -> 2.0). - Additionally, we may want to consider making key lookup Unicode-aware. It makes sense, but AFAIK is unprecedented in other libraries and so may be surprising. But it does make a lot of sense. Feedback is welcome. +Swift [_has_ non-exhaustive enums][swift-nonfrozen-enum], which force `switch` statements to handle `@unknown default:` patterns and allowing libraries to add new cases later. You see that in SDK libraries, but unfortunately, it is not available to source packages. This API really needs to be an enum to be usable though; so, painful as it is, we're accepting that limitation. We consider it unlikely that the URL Standard would add new kinds of host any time soon, though; it's not a common occurrence. -Looking beyond v1.0, the other features I'd like to add are: +Other than that, I'm relatively happy with how things are. -2. Better APIs for `data:` URLs. - - WebURL already supports them as generic URLs, but it would be nice to add specialized APIs for extracting the MIME type and decoding base64-encoded data. - -3. APIs for relative references. - - All `WebURL`s are absolute URLs (following the standard), and relative references are currently only supported as strings via the [`WebURL.resolve(_:)` method][weburl-resolve]. - - It could be valuable to some applications to add richer APIs for reading and manipulating relative references, instead of using only strings. We may also want to calculate the difference between 2 URLs and return the result as a relative reference. It depends on what people actually need, so please do leave feedback. - -4. Support Internationalized Domain Names (IDNA). - - This is part of the URL Standard, and its position on this list shouldn't be read as downplaying its importance. It is a high-priority item, but is currently blocked by other things. - - There is reason to hope this may be implementable soon. Native Unicode normalization was [recently](https://github.com/apple/swift/pull/38922) implemented in the Swift standard library for String, and there is a desire to expose this functionality to libraries such as this one. Once those APIs are available, we'll be able to use them to implement IDNA. +If there's anything _you think_ could be improved, this is a great time to let me know! Open a GitHub issue or post to the [Swift forums][swift-forums-weburl].

# ๐Ÿ’ Sponsorship -I'm creating this library because I think that Swift is a great language, and it deserves a high-quality, modern library for handling URLs. I think it's a really good, production-quality implementation, it has taken a lot of time to get to this stage, and there is still an exciting roadmap ahead. So if you (or the company you work for) benefit from this project, consider donating a coffee to show your support. You don't have to; it's mostly about letting me know that people appreciate it. That's ultimately what motivates me. +I'm creating this library because I think that Swift is a great language, but it lacked a high-quality, modern library for handling URLs. It has taken a lot of time to get WebURL to this stage, and there is still a lot that can be done with it. If you'd like to show support for the project, consider donating a coffee or something.

@@ -238,7 +251,7 @@ I'm creating this library because I think that Swift is a great language, and it ## How do I leave feedback? -Either open a GitHub issue or post to the [Swift forums][swift-forums-weburl]. +Open a GitHub issue or post to the [Swift forums][swift-forums-weburl]. ## Are pull requests/review comments/questions welcome? @@ -246,15 +259,13 @@ Most definitely! ## Is this production-ready? -Yes. With the caveat that the API might see some minor adjustments between now and 1.0. - -The implementation is extensively tested, including against the shared `web-platform-tests` used by the major browsers and other libraries, and which we've made _a lot_ of contributions to. As mentioned above, having that shared test suite across the various implementations is a really valuable resource and should give you confidence that WebURL will actually interpret URLs according to the standard. +I think so. With the caveat that the API is not yet stable across minor versions, only patch versions. -We also verify a lot of things by regular fuzz-testing (e.g. that parsing and serialization are idempotent, that Foundation conversions are safe, etc), so we have confidence that the behavior is well understood. +Testing and reliability have been taken extremely seriously from the beginning. The implementation is extensively tested, including against the shared `web-platform-tests` and Unicode conformance tests, used by the major browsers and other libraries. Having those shared test suites is a really valuable resource and should give you confidence that WebURL is actually interpreting the standard as other projects such as WebKit do. -Additionally, the benchmarks package available in this repository helps ensure we deliver consistent, excellent performance across a variety of devices. +The other library features are also extensively tested, with ~90% coverage, and many difficult-to-test assertions are checked using fuzz-testing (for example, that parsing and serialization are idempotent, that Foundation/WebURL conversions are safe, etc), so the behavior is well understood. -We've taken testing and reliability extremely seriously from the very beginning, which is why we have confidence in claiming that this is the best-tested URL library available for Swift. To be quite frank, Foundation does not have anything even close to this. +The benchmarks package available in this repository helps ensure we have consistent performance across a variety of devices and can measure any regressions. We are also mindful of code-size and do what we can to keep it down, while implementing the entire standard. ## Why the name `WebURL`? @@ -262,29 +273,9 @@ We've taken testing and reliability extremely seriously from the very beginning, 2. The WHATWG works on technologies for the web platform. By following the WHATWG URL Standard, `WebURL` could be considered a kind of "Web-platform URL". -## What is the WHATWG URL Living Standard? - -It may be surprising to learn that there many interpretations of URLs floating about - after all, you type a URL in to your browser, and it just works! Right? Well, sometimes... - -URLs were first specified in 1994, and were repeatedly revised over the years, such as by [RFC-2396](https://datatracker.ietf.org/doc/html/rfc2396) in 1998, and [RFC-3986](https://www.ietf.org/rfc/rfc3986.txt) in 2005. So there are all these different standards floating around - and as it turns out, they're **not always compatible** with each other, and are sometimes ambiguous. - -While all this was going on, browsers were doing their own thing, and each behaved differently to the others. The web in the 90s was a real wild west, and standards-compliance wasn't a high priority. Now, that behavior has to be maintained for compatibility, but having all these different standards can lead to severe misunderstandings and even exploitable security vulnerabilities. Consider these examples from [Orange Tsai's famous talk](https://www.youtube.com/watch?v=voTHFdL9S2k) showing how different URL parsers (sometimes even within the same application) each think these URLs point to a different server. - -![](abusing-url-parsers-example-orange-tsai.png) ![](abusing-url-parsers-example-orange-tsai-2.png) - -_Images are Copyright Orange Tsai_ - -So having all these incompatible standards is a problem. Clearly, there was only one answer: yet another standard! ๐Ÿ˜… But seriously, this time, it had to have browsers adopt it. For a URL standard, matching how browsers behave is kinda a big deal, you know? And they're not going to break the web, so it needs to document what it means to be "web compatible". It turns out, most URL libraries already include ad-hoc collections of hacks to try to guess what web compatibility means. - -This is where the WHATWG comes in to it. The WHATWG is an industry association led by the major browser developers (currently, the steering committee consists of representatives from Apple, Google, Mozilla, and Microsoft), and there is high-level approval for their browsers to align with the standards developed by the group. The latest WebKit (Safari 15) is already in compliance. The WHATWG URL Living Standard defines how **actors on the web platform** should understand and manipulate URLs - how browsers process them, how code such as JavaScript's `URL` class interprets them, etc. And this applies at all levels, from URLs in HTML documents to HTTP redirect requests. This is the web's URL standard. - -By aligning to the URL Living Standard, this project aims to provide the behavior you expect, with better reliability and interoperability, sharing a standard and test-suite with your browser, and engaging with the web standards process. And by doing so, we hope to make Swift an even more attractive language for both servers and client applications. - - - - [swift-forums-weburl]: https://forums.swift.org/c/related-projects/weburl/73 [weburl-docs]: https://karwa.github.io/swift-url/main/documentation/weburl/ -[weburl-resolve]: https://karwa.github.io/swift-url/main/documentation/weburl/weburl/resolve(_:) [using-weburl-with-foundation]: https://karwa.github.io/swift-url/main/documentation/weburl/foundationinterop +[weburl-docs-host]: https://karwa.github.io/swift-url/main/documentation/weburl/weburl/host-swift.enum +[swift-nonfrozen-enum]: https://docs.swift.org/swift-book/ReferenceManual/Statements.html#ID602 \ No newline at end of file