Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add Natvis visualizations for core indexmap types #239

Open
wants to merge 2 commits into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 12 additions & 1 deletion .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -12,9 +12,9 @@ env:

jobs:
tests:
runs-on: ubuntu-latest
strategy:
matrix:
os: [ubuntu-latest]
include:
- rust: 1.56.0 # MSRV
features:
Expand All @@ -30,6 +30,13 @@ jobs:
features:
- rust: nightly
bench: test build benchmarks
- rust: nightly
features: debugger_visualizer
- rust: nightly
features: debugger_visualizer
os: windows-latest

runs-on: ${{ matrix.os }}

steps:
- uses: actions/checkout@v3
Expand All @@ -46,6 +53,10 @@ jobs:
if: matrix.features == 'serde'
run: |
cargo test --verbose -p test-serde
- name: Tests (debugger_visualizer)
if: matrix.features == 'debugger_visualizer'
run: |
cargo test --verbose --test debugger_visualizer --features "${{ matrix.features }}" -- --test-threads=1
- name: Test run benchmarks
if: matrix.bench != ''
run: cargo test -v --benches
Expand Down
12 changes: 12 additions & 0 deletions Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -35,6 +35,8 @@ fnv = "1.0"
lazy_static = "1.3"
fxhash = "0.2.1"
serde_derive = "1.0"
debugger_test = "0.1"
debugger_test_parser = "0.1"

[features]
default = ["std"]
Expand All @@ -43,6 +45,10 @@ std = []
# for testing only, of course
test_debug = []

# UNSTABLE FEATURES (requires Rust nightly)
# Enable to use the #[debugger_visualizer] attribute.
debugger_visualizer = []

[profile.bench]
debug = true

Expand All @@ -55,3 +61,9 @@ features = ["serde", "rayon"]

[workspace]
members = ["test-nostd", "test-serde"]

[[test]]
path = "tests/debugger_visualizer.rs"
name = "debugger_visualizer"
required-features = ["debugger_visualizer"]
test = false
111 changes: 111 additions & 0 deletions debug_metadata/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,111 @@
## Debugger Visualizers

Many languages and debuggers enable developers to control how a type is
displayed in a debugger. These are called "debugger visualizations" or "debugger
views".

The Windows debuggers (WinDbg\CDB) support defining custom debugger visualizations using
the `Natvis` framework. To use Natvis, developers write XML documents using the natvis
schema that describe how debugger types should be displayed with the `.natvis` extension.
(See: https://docs.microsoft.com/en-us/visualstudio/debugger/create-custom-views-of-native-objects?view=vs-2019)
The Natvis files provide patterns which match type names a description of how to display
those types.

The Natvis schema can be found either online (See: https://code.visualstudio.com/docs/cpp/natvis#_schema)
or locally at `<VS Installation Folder>\Xml\Schemas\1033\natvis.xsd`.

The GNU debugger (GDB) supports defining custom debugger views using Pretty Printers.
Pretty printers are written as python scripts that describe how a type should be displayed
when loaded up in GDB/LLDB. (See: https://sourceware.org/gdb/onlinedocs/gdb/Pretty-Printing.html#Pretty-Printing)
The pretty printers provide patterns, which match type names, and for matching
types, descibe how to display those types. (For writing a pretty printer, see: https://sourceware.org/gdb/onlinedocs/gdb/Writing-a-Pretty_002dPrinter.html#Writing-a-Pretty_002dPrinter).

### Embedding Visualizers

Through the use of the currently unstable `#[debugger_visualizer]` attribute, the `indexmap`
crate can embed debugger visualizers into the crate metadata.

Currently the two types of visualizers supported are Natvis and Pretty printers.

For Natvis files, when linking an executable with a crate that includes Natvis files,
the MSVC linker will embed the contents of all Natvis files into the generated `PDB`.

For pretty printers, the compiler will encode the contents of the pretty printer
in the `.debug_gdb_scripts` section of the `ELF` generated.

### Testing Visualizers

The `indexmap` crate supports testing debugger visualizers defined for this crate. The entry point for
these tests are `tests/debugger_visualizer.rs`. These tests are defined using the `debugger_test` and
`debugger_test_parser` crates. The `debugger_test` crate is a proc macro crate which defines a
single proc macro attribute, `#[debugger_test]`. For more detailed information about this crate,
see https://crates.io/crates/debugger_test. The CI pipeline for the `indexmap` crate has been updated
to run the debugger visualizer tests to ensure debugger visualizers do not become broken/stale.

The `#[debugger_test]` proc macro attribute may only be used on test functions and will run the
function under the debugger specified by the `debugger` meta item.

This proc macro attribute has 3 required values:

1. The first required meta item, `debugger`, takes a string value which specifies the debugger to launch.
2. The second required meta item, `commands`, takes a string of new line (`\n`) separated list of debugger
commands to run.
3. The third required meta item, `expected_statements`, takes a string of new line (`\n`) separated list of
statements that must exist in the debugger output. Pattern matching through regular expressions is also
supported by using the `pattern:` prefix for each expected statement.

#### Example:

```rust
#[debugger_test(
debugger = "cdb",
commands = "command1\ncommand2\ncommand3",
expected_statements = "statement1\nstatement2\nstatement3")]
fn test() {

}
```

Using a multiline string is also supported, with a single debugger command/expected statement per line:

```rust
#[debugger_test(
debugger = "cdb",
commands = "
command1
command2
command3",
expected_statements = "
statement1
pattern:statement[0-9]+
statement3")]
fn test() {

}
```

In the example above, the second expected statement uses pattern matching through a regular expression
by using the `pattern:` prefix.

#### Testing Locally

Currently, only Natvis visualizations have been defined for the `indexmap` crate via `debug_metadata/indexmap.natvis`,
which means the `tests/debugger_visualizer.rs` tests need to be run on Windows using the `*-pc-windows-msvc` targets.
To run these tests locally, first ensure the debugging tools for Windows are installed or install them following
the steps listed here, [Debugging Tools for Windows](https://docs.microsoft.com/en-us/windows-hardware/drivers/debugger/).
Once the debugging tools have been installed, the tests can be run in the same manner as they are in the CI
pipeline.

#### Note

When running the debugger visualizer tests, `tests/debugger_visualizer.rs`, they need to be run consecutively
and not in parallel. This can be achieved by passing the flag `--test-threads=1` to rustc. This is due to
how the debugger tests are run. Each test marked with the `#[debugger_test]` attribute launches a debugger
and attaches it to the current test process. If tests are running in parallel, the test will try to attach
a debugger to the current process which may already have a debugger attached causing the test to fail.

For example:

```
cargo test --test debugger_visualizer --features debugger_visualizer -- --test-threads=1
```
29 changes: 29 additions & 0 deletions debug_metadata/indexmap.natvis
Original file line number Diff line number Diff line change
@@ -0,0 +1,29 @@
<AutoVisualizer xmlns="http://schemas.microsoft.com/vstudio/debugger/natvis/2010">
<Type Name="indexmap::set::IndexSet&lt;*,*&gt;">
ridwanabdillahi marked this conversation as resolved.
Show resolved Hide resolved
<DisplayString>{{ len={map.core.entries.len} }}</DisplayString>
<Expand>
<ExpandedItem>map</ExpandedItem>
</Expand>
</Type>
<Type Name="indexmap::map::IndexMap&lt;*,*,*&gt;">
<DisplayString>{{ len={core.entries.len} }}</DisplayString>
<Expand>
<ExpandedItem>core.entries</ExpandedItem>
</Expand>
</Type>
<Type Name="indexmap::Bucket&lt;*,tuple$&lt;&gt;&gt;">
<DisplayString>{key}</DisplayString>
<Expand>
<Item Name="[key]">key</Item>
<Item Name="[hash]">hash.__0,d</Item>
</Expand>
</Type>
<Type Name="indexmap::Bucket&lt;*,*&gt;">
<DisplayString>{{ key={key}, value={value} }}</DisplayString>
<Expand>
<Item Name="[key]">key</Item>
<Item Name="[value]">value</Item>
<Item Name="[hash]">hash.__0,d</Item>
</Expand>
</Type>
</AutoVisualizer>
5 changes: 5 additions & 0 deletions src/lib.rs
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,11 @@
#![warn(rust_2018_idioms)]
#![doc(html_root_url = "https://docs.rs/indexmap/1/")]
#![no_std]
#![cfg_attr(
feature = "debugger_visualizer",
feature(debugger_visualizer),
debugger_visualizer(natvis_file = "../debug_metadata/indexmap.natvis")
)]
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm personally very wary of using any unstable features in a stable crate, even behind an optional feature.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I understand, the goal is to push the debugger_visualizer feature to stabilization relatively soon by having it provide value to crates in the wild today as with any unstable feature.

As you said it is gated behind an optional feature but you as the crate maintainer would have to potentially take on an extra cost which I fully understand could be worrisome.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Well, I gave approval for CI to run, at least, so it can be a proof of concept for the feature.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great! I see the CI was green and tests passed. Thanks for approving the CI job.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@cuviper Do you have any reservations about taking in this change now that the CI shows as green?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Passing CI is necessary, but doesn't change the maintenance aspect. If the unstable feature changes, as it is allowed, then we'll have to update again to fix our own CI. That's especially true if any users (on nightly) start depending on it (providing "value in the wild").

For ad hoc demonstrations in the meantime, you could use cargo's patch mechanisms to point at your git fork.


//! [`IndexMap`] is a hash table where the iteration order of the key-value
//! pairs is independent of the hash values of the keys.
Expand Down
89 changes: 89 additions & 0 deletions tests/debugger_visualizer.rs
Original file line number Diff line number Diff line change
@@ -0,0 +1,89 @@
use debugger_test::debugger_test;
use indexmap::IndexMap;
use indexmap::IndexSet;

#[inline(never)]
fn __break() {}

#[debugger_test(
debugger = "cdb",
commands = r#"
.nvlist
dx set
dx map
"#,
expected_statements = r#"
set : { len=0x19 } [Type: indexmap::set::IndexSet<char,std::collections::hash::map::RandomState>]
[len] : 0x19 [Type: unsigned __int64]
[capacity] : 0x1c [Type: unsigned __int64]
[0] : 0x74 't' [Type: indexmap::Bucket<char,tuple$<> >]
[1] : 0x68 'h' [Type: indexmap::Bucket<char,tuple$<> >]
[2] : 0x65 'e' [Type: indexmap::Bucket<char,tuple$<> >]
[3] : 0x20 ' ' [Type: indexmap::Bucket<char,tuple$<> >]
[4] : 0x71 'q' [Type: indexmap::Bucket<char,tuple$<> >]
[5] : 0x75 'u' [Type: indexmap::Bucket<char,tuple$<> >]
[6] : 0x69 'i' [Type: indexmap::Bucket<char,tuple$<> >]
[7] : 0x63 'c' [Type: indexmap::Bucket<char,tuple$<> >]
[8] : 0x6b 'k' [Type: indexmap::Bucket<char,tuple$<> >]
[9] : 0x72 'r' [Type: indexmap::Bucket<char,tuple$<> >]
[10] : 0x6f 'o' [Type: indexmap::Bucket<char,tuple$<> >]
[11] : 0x77 'w' [Type: indexmap::Bucket<char,tuple$<> >]
[12] : 0x6e 'n' [Type: indexmap::Bucket<char,tuple$<> >]
[13] : 0x66 'f' [Type: indexmap::Bucket<char,tuple$<> >]
[14] : 0x78 'x' [Type: indexmap::Bucket<char,tuple$<> >]
[15] : 0x6a 'j' [Type: indexmap::Bucket<char,tuple$<> >]
[16] : 0x6d 'm' [Type: indexmap::Bucket<char,tuple$<> >]
[17] : 0x70 'p' [Type: indexmap::Bucket<char,tuple$<> >]
[18] : 0x64 'd' [Type: indexmap::Bucket<char,tuple$<> >]
[19] : 0x76 'v' [Type: indexmap::Bucket<char,tuple$<> >]
[20] : 0x6c 'l' [Type: indexmap::Bucket<char,tuple$<> >]
[21] : 0x61 'a' [Type: indexmap::Bucket<char,tuple$<> >]
[22] : 0x7a 'z' [Type: indexmap::Bucket<char,tuple$<> >]
[23] : 0x79 'y' [Type: indexmap::Bucket<char,tuple$<> >]
[24] : 0x67 'g' [Type: indexmap::Bucket<char,tuple$<> >]

map : { len=0x19 } [Type: indexmap::map::IndexMap<char,i32,std::collections::hash::map::RandomState>]
[len] : 0x19 [Type: unsigned __int64]
[capacity] : 0x1c [Type: unsigned __int64]
[0] : { key=0x74 't', value=2 } [Type: indexmap::Bucket<char,i32>]
[1] : { key=0x68 'h', value=2 } [Type: indexmap::Bucket<char,i32>]
[2] : { key=0x65 'e', value=4 } [Type: indexmap::Bucket<char,i32>]
[3] : { key=0x20 ' ', value=8 } [Type: indexmap::Bucket<char,i32>]
[4] : { key=0x71 'q', value=1 } [Type: indexmap::Bucket<char,i32>]
[5] : { key=0x75 'u', value=2 } [Type: indexmap::Bucket<char,i32>]
[6] : { key=0x69 'i', value=1 } [Type: indexmap::Bucket<char,i32>]
[7] : { key=0x63 'c', value=1 } [Type: indexmap::Bucket<char,i32>]
[8] : { key=0x6b 'k', value=1 } [Type: indexmap::Bucket<char,i32>]
[9] : { key=0x67 'g', value=1 } [Type: indexmap::Bucket<char,i32>]
[10] : { key=0x72 'r', value=2 } [Type: indexmap::Bucket<char,i32>]
[11] : { key=0x6f 'o', value=4 } [Type: indexmap::Bucket<char,i32>]
[12] : { key=0x77 'w', value=1 } [Type: indexmap::Bucket<char,i32>]
[13] : { key=0x6e 'n', value=1 } [Type: indexmap::Bucket<char,i32>]
[14] : { key=0x66 'f', value=1 } [Type: indexmap::Bucket<char,i32>]
[15] : { key=0x78 'x', value=1 } [Type: indexmap::Bucket<char,i32>]
[16] : { key=0x6a 'j', value=1 } [Type: indexmap::Bucket<char,i32>]
[17] : { key=0x6d 'm', value=1 } [Type: indexmap::Bucket<char,i32>]
[18] : { key=0x70 'p', value=1 } [Type: indexmap::Bucket<char,i32>]
[19] : { key=0x64 'd', value=2 } [Type: indexmap::Bucket<char,i32>]
[20] : { key=0x76 'v', value=1 } [Type: indexmap::Bucket<char,i32>]
[21] : { key=0x6c 'l', value=1 } [Type: indexmap::Bucket<char,i32>]
[22] : { key=0x61 'a', value=1 } [Type: indexmap::Bucket<char,i32>]
[23] : { key=0x7a 'z', value=1 } [Type: indexmap::Bucket<char,i32>]
[24] : { key=0x79 'y', value=1 } [Type: indexmap::Bucket<char,i32>]
"#
)]
fn test_debugger_visualizer() {
let mut set = IndexSet::new();
let mut map = IndexMap::new();

for ch in "the quick brown fox jumped over the lazy dog".chars() {
set.insert(ch);
*map.entry(ch).or_insert(0) += 1;
}

let b = 'b';
assert!(set.shift_remove(&b));
assert_eq!(Some(1), map.remove(&b));

__break();
}